Since I updated to Windows 8.1 (Enterprise) I have a sporadic issue with RDP connections. They're very slow. So it looks like the the machine runs at 100% with a high ping. But it's not.
The host itself is irrelevant in this case, whether it's Server 2012 or Server 2003. Having a look in the task manager, no process runs at 100% CPU nor consumes all the memory.
To workaround this issue, I'm running a Windows 7 VM and connect via RDP through this machine.
The problem is, I can't really reproduce the problem. My colleague has this issue in the office and at the customer's network. I just had it while being at two different customers.
We use TrendMicro OfficeScan which I thought might be the problem, but after disabling it, the problem persists.