There are various architectural features in Server Class processors that are critical for performance and the CPU frequency is just one aspect in that.
- Number of cores - This is one of the biggest factors that influences its applicability for HPC. The total cores = number of sockets * cores per die.
- Memory Bandwidth achievable - The Xeons have huge caches when compared to the desktop counterparts and achieve higher throughputs as well. The memory Bandwidth in the latest Xeons (Q2' 15) is more than 100GB when compared to <30GB/s on the i7
- Security features like the Trusted execution technology
- Higher PCI express lanes which enable further extension and performance from the PCI connected devices.
A quick comparison on Intel ark gives us the high level overview.
Ultimately an architect has to tune the system to suit its needs, these advantages on the server front is weighed against other features on the desktop front like the support for Graphics.