This question on performance can be separated into two categories:
Performance of 2.4GHz vs. 5GHz and Performance of allowing backwards compatibility (e.i. allowing b devices, even if you don't own any). I assume that your devices are all N capable.
Performance of 2.4GHz vs. 5GHz
N devices can connect to both the 2.4 and 5GHz range. The two ranges are different in several different ways:
- 5GHz will always have a shorter range than 2.4.
- 5GHz will have trouble going through objects (anything with water:
wood, cement, etc) while 2.4 will not.
- 5GHz will have a greater throughput at its smaller range than 2.4
- 2.4GHz is over used by the industry, more usage leads to noise, more noise leads to congestion, congestion leads to decreased performance.
Therefore if you are close to the router the majority of the time using only the 5GHz will not take away from the potential performance of using both. If, on the other hand, the router is usually far away, using the 2.4 band will offer less power usage will no decrease of performance.
Performance of allowing backwards compatibility
There is no significant performance degradation of allowing backwards compatibility. However, when a device connects to the network and the router is forced to use backwards compatibility with the device there could be a significant performance difference.
- With devices using the B standard, many routers must employ "hacks" to prevent the B device from messing with the other N/G/A devices. This performance drop could be over 80% for all other devices on the network.
- With devices using the G standard, many routers must reduce the amount of frames (hence data) being transferred by the network to the G standard. Therefore, any N devices will be impacted.
So in the end allowing older devices to connect will not impact performance by itself. However, having these devices connect could reduce the performance of the whole network.