Your main concern shouldn't be G vs. N. You really want to pay attention to what signaling rates your clients are getting, regardless of which technologies and max rates they're capable of.
Having a G device connect to an N network doesn't really slow the network down much at all. It certainly doesn't "drop the whole network down to G speeds" as the oft-repeated myth goes. When there are no B or G devices present on the current network, or even on an overlapping network on the same channel or an overlapping channel, an N network can enable a few minor speed optimizations, but they probably only amount to single digit percentage speed increases.
A bigger worry is how much of the airtime a given client is taking up. But an N client that's far enough away from your access point that it has to fall back to the lower signaling rates will take up just as much airtime as a G client that's using similar rates.
So the big thing to pay attention to is what signaling rates your router is using when transmitting to each client, and what signaling rates each client is using when transmitting to the router. This information is available on the routers I use the most.