No it is not a waste of time to use more than one key.
More diversity == less risk.
That statement of Spiff's is incorrect.
The point is the public key grants access to the private key holder and no one else.
The risk to be concerned about here is authentication.
A rogue site forwards authentication requests to your agent task.
If you use only one key, then even when only one key is loaded in
your agent, all sites are open to the rogue.
This has nothing to do with the passphrases, you could have several keys
with the same passphrase that would make no difference here. Because
it is not the passphrase that is compromised.
The rogue forwards challenges to your agent and can connect to all sites
for which you have keys loaded.
With different keys, one key loaded -> one site at risk.
I say good for you, you picked other peoples privacy over your own laziness.
P.S. the moral of the story is be wary of agent forwarding