It's almost certainly the downloading. If you guys are going to share Internet, you have to actually share it. That means that when one person is doing something that requires low latency (like gaming) someone else can't do something that destroys latency (like running clustering downloaders).
The usual simple solution is to talk to the person downloading and explain to them that they need to limit their bandwidth in each direction to at most 80% of the maximum bandwidth they ever see. This will only reduce their transfer rates by 20% or so and it will keep the queues on the connection nearly empty rather than nearly full, preserving latency for applications like gaming.