If saturating either your upstream or your downstream network bandwidth causes latency to spike, it means some box in your network path (probably your broadband modem or your ISP's CMTS or DSLAM) has a well-known bug called bufferbloat, where excessive buffering on a box increases latency without benefit.
The fix for bufferbloat is to upgrade the queuing algorithm (a.k.a. queue discipline, network scheduler) of the buggy device to a latency-aware smart queueing algorithm such as FQ-CoDel.
If you can't fix the actual box with the problem, you can work around it by setting up a box with FQ-CoDel at the head of your network and tune its traffic shaping to make it a slight bottleneck in both the upstream and downstream directions. That way FQ-CoDel gets to kick in and allow TCP congestion control to work, before bloated buffer queues can build up on the buggy box.
You can do this yourself with open source router firmware distros such as LEDE (formerly OpenWrt), but if you want a turn-key solution, look at IQrouter from evenroute.com. It apparently auto-tunes its bandwidth shaping throughout the day, maximizing throughput while minimizing latency.
Many people who haven't learned about bufferbloat wrongly assume latency spikes are a natural result of saturated network links. Many people also make problematic attempts to work around bufferbloat by tweaking QoS, trying to prioritize some flows over the large flows that are triggering bloat. But solving bufferbloat directly is much better, because it improves all flows, even the big ones that were triggering bloat.