For almost 15 years I've suffered the following problem (sometimes chronically and sometimes only rarely), across multiple machines, router hardware, and ISPs at both ends of the connection:

  1. My home computer H in city A regularly attains excellent download speeds of X on internet speed tests.

  2. My web server W in city B regularly attains excellent upload speeds of Y on internet speed tests. (By coincidence, because city B is not too far from city A, sometimes the speed test service even assigns a server in city A to perform B's upload test.)

  3. Sometimes when transferring files from W to H, I attain speeds approaching the limit that would be logical given the speed tests: min(X, Y). Sometimes I attain speeds that are only 25% or 10% of that logical limit (which is still fine).

  4. Often, however, files are transferred at only 1% or even 0.1% of the logical limit; these have been as low as 1990s dial-up speeds. The perplexing thing is that during these episodes -- even WHILE A SLOW TRANSFER IS HAPPENING -- speed tests on both H and W continue to report the same high-quality speeds as always (X and Y). In addition, random transfers to/from commercial sites (like Google Drive or Proton Drive) also continue to show high-speed downloads at H and high-speed uploads at W. In other words, the issue (as far as I can see) only seems to affect transfers from W to H, and no other transfers involving those hosts. So I can't call either service provider and say "my transfer speeds are not as advertised" because they are as advertised for all other endpoints.

The episodes described in (4) are sometimes chronic (lasting for months), sometimes periodic (lasting for a day, once or twice per week), and sometimes transient (lasting 2-12 hours and then not coming back for weeks).

The episodes do not seem to happen at any specific time of day or day of the week; even the ones that occur a few days per week are not always on the same days.

The episodes do not seem to happen in response to any sort of transfer behavior on my part. Sometimes I can go several days without transferring any files and then have my first transfer attempt be slow, whereas other times I can transfer files near the logical speed limit for hours at a time without a dip in quality.

During the slow episodes I have tried routing the transfers through a VPN (on H's side), but that has not helped. So it does not seem like H's ISP is specifically targeting W.

All my transfers are performed over SSH, so in theory no file details can be read by either provider.

The files being transferred run the gamut from very small to gigantic. The times and days of the transfers are irregular and are not correlated to either the file size or the numbers of files being transferred.

This has been going on for around 15 years, including 5 separate ISPs at H's end (Comcast when it was still Comcast, a local provider, AT&T U-Verse, Xfinity bulk, and Xfinity individual), 2 hosting providers at W's end, various versions of Linux running on W, various versions of MacOS/OpenBSD running on H, and multiple types of router hardware at H's end.

Over the years my solution to the problem has essentially become to massively parallelize my file transfers when a slowness episode is occurring. So instead of doing one rsync, I will launch 10 or 20 rsyncs in parallel. Although I've never forensically examined the speed results of doing this, my general impression is that this speeds up the aggregate transfer rate linearly. In other words, my transfers appear to be throttled individually but not aggregately, and if I were to launch scores of them in parallel, I could use all of my expected bandwidth.

What in the world is going on? Is this traffic-shaping (by either my home ISP or my web-hosting provider)? Do they traffic-shape at 4:00am? Do they traffic-shape to unusably slow speeds?

::edit:: So ... no questions or suggestions on how to improve the post? Just a close vote because the problem is difficult, despite containing significant detail?

  • Beware of ISPs prioritising links to speed test sites.
    – Bib
    Apr 3 at 9:51
  • Massive parallelization is the normal way of maximizing throughput. ISP speed readings are known to not be accurate. Different transfer methods have different strengths and weaknesses. A given transfer will only run at the speed of the slowest element. You're running your connection through a variety of consumer-grade ISP connections which all tend to throttle upload (one side of your connection). What you are describing appears entirely within the bounds of normal. Apr 3 at 21:20
  • What is the media used on H side? DSL? Fiber? Are some segments the same between ISP? Apr 3 at 22:37


Browse other questions tagged .