As a user in Europe, what is the download speed I should expect from a service powered by AWS/CloudFront? At what point should I report a slowness and to whom?
For WeTransfer, from an example link I get to an example download URL (found in network console, F12). I then use iftop
to see what host is serving the file to me and mtr
to see if any obvious problem stands out (though the traceroute from their host to my machine can be different from the other way).
Yesterday, the file was served from CloudFront's Madrid edge, something like server-54-192-61-242.mad50.r.cloudfront.net
, and my download speed didn't go beyond 300 KiB/s, staying at 150-200 KiB/s most of the time. That's terribly slow.* I did not save the traceroute but there was no obvious packet loss or latency; IIRC packets went through Telia.
Today, the file is served from server-54-240-166-250.lhr5.r.cloudfront.net
(London) and I get 1.1 MiB/s at home, 13 MiB/s average (and 25 MiB/s peak) on a Northern Europe server. This is what I expect.
Given Amazon/AWS changed the host from yesterday and now things work, it seems even more likely that the problem was with them. However, the AWS client on The download speed is slow says they won't do anything. CloudFront docs and AWS forums have no information on how to report networking/routing/peering issues. What to do in such cases then? I guess only the AWS client is in the position to get something done, but only if the person who receives the report is able to understand networking.
My traceroute to CloudFront Madrid is something like this:
10.|-- 62-101-124-129.fastres.net 0.0% 50 4.6 13.8 3.5 101.1 20.3
11.|-- 89.96.200.21 0.0% 50 17.6 16.6 2.6 92.9 22.0
12.|-- mno-b2-link.telia.net 4.0% 50 52.6 26.3 13.1 69.2 13.7
13.|-- mei-b1-link.telia.net 0.0% 50 23.7 30.3 20.4 87.7 11.3
14.|-- bcn-b2-link.telia.net 0.0% 50 47.5 53.7 30.2 92.9 16.4
15.|-- mad-b2-link.telia.net 0.0% 50 62.7 57.7 36.1 102.2 14.4
16.|-- mad-b1-link.telia.net 0.0% 50 37.7 42.1 34.3 59.8 5.6
17.|-- a100-ic-314004-mad-b1.c.telia.net 0.0% 50 70.2 58.5 39.7 87.2 12.5
18.|-- ??? 100.0 50 0.0 0.0 0.0 0.0 0.0
19.|-- ??? 100.0 50 0.0 0.0 0.0 0.0 0.0
20.|-- ??? 100.0 50 0.0 0.0 0.0 0.0 0.0
21.|-- server-54-192-61-242.mad50.r.cloudfront.net 2.0% 50 71.1 83.5 56.4 156.2 19.5
The traceroute is now something like this:
10.|-- 62-101-124-94.fastres.net 0.0% 50 68.6 79.5 36.1 108.8 15.4
11.|-- 89.96.200.110 0.0% 50 75.9 94.8 46.0 141.8 17.6
12.|-- ??? 100.0 50 0.0 0.0 0.0 0.0 0.0
13.|-- 54.239.4.248 2.0% 50 107.2 112.9 71.6 146.7 18.2
14.|-- 54.239.41.135 0.0% 50 112.8 108.7 72.8 147.6 15.0
15.|-- 178.236.3.22 0.0% 50 115.8 102.3 58.4 127.9 16.9
16.|-- 176.32.106.11 4.0% 50 95.8 103.2 73.7 130.7 14.2
17.|-- 176.32.106.11 40.0% 50 110.6 108.6 80.4 136.1 14.7
18.|-- ??? 100.0 50 0.0 0.0 0.0 0.0 0.0
19.|-- ??? 100.0 50 0.0 0.0 0.0 0.0 0.0
20.|-- server-54-240-166-250.lhr5.r.cloudfront.net 60.0% 50 88.7 100.0 57.6 131.9 18.0
As the first answer notes, it matters a lot whether the file was already cached on the CloudFront edge or not. Here is an example of a cache miss (which right now manages to saturate my bandwidth):
$ LANG='en' wget -S 'https://download.wetransfer.com/wetransfer-eu1/f7a2031249f56fdeeda9040adda5a26f20160224143804/wetransfer-f7a203.zip?expiration=1456605646&escaped=false&signature=3d916716d49e415f637b4f824c7709f7483b67a8f02588caece30d6c2a3ed0ea&filename=wetransfer-f7a203.zip'
--2016-02-27 21:34:39-- https://download.wetransfer.com/wetransfer-eu1/f7a2031249f56fdeeda9040adda5a26f20160224143804/wetransfer-f7a203.zip?expiration=1456605646&escaped=false&signature=3d916716d49e415f637b4f824c7709f7483b67a8f02588caece30d6c2a3ed0ea&filename=wetransfer-f7a203.zip
Resolving download.wetransfer.com (download.wetransfer.com)... 54.192.61.62, 54.192.61.196, 54.192.61.80, ...
Connecting to download.wetransfer.com (download.wetransfer.com)|54.192.61.62|:443... connected.
HTTP request sent, awaiting response...
HTTP/1.1 200 OK
Content-Type: application/octet-stream
Content-Length: 1449534395
Connection: keep-alive
Server: nginx
Date: Sat, 27 Feb 2016 20:34:39 GMT
Content-Transfer-Encoding: binary
Content-Encoding: none
Cache-Control: private, no-transform, no-store
Allow: GET, HEAD
Accept-Ranges: bytes
Content-Disposition: attachment; filename="wetransfer-f7a203.zip"
X-Transfer-Id: f7a2031249f56fdeeda9040adda5a26f20160224143804
X-Cache: Miss from cloudfront
Via: 1.1 943ab292a0096b706fe263560805857e.cloudfront.net (CloudFront)
X-Amz-Cf-Id: 4hEZcZL56GWMBn8z1T2txF-O3TTdrAC6OxCtqVDZUoJREUd9_EBo6A==
Length: 1449534395 (1.3G) [application/octet-stream]
On further testing, I always gewt X-Cache: Miss from cloudfront
, even at the 6th time I request the same resource, so it seems that WeTransfer isn't caching anything at CloudFront (or not files of this size). Interestingly, the X-Transfer-Id: f7a2031249f56fdeeda9040adda5a26f20160224143804
header is always the same although the actual download URL I get from clicking the download button varies; the Via
and X-Amz-Cf-Id
headers also vary. As of this update, the first time I request a given download URL is very fast, the second very slow, the third 404s. I tried and I can have two simultaneous downloads, one at the second attempt and one at the first attempt: the first will be very slow and the latter very fast, although the networking conditions are clearly the same.
See https://paste.debian.net/408552/ for a test from my Northern Europe server: download A* are one URL, B* another; A-2 is after A-1 and B-2 is after B-1, but B* started while A-2 was running. Yet A-1 and B-1 were very fast, A-2 and B-2 very slow.
This is increasingly looking like an issue with quality of service/QoS aka throttling. Can CloudFront throttle me with cache misses, or should we only blame their client?
(*) Note: I have a 10/10 Mb/s FTTH connection with Fastweb. The available bandwidth never goes under this guaranteed speed. The ISP is not known to apply QoS throttling, but does sometimes have some routing issues outside Italy. When I observed the problem, I didn't have any problem saturating my bandwidth with other services.
25MiB/s
if you have a10Mb/s
line?