Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I've got a problem with syncing big amount of data between two data centers. Both machines have got a gigabit connection and are not fully occupied but the fastest that I am able to get is something between 6 and 10 Mbit => not acceptable!

Yesterday I made some traceroute which indicates huge load on a LEVEL3 router but the problem exists for weeks now and the high response time is gone (20ms instead of 300ms).

How can I trace this to find the actual slow node? Thought about a traceroute with bigger packages but will this work?

In addition this problem might not be related to one of our servers as there are much higher transmission rates to other servers or clients. Actually office => server is faster than server <=> server!

Any idea is appreciated ;)

We actually use rsync over ssh to copy the files. As encryption tends to have more bottlenecks I tried a HTTP request but unfortunately it is just as slow.

We have a SLA with one of the data centers. They said they already tried to change the routing because they say this is related to a cheap network where the traffic gets routed through. It is true that it will route through a "cheapnet" but only the other way around. Our direction goes through LEVEL3 and the other way goes through lambdanet (which they said is not a good network). If I got it right (I'm a network intermediate) they simulated a longer path to force routing through LEVEL3 and they announce LEVEL3 in the AS path.

I basically want to know if they're right or they're just trying to abdicate their responsibility. The thing is that the problem exists in both directions (while different routes), so I think it is in the responsibility of our hoster. And honestly, I don't believe that there is a DC2DC connection which only can handle 600kb/s - 1,5 MB/s for weeks! The question is how to detect WHERE this bottleneck is

share|improve this question
up vote 4 down vote accepted

If you are being routed through a slow link on the public Internet, pretty much your only options are to forcibly route yourself around it. The simplest way to do this is to attempt a file transfer between two endpoints, one of them being "point A" (the origin of the data) and an intermediate site that is not geographically co-located with your destination, "point B".

Once you find a "point C" which is a server that does not get routed over the slow Internet router you're facing, you can set up a VPN between point A and point C, so that the traffic gets "routed around" the slow node.

If you have high business value ($$$$$$) or clout with the ISP, you could also take up the issue directly with Level 3. However, L3 is a Tier 1 ISP and might not be particularly receptive to complaints about service quality or network saturation, since there is very little they can do about this if they can't, won't, or are unable to expand their peering agreements with the downstream or other Tier 1 providers who are creating the contention on their node.

Since you said the "office to server" link is faster, you could try setting up the VPN at the "office" site with a moderately powerful computer (a dual core server-grade system should be fine).

Oh, also! If the latency (end to end) between "point A" and "point B" is very high (greater than 100 ms is high in the server world), you should make sure that you are not using a chatty network protocol. Samba (also known as SMB or Windows File Sharing) is extremely chatty; other "sync" protocols may also be chatty.

Chatty protocols are those which require a lot of synchronous "back and forth" round trips in order to transfer data. If your protocol is too chatty, then latency alone can bottleneck your transfer, regardless of how fast the link is capable of.

One thing you can do to determine whether chattiness is really affecting your throughput is by using a known un-chatty protocol, such as HTTP, for a test transfer. So, try regular old HTTP from "point A" to "point B" over the "slow" Level3 router, and if the latency is high but the throughput is still good, then you know that the reason your transfer is slow is that your protocol is too chatty, so you need to change protocol.

So let me round out the discussion by briefly defining and explaining the three network impairments and why any one of them can be responsible for this issue:

  • Latency -- How long a datagram takes to get from your end to the other end. You can't directly improve latency in most cases, unless one of your computers is so overloaded that its networking stack, kernel, or applications are generating significant additional latency. Most latency over the public Internet originates from Internet routers, not from your computer or the endpoint.

  • Bandwidth -- Bandwidth is the maximum throughput of the slowest link between your computer and the endpoint. In most modern networks, bandwidth is not a real restriction, because other network impairments set in and slow down the network long before bandwidth is a real issue.

  • Packet Loss -- Packet loss can increase perceived latency for reliable datagrams (such as TCP), and is often the result of heavily saturated links having to drop your packet out of the TCP transmit or receive buffer due to the buffer being too full already. Also, packet loss can occur with "time-sensitive" packets, as is the case with almost all TCP packets, because if the packet arrives after the deadline, then it is discarded. This occurs if a larger TCP packet is fragmented into multiple IP datagrams, and the TCP protocol on the receiving side can only wait a fixed amount of time for all the fragments to arrive, before deciding to abort receipt of the packet. So packet loss is indirectly derived from issues of saturation (which is a bandwidth problem), or also from hardware problems or failures.

Derived from the fundamental network impairments, are mitigations you can take to improve the reliability of your programs without changing the fundamental impairments, because most of the time, there is little or nothing you can do to control them:

Mitigation one is to make your protocol less chatty (or, from a systems integration perspective, use an existing protocol that is less chatty than your current solution). The fewer "round trips" are required to synchronize data between the endpoints, the better off you'll be -- period. Some protocols can be engineered to require a variable frequency of synchronization -- if this is the case, you should dynamically back off the synchronization frequency as much as possible if you detect high latency or packet loss. Reducing chattiness helps mitigate latency and packet loss, but not bandwidth ceiling problems.

Mitigation two is to configure all of your hops (the ones that you directly control at an administrative / hardware level) to use the best available Active Queue Management (AQM) algorithm, which at present is Fair Queue Controlled Delay AQM. This is available in Linux kernel 3.5 or later as the fq_codel qdisc implementation, and what it does is it dynamically reduces the size of the transmit and receive buffers, in order to reduce the latency that these buffers invariably produce. This can reduce packet loss and help cope with latency using the TCP protocol, because your fragmented packets are less likely to expire if you minimize the amount of waiting that the packet has to go through before it is sent over the link. Note that this mitigation only makes any difference if the node is "saturated" (i.e., if the TCP buffer is empty, it has no effect). A node is saturated whenever the rate of data writes into the network socket exceeds the transmission rate of the uplink. The typical response of the TCP stack to this situation is to make the buffer larger, which actually has a negative effect, because it increases latency, and that causes all kinds of problems -- so fq_codel helps to mitigate that.

Both of these mitigations help with all three of the fundamental network impairments, without routing around the faulty node, and without changing any hardware or contacting the ISP.

share|improve this answer
First, thank you for this detailed answer! I edited my question and added some more info (forgive me that I didn't directly). So the protocol isn't the problem. We can upload with 12 Mbit from our office so it isn't really faster. 1 MB/s is too slow anyway as we move gigabytes per day so we would like to use at least 10% of our gigabit capacity ;) – 2called-chaos Nov 2 '12 at 16:33
From an endpoint, you probably can't really tell which exact node is responsible, because intermediate routers often block ICMP or otherwise restrict your ability to probe them (to protect themselves from flooding, of course). You'd have to be in the internet infrastructure business itself. – allquixotic Nov 2 '12 at 17:25

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.