Tell me more ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.
up vote 7 down vote favorite

There is one large file in my server. I find that multi thread download can get 20Mbs, but single thread can get 10Mbps, can anyone explain this?

share|improve this question
Multiple threads servicing the same TCP connection, or multiple threads each with separate TCP connections? Also are you saying the server is multithreaded, or the client is multithreaded, or both? – Spiff Mar 25 '11 at 5:40

migrated from serverfault.com Mar 24 '11 at 3:01

4 Answers

up vote 8 down vote

Usually this is because somewhere between you and the other server there's a firewall limiting each HTTP stream to 10Mbps. When you use multi-thread, you get 2x 10Mb (one for each thread).

share|improve this answer
1  
I use FTP and there is no limiting on my server – why Mar 24 '11 at 6:17
@why: maybe it's your ISP that is capping each connection to 10mbps? Can you get more than that in a speed tester? – André Paramés Mar 24 '11 at 12:33
up vote 3 down vote

This is due to your ping between you and the server and packet size/tcpip window size used by your downloading software.

Basically, if you have 100ms ping to the server, and request packets of 100kb, you can only get 10 packets per second using 1 connection, even if your internet speed is infinite.

share|improve this answer
You don't need to ACK each packet, as long as receiver is emptying its buffer at a reasonable rate, the sender should be able to pump them continually. – André Paramés Mar 24 '11 at 12:30
That's right. But even with 256kb buffer, ping still causes massive slowdown – BarsMonster Mar 24 '11 at 13:15
up vote 2 down vote

Well, that's probably because you can only transfer so much data over one connection. However in a multi-threaded program you can have two connections receiving data at the same time and doubling the amount of information you can obtain. There are some limitations to this for example the speed of the server you are downloading from... Hats off two whoever wrote the multi-threaded downloader, those are not easy to write.

share|improve this answer
Why is it so difficult? You just need to allocate an individual section for each thread and let it write in the appropriate section of the result file. The source of axel seems pretty straightforward to me. – André Paramés Mar 24 '11 at 12:19
up vote 1 down vote

TCP works best when you "keep the pipe full" — when the sending app keeps sending buffers quickly enough to keep the sender TCP stack constantly supplied with data so that it can always have data "in flight" on the network, and when the receiver app keeps reading from the receiver TCP stack fast enough that the receiver TCP window never fills up (again, so the sending TCP stack can always keep data "in flight" on the network).

I could imagine a poorly-written single-threaded sender app that passes one buffer to the TCP stack, waits to hear that it's been fully Acked, and then passes another buffer. That means that once the end of the first buffer is "in flight" on the network, the sending TCP stack is starved for data to send, which means the pipe drains and isn't refilled until after the Ack comes back and the sending app passes it a new buffer.

I could also imagine a poorly-written single-threaded receiver app that doesn't read from the receiving TCP stack fast enough and thus lets the TCP stack's buffers fill up, which means the TCP window fills up, which causes the sending TCP stack to stop sending until the window opens up some. Increasing the receiver's TCP window size may help a little, but the real solution on this end is to read the data faster.

share|improve this answer
So, it might have nothing to do with being single threaded? – Ape-inago Mar 25 '11 at 7:23
@Ape-inago Sure, a well-written single-threaded app could keep the pipe full, yes. – Spiff Mar 25 '11 at 9:02

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.