I would like to transfer files between two places over the internet. Right now I have a VPN and I am able to browse, download and transfer files. So my question is not really how to transfer the files; Instead, I would like to use the most efficient approach because the two places constantly share a lot of data.

The reason why I want to get rid of the VPN is because it is two slow. Having high upload speed is very expensive/impossible in residential places so I would like to use a different approach.

I was thinking about using programs such as http://www.dropbox.com . The problem with Dropbox is that the free version comes with only 2 GB of storage. I think the deals they offer are OK and I might be willing to pay to get that increase in speed. But I am concerned with the speed of transferring data. Dropbox will upload the file to their server then send it from the server to the other location. I would like it to be even faster.

Anyway I was thinking why not create a program myself. This is the algorithm that I was thinking of. Let me know if it sounds too crazy.

(Remember my goal is to transfer files as fast as possible)

Things that I will use in this algorithm:

  • Server on the internet called S (Has fast download and upload speed. I pay to host a website and some services in there. I want to take advantage of it.)
  • Client A at location 1
  • Client B at location 2

So lets say at location 1, 20 large files are created and need to be transferred to location 2.

  • Client A compresses the files with the highest compression ratio possible.
  • Client A starts sending data via UDP to client B.
  • Because I am using UDP I will include the sequence number on each packet.
  • Have server S help speed up things. For example every time a packet is lost we can use Server S to inform client A that it needs to resend a packet.

Anyways I think this approach will increase the transfer rate. I do not know if it is possible to start sending data while it is being compressed. Or if it is possible to start decompressing data even if we are not done receiving the whole file. Maybe it will be faster to start sending the files right away without compressing. If I knew that I will always be sending large text files then I will obviously use the compression. I need this as a general algorithm.

So I guess my question is could I increase performance by using UDP instead of TCP and by using an extra server to keep track of lost packets? And how should I compress files before sending? Compressing a 1 GB file with the highest compression ratio takes about 1 hour! I would like to take advantage of that time by sending it as it is being compressed.

  • Why is the VPN slow? How would getting rid of the VPN speed up your internet upload at home? Also Dropbox doesn't offer faster speeds for paid accounts, only more data. Furthermore, you don't have to care about TCP or UDP in this case, the speed difference is probably negligible on a network like the internet. rsync or a similar tool is a good start as stated below. – Tim Sep 5 '12 at 10:01

Just use rsync. It's very efficient in its use of TCP, it transfers just the files that have changed, and you can choose to use compression if you want.

You're not going to find a way to beat an app that uses TCP efficiently. TCP has been defined and refined over the last ~40 years by a lot of really, really smart people, so it's very hard to beat.

Modern TCP stacks that support SACK are very efficient at detecting and reporting lost packets to each other so just the lost packets get retransmitted. Putting a server in the middle won't speed anything up, it'll just add latency.

The only way you can really beat TCP at data transfer performance is by making network congestion worse. TCP goes as fast as possible, but backs off momentarily when there are signs of congestion. If you created a UDP-based protocol that didn't care how much it added to the congestion of any links it traverses, then you could blast out packets that cause lots of congestion problems but might on average get slightly higher throughput than TCP can.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.