Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

So why are FTP uploads soo slow? I am using filezila as a client.

I have like 10 mb in 1000 files and I can upload each individual file with 300-500kb/s yet the whole upload is incredibily slow due to the queueing process that occurs as files are uploaded. For every singe file the client performs all kind of commands and connection operations before actually uploading.

Is there no way to skip over these commands? I am new to ftp clients/uploads/websites etc Is this standard practice? Is this just the way ftp uploads work? Don't you get bored waiting like 20 minutes for 8-10 mb?

How can I efficiently upload 100 mb or more?

share|improve this question

migrated from stackoverflow.com Jun 13 '11 at 18:07

This question came from our site for professional and enthusiast programmers.

    
Transferring a large number of small files is much less efficient for any protocol, not just FTP. It's just much more noticeable over the internet (or a WAN) because the latency makes the command overhead swamp the actual transfer rate. –  afrazier Jun 14 '11 at 17:59
    
@afrazier theoretically not. It depends on the nature of the protocol in use, which doesn't support pipelining. For example, pipelined http was designed to reduce latency –  djechelon Oct 21 at 20:05
    
@djechelon: Pipelining helps, but you still have the problem of command overhead and most pipeline implementations limit the number of active simultaneous requests (because overhead can still swamp the payload eventually). –  afrazier Oct 21 at 20:14

4 Answers 4

Sadly, this is the way that FTP functions. To efficiently transfer lots of small files, either archive them locally, transmit the entire archive via ftp, and then unarchive the files on the remote machine, or turn on simultaneous uploads, where the client is uploading 10 files at once. This will help to fully saturate your upload link.

share|improve this answer
    
with an avg. of 100kb per file they are not small files. –  Captain Giraffe Jun 13 '11 at 17:53
    
@Captain Giraffe: On most WAN/Internet links, 100 kB is still small enough that latency swamps throughput when it comes to uploading. –  afrazier Jun 14 '11 at 17:56

Have you tried compressing the files locally then uncompressing them on the server? Then you'd only have to transfer one small(er) file.

If it's applicable, you could also only copy files that have changed since your previous upload. Tools like rsync (if you have ssh access) and robocopy (if it's a windows server) could help you do this.

share|improve this answer

I use Auto FTP Manager. It runs multiple simultaneous FTP transfers so the entire bandwidth link can be used:

Auto FTP Manager makes it easy to schedule and automate your FTP transfers. Use Auto FTP Manager to connect to any FTP server and automatically upload and download files. Plan and automate your workflow. Let your PC move or synchronize files between PC to FTP Server, PC to PC, and FTP Server to FTP Server, automatically according to a schedule...

... Auto FTP Manager is multi-threaded, allowing you to open connections to multiple FTP servers at the same time. The program can transfer files in the background while you work on other tasks.

share|improve this answer

I know this is an old post, and there isn't a lot you can do at the application layer since FTP relies on TCP for sending the bytes. There are few things though

  1. Enable simultaneous uploads as mentioned above (this helps utilize more of your bandwidth since it takes time for TCP to "ramp up" each connection, so doing it in parallel is more efficient)
  2. Archive the file as mentioned above (again, because it allows TCP to ramp up over time instead of individually for each file)
  3. Check out SuperTCP: it's a new product that (full disclosure) I'm helping build that will optimize TCP upload, which can help FTP go faster in a lot of cases. We're launching a Beta soon and would love to have you help us test it!
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.