Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'm downloading a large number of small files (over 800, each around 5k). I'm doing this with lftp's mget command. But it's rather slow. Each file takes roughly 1s. Is there a tool/way to speed this up?

Thanks for all the reply but I only have ftp access to this server

share|improve this question
    
If it is slow due to the number of files: Either tar/zip them and download them in one go, or use a stream (e.g. tar piped though netcat to untar) –  Hennes Apr 7 '13 at 11:04
    
@Hennes: can you tar them on the ftp server? How can you do ftp download through netcat? –  lang2 Apr 7 '13 at 11:05
    
Can you log in on the server running FTP? If you can try this: On the receiving host: nc -l 7654 | tar x, on the server which also runs ftp: tar cf - | nc -p receiving_host 7654. Mind you, that is not over FTP, but FTP needs do die. It is old, and should have been replaced decades ago. –  Hennes Apr 7 '13 at 11:09

1 Answer 1

If you have command line access to the server, log on and bundle them into one file with zip, gzip, bzip or tar (whichever is available, and can be reversed on the destination -- you didn't say what OSs are involved). This reduces the number of files _and _ the total number of bytes to be transferred.

If you only have FTP access to the server: some servers are configured to deliver a zip/gzip archive (which they build on the fly) if you just ask for it. E.g., if your files are in ftp://example.com/some/folder/, try fetching ftp://example.com/some/folder.zip. It might not work, but it's worth a try.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.