Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I often have to transfer 4000 small files to a remote server (around 20 MB of data), which I currently only have FTP access to. This operation takes 30 minutes but if I were to send one file sized 20 MB it would take 2 seconds or so. I am looking for an alternative which is faster, and which for example zips the 4000 files into one file, sends the zip file, and unzips it at the remote machine. Are there any file transfer servers that do this?

Note that I have access to installing the alternative server on the remote machine, so i am not limitted to the FTP server.

Thank you very much in advance Jeeji

share|improve this question

migrated from stackoverflow.com Oct 20 '10 at 18:08

This question came from our site for professional and enthusiast programmers.

1  
This should really be on Superuser, unless you are trying to do this in code - in which case, you need to put your code in the question. –  adam Oct 20 '10 at 13:32
1  
This should probably be moved to superuser.com indeed. It would be useful to say what kind of client/server environment you're using (Windows, Linux, ...)? –  Bruno Oct 20 '10 at 13:36
add comment

5 Answers

Zip them up and send them to the server and then use a scheduled task on the receiving side to find the zips and unzip them locally. Using a file system watcher you can detect when a zip is created, wait till it's done uploading (i.e., is no longer modified) and then unzip it.

share|improve this answer
add comment

You say you only have FTP access but then say you have the ability to install an alternate server. Can you install and use rsync?

Rsync will run over ssh and was designed to handle the "lots of tiny files" problem by sending them as a continuous stream of data and not as discrete chunks each with their own overhead.

share|improve this answer
add comment

Thanks for your suggestions. I thought of solving the problem myself as you suggest using a file system monitor that triggers when a zip file is copied. However, it requires some programming, since I need it to run as a service (so that it starts with no user logged on), and I need it to return some status of the result. All this is possible to program, but it would however be a lot easier if there is an off the shelf solution.

I will look into rsync. It sounds exactly as the application I am looking for.

share|improve this answer
    
Jihad - associate your Stack Overflow and Super User accounts and you'll regain ownership of the question –  ChrisF Oct 22 '10 at 18:29
add comment

You should have a look at weex or use rsync over the mountpoint created with CurlFtpFS

share|improve this answer
add comment

A simple shell script on the server end will take care of this for you. Pick a folder in which every zip file that goes in gets uncompressed. Write a script to find the zip files, and unzip them.

share|improve this answer
    
For flexibility, I'd write it so that it'd wait until it found a foo.zip and a foo.txt, where the foo.txt contains an absolute path stating where to unload the .zip. Put some marker in the .txt file so the script won't try to read an only half-uploaded txt and put the zip in some screwed up location. –  Jonathan Hobbs Oct 21 '10 at 5:09
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.