0

I often have to transfer 4000 small files to a remote server (around 20 MB of data), which I currently only have FTP access to. This operation takes 30 minutes but if I were to send one file sized 20 MB it would take 2 seconds or so. I am looking for an alternative which is faster, and which for example zips the 4000 files into one file, sends the zip file, and unzips it at the remote machine. Are there any file transfer servers that do this?

Note that I have access to installing the alternative server on the remote machine, so i am not limitted to the FTP server.

Thank you very much in advance Jeeji

2
  • 1
    This should really be on Superuser, unless you are trying to do this in code - in which case, you need to put your code in the question.
    – adam
    Oct 20, 2010 at 13:32
  • 1
    This should probably be moved to superuser.com indeed. It would be useful to say what kind of client/server environment you're using (Windows, Linux, ...)?
    – Bruno
    Oct 20, 2010 at 13:36

5 Answers 5

2

You say you only have FTP access but then say you have the ability to install an alternate server. Can you install and use rsync?

Rsync will run over ssh and was designed to handle the "lots of tiny files" problem by sending them as a continuous stream of data and not as discrete chunks each with their own overhead.

1

Zip them up and send them to the server and then use a scheduled task on the receiving side to find the zips and unzip them locally. Using a file system watcher you can detect when a zip is created, wait till it's done uploading (i.e., is no longer modified) and then unzip it.

1

You should have a look at weex or use rsync over the mountpoint created with CurlFtpFS

1
  • 1
    Good idea, I tried the same solution yesterday. CurlFtpFS is painfully slow, which makes very impratical to use cp or rsync (at least the first full copy) over there. But the solution is clean and worth a try anyway.
    – Paolo
    Aug 25, 2015 at 18:44
0

Thanks for your suggestions. I thought of solving the problem myself as you suggest using a file system monitor that triggers when a zip file is copied. However, it requires some programming, since I need it to run as a service (so that it starts with no user logged on), and I need it to return some status of the result. All this is possible to program, but it would however be a lot easier if there is an off the shelf solution.

I will look into rsync. It sounds exactly as the application I am looking for.

1
  • 2
    Jihad - associate your Stack Overflow and Super User accounts and you'll regain ownership of the question
    – ChrisF
    Oct 22, 2010 at 18:29
-1

A simple shell script on the server end will take care of this for you. Pick a folder in which every zip file that goes in gets uncompressed. Write a script to find the zip files, and unzip them.

1
  • For flexibility, I'd write it so that it'd wait until it found a foo.zip and a foo.txt, where the foo.txt contains an absolute path stating where to unload the .zip. Put some marker in the .txt file so the script won't try to read an only half-uploaded txt and put the zip in some screwed up location. Oct 21, 2010 at 5:09

You must log in to answer this question.