1

I have a folder being synchronised (mirrored) remotely, using the synchronize -mirror -delete remote command, over FTP.

The folder I am synchronising contains 209,068 files, and 19,079 directories, and thus, even just to check that everything is synced, it takes ages.

The way it seems that WinSCP does it, is it takes 1 file a time, and checks if it is up to date. If it is not, it uploads it.

This process takes forever. 210K files, with a ping of 50ms at least, which means around 3 hours of just comparing files! Not even syncing. (not accounting for actual handling time even)

So I was wondering if perhaps there is a way to make it batch compare (ask for meta-data for like a 100 files at a time), or perhaps create a buffer of a 100 requests a time, that would go much faster (around 100 times faster), which would be 1.8 minutes instead of 3 hours.

If not, perhaps there is a different software I could use that performs better in this kind of task?

Edit: Why I think it does 1 file at a time? When running the sync command, this is the output I am seeing:

E:\sync\a.txt -> \Sync\a.txt

Less than a second after:

E:\sync\b.txt -> \Sync\b.txt

And so on, every file shows, in order of the directory, a part of a second after another

Edit2: Correction

  • When syncing a folder, it does go file by file.
  • When checking if a folder is synced, it goes folder by folder.

My previous information was incorrect, my apologies.

  • Keep in mind, you are still limited to FTP connection/command semantics when using WinSCP for FTP access. FTP commands are focused on a single file at a time. The only way to avoid that would be to host the files on another kind of server (perhaps SFTP which is not related to FTP) that WinSCP can work with. – Frank Thomas Feb 6 '17 at 20:59
  • @FrankThomas Ok, good to know, but can't it open several connections in parallel? About SFTP - I have no control over the device's firmware, as it is a WDMyCloud box, and thus I can not use SFTP – Amit Feb 6 '17 at 21:02
  • @FrankThomas Correction: I do have a way to use SFTP. Will it help for sure, or was it just a direction to try? – Amit Feb 6 '17 at 21:09
  • "takes 1 file a time" - Why do you think so? WinSCP should get the file data from directory listing. So there should be one request per whole directory, not per file. Show us a sample log, if you think otherwise. – Martin Prikryl Feb 6 '17 at 22:40
  • FTP uses command semantics like GET and PUT which operate on a single file. When WinSCP uses FTP as its target protocol, it strings together many FTP commands to do its job. The only exception is the MGET command which works well for files within a given directory that match a pattern, but a synch task would not be able to operate in this fashion. – Frank Thomas Feb 6 '17 at 23:30
1

Both ftp and sftp are very slow when transmitting large number of small files.

If it is possible (that is, if you have access to ssh shell), please try to use rsync, it should be way faster, especially, when some files were transferred earlier and only few of them were changed.

There are several implementations of rsync for Windows. I would recomment grsync, as it has a graphical interface.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.