Folder transfer over poor sat link

Question

Does anyone know of a tool (exe or script etc.) to manage transfer of large folders over a poor network connection? It would need to be able to handle very slow speeds and network outages, and keep trying until it succeeeded.

The scenario being I have a CD (in the CD drive or copied to a HARD Drive) and I need to be able to copy to a remote computer. I can then leave it overnight or whatever, and in the morning it has copied all the data across.

The facilities I need are:

1. Standalone, i.e. no installation required
2. Command line operation, returning an error code, optionally silent
3. Allow Transfer folder or Drive letter
4. No reasonable restriction on folder size (at least up to 100GB)
5. Split large files into smaller chunks (max size specified by user)
6. Resumable (if stopped, the tool will be able to continue where it left off, not necessarily files copied in resumable mode)
7. Another command line tool to reconstitute the split files the other end
8. Must use standard windows copying
9. Checksums produced at either end on completion of copy

To copy the D drive on this machine to the share on remotepc splitting files larger than 1000K, I'd imagine a command line like:

SlowButSureCopy.exe D:\ \\remotepc\share 1000

Any ideas, or will I have to write it?

-Frink

Answer 1

I don't know of any tools that will do exactly what you want.

What I can tell you though, is that there are other ways to get where you want to go, but you'll have to decide which items are most important to you. Since it looks like what you really need is a way to transfer files reliably over an unreliable network link, the most important pieces of the puzzle are hashing/checksumming, resumability, and partial-file granularity. The top 2 options I can think of off the top of my head are:

1. rsync - There are portable/standalone implementations available, can handle copying large amounts of data efficiently, and you can loop it manually via a batch file to keep copying until it finishes successfully.
2. BitTorrent - What you can do is use uTorrent to create a .torrent file of the files you want to use using uTorrent's built-in tracker, send the .torrent file to the other system, and use uTorrent on the other system to download the contents.

Both of these options sacrifice some raw speed for higher reliability, but it sounds like making sure that all the bits get copied correctly is more important than saving a few minutes of transfer time.

RoboCopy and RichCopy are your best options that use the "normal" Windows copying routines, but I don't think that either one would necessarily be the best option in this situation.

Answer 2

You might be able to write a batch file around some of the features of the Swiss File Knife command-line program. I've used it to split and transfer large MS-SQL database dumps (9-20GB)

Some of the inbuilt commands/features include:

• sfk copy - copy directory trees additively
• sfk sync - mirror tree content with deletion
• sfk partcopy - copy part from a file into another one
• sfk script - run many sfk commands in a script file
• sfk split - split large files into smaller ones
• sfk join - join small files into a large one
• sfk ftpserv - run an instant FTP server
• sfk ftp - instant anonymous FTP client
• sfk md5gento - create list of md5 checksums over files
• sfk md5check - verify list of md5 checksums over files
• sfk md5 - calc md5 over a file, compare two files