Single unix command to correctly transfer a large file?

Question

I am trying to transfer a large (~3GB) file between two unix machines.

I can use scp or rsync, but sometimes the transfer is corrupted. (I have to check manually.) I can split the file into pieces and transfer them and then checksum and then recombine, but this is tedious.

Is there a single command to correctly transfer a large file between two Unix machines? I want it to automatically checksum both copies, and keep redoing the transfer (or pieces thereof) until it gets all bytes across the wire correctly.

Answer 1

Rsync works by dividing a file into chunks and only transferring the chunks if they're different. By its very nature, it should be able to detect and fix corruption. Have you tried using the --partial option, which will let it continue if it gets interrupted, and re-running the command repeatedly until it no longer transfers any data?

Are you using an rsync server on the remote end of the connection? If not, then you're not actually using rsync to transfer the file, it's just using whatever underlying transport method you're using, so you won't have the error detection.

Answer 2

Use the -c option (checksum) in rsync.

rsync -azcvPh file1 user@remotehost:/tmp/

The -P option display progress stats and will help you understand where/when your file transfer breaks. The -h makes it "human readable" and the -z compresses.

Answer 3

You are using the best commands but your computer is broken. I highly recommend to run memtest86+ on both machines over night to check you RAM.

Answer 4

You could always bittorrent it between the hosts, but I'm not sure how to automate it offhand.

I don't do this kind of thing often so I wouldn't script it. Instead I'd build up a big one-liner to do the job. So technically this isn't one command, but it's all on one line. It's not hard to script-ify if you do this often.

$ md5sum bigfile > bigfile.md5 ; export BIGFILE="notdone" ; while [ "$BIGFILE" eq "notdone" ] ; do rsync --checksum --partial bigfile* user@remotehost:path/to/put/it/in/ ; ssh user@remotehost "cd path/to/put/it/in/; md5sum -c < bigfile.md5" | grep -Ev 'OK$' | [ `wc -l` == "0" ] && BIGFILE="done" ; done

This splits out into:

$ md5sum bigfile > bigfile.md5 ;         \  # create our own checksum
  export BIGFILE="notdone" ;             \  # set our check variable
  while [ "$BIGFILE" == "notdone" ]; do  \  # recheck variable state after each pass
     rsync --checksum --partial bigfile* \  # call rsync to copy
          user@remotehost:path/to/put/it/in/ ; \  # and call ssh to check
     ssh user@remotehost                 \  # connect with ssh
       "cd path/to/put/it/in/; md5sum -c < bigfile.md5" \ # and run the check
          | grep -Ev 'OK$'               \  # ignore good output
          | [ `wc -l` == "0" ]           \  # if we didn't find one
            && BIGFILE="done" ;          \  # set our get-out-of-jail card
  done                                   \  # and we're done

You have to setup SSH to login to your host with key authorization to run it without interaction. If you do, put an echo statement in there to tell you where it is.

Tested, but I expect the rsync options could be tweaked.