Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am trying to transfer a large (~3GB) file between two unix machines.

I can use scp or rsync, but sometimes the transfer is corrupted. (I have to check manually.) I can split the file into pieces and transfer them and then checksum and then recombine, but this is tedious.

Is there a single command to correctly transfer a large file between two Unix machines? I want it to automatically checksum both copies, and keep redoing the transfer (or pieces thereof) until it gets all bytes across the wire correctly.

share|improve this question

Rsync works by dividing a file into chunks and only transferring the chunks if they're different. By its very nature, it should be able to detect and fix corruption. Have you tried using the --partial option, which will let it continue if it gets interrupted, and re-running the command repeatedly until it no longer transfers any data?

Are you using an rsync server on the remote end of the connection? If not, then you're not actually using rsync to transfer the file, it's just using whatever underlying transport method you're using, so you won't have the error detection.

share|improve this answer
If you are using rsync then you are using rsync - if there is no server daemon listening then it will try login via SSH, start an rsync process to act temporarily as the "server" end, and tunnel the rsync protocol the the SSH stream. Remember: when using rsync to recheck possible corrupt files against a (presumably) known-good master copy, use the --checksum option to force it to check the contents of everyfile instead of taking the shortcut of assuming files with the same size+timestamp are OK. – David Spillett Oct 10 '09 at 11:01
That's very true, unless he's NFS or SAMBA mounted the remote disk and is specifying it as if it's a local file, in which case any corruption below NFS or SAMBA won't be detected. But he didn't specify. – Randy Orrison Oct 10 '09 at 13:44
There is NFS on the local client, but I am copying through rsync ssh to the remote client (which is not NFS mounted). – user13798 Oct 10 '09 at 18:22
In which case try --checksum as David Spillett suggests. ~quack's script that passes the md5sum also looks good. – Randy Orrison Oct 10 '09 at 19:13

Use the -c option (checksum) in rsync.

rsync -azcvPh file1 user@remotehost:/tmp/

The -P option display progress stats and will help you understand where/when your file transfer breaks. The -h makes it "human readable" and the -z compresses.

share|improve this answer
-P also implies --partial, and that is good to have in this case as well. – amarillion Jan 8 '13 at 10:50

You are using the best commands but your computer is broken. I highly recommend to run memtest86+ on both machines over night to check you RAM.

share|improve this answer
May want to check for failing harddrives too... – retracile Oct 10 '09 at 17:53

You could always bittorrent it between the hosts, but I'm not sure how to automate it offhand.

I don't do this kind of thing often so I wouldn't script it. Instead I'd build up a big one-liner to do the job. So technically this isn't one command, but it's all on one line. It's not hard to script-ify if you do this often.

$ md5sum bigfile > bigfile.md5 ; export BIGFILE="notdone" ; while [ "$BIGFILE" eq "notdone" ] ; do rsync --checksum --partial bigfile* user@remotehost:path/to/put/it/in/ ; ssh user@remotehost "cd path/to/put/it/in/; md5sum -c < bigfile.md5" | grep -Ev 'OK$' | [ `wc -l` == "0" ] && BIGFILE="done" ; done

This splits out into:

$ md5sum bigfile > bigfile.md5 ;         \  # create our own checksum
  export BIGFILE="notdone" ;             \  # set our check variable
  while [ "$BIGFILE" == "notdone" ]; do  \  # recheck variable state after each pass
     rsync --checksum --partial bigfile* \  # call rsync to copy
          user@remotehost:path/to/put/it/in/ ; \  # and call ssh to check
     ssh user@remotehost                 \  # connect with ssh
       "cd path/to/put/it/in/; md5sum -c < bigfile.md5" \ # and run the check
          | grep -Ev 'OK$'               \  # ignore good output
          | [ `wc -l` == "0" ]           \  # if we didn't find one
            && BIGFILE="done" ;          \  # set our get-out-of-jail card
  done                                   \  # and we're done

You have to setup SSH to login to your host with key authorization to run it without interaction. If you do, put an echo statement in there to tell you where it is.

Tested, but I expect the rsync options could be tweaked.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .