3

I'm working on transferring a lot of files (>100GB, several thousands of files) over my network to a new Mac. Once the transfers are done I'd like to be able to verify that all of the files were successfully transferred and that no corruption occurred during the process.

The files are coming from a FreeNAS (open source NAS based on FreeBSD) server share. They're being loaded onto the Mac file system.

So far the best solution I can think of is running ls -aR into a file for the share and then the local disk, then diffing the two files. Are there better solutions? Optionally, but even better, is there a way to do this which would hash the files to make sure that all the data was successfully transferred?

In terms of my computer skills, I'm comfortable using terminal applications so there's no need to recommend only GUI tools.

Improve this question
6

One word. rsync.

Improve this answer
3
  • 1
    rsync is the answer. You especially want the checksum option, -c, which you can use to do checksum validations after you have copied the file. – ChrisInEdmonton Dec 21 '09 at 4:17
  • +1 for rsync, as soon as I read the subject of this post, I thought rsync ;) – Jakub Dec 21 '09 at 16:04
  • @ChrisInEdmonton -c is for a checksum check BEFORE the file is copied, to see if it actually has to be copied or not (i.e. it already exists), right? – David Doria Mar 18 '14 at 14:13
1

As SleighBoy mentioned, Rsync is the standard tool for this sort of thing.

Your suggestion for diffing the output of ls on both systems wouldn't work, because while it would verify that the files were present in both locations it would do nothing to verify the integrity of the data. If you were to do this manually, you would need to generate a checksum for each file on both systems and then verify that the checksums match. There are tools out there to do this for you (radmind has tools that make this easy), but it's easier to just use Rsync.

This article describes a clever use of BitTorrent to achieve roughly the same thing. I don't recommend this solution, but it's an interesting read.

Improve this answer
1

If you have the files on both sides, you can run md5sum on each end and compare the hash. The way to do this on a bunch of files is to tar them to stdout and pipe the output to md5sum.

rsync and plain scp -r or rcp -r are also your friends.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.