0

I'm setting up a NAS box for the first time. At the moment, I have most of my data backed up to a few local hard drives, and I intend to transfer all the data to my NAS over ethernet once the RAID array is setup. Since this is all happening over the network, I'm a bit worried about my data getting corrupted silently during transfer. From what I understand, data generally doesn't get corrupted without notice on local transfers because a checksum is performed at some point by the drive or the OS. (This could be totally wrong.) Does the same thing happen with SMB, or is it up to the transferrer to check the integrity of their data? And if it doesn't happen with SMB, is there a protocol that does ensure data integrity? I know that rsync can checksum a transfer, but I'm on Windows and I already have a robocopy configuration that I like. Will my data be safe or do I have to use an external checksum tool to make sure?

2 Answers 2

2

Data Corruption is a probability and the odds of it happening increase with the amount of data

Please note that xcopy /v does not verify data integrity

"This verification process consists of confirming that the data just written can be read (for example, that the data was not written to a bad sector on the disk). No comparison of the source and destination data occurs. "

https://support.microsoft.com/en-us/kb/126457

I am currently developing a data migration plan (personal data) for 5Tb from an old storage array to my new storage server. My research, incomplete, has shown that Ethernet uses CRC16 to verify data integrity of the data that was transferred.

There is a cool link but I cannot post it due to reputation points. just put http in front of it

doc.utwente.nl/64267/1/schiphorst.pdf

As soon as I learn the nuances of the MS File Checksum Integrity Verifier, I will script a MD5 hash of my source and compare it to the destination. It took fciv.exe about 9 hours to generate the hash database on ~1Tb. This is the most time consuming method but also the most complete. Some of my data cannot be replaced and is important to me. The odds of it not catching an error (Silent corruption) are very small (like 10^-16 I think but cannot find my source for that)

https://support.microsoft.com/en-us/kb/841290

1
  • FYI, I ended up using hashdeep/md5deep to verify the integrity of all my files. Works really well.
    – Archagon
    Dec 3, 2015 at 22:36
1

Read your docs - everyone has a verify option these days.

I strongly suggest rsync with verify. You can get rsync on any OS these days, and all you need is the client. Further, It's unlikely that your data will be corrupted during copy by the network layer. Maybe your drives are junk, or your motherboard is dying, but the network is pretty robust.

Also, if you mount the drives up, fire up a cmd window (aka dos window) and try xcopy /V to get a verified copy (use xcopy /? to see all the options) - and make sure your data lives on multiple drives, or you have copies spread out. Don't ever trust a single point of failure!

1
  • Robocopy does not, in fact, have a verify option. I'm surprised that xcopy does, given that robocopy is meant to be a replacement.
    – Archagon
    Dec 20, 2012 at 2:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .