9

I am a GIS specialist, part of this is to routinely copy vast quantities of data to disc - backup, archiving and delivery. Occasionally the files on the other disc do not match the source disc, in the order of for every 4TB copied there will be one error in one file (that's 0.000000025%).

Currently I am using RoboCopy or Windows Explorer copy to duplicate the files onto the other discs and WinDiff to verify the contents, which is how I know that there's a very small error rate. This process works but WinDiff is slow, sometimes taking more time to verify than it took to originally copy; RoboCopy does not have a and verify switch like XCopy.

I have previously used XCopy with the /v switch but was advised that that utility was depreciated in Windows 7 and I shouldn't be using it. Individually creating MD5, checksum or hash of files is not an option unless there's an efficient library/method/algorithm that I can implement in a programming language as there would be hundreds of thousands of individual files.

The question is: Is there a method that will verify the files as they are copied or a better verification program than WinDiff, or even better yet a guaranteed way to copy these files with 100% integrity so I don't have to verify them?

I can code for C# / VB.net / C++ / Python / DOS Batch provided the required libraries are free or built in. I'm not scared by command line, I can remember when there was only command line, so DOS utilities/instructions are fine.

So why is this now a problem if I have a working method? It's not really, I have a working method but have been doing it the same way since Windows NT in the late 90's; in the late 90's it was Exabyte tapes and 100MB or less, then there was Optical media which could be verified in less than half an hour. Now, with media being high capacity and relatively inexpensive, I find that I am copying and verifying up to 16 TB for a single delivery and the verification time is overrunning the delivery date.... I need to at least look for a better way!

7
  • EVERYTHING about computers becomes dated, there are still questions on this site about verifying ISO files are written to CD correctly! When was the last time you wrote a CD. How about instead of down voting you tell me how to do this with windows ONLY without resorting to 3rd party software. – Michael Stimson Jun 25 '14 at 5:19
  • All you need to do is re-word it carefully. Instead of asking for "software", state only what you need to accomplish, if it is changes to the system, or built in tools, or a piece of software that does it then the answer will be the same. Also softwarerecs.stackexchange.com will be the place to directally ask for software. – Psycogeek Jun 25 '14 at 9:07
  • 1
    I quite like TeraCopy for this. – ChrisInEdmonton Jul 3 '14 at 13:33
  • 3
    According to the /? help with xcopy, the /v argument only verifies the file size, not the contents, so if that's true, I guess it is probably not appropriate to your task! – Mike Simpson Aug 18 '19 at 20:56
  • 1
    @Shayan there is not an option in RoboCopy, see superuser.com/questions/671182/… for more info. You can use WinDiff from the command line after copying support.microsoft.com/en-au/help/159214/… to verify the contents if integrity is paramount and time is only a secondary consideration. – Michael Stimson Sep 30 '19 at 4:36
5

The program you've been looking for is xxcopy. I too had this desire to do a binary file compare when copying oh, a few hundred thousand files for forensic analysis. I didn't care it took a little longer, I didn't want to have to create a SFV or MD5 file and then run another program on the files afterward, I wanted something that did the DOS "fc" command on the fly.

http://xxcopy.com/xxtb_027.htm#tag_319

You want switch /V2

Have a good day.

3
  • Thank you, I'll give that a try when I get back from holidays. – Michael Stimson Jul 4 '14 at 4:27
  • It is unfortunate that the company has ceased trading. – wwkudu Mar 6 '18 at 14:16
  • @wwkudu This is true, but the free version is still available on the download tab on their site as of today. – mbmast Apr 6 '18 at 18:18
2

I have used Fastcopy and terracopy for years in windows, either of them make good replacements for the windows copy paste methods, both have a comparison routine. I think someone said the compare method for fastcopy is a "better" method. Both of them have caught very minor errors I caused myself by having ram timed incorrectaly.

There are other differances that make FC and TC 2 totally different acting programs, it is better to experience and adjust the options in both to be able to understand them. Like on moves FC will check each item prior to deletion one by one moves. TC can be easily set to compare all as a second action, or repeated action. TC might have issues with network drives, less people had issues with FC when using networked drives.

I prefer to use FC, but something like a single error in FC can cause more manuel effort, TC having all the items still in the list, and having a visable listing shown always has its advantages. They are both GUI style programs, not CMD style although they both will work in a command prompt or batch.
Both can isure better sequentiality of data by waiting till one copy operation is completed, prior to starting the other operation, so making fragmented messes on disks is reduced.

Neither of them make "fixing those last few files" that did not copy right like Windiff.exe, so windiff was a lot better when everything is failing :-)

Both will integrate right into the system as a replacement for the windows copy, and integrate themselves into the right click context menu also. Both can also work as fully portable. I no longer have either of them replace the systems copy, I have them in the context menu only to use them as desired.

Conclusion, try them both, see what you think.

5
  • Thank you psycogeek. I'll give them a try. Would you happen to have links to them handy? – Michael Stimson Jun 25 '14 at 4:15
  • majorgeeks.com/files/details/teracopy.html & majorgeeks.com/files/details/fastcopy.html should have legitimate untainted downloads of them today if you do not hit the wrong buttons. – Psycogeek Jun 25 '14 at 4:35
  • Oh by the way what is GIS ? – Psycogeek Jun 25 '14 at 4:37
  • Geospatial Information Systems. See gis.stackexchange.com. I deal with data that has a spatial component, like a map but smarter. The large files are LiDAR data (airborne laser imaging) and rasters upward of 100GB each derived from airborne LiDAR data. – Michael Stimson Jun 25 '14 at 4:41
  • Probably I shouldn't comment this here... but I'll do it anyway. I was looking for "copy and verify tool" and this answer popped up on google. Guess what I was copying... tifs and LiDAR data! – Andrei Feb 24 '18 at 19:26
0

You can read about the Windows FCIV tool here. That link will explain how to install the tool and also explain its usage. For your purpose, you can use the tool's recursive option and run these two commands:

fciv D:\Path\To\Source -r -xml db.xml
fciv -v -bp E:\Path\To\Destination -r -xml db.xml

Replace D:\Path\To\Source with the path of your source files and replace E:\Path\To\Destination with the path of the files that you copied.

The first command will create a database called "db.xml" in the folder where you ran the command. The second command will use that database to verify the copied files using the md5 hash checksum.

If all went well, you'll receive a message that says "All files verified successfully".

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.