1

I have over 1 million (small size) files(<500kB jpg) in a structure like this:

H:\main_folder\folder\sub_folder\sub_sub_folder\sub_sub_sub_folder

Every sub_sub_sub_folder have 10-15 files, sub_sub_folder could be a date/time stamp, sub_folder is a machine_name and folder is another incremented number.

I need to have a copy of main_folder with the same structure but I need to verify that my copy is 100% the same as the source. Windows Explorer will block after ~1000 copies, doesn't even show the properties (size, size on disk, contains).

I need a batch that will go to H:\main_folder\folder\sub_folder\sub_sub_folder\sub_sub_sub_folderA read&copy all the 14 files attributes and paste them in Z:\main_folder\folder\sub_folder\sub_sub_folder\sub_sub_sub_folderA, open a file log.txt, write:

"file1.jpg 490kB copied from H:\main_folder\folder\sub_folder\sub_sub_folder\sub_sub_sub_folderA to Z:\main_folder\folder\sub_folder\sub_sub_folder\sub_sub_sub_folderA successfully ..."

and

"file15.jpg 470kB copied from H:\main_folder\folder\sub_folder\sub_sub_folder\sub_sub_sub_folderA to Z:\main_folder\folder\sub_folder\sub_sub_folder\sub_sub_sub_folderA successfully"

e.t.c.

mark the end of the log(one line of ====== or something) and only then proceed to the next operation(copy content and folder structure of sub_sub_sub_folderB). If copy operation not successfully terminated stop, create a new log error_sub_sub_sub_folderB.txt (the next log will be error_sub_sub_sub_folderZ.txt) and only after the log is created move to the next sub_sub_sub_folder

In a few words: a step by step logging copy batch that doesn't kill the system or the machine hardware resources.

  • try whatever to do the copy, and then use "Beyond Compare" to compare the source and dest. I don't see how making a log of the copy will help you, as you're not going to want to read through it all. So I suggest a program to check the src and dst are the same in content and structure. – barlop Jan 22 '15 at 6:14
  • you can also do dir /a-d/s and you will get all files listed with their sizes. Do that at the dest, and you will see if all the files are at the dest, and the size isn't funny e.g. size isn't 0 bytes or all sizes are within what you expect. What i'm suggesting doesn't involve a batch file though. Just a copy command and one or two commands after to check that it copied. And you could use "beyond compare" too or any folder sync program to check. – barlop Jan 22 '15 at 6:20
  • Maybe robocopy would be a good tool – Ivan Viktorovic Jan 22 '15 at 11:43
1

Both xcopy and robocopy will do the job, but robocopy's output is closer to what you described.

xcopy src dst /i /e /f /v /c > log.txt

/F will output the full source and destination paths.

/V is claimed to verify the file sizes, but doesn't print them in the output.

Apart from not having the file sizes logged, this mostly achieves what you described. There are other switches that can control whether ACLs and attributes are copied. Being a native exe, it will be faster than a script loop. But you won't get a separate error log, and you can't abort a subfolder and continue with next subfolder on error (/C continues all files on error).

Sample output:

C:\Users\Celery\test>xcopy src dst /i /e /f /v /c
C:\Users\Celery\test\src\a\blah.txt -> C:\Users\Celery\test\dst\a\blah.txt
C:\Users\Celery\test\src\a\blah2.txt -> C:\Users\Celery\test\dst\a\blah2.txt
C:\Users\Celery\test\src\b\blah.txt -> C:\Users\Celery\test\dst\b\blah.txt
C:\Users\Celery\test\src\b\blah2.txt -> C:\Users\Celery\test\dst\b\blah2.txt
4 File(s)

Robocopy produces a fairly detailed log, including file sizes. Try:

robocopy src dst /e /np /log:log.txt [/tee]

Sample output:

C:\Users\Celery\test>robocopy src dst /e

-------------------------------------------------------------------------------
   ROBOCOPY     ::     Robust File Copy for Windows

-------------------------------------------------------------------------------

  Started : Thursday, 22 January 2015 10:58:55 PM
   Source : C:\Users\Celery\test\src\
     Dest : C:\Users\Celery\test\dst\

    Files : *.*

  Options : *.* /S /E /DCOPY:DA /COPY:DAT /R:1000000 /W:30

------------------------------------------------------------------------------

          New Dir          0    C:\Users\Celery\test\src\
          New Dir          2    C:\Users\Celery\test\src\a\
            New File                  17        blah.txt
            New File                  17        blah2.txt
          New Dir          2    C:\Users\Celery\test\src\b\
            New File                  17        blah.txt
            New File                  17        blah2.txt

------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :         3         3         0         0         0         0
   Files :         4         4         0         0         0         0
   Bytes :        68        68         0         0         0         0
   Times :   0:00:00   0:00:00                       0:00:00   0:00:00


   Speed :                2193 Bytes/sec.
   Speed :               0.125 MegaBytes/min.
   Ended : Thursday, 22 January 2015 10:58:55 PM

Otherwise, to script the iteration yourself, you would need four nested loops, like:

for /d %%a in (main_folder\*) do (
  for /d %%b in (%%a\*) do (
    for /d %%c in (%%b\*) do (
      for /d %%d in (%%c\*) do (
        :: code to copy files and write logs goes here
      )
      echo ================= end of %%d =============== >> log.txt
    )
  )
)

That will only copy files in the sub_sub_sub_folders: any files in higher level folders will be ignored, because for /d only iterates over directories.

  • I will definitely try it. Somebody else suggest me the TeraCopy (has GUI and CRC check)program but ... I will see. – Lucs Jan 23 '15 at 5:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.