Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I need a way to compare files in two different sets of locations, preferably in a way that is a little more thorough than just the file name.

We recently got a new NAS for the office and have been moving data from various USB hard drives on to it. I would like to be able to confirm that all the files have been moved over successfully.

I've had a look at many file compare programs but most of them are sensitive to the directory structure of where the files are. Ideally, I'd just want to hash (MD5 or similar) all the files on drives USB1, USB2 and USB3 and then hash all the files on NAS-VOLUME1 and NAS-VOLUME2 and compare the lists to see which files are missing from which side.

I suspect this could be done with a script or from the command line but I'm not overly familiar with working on the command line in OSX (normally a windows type of guys).

Any pointers much appreciated

share|improve this question
    
This is easy with python. I'm not sure what options are available in OSX. –  Eric Roper Feb 5 '13 at 11:30
    
I actually bashed out something in c# but I keep running out of RAM. Any examples of Python scripts which will do this? –  mynameisdev Feb 6 '13 at 9:47
add comment

3 Answers

fdupes could do it, if it's available on the Mac. You'd mount all your disks and let fdupes run over all the directories. It would be most efficient about it too (only comparing same sized files, as files with a unique filesize can not possibly be duplicates, etc.). Be careful as fdupes is often used to delete unwanted duplicates so a lot of examples may have delete options in them.

share|improve this answer
    
Can't seem to find a sensible install route for fdupes on this Mac. I've got Ubuntu on a laptop somewhere but in the meantime, any other ideas, something with a GUI would be great... –  mynameisdev Feb 5 '13 at 13:59
add comment

This version works if the filenames and subdirectory structures are the same in both locations. The advantage of this version is that it should be memory friendly. This has been lightly tested against simple errors but it is likely there is more to be accounted for. Also, a bash approach would be a lot more efficient as python has a lot of overhead. Depending on the size of data this could take a long time.

import os
import hashlib

def hash(file):
  f = open(file,'rb')
  h = hashlib.md5()
  checkEOF = b' '
  while checkEOF != b'':
    checkEOF = f.read(1024)
    h.update(checkEOF)
  f.close()
  return h.hexdigest()


def Hashwalk(d1, d2):
  errlist = []
  log = []
  walkobject1 = os.walk(d1)
  walkobject2 = os.walk(d2)
  try:
    for walks in zip(walkobject1,walkobject2):
      dir1 = walks[0][0]
      dir2 = walks[1][0]
      files1 = walks[0][2]
      files2 = walks[1][2]
      for files in zip(files1,files2):
        try:
          pathfile1 = os.path.join(dir1,files[0])
          pathfile2 = os.path.join(dir2,files[1])
          digest1 = hash(pathfile1)
          digest2 = hash(pathfile2)
          if digest1 != digest2:
            log.append((pathfile1, digest1, pathfile2, digest2))
        except PermissionError as error:    
          errlist.append((pathfile1,error))
        except FileNotFoundError as error:
          errlist.append((pathfile1,error))        
  except KeyboardInterrupt:
    print('Program terminated, results may be incomplete')
  return (log,errlist)

def ToDisk(hw):
  diff = open('differenthashes.txt','w',encoding='utf-8')
  for pair in hw[0]:
    diff.write(pair[0]+','+pair[1]+'\n')
    diff.write(pair[2]+','+pair[3]+'\n')
  if hw[1]:  
    diff.write('\nerrors\n')
    for error in hw[1]:
      diff.write(error[0]+','+error[1]+'\n')
  else:
    diff.write('no errors detected')
  diff.close()

ToDisk(Hashwalk('test1','test2'))
share|improve this answer
add comment

Currently the only errors this script accounts for is PermissionError and FileNotFoundError. Certain characters cannot properly be handled since they are represented using their encoding string and this resulted in a FileNotFoundError. I added a KeyboardInterrupt exception in case the script is running long and you want to see the accumulated results. The directory this script is ran from will contain a file called differenthashes.txt.

To execute just replace 'path1' and 'path2' in the compare() call at the bottom. Let me know if you have any suggestions or don't feel this properly accounts for your need.

import os
import hashlib
import time

def hash(file):
  f = open(file,'rb')
  h = hashlib.md5()
  checkEOF = b' '
  while checkEOF != b'':
    checkEOF = f.read(1024)
    h.update(checkEOF)
  f.close()
  return h.hexdigest()

def hashwalk(d = './'):
  errlist = []
  hashes = []
  cwd = os.getcwd()
  os.chdir(d)
  walkobject = os.walk('./')
  try:
    for directory in walkobject:
      dir = directory[0]
      files = directory[2]
      for file in files:
        try:
          pathfile = os.path.join(dir,file)
          digest = hash(pathfile)
          hashes.append((pathfile,digest))
        except PermissionError as error:    
          errlist.append((pathfile,error))
        except FileNotFoundError as error:
          errlist.append((pathfile,error))
  except KeyboardInterrupt:
    print('Program terminated, results may be incomplete')
  os.chdir(cwd)
  return [hashes,errlist]

def compare(path1,path2,logerrors = False):
  loc1 = hashwalk(path1)
  loc2 = hashwalk(path2)
  differenthash = set(loc1[0]).symmetric_difference(set(loc2[0]))
  log = open('differenthashes.txt','w',encoding='utf-8')
  log.write('path                                          hash                                 date modified\n')
  for f,h in sorted(differenthash):
    if (f,h) in loc1[0]:
      print(path1+'\\'+f[2:],h,time.ctime(os.stat(path1+'\\'+f[2:]).st_mtime))
      log.write(path1 + ' ' +f[2:] + ' ' + h + ' ' + time.ctime(os.stat(path1+'\\'+f[2:]).st_mtime)+'\n')
    else:
      print(path2+'\\'+f[2:],h,time.ctime(os.stat(path2+'\\'+f[2:]).st_mtime))
      log.write(path2 + ' ' +f[2:] + ' ' + h + ' ' + time.ctime(os.stat(path2+'\\'+f[2:]).st_mtime)+'\n')
  if logerrors:
    log.write('\n\n'+path1+' errors\n')
    for error in loc1[1]:
      log.write(str(error) + '\n')
    log.write('\n'+path2+' errors\n')
    for error in loc2[1]:
      log.write(str(error) +'\n')
  log.close()

compare('path1', 'path2' ,logerrors=True)
share|improve this answer
    
If this gives you a MemoryError and the subdirectory structure is the same in both locations, you can move the comparisons directly into hashwalk() and compare without lists. I'll try this version first and edit my post if it works. –  Eric Roper Feb 6 '13 at 11:00
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.