Binary/Hash comparing files in different locations while ignoring folder structure in OSX

Question

I need a way to compare files in two different sets of locations, preferably in a way that is a little more thorough than just the file name.

We recently got a new NAS for the office and have been moving data from various USB hard drives on to it. I would like to be able to confirm that all the files have been moved over successfully.

I've had a look at many file compare programs but most of them are sensitive to the directory structure of where the files are. Ideally, I'd just want to hash (MD5 or similar) all the files on drives USB1, USB2 and USB3 and then hash all the files on NAS-VOLUME1 and NAS-VOLUME2 and compare the lists to see which files are missing from which side.

I suspect this could be done with a script or from the command line but I'm not overly familiar with working on the command line in OSX (normally a windows type of guys).

Any pointers much appreciated

Answer 1

fdupes could do it, if it's available on the Mac. You'd mount all your disks and let fdupes run over all the directories. It would be most efficient about it too (only comparing same sized files, as files with a unique filesize can not possibly be duplicates, etc.). Be careful as fdupes is often used to delete unwanted duplicates so a lot of examples may have delete options in them.

Answer 2

This version works if the filenames and subdirectory structures are the same in both locations. The advantage of this version is that it should be memory friendly. This has been lightly tested against simple errors but it is likely there is more to be accounted for. Also, a bash approach would be a lot more efficient as python has a lot of overhead. Depending on the size of data this could take a long time.

import os
import hashlib

def hash(file):
  f = open(file,'rb')
  h = hashlib.md5()
  checkEOF = b' '
  while checkEOF != b'':
    checkEOF = f.read(1024)
    h.update(checkEOF)
  f.close()
  return h.hexdigest()


def Hashwalk(d1, d2):
  errlist = []
  log = []
  walkobject1 = os.walk(d1)
  walkobject2 = os.walk(d2)
  try:
    for walks in zip(walkobject1,walkobject2):
      dir1 = walks[0][0]
      dir2 = walks[1][0]
      files1 = walks[0][2]
      files2 = walks[1][2]
      for files in zip(files1,files2):
        try:
          pathfile1 = os.path.join(dir1,files[0])
          pathfile2 = os.path.join(dir2,files[1])
          digest1 = hash(pathfile1)
          digest2 = hash(pathfile2)
          if digest1 != digest2:
            log.append((pathfile1, digest1, pathfile2, digest2))
        except PermissionError as error:    
          errlist.append((pathfile1,error))
        except FileNotFoundError as error:
          errlist.append((pathfile1,error))        
  except KeyboardInterrupt:
    print('Program terminated, results may be incomplete')
  return (log,errlist)

def ToDisk(hw):
  diff = open('differenthashes.txt','w',encoding='utf-8')
  for pair in hw[0]:
    diff.write(pair[0]+','+pair[1]+'\n')
    diff.write(pair[2]+','+pair[3]+'\n')
  if hw[1]:  
    diff.write('\nerrors\n')
    for error in hw[1]:
      diff.write(error[0]+','+error[1]+'\n')
  else:
    diff.write('no errors detected')
  diff.close()

ToDisk(Hashwalk('test1','test2'))

Answer 3

Currently the only errors this script accounts for is PermissionError and FileNotFoundError. Certain characters cannot properly be handled since they are represented using their encoding string and this resulted in a FileNotFoundError. I added a KeyboardInterrupt exception in case the script is running long and you want to see the accumulated results. The directory this script is ran from will contain a file called differenthashes.txt.

To execute just replace 'path1' and 'path2' in the compare() call at the bottom. Let me know if you have any suggestions or don't feel this properly accounts for your need.

import os
import hashlib
import time

def hash(file):
  f = open(file,'rb')
  h = hashlib.md5()
  checkEOF = b' '
  while checkEOF != b'':
    checkEOF = f.read(1024)
    h.update(checkEOF)
  f.close()
  return h.hexdigest()

def hashwalk(d = './'):
  errlist = []
  hashes = []
  cwd = os.getcwd()
  os.chdir(d)
  walkobject = os.walk('./')
  try:
    for directory in walkobject:
      dir = directory[0]
      files = directory[2]
      for file in files:
        try:
          pathfile = os.path.join(dir,file)
          digest = hash(pathfile)
          hashes.append((pathfile,digest))
        except PermissionError as error:    
          errlist.append((pathfile,error))
        except FileNotFoundError as error:
          errlist.append((pathfile,error))
  except KeyboardInterrupt:
    print('Program terminated, results may be incomplete')
  os.chdir(cwd)
  return [hashes,errlist]

def compare(path1,path2,logerrors = False):
  loc1 = hashwalk(path1)
  loc2 = hashwalk(path2)
  differenthash = set(loc1[0]).symmetric_difference(set(loc2[0]))
  log = open('differenthashes.txt','w',encoding='utf-8')
  log.write('path                                          hash                                 date modified\n')
  for f,h in sorted(differenthash):
    if (f,h) in loc1[0]:
      print(path1+'\\'+f[2:],h,time.ctime(os.stat(path1+'\\'+f[2:]).st_mtime))
      log.write(path1 + ' ' +f[2:] + ' ' + h + ' ' + time.ctime(os.stat(path1+'\\'+f[2:]).st_mtime)+'\n')
    else:
      print(path2+'\\'+f[2:],h,time.ctime(os.stat(path2+'\\'+f[2:]).st_mtime))
      log.write(path2 + ' ' +f[2:] + ' ' + h + ' ' + time.ctime(os.stat(path2+'\\'+f[2:]).st_mtime)+'\n')
  if logerrors:
    log.write('\n\n'+path1+' errors\n')
    for error in loc1[1]:
      log.write(str(error) + '\n')
    log.write('\n'+path2+' errors\n')
    for error in loc2[1]:
      log.write(str(error) +'\n')
  log.close()

compare('path1', 'path2' ,logerrors=True)