0

I have a folder with thousands (70G folder/to/check/) images that I, for each file want check if its already existing in another directory with thousands of thousands images (414G folder/to/search/).

This is the script I came up with, but its taking a very long time to run so I think the task could be optimized.

find folder/to/check/ -type f -exec md5sum {} + | awk -F" " '{ print $1,$2 }' | while read -r p1 p2; do find folder/to/search/ -type f -exec md5sum {} + | grep "^$p1" && echo "Found: $p1 $p2" || echo "Not found: $p1 $p2"; done > out.txt

Does anyone have an idea how to or could point me in the right direction how to optimize the script? My skills is very basic tho.

3 Answers 3

1

Do this.

  • Compute md5 sum of each image in "to-check" folder, write it to a text file.

  • Compute md5 sum of each image in "to-search" folder, append to 1st file.

  • Sort this giant file to a new file.

  • Use something like uniq -c to get a count of the number of times each line is duplicated.

If I got your command right (I might not), I think right now you're executing md5sum a total of X * Y times, but with the above, that's reduced to X + Y times.

2
  • Using a file size criteria first avoids computing (and performing the corresponding I/O) the MD5 of 99.5% of the files (on my current "live" photo library, 9860 JPG, only 50 have a file of the same size).
    – xenoid
    Oct 1, 2019 at 0:17
  • Yeah you are right. Mine is unnecessary slow. Unfortunately I cannot concatenate the files as I need to know which files I need to move from folder/to/check/ into folder/to/search/ before deleting folder/to/check/
    – Staplerz
    Oct 1, 2019 at 5:50
0

mtree is a handy utility whenever you need to compare two file hierarchies looking for same/different timestamps or same/different MD5/SHA256 hashes.

Let's set up a simple test hierarchy:

$ mkdir -p folder/to/check
$ mkdir -p folder/to/search
$ for N in $(jot -w %03d 10); do touch `/image-$N.jpg; done
$ for N in $(jot -w %03d 10 2); do touch folder/to/search/image-$N.jpg; done

folder/to/check has files image-001 through -010, while folder/to/search has image-002 through -011. All files are identical (zero bytes long), but have varying timestamps.

If you only want to check the MD5 hash (and the presence/absence of each file), then:

$ mtree -kmd5 -cp folder/to/check | mtree -p folder/to/search 
extra: image-011.jpg
./image-001.jpg missing

The extra line says that image-011.jpg was present in folder/to/search but wasn't required to be (since it isn't in folder/to/check), and the missing line says that image-001.jpg was found in the source folder (folder/to/check) but not in the target folder (folder/to/search).

If you want to check the timestamp in addition to the MD5 hash, then:

$ mtree -Kmd5 -cp folder/to/check | mtree -p folder/to/search 
.:      modification time (Mon Sep 30 16:16:15 2019, Mon Sep 30 16:16:24 2019)
image-010.jpg: 
        modification time (Mon Sep 30 16:16:15 2019, Mon Sep 30 16:16:24 2019)
image-008.jpg: 
        modification time (Mon Sep 30 16:16:15 2019, Mon Sep 30 16:16:24 2019)
image-006.jpg: 
        modification time (Mon Sep 30 16:16:15 2019, Mon Sep 30 16:16:24 2019)
extra: image-011.jpg
image-007.jpg: 
        modification time (Mon Sep 30 16:16:15 2019, Mon Sep 30 16:16:24 2019)
image-009.jpg: 
        modification time (Mon Sep 30 16:16:15 2019, Mon Sep 30 16:16:24 2019)
image-004.jpg: 
        modification time (Mon Sep 30 16:16:15 2019, Mon Sep 30 16:16:24 2019)
image-003.jpg: 
        modification time (Mon Sep 30 16:16:15 2019, Mon Sep 30 16:16:24 2019)
image-002.jpg: 
        modification time (Mon Sep 30 16:16:15 2019, Mon Sep 30 16:16:24 2019)
image-005.jpg: 
        modification time (Mon Sep 30 16:16:15 2019, Mon Sep 30 16:16:24 2019)
./image-001.jpg missing

That output still shows the same extra and missing lines, plus a number of files that are identical in MD5 hash, but have different timestamps. Finally, if any of the files has a different MD5 hash than its counterpart, mtree will tell you which ones:

$ date > folder/to/search/image-007.jpg
$ mtree -Kmd5 -cp folder/to/check | mtree -p folder/to/search
.:      modification time (Mon Sep 30 16:16:15 2019, Mon Sep 30 16:16:24 2019)
image-010.jpg: 
        modification time (Mon Sep 30 16:16:15 2019, Mon Sep 30 16:16:24 2019)
image-008.jpg: 
        modification time (Mon Sep 30 16:16:15 2019, Mon Sep 30 16:16:24 2019)
image-006.jpg: 
        modification time (Mon Sep 30 16:16:15 2019, Mon Sep 30 16:16:24 2019)
extra: image-011.jpg
image-007.jpg: 
        size (0, 29)
        modification time (Mon Sep 30 16:16:15 2019, Mon Sep 30 16:25:05 2019)
        md5digest (0xd41d8cd98f00b204e9800998ecf8427e, 0x6fd624d18a8fe0c096b19ff858a2acf7)
image-009.jpg: 
        modification time (Mon Sep 30 16:16:15 2019, Mon Sep 30 16:16:24 2019)
image-004.jpg: 
        modification time (Mon Sep 30 16:16:15 2019, Mon Sep 30 16:16:24 2019)
image-003.jpg: 
        modification time (Mon Sep 30 16:16:15 2019, Mon Sep 30 16:16:24 2019)
image-002.jpg: 
        modification time (Mon Sep 30 16:16:15 2019, Mon Sep 30 16:16:24 2019)
image-005.jpg: 
        modification time (Mon Sep 30 16:16:15 2019, Mon Sep 30 16:16:24 2019)
./image-001.jpg missing
3
  • Wow. This sounds very promising! I'll give it a try tonight when I'll get back home, then I can answer if this is what I'm looking for :)
    – Staplerz
    Oct 1, 2019 at 5:55
  • It seems I don't have mtree on my machine (Debian). How can I install it? And is there a way to ignore the depth/folder location of a file?
    – Staplerz
    Oct 3, 2019 at 12:04
  • @Staplerz This post has a rough outline of how to install mtree. I'm not aware of any way to ignore the folder location of the files.
    – Jim L.
    Oct 3, 2019 at 15:38
0

There are utilities around to do this, but it you want to it yourself:

  • Get the size in bytes of all the files, and make lists of all the files with the exact same size.
  • For lists that have more than one element (not that many, especially among big files):
    • compute the MD5 of all elements of the list
    • check for duplicates

In essence this makes you compute the MD5 hash once and only for files that you cannot discriminate on size.

This of course assumes that you have no duplicates among the existing files or among the new files (but can easily be tweaked to check this too).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .