1

I have a list of 2000 files from a database that looks like this:

./aa/0f/unique-string/IMG_0987.JPG
./ab/cf/unique-string/IMG_0987.JPG

I want to compare that list to the actual directory contents in order to identify missing files.

The following command works individually, but not when I script it...

find . -false -samefile ./ab/cf/unique-string/IMG_0987.JPG

The closest I have come is the following:

#!/bin/bash
TEST=`cat ./list.lst`
find . -false -samefile "$TEST"

I am doing it wrong. what is the right way?

  • Pls take a look at my answer, for a faster solution. – MariusMatutiae Jun 26 '15 at 9:15
0

Your find command is using the entire contents of list.lst because you aren't feeding it through 1 line at a time.

while read f; do
find . -false -samefile "$f"
done<./list.lst

This reads the file list.lst 1 line at a time.

| improve this answer | |
  • This is correct, but much slower than it need be. Pls see my answer. – MariusMatutiae Jun 26 '15 at 9:16
0

By following your strategy, you will be making about 2000x2000 = 4,000,000 comparisons. You can do better than this.

Suppose the list is in file_t1; now we first generate a list of all files on the pc by means of

      find . -type f > file_t2

Then we sort both files:

      sort -n file_t1 > file1
      sort -n file_t2 > file2

Now we use comm to generate a list of differences:

      comm -X file1 file2

where:

      X = 12 -> lines that appear in **both** files
      X = 13 -> lines unique to file 2
      X = 23 -> lines unique to file1

This could be done with a one-liner, at the expense of clarity.

If you are interested: this is much faster because the files are already sorted (a strict requirement for comm), so they take of order N steps to compare, if the file size is N. Sorting requires N log N operations, which is thus the most expensive part of this solution, while the one you proposed requires N^2 operations, which is significantly longer for your file sizes.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.