1

I have been given several hundred directories with millions of files in each of them. E.g.

DirA
+----+ A_000001.txt
     |
     + A_000002.txt
     |
     + ... etc..

I want to ensure that the files exists for every number from 000001 until 999999. So if the file A_678123.txt is missing, the tool should report that. Currently I am creating a bit array & then it takes a ls output and sets a 1 for every bit position that corresponds to the parsed file number.

Is there a tool (or a combination of tools) that can replace my hacky python script ?

1 Answer 1

1

One way is to use a combination of seq, sort, and uniq:

# Get a list of all the files in DirA
cd DirA
ls . > files.txt
# Get a list of all expected files
seq -f "A_%06.0f.txt" 0 999999 > expected.txt
# Find unique entries
sort files.txt expected.txt | uniq -u

You may have to adjust the arguments to seq to match the names of your files.

Once that is working, you can try doing it without using temporary files:

cd DirA
sort <(ls .) <(seq -f "A_%06.0f.txt" 0 999) | uniq -u

Then, once you're ready to search through all your directories:

for d in ./*/; do
  pushd "$d"
  sort <(ls .) <(seq -f "A_%06.0f.txt" 0 999) | uniq -u
  popd
done

You should get an output similar to this, which lists each directory and every missing file in that directory:

./DirA/
A_000371.txt
./DirB/
A_000881.txt
./DirC/
A_000101.txt
A_000191.txt

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .