0

I need to search a large directory tree for a large number of possible file names, i.e. I have an input file with a long list of strings that I need to find in the tree (or as below: print all those that are not in the tree)

As a file system based solution I can do something like this:

while read a;
do
    count=`find /path/to/dir -name "*$a*" | wc -l`; ;
    if [ $count -eq 0 ];
    then
    echo $a;
    fi
done < inputnames.txt

which is wildly inefficient given the size of the tree and the size of the input file. I then figured I could simply dump the output of find into a file

find /path/to/dir > pathtodir_tree.txt

and then grep the file.

while read a;
do
    count=`grep $a pathtodir_tree.txt | wc -l`;
    if [ $count -eq 0 ];
    then
    echo $a;
    fi
done < inputnames.txt

which is much faster. My questions are: am I missing a more efficient file system based solution? If not, is there another solution that is more efficient (or generally better) than the grep based solution I am proposing?

  • Dumping the tree to a file is the way to go, I'd say. Did you try the -f option of grep (or am I missing something)? That could reduce your problem to a dump of the tree with find and a grep -f. – agtoever Aug 12 '14 at 11:05
  • @agtoever I didn't play with grep -f ( I am also always forgetting about its very existence ;) ) but I am not sure I see how I can then simply extract the list of items in the input file which are (or not) present ; I don't need the actual matches in the tree, I only need to know if there are any (or not). – Erik Aug 12 '14 at 13:14
0

This works for me (if you want to find that files not matching; otherwise, omit the v from the grep option):

find /path/to/dir > tree.txt
grep -vf inputnames.txt tree.txt

And if you want to know how many there are: grep -vf inputnames.txt tree.txt | wc -l

This finds/counts all occurences in tree.txt that do not match any of the string in inputnames.txt.

  • hm, not really, this will give me all hits for everything that is in the file, but I need to know on a line-by-line basis if there are any hits. – Erik Aug 12 '14 at 15:31
  • Not sure I understand. You want to see the matching string, not the filename? Try the "-o" option of grep. – agtoever Aug 12 '14 at 21:42
  • as stated in my question: go line by line through the input file. For a given record read from the file, count the number of occurrences in the directory tree. If you find none, print the record that was read from the input file. -o from grep prints the match from the directory tree file, not just the line that was used as 'to be matched' string. – Erik Aug 12 '14 at 22:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.