0

I need to search a large directory tree for a large number of possible file names, i.e. I have an input file with a long list of strings that I need to find in the tree (or as below: print all those that are not in the tree)

As a file system based solution I can do something like this:

while read a;
do
    count=`find /path/to/dir -name "*$a*" | wc -l`; ;
    if [ $count -eq 0 ];
    then
    echo $a;
    fi
done < inputnames.txt

which is wildly inefficient given the size of the tree and the size of the input file. I then figured I could simply dump the output of find into a file

find /path/to/dir > pathtodir_tree.txt

and then grep the file.

while read a;
do
    count=`grep $a pathtodir_tree.txt | wc -l`;
    if [ $count -eq 0 ];
    then
    echo $a;
    fi
done < inputnames.txt

which is much faster. My questions are: am I missing a more efficient file system based solution? If not, is there another solution that is more efficient (or generally better) than the grep based solution I am proposing?

2
  • Dumping the tree to a file is the way to go, I'd say. Did you try the -f option of grep (or am I missing something)? That could reduce your problem to a dump of the tree with find and a grep -f.
    – agtoever
    Aug 12, 2014 at 11:05
  • @agtoever I didn't play with grep -f ( I am also always forgetting about its very existence ;) ) but I am not sure I see how I can then simply extract the list of items in the input file which are (or not) present ; I don't need the actual matches in the tree, I only need to know if there are any (or not).
    – Erik
    Aug 12, 2014 at 13:14

1 Answer 1

0

This works for me (if you want to find that files not matching; otherwise, omit the v from the grep option):

find /path/to/dir > tree.txt
grep -vf inputnames.txt tree.txt

And if you want to know how many there are: grep -vf inputnames.txt tree.txt | wc -l

This finds/counts all occurences in tree.txt that do not match any of the string in inputnames.txt.

3
  • hm, not really, this will give me all hits for everything that is in the file, but I need to know on a line-by-line basis if there are any hits.
    – Erik
    Aug 12, 2014 at 15:31
  • Not sure I understand. You want to see the matching string, not the filename? Try the "-o" option of grep.
    – agtoever
    Aug 12, 2014 at 21:42
  • as stated in my question: go line by line through the input file. For a given record read from the file, count the number of occurrences in the directory tree. If you find none, print the record that was read from the input file. -o from grep prints the match from the directory tree file, not just the line that was used as 'to be matched' string.
    – Erik
    Aug 12, 2014 at 22:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .