1

I need to find every folder with more than one PDF in a folder structure but hide the pesky hidden files such as .HResource files that infiltrate our file system.

I only want to filter folders that have HiRes in the filename.

Ideally I want to run this at top level so it searches every folder with HiRes and spits out the folders with more than one PDF inside it.

The OS is RHEL 7 and this command can be run as user/root as required.

I have had a play with

find -type f -iname '*.pdf' -iname '*HiRes*' ! -iname '.*' -printf '%h\n' | sort | uniq -d

but have had very little luck so far as the hidden files keep showing and thus in a 11,000 folder system giving thousands of false positives. For example, given the following directory structure,

.
|--Apples.HiRes1
|     |--Gala.pdf
|     |--Granny Smith.pdf
|     |--McIntosh.pdf
|     |--Red Delicious.pdf
|--Banana-HiRes2
|     |--upper.pdf
|     |--subdir
|     |    |--lower.pdf
|--Cherry_HiRes3
|     |--Bing.pdf
|     |--Blossom.jpg
|     |--.pesky
|     |--.hidden
|     |--.files
|     |--.HResource
|--Dates (HiRes4)
|     |--Midsummer.Pdf
|     |--New Year’s Eve.PDF
|--Employees
|     |--Fred.pdf
|     |--Ginger.pdf
|     |--New Hires
|     |     |--Sam Malone.pdf
|     |     |--Woody Boyd.pdf
|--Gemstones
|     |--Rubies
|     |     |--1.pdf
|     |     |--2.pdf
|     |--Sapphires
|     |     |--3.pdf
|     |     |--4.pdf
|--Tomato.HiRes
|     |--Bacon
|     |--Lettuce
|     |     |--5.pdf
|     |     |--6.pdf
|--Zucchini.LoRes
|     |--Bread.pdf
|     |--Squash.pdf

the output should be

./Apples.HiRes1
./Dates (HiRes4)

Note that

  • Cherry_HiRes3 is excluded because it has only one PDF file in it.
  • Banana-HiRes2 and Tomato.HiRes are excluded because they have two PDF files under them, but not in them.
  • Zucchini.LoRes is excluded (obviously) because its name has LoRes rather than HiRes.
3
  • (1) One of the primary rules of debugging is to break things into pieces.  Have you tried running your find command by itself, and looking at the output?  Is it what you expect?  Does that help you?  (2) Another debugging technique is to explain your problem to a rubber duck.  Can you explain how you believe your solution should work, and what output you expect to get from the find command?  (3) Do you have files with names like .MRducks.pdf?  If not, I don’t understand what issue you could possibly be having with “pesky hidden files” such as .HResource.  Can you explain?  … (Cont’d) Jul 31, 2019 at 20:38
  • (Cont’d) …  Please do not respond in comments; edit your question to make it clearer and more complete.  … (Cont’d) Jul 31, 2019 at 20:38
  • P.S. (4) I’ve taken the liberty of adding example data to your question.  Please verify that my edit makes sense.  In particular, would you want the “Banana” and “Tomato” directories to be listed?  (5) Do you really want the *HiRes* test to be case-insensitive?  Note that, if you do, “New hires” and “Sapphires” will be listed. Jul 31, 2019 at 20:38

3 Answers 3

0

A solution that’s close to what you tried is

find . -ipath '*HiRes*/*.pdf' -printf '%h\n' | sort | uniq -d

As you seem to (at least partially) understand, the -name and -iname tests operate on only one filename; i.e., only one level of a pathname.  So, when you say -iname '*.pdf' -iname '*HiRes*', that’s equivalent to -iname '*HiRes*.pdf'; i.e., you’re looking for files whose names match both *.pdf and *HiRes*.  However, the -path, -ipath, -regex and -iregex tests operate on the entire path, so this lists files whose pathname contains HiRes and ends with .pdf.  We put a / between them to make sure that the HiRes is in a directory names, so we don’t find files called abcHiRes123.pdf.

We can do the exact same thing with

find . -iregex '.*HiRes.*/.*\.pdf' -printf '%h\n' | sort | uniq –d

Note the difference between glob syntax (as used by -ipath) and regex syntax.  Dot (.) means any character, so we need to use .* to match “anything” and \. to match a literal ..

Both of the above will find the Tomato.HiRes directory in the example in your question, because  . matches /, so .*/.* matches /Lettuce/5, so .*HiRes.*/.*\.pdf matches  ./Tomato.HiRes/Lettuce/5.pdf .  If that’s OK with you, great.  But, if you want to count only the PDF files that are in*HiRes directory, use

find . -iregex '.*HiRes[^/]*/[^/]*\.pdf' -printf '%h\n' | sort | uniq -d

in which we use [^/] to match any character other than a /.


A substantially different approach, which may be easier to understand, is

find . -iname '*HiRes*' -type d -exec sh -c \
                    'shopt -s nocaseglob; for dir do files=("$dir"/*.pdf);
                    if [ "${#files[@]}" -gt 1 ]; then echo "$dir"; fi; done' sh {} +

This only uses find to find directories called *HiRes*.  It then turns those directory names over to a short shell script, which uses glob to enumerate all the PDF files in each directory and count them.

3
  • Thanks for sharing.  Yeah, I tbought it was borderline, but I tbtally missed tbat typo.  The person who proposed that edit seems to be trying too hard (seemingly editing just for the sake of editing) and simultaneously not trying hard enough (missing low-hanging fruit). Nov 11, 2019 at 0:58
  • In other news, I rejected this suggested edit for this reason. Yeah; I should have written that as a custom close reason; I lost my keys. Nov 11, 2019 at 21:05
  • Today’s edition: I voted to reject this edit because it actually changed the answer.   (I subsequently repaired the damage.) Dec 2, 2019 at 5:29
0

EDIT: This actually doesn't work either. See the comments below.


Have you tried adding -not -iname '.*' to your find command? The ! is probably not doing what you think it is.

This would result in find -type f -iname '*.pdf' -iname 'HiRes' -not -iname '.*' -printf '%h\n' | sort | uniq -d

2
  • In GNU find, -not and ! mean the same thing.  In POSIX find, -not is not defined.  What do you think the ! is doing in the OP’s command?  What do you think -not would do differently?  What do you think the OP thinks the ! is doing in their command? Jul 31, 2019 at 20:42
  • Given that it's on a Linux host (I know OP and know this to be true), I had issues with the same command on a test host. The ! is handled by the shell (bash) as an event. I've previously had luck with -not, but apparently something has changed as you pointed out. I'll update my post.
    – Skudd
    Aug 1, 2019 at 14:44
-1

Okay, so I took another stab at this after realizing my previous answer was wrong. JandP, you are so close to having the right command.


The Setup

  • 10 directories, 1-10, referred to as $s hereout
  • $n files in each, where $n is the number of the directory, named $s-$n.pdf
  • $n dotfiles in each, .fake-$s-$n.pdf

Command I used to create this tree of files: for s in `seq 1 10`; do mkdir $s; cd $s; for n in `seq 1 $s`; do touch $s-$n.pdf; touch .fake-$s-$n.pdf; done; cd ..; done

Produced this tree:

.
├── 1
│   ├── 1-1.pdf
│   └── .fake-1-1.pdf
├── 10
│   ├── 10-10.pdf
│   ├── 10-1.pdf
│   ├── 10-2.pdf
│   ├── 10-3.pdf
│   ├── 10-4.pdf
│   ├── 10-5.pdf
│   ├── 10-6.pdf
│   ├── 10-7.pdf
│   ├── 10-8.pdf
│   ├── 10-9.pdf
│   ├── .fake-10-10.pdf
│   ├── .fake-10-1.pdf
│   ├── .fake-10-2.pdf
│   ├── .fake-10-3.pdf
│   ├── .fake-10-4.pdf
│   ├── .fake-10-5.pdf
│   ├── .fake-10-6.pdf
│   ├── .fake-10-7.pdf
│   ├── .fake-10-8.pdf
│   └── .fake-10-9.pdf
├── 2
│   ├── 2-1.pdf
│   ├── 2-2.pdf
│   ├── .fake-2-1.pdf
│   └── .fake-2-2.pdf
├── 3
│   ├── 3-1.pdf
│   ├── 3-2.pdf
│   ├── 3-3.pdf
│   ├── .fake-3-1.pdf
│   ├── .fake-3-2.pdf
│   └── .fake-3-3.pdf
├── 4
│   ├── 4-1.pdf
│   ├── 4-2.pdf
│   ├── 4-3.pdf
│   ├── 4-4.pdf
│   ├── .fake-4-1.pdf
│   ├── .fake-4-2.pdf
│   ├── .fake-4-3.pdf
│   └── .fake-4-4.pdf
├── 5
│   ├── 5-1.pdf
│   ├── 5-2.pdf
│   ├── 5-3.pdf
│   ├── 5-4.pdf
│   ├── 5-5.pdf
│   ├── .fake-5-1.pdf
│   ├── .fake-5-2.pdf
│   ├── .fake-5-3.pdf
│   ├── .fake-5-4.pdf
│   └── .fake-5-5.pdf
├── 6
│   ├── 6-1.pdf
│   ├── 6-2.pdf
│   ├── 6-3.pdf
│   ├── 6-4.pdf
│   ├── 6-5.pdf
│   ├── 6-6.pdf
│   ├── .fake-6-1.pdf
│   ├── .fake-6-2.pdf
│   ├── .fake-6-3.pdf
│   ├── .fake-6-4.pdf
│   ├── .fake-6-5.pdf
│   └── .fake-6-6.pdf
├── 7
│   ├── 7-1.pdf
│   ├── 7-2.pdf
│   ├── 7-3.pdf
│   ├── 7-4.pdf
│   ├── 7-5.pdf
│   ├── 7-6.pdf
│   ├── 7-7.pdf
│   ├── .fake-7-1.pdf
│   ├── .fake-7-2.pdf
│   ├── .fake-7-3.pdf
│   ├── .fake-7-4.pdf
│   ├── .fake-7-5.pdf
│   ├── .fake-7-6.pdf
│   └── .fake-7-7.pdf
├── 8
│   ├── 8-1.pdf
│   ├── 8-2.pdf
│   ├── 8-3.pdf
│   ├── 8-4.pdf
│   ├── 8-5.pdf
│   ├── 8-6.pdf
│   ├── 8-7.pdf
│   ├── 8-8.pdf
│   ├── .fake-8-1.pdf
│   ├── .fake-8-2.pdf
│   ├── .fake-8-3.pdf
│   ├── .fake-8-4.pdf
│   ├── .fake-8-5.pdf
│   ├── .fake-8-6.pdf
│   ├── .fake-8-7.pdf
│   └── .fake-8-8.pdf
└── 9
    ├── 9-1.pdf
    ├── 9-2.pdf
    ├── 9-3.pdf
    ├── 9-4.pdf
    ├── 9-5.pdf
    ├── 9-6.pdf
    ├── 9-7.pdf
    ├── 9-8.pdf
    ├── 9-9.pdf
    ├── .fake-9-1.pdf
    ├── .fake-9-2.pdf
    ├── .fake-9-3.pdf
    ├── .fake-9-4.pdf
    ├── .fake-9-5.pdf
    ├── .fake-9-6.pdf
    ├── .fake-9-7.pdf
    ├── .fake-9-8.pdf
    └── .fake-9-9.pdf

10 directories, 110 files

The Lookup

Using JandP's basic find command, and the knowledge revealed in the comments from my other (incorrect) answer, produced an output showing the number of matching files in each directory, excluding any dotfiles:

$ find -type f -iname '*.pdf' ! -iname '.*' -printf '%h\n' | sort | uniq -c
      1 ./1
     10 ./10
      2 ./2
      3 ./3
      4 ./4
      5 ./5
      6 ./6
      7 ./7
      8 ./8
      9 ./9

Throwing a simple awk comparison on the end, produced the expected result:

$ find -type f -iname '*.pdf' ! -iname '.*' -printf '%h\n' | sort | uniq -c | awk '$1 > 1'
     10 ./10
      2 ./2
      3 ./3
      4 ./4
      5 ./5
      6 ./6
      7 ./7
      8 ./8
      9 ./9

The Conclusion

JandP was oh so close. This answer took a little bit of Google-fu, but came together pretty easily from there. My recommendation is to study this scenario and answer, and apply it to your situation at hand. Then, pick it apart. Understand what each command does, why the arguments are stated the way they are, and what their produced output is. Learn why piping them together has the behavior it does. This will take you far in your career in systems administration.

3
  • I realize this doesn't fix the subdir issues. Perhaps a -maxdepth argument to find would fix that.
    – Skudd
    Aug 2, 2019 at 3:14
  • “I only want to filter folders that have HiRes in the filename.” Aug 2, 2019 at 17:54
  • Then change the first -iname '*.pdf' to include the appropriate pattern.
    – Skudd
    Aug 3, 2019 at 18:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.