How can I find invalid images?

Question

I have a directory with sub-directories. In the directories, there are a lot of images, crawled from the web.

How do I loop through every file and show those files which are not valid image files?

It should not be based on file extension.

I came up with this script:

find . -name '*.jpg' -o -name '*.jpeg' -o -name '*.gif' -o -name '*.png' | while read FILE; do
    if ! identify "$FILE" &> /dev/null; then
         echo "$FILE"
    fi  
done

But this is not working, because it outputs valide images, too.

Answer 1

find . -type f \
       \( -name '*.jpg' -o -name '*.jpeg' -o -name '*.gif' -o -name '*.png' \) \
       -exec sh -c '! file -b --mime-type "$1" | grep -q "^image/"' sh {} \; \
       -print

My approach uses -exec to perform a custom test on files. A shell is needed to construct a pipe. A separate shell is run for every file with the right extension, therefore the solution performs rather poorly.

The shell runs file -b --mime-type, then grep checks if the result begins with image/. ! at the beginning of the pipe negates its exit status, so the entire -exec test succeeds iff the file is not really an image. The path is then printed.

Notes:

1. Omit -name tests to check all files.
2. Or you may want to use -iname instead of -name.
3. -iname is not required by POSIX though. Neither is -b nor --mime-type option of file.

4. The following yields a slightly different output and it's faster:

   find . -type f \
          \( -name '*.jpg' -o -name '*.jpeg' -o -name '*.gif' -o -name '*.png' \) \
          -exec file --mime-type {} + \
   | grep -v "\bimage/"
   

   but some filenames (e.g. with newlines) or paths (with image/) will break the logic.

Answer 2

You must do this by checking the file type signatures. It can be found here, or you can find it by trial and error checking your files.

For example, JPG signature is FF D8 FF E0, which means its first four bytes must be equal to FF D8 FF E0.

All you need is a tool to match these bytes with those of the file. As an example, hexdump -n 4 -C file.jpg| awk '{print $2 $3 $4 $5}' returns those bytes in hexadecimal format, which can be compared with the desired signature.

If image files you have may be broken, advanced information about file signatures and recovery can be found here. For example, when only a fraction of an image is downloaded.

Answer 3

Checking file extension and magic bytes can easily be spoofed. See https://unix.stackexchange.com/questions/189364/script-to-determine-if-apparent-image-files-are-real-image-files/189367#189367 for inspiration, basically use imagemagick to check if the image is valid, but even then that can be spoofed! So no perfect way to check.