1

I've been battling with this issue for a couple of days and haven't been successful in finding a solution on StackExchange nor any Google searches.

I have the following directory structure (simplified because in reality there are 20,000+ files). Each year has a directory for each month. Each month has a directory for each day.

2017
  01
    01
    ...
    31
  ...
  12
2018
2019

Within these directories are images:

123_43e6d929fcdbfa00aee1892893127b34.png
4567_24a847285bae9ddb6d3c33d237c6d481.jpg

What I want to achieve

I would like to create thumbnail versions (resized and cropped) of each file saved in the same destination but with suffixes. So the above example would be:

123_43e6d929fcdbfa00aee1892893127b34.png
123_43e6d929fcdbfa00aee1892893127b34-120x160.png
123_43e6d929fcdbfa00aee1892893127b34-300x300.png
123_43e6d929fcdbfa00aee1892893127b34-800x800.png
4567_24a847285bae9ddb6d3c33d237c6d481.jpg
4567_24a847285bae9ddb6d3c33d237c6d481-120x160.jpg
4567_24a847285bae9ddb6d3c33d237c6d481-300x300.jpg
4567_24a847285bae9ddb6d3c33d237c6d481-800x800.jpg

What I've tried

for file in *.jpg; do convert $file -set filename:base "%[base]" -resize 120x160^ -gravity center -extent 120x160 -quality 90\> "%[filename:base]-120x160.jpg"; done

I think this causes a bit of a loop because we're creating new images as we're looping through. So I then created a PHP script to add a check and came up with this:

foreach ($all_files as $file) {
    $path = $file->getPathname();

    $resized = str_replace_last('.', '-120x160.', $path);             
    if (preg_match('/\-\d{3}x\d{3}\./', $path) || file_exists($resized) || substr($path,-4) != '.jpg') continue;

    exec("convert \"$path\" -set filename:base \"%d/%[base]-120x160.jpg\" -resize 120x160^ -gravity center -extent 120x160 -quality 95\> \"%[filename:base]\";
          convert \"$path\" -set filename:base \"%d/%[base]-300x300.jpg\" -resize 300x300^ -gravity center -extent 300x300 -quality 95\> \"%[filename:base]\";
          convert \"$path\" -set filename:base \"%d/%[base]-800x800.jpg\" -resize 800x800^ -gravity center -extent 800x800 -quality 95\> \"%[filename:base]\"");
}

This works great, however it's going to take an eternity to create all 66,000 thumbnails as well as kill server performance. I thought maybe mogrify would be the answer but has to be run in each directory and there's thousands of directories.

Could anybody please suggest a solution - shell, bash, php, anything that can achieve this?

Thanks

1

I think this causes a bit of a loop because we're creating new images as we're looping through.

No, the list of files is created, then iterated, so you are not going to reprocess the files you create. If you want to avoid reprocessing the file if you rerun the script it's another matter:

  • You can avoid processing the thumbs by excluding them with a negative match (or a more stringent positive match),
  • But you should also perhaps have a some logic to avoid processing a file if the thumbs are already there (I assume the source image is never updated)

... but as far as I can tell this is what you PHP code does.

This works great, however it's going to take an eternity to create all 66,000 thumbnails as well as kill server performance.

Producing three thumbs from 66K files is indeed going to take some time. But whatever you do, you have to perform the scaling and re-encoding for each thumb and this is what is going to take some CPU. You could envision to avoid reading the source file three times, but file caching in Linux is going to take care of that.

If you don't want the process to hog your server, you can add a 1-2 second sleep command after each image (or each day directory).

| improve this answer | |
  • Thanks for your help @xenoid. I have decided to stick with the PHP approach as this is within my comfort zone. I'm running the script with 'nice' such that it shouldn't hog the CPU – kinggs Mar 22 '19 at 8:34
0

Here's an example using Make. The advantage here is that it won't try to make anything you already have. So it's safe to run it for a bit and then stop it, then run it again later when you have time. You can also give the -j flag, eg "make -j 4" will use up to 4 CPU cores at a time (which will be quicker, but will use more resources.)

ifeq (${MAKELEVEL}, 0)
   # path to this Makefile
   mfile = Makefile
endif

subdirs = $(shell find * -type d -print -prune)
files = $(wildcard *.jpg) $(wildcard *.png)
thumbs = $(wildcard *-120x160.*) $(wildcard *-300x300.*) $(wildcard *-800x800.*)
notthumbs = ${basename $(filter-out $(thumbs), $(files) ) }
neededthumbs = $(foreach f, ${notthumbs}, ${f}-120x160.jpg ${f}-300x300.jpg ${f}-800x800.jpg)

all: thumbs recurse

thumbs: ${neededthumbs}

recurse:
        $(foreach f, ${subdirs}, $(MAKE) -C ${f} -f ../${mfile} mfile=../${mfile} all ; )

%-120x160.jpg: %.jpg
        echo convert $< -resize 120x160 $@
%-300x300.jpg: %.jpg
        echo convert $< -resize 300x300 $@
%-800x800.jpg: %.jpg
        echo convert $< -resize 800x800 $@
%-120x160.jpg: %.png
        echo convert $< -resize 120x160 $@
%-300x300.jpg: %.png
        echo convert $< -resize 300x300 $@
%-800x800.jpg: %.png
        echo convert $< -resize 800x800 $@

Copy the code into a file (note that the indents MUST be TABS! I know, right?) called "Makefile" at the root of your tree. cd to that directory and type "make" (or "make -j 4" etc.) To make it run slower, you can use "nice" as in "nice make"

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.