I have 3 million JPG files stored in a Linux CentOS 6 server.

I want to change quality to %50 file size over 1 megabyte. I wrote this command but got an "argument list too long" error:

$ find -type f -name "*..jpg" -size +1M | xargs mogrify -quality 50 *.jpg
bash: /usr/bin/xargs: Argument list too long

How can I change the quality of millions of files?

  • 2
    Why do you add *.jpg to xargs? It will get the files from find. – choroba Apr 24 '14 at 13:31

xargs supports a -n argument to limit the amount of arguments passed to whatever it calls:

find -type f -name '*.jpg' -size +1M -print0 | xargs -0 -n1 mogrify -quality 50

This will launch mogrify once per image. As mogrify can only process one file at the time, this is the way to go.

  • mogrify's documentation shows examples with *.jpg. – choroba Apr 24 '14 at 13:48
  • Ah, I blindly trusted the manpage: "SYNOPSIS mogrify [options] input-file" - indicating it only supports one. – Dennis Kaarsemaker Apr 24 '14 at 14:29

When using find and xargs, you don't need to name the files for xargs. It will get the list of files from find:

find -print0 -type f -name '*.jpg' -size +1M | xargs -0 -n100 mogrify -quality 50

-n100 will process the images by 100s. -print0 and -0 will make the pipe work even if the filenames contain whitespace.

You can also call mogrify directly from find, ideally if it supports the + ending for exec:

find  -type f -name '*.jpg' -size +1M -exec mogrify -quality 50 {} +
  • This will still make the argument list too long in the first command, if all are passed to xargs in one. Also, you should pipe from find to xargs with find ... -print0 | xargs -0. – slhck Apr 24 '14 at 13:39
  • Well, luckily, find can do it all. :D Instead of ending the -exec command with +, it can also be terminated with ;. This executes the command once per result. Don’t forget to escape ;! – Daniel B Apr 24 '14 at 13:49
  • @DanielB: What is the advantage of ; over + in this case? – choroba Apr 24 '14 at 13:51
  • 1
    It doesn’t result in a command line that’s too long. Although find might already work around that, who knows. It’s also suitable for commands that don’t accept batch input. – Daniel B Apr 24 '14 at 13:57
  • Well, although it may be a bit spammy, I discovered xargs and find are, in fact, smart enough to split the arguments collection automatically. – Daniel B Apr 24 '14 at 14:02

A cross-platform solution with Python+convert: it will convert all PDF files of the current directory into PNG files (you can change to JPG if you prefer) multithreadedly.

from __future__ import print_function
import os
import glob
import multiprocessing      

def convert_to_png(pdf_filepath):
    '''
    Convert PDF file to PNG file
    '''
    png_filepath = '{0}.png'.format(pdf_filepath[:-4])
    print('pdf_filepath: {0}'.format(pdf_filepath))
    print('png_filepath: {0}'.format(png_filepath))
    command = 'convert -background white -alpha off -geometry 1600x1600 -density 200x200 -quality 100 -resize 800x {0} {1}'.format(pdf_filepath, png_filepath)
    print(command)
    os.system(command)

def main():
    pdf_filepaths = glob.iglob(os.path.join('.','*.pdf'))
    pool = multiprocessing.Pool(processes=4)
    pool.map(convert_to_png, pdf_filepaths)
    pool.close()
    pool.join()   
    print('done')

if __name__ == "__main__":
    main()
    #cProfile.run('main()') # if you want to do some profiling

This requires Imagemagick and Ghostscript to be installed. Works on Linux/Mac OS X/Microsoft Windows.

If you prefer to add the filename on each image, you can replace the command in convert_to_png() by:

command = 'convert  -background white -alpha off -geometry 1600x1600 -density 200x200 -quality 100 -annotate +50+50 {2} -resize 800x {0} {1}'.format(pdf_filepath, png_filepath, os.path.basename(pdf_filepath))

(See -annotate documentation)

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.