I have a lot of—hundreds of thousands—JPEG images, and most of them are very similar to each other. While they aren’t of high quality and take about 100+ kB each, this adds up to a lot of storage. Are there any options to compress them which take the similarity into account?

I thought about something like using a video compressor which should perform well, but the images—not all of them, of course, but blocks of several thousands sequential ones—are often needed as separate files for analysis, with correct file names. Also new images are added at a rate of about 100/hour. So, just converting them all to a video once doesn’t suffice. I suppose using Fuse can help somehow? There doesn’t seem to be a ready-to-use solution, so any suggestions are welcome.

  • What do you mean by “I suppose using fuse…” What “fuse” do you mean? Perhaps this piece of software called FuseCompress? Mar 24 '15 at 6:32
  • @JakeGould by fuse I mean just FUSE: Filesystem in Userspace. The one you referred to compresses files individually, and it doesn't seem to help in my case.
    – aplavin
    Mar 24 '15 at 6:35
  • Some flavour of block level deduplication might be useful here. Since jpeg is already compressed, recompressing them isn't likely to work too well I suspect.
    – Journeyman Geek
    Mar 24 '15 at 6:41
  • @JourneymanGeek: note that images come already as jpegs, so there are probably no regions which are 100% similar. I actually tried compressing a subset of them to a video (mkv) using ffmpeg, and the size dropped more than 10 times.
    – aplavin
    Mar 24 '15 at 6:44
  • How are you doing the analysis? Would it be possible to adapt your routines to use video frames instead? You could store the filename for each frame somewhere else (though presumably these are autogenerated anyway). Mar 24 '15 at 8:23

To find similar images you can use software like SimilarImages: https://tn123.org/simimages/

or Image Comparer: http://image-comparer.software.informer.com/

or many others: http://similar-images.software.informer.com/

Generally speaking, JPEGS are already highly compressed images so compressing them with a standard WINRAR\WINZIP\7-ZIP program would give little to no further compression. I'd recommend what was once used in Windows XP, called imageresizer, from Microsoft Power Tools, is now available for Vista and 7, here: https://imageresizer.codeplex.com/

It does batch resizing and the image looks great even after the size is reduced to almost 10 times the original size.

  • I thought it was clear. You use a program to get all the similar images and then you can use the "Resize Images" to compress them further. That's exactly what the asker needs. One doesn't need to downvote an answer because he didn't understand it. He can just ask - like you did. Mar 24 '15 at 9:26
  • Some people might not understand just because they are not as smart as others. Don't rush to judge someone because you THINK you don't understand. My answer is as clear as Cristal if you read the question and answer. Maybe the voter had a different opinion on the askers needs. Stop commenting on my answers and focus on answering others. Mar 24 '15 at 11:40

I was once using Antelope by Voralent, is can save up to 20% of space or something like that, as far as I remember

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.