1

I use Linux. I coded a screenshot program some time ago and now I have 9 GIG of screenshots, 60000 JPEGs, most of them look pretty similar, and I have 300 MB of disk space remaining.

What are some good ways to start to compress batches of them (or all of them) in the background given the limited space?

The problem with compressing the folder all at once is that I wouldn't have enough disk space for that. It seems the process needs to be broken down into chunks.

So maybe something like:

  1. Get a list of all the files
  2. Add a chunk of the files (say, 20) to a compressed archive.
  3. Once it is done and saved successfully, delete the chunk of files
  4. Process the next chunk.
13
  • When you say 'compress' do you mean you want to run them through a normal compression engine (eg. gzip, bzip2) (relatively ineffective) or do you want to re-encode the JPEGs with lower quality? Jul 18, 2011 at 0:45
  • I just did a LZMA on a tarred portion and it reduced the size from 4MB to 2.3MB. The individual images are fairly compressed themselves so I don't want any more lossy compression / re-encoding. Jul 18, 2011 at 0:52
  • I figured it out myself with a Python script and 7za: pastebin.com/FZGAmpXB I now have almost 400MB of space already, it was 300MB earlier, and it keeps churning away. My rep is too low to post my answer at the moment. NOTE: It does NOT check the compression worked, ('Everything is Ok'). That part is DIY ATM, disable rm if you like. Jul 18, 2011 at 2:16
  • If "most of them look pretty similar" why do you need to keep them all? Jul 18, 2011 at 3:32
  • Glenn, if anyone really needs to compress files it is people with low disk space - this is an important problem to solve. Questioning the notability of the files is not very productive, it is 2011, we live in the GMail era. Jul 18, 2011 at 4:02

2 Answers 2

0

If the files you want to compress are in the directory <directory> and the archive you want to create is <archive>, you could:

find <directory> -type f -exec zip -g <archive> {} \; -exec rm {} \;

or with 7za:

find <directory> -type f -exec 7za a <archive> {} \; -exec rm {} \;
4
  • This doesn't break the task down (NEEDED IF LOW ON SPACE!) OR error check. Jul 19, 2011 at 23:29
  • Actually, it does break the task down as the files are added one by one to the archive (1st exec) and removed after being added and before adding the next file (2nd exec). For error checking, the 1st exec will return true if zip of 7za return an error code of 0, false otherwise. If false is returned, the next exec should not be executed. Obviously, you should test it before trusting it.
    – jfg956
    Jul 20, 2011 at 8:05
  • Maybe by 'breaking the task down' you meant creating many archives. However this makes finding the JPG you want more difficult as you will have to check many archives or build an index. Have you tried compressing the file one by one and does this save you enough space ? Maybe a better solution would be filesystem compression...
    – jfg956
    Jul 20, 2011 at 8:13
  • I think the point missed here is that as the files are quite similar, some redundancy may be identified by grouping some of them together in a tar and then compressing that. Unfortunately, as jpeg is itself compressed, one bit difference early in a file will make the rest of it incomparable with another. Otherwise, the command xz -9 * would compress all the files individually. PNG may have been better as a format for screenshots that don't contain anything photographic. Jul 27, 2011 at 15:15
0

I figured it out myself, this is what I did, with a little Python: http://pastebin.com/w6mSmFTW

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .