Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I use Linux. I coded a screenshot program some time ago and now I have 9 GIG of screenshots, 60000 JPEGs, most of them look pretty similar, and I have 300 MB of disk space remaining.

What are some good ways to start to compress batches of them (or all of them) in the background given the limited space?

The problem with compressing the folder all at once is that I wouldn't have enough disk space for that. It seems the process needs to be broken down into chunks.

So maybe something like:

  1. Get a list of all the files
  2. Add a chunk of the files (say, 20) to a compressed archive.
  3. Once it is done and saved successfully, delete the chunk of files
  4. Process the next chunk.
share|improve this question
    
When you say 'compress' do you mean you want to run them through a normal compression engine (eg. gzip, bzip2) (relatively ineffective) or do you want to re-encode the JPEGs with lower quality? –  Mike Fitzpatrick Jul 18 '11 at 0:45
    
I just did a LZMA on a tarred portion and it reduced the size from 4MB to 2.3MB. The individual images are fairly compressed themselves so I don't want any more lossy compression / re-encoding. –  Luke Stanley Jul 18 '11 at 0:52
    
I figured it out myself with a Python script and 7za: pastebin.com/FZGAmpXB I now have almost 400MB of space already, it was 300MB earlier, and it keeps churning away. My rep is too low to post my answer at the moment. NOTE: It does NOT check the compression worked, ('Everything is Ok'). That part is DIY ATM, disable rm if you like. –  Luke Stanley Jul 18 '11 at 2:16
    
If "most of them look pretty similar" why do you need to keep them all? –  glenn jackman Jul 18 '11 at 3:32
    
Glenn, if anyone really needs to compress files it is people with low disk space - this is an important problem to solve. Questioning the notability of the files is not very productive, it is 2011, we live in the GMail era. –  Luke Stanley Jul 18 '11 at 4:02
show 8 more comments

2 Answers 2

If the files you want to compress are in the directory <directory> and the archive you want to create is <archive>, you could:

find <directory> -type f -exec zip -g <archive> {} \; -exec rm {} \;

or with 7za:

find <directory> -type f -exec 7za a <archive> {} \; -exec rm {} \;
share|improve this answer
    
This doesn't break the task down (NEEDED IF LOW ON SPACE!) OR error check. –  Luke Stanley Jul 19 '11 at 23:29
    
Actually, it does break the task down as the files are added one by one to the archive (1st exec) and removed after being added and before adding the next file (2nd exec). For error checking, the 1st exec will return true if zip of 7za return an error code of 0, false otherwise. If false is returned, the next exec should not be executed. Obviously, you should test it before trusting it. –  jfgagne Jul 20 '11 at 8:05
    
Maybe by 'breaking the task down' you meant creating many archives. However this makes finding the JPG you want more difficult as you will have to check many archives or build an index. Have you tried compressing the file one by one and does this save you enough space ? Maybe a better solution would be filesystem compression... –  jfgagne Jul 20 '11 at 8:13
    
I think the point missed here is that as the files are quite similar, some redundancy may be identified by grouping some of them together in a tar and then compressing that. Unfortunately, as jpeg is itself compressed, one bit difference early in a file will make the rest of it incomparable with another. Otherwise, the command xz -9 * would compress all the files individually. PNG may have been better as a format for screenshots that don't contain anything photographic. –  Andy Lee Robinson Jul 27 '11 at 15:15
add comment
up vote 0 down vote accepted

I figured it out myself, this is what I did, with a little Python: http://pastebin.com/w6mSmFTW

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.