2

I want to gzip the contents of a couple of thousand of tiny files into a single file. While I could do it with something like for file in $(find . -iname 'pattern'); do; cat $file | gzip - >> zipped.gz; done;, this achieves pretty bad compression on the first go. While rezipping it is quite easy with zcat zipped.gz | gzip --best > rezipped.gz I'd like to know if someone knows a nice way to do this in a single pass.

  • Does it have to be gzip? See here. – Kenster Sep 26 '14 at 14:51
  • The end goal is to be able to search, fast, through the contents of the files. The files compress really well, so what used to be a grep over 2GB over a lot of files is now a grep over 14MB of zipped data, which is a lot faster. It doesn't have to be gzip, but with bzip2 being slower (afaik) at decompression, I think it's the better alternative. Unless bzip has an option to compress all the files listed in another file? That'd be nice. – Nick Sep 29 '14 at 9:33
3

When it turned out that best behaviour is when you cat all files into a single stream, I was figuring things out with a loop. But then I realized that there's an even easier (and better) way:

find . -iname 'pattern' -exec cat {} \; | gzip --best - > file.gz

All the invocations of cat will go to the stdout of find, there's only one invocation of gzip. Rezipping the result yields the same file size. You may be able to get an even better result (in terms of no. of invocations of cat, not in file size) if you use the + version of -exec (see the find man page) but I haven't tested that.

|improve this answer|||||
  • A winner is you! The {} + version works fine for my 50k files. – Nick Oct 1 '14 at 12:11
1
find . -iname 'pattern' | xargs gzip -9 -v

EDIT

It seems that when you cat the file gzip is better able to zip it.

This may work:

for TXT in $(find /PATH/TO/TXT/FILES -iname '*.txt'); do cat ${TXT} | gzip -9 > ${TXT}.gz; done

On my mac, the original text file was not removed. Thus, both the original text file and the zipped file were present after running the script.

You could easily add

rm -f /PAHT/TO/TXT/FILES/${TXT}

to the loop to get rid of the plain text files.

|improve this answer|||||
  • Of course! I haven't even tried it, because I didn't think it would work for 50k files, but it works fine... – Nick Sep 26 '14 at 8:39
  • I'm sorry, but this doesn't achieve the best possible compression. Not sure why. After find . -iname '*.gtr' | xargs gzip --best -vkc > dazip.gz, ls -la dazip.gz says it's 19945871 bytes, and then after zcat dazip.gz | gzip --best > darezip.gz, darezip is just 8641529 bytes, so unmarking as 'the' answer, sorry. – Nick Sep 26 '14 at 10:12
  • Before the edit I think you had a pretty good answer (I removed my 'mark as answer', but not the upvote!), so could you bring that back? – Nick Sep 29 '14 at 9:29
  • I left my original comment in place. Everything underneath "EDIT" is what I added later. – Vincent Sep 30 '14 at 2:03
  • Huurrrr. You are right, haha. Sorry. But yeah, that edit doesn't solve the problem of getting all the contents of those files into a single file. – Nick Sep 30 '14 at 9:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.