2

I want to gzip the contents of a couple of thousand of tiny files into a single file. While I could do it with something like for file in $(find . -iname 'pattern'); do; cat $file | gzip - >> zipped.gz; done;, this achieves pretty bad compression on the first go. While rezipping it is quite easy with zcat zipped.gz | gzip --best > rezipped.gz I'd like to know if someone knows a nice way to do this in a single pass.

2
  • Does it have to be gzip? See here.
    – Kenster
    Sep 26, 2014 at 14:51
  • The end goal is to be able to search, fast, through the contents of the files. The files compress really well, so what used to be a grep over 2GB over a lot of files is now a grep over 14MB of zipped data, which is a lot faster. It doesn't have to be gzip, but with bzip2 being slower (afaik) at decompression, I think it's the better alternative. Unless bzip has an option to compress all the files listed in another file? That'd be nice.
    – Nick
    Sep 29, 2014 at 9:33

2 Answers 2

3

When it turned out that best behaviour is when you cat all files into a single stream, I was figuring things out with a loop. But then I realized that there's an even easier (and better) way:

find . -iname 'pattern' -exec cat {} \; | gzip --best - > file.gz

All the invocations of cat will go to the stdout of find, there's only one invocation of gzip. Rezipping the result yields the same file size. You may be able to get an even better result (in terms of no. of invocations of cat, not in file size) if you use the + version of -exec (see the find man page) but I haven't tested that.

1
  • A winner is you! The {} + version works fine for my 50k files.
    – Nick
    Oct 1, 2014 at 12:11
1
find . -iname 'pattern' | xargs gzip -9 -v

EDIT

It seems that when you cat the file gzip is better able to zip it.

This may work:

for TXT in $(find /PATH/TO/TXT/FILES -iname '*.txt'); do cat ${TXT} | gzip -9 > ${TXT}.gz; done

On my mac, the original text file was not removed. Thus, both the original text file and the zipped file were present after running the script.

You could easily add

rm -f /PAHT/TO/TXT/FILES/${TXT}

to the loop to get rid of the plain text files.

5
  • Of course! I haven't even tried it, because I didn't think it would work for 50k files, but it works fine...
    – Nick
    Sep 26, 2014 at 8:39
  • I'm sorry, but this doesn't achieve the best possible compression. Not sure why. After find . -iname '*.gtr' | xargs gzip --best -vkc > dazip.gz, ls -la dazip.gz says it's 19945871 bytes, and then after zcat dazip.gz | gzip --best > darezip.gz, darezip is just 8641529 bytes, so unmarking as 'the' answer, sorry.
    – Nick
    Sep 26, 2014 at 10:12
  • Before the edit I think you had a pretty good answer (I removed my 'mark as answer', but not the upvote!), so could you bring that back?
    – Nick
    Sep 29, 2014 at 9:29
  • I left my original comment in place. Everything underneath "EDIT" is what I added later.
    – Vincent
    Sep 30, 2014 at 2:03
  • Huurrrr. You are right, haha. Sorry. But yeah, that edit doesn't solve the problem of getting all the contents of those files into a single file.
    – Nick
    Sep 30, 2014 at 9:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .