I want to gzip the contents of a couple of thousand of tiny files into a single file. While I could do it with something like for file in $(find . -iname 'pattern'); do; cat $file | gzip - >> zipped.gz; done;
, this achieves pretty bad compression on the first go. While rezipping it is quite easy with zcat zipped.gz | gzip --best > rezipped.gz
I'd like to know if someone knows a nice way to do this in a single pass.
-
Does it have to be gzip? See here.– KensterSep 26, 2014 at 14:51
-
The end goal is to be able to search, fast, through the contents of the files. The files compress really well, so what used to be a grep over 2GB over a lot of files is now a grep over 14MB of zipped data, which is a lot faster. It doesn't have to be gzip, but with bzip2 being slower (afaik) at decompression, I think it's the better alternative. Unless bzip has an option to compress all the files listed in another file? That'd be nice.– NickSep 29, 2014 at 9:33
2 Answers
When it turned out that best behaviour is when you cat all files into a single stream, I was figuring things out with a loop. But then I realized that there's an even easier (and better) way:
find . -iname 'pattern' -exec cat {} \; | gzip --best - > file.gz
All the invocations of cat will go to the stdout of find, there's only one invocation of gzip. Rezipping the result yields the same file size. You may be able to get an even better result (in terms of no. of invocations of cat, not in file size) if you use the + version of -exec (see the find man page) but I haven't tested that.
find . -iname 'pattern' | xargs gzip -9 -v
EDIT
It seems that when you cat the file gzip is better able to zip it.
This may work:
for TXT in $(find /PATH/TO/TXT/FILES -iname '*.txt'); do cat ${TXT} | gzip -9 > ${TXT}.gz; done
On my mac, the original text file was not removed. Thus, both the original text file and the zipped file were present after running the script.
You could easily add
rm -f /PAHT/TO/TXT/FILES/${TXT}
to the loop to get rid of the plain text files.
-
Of course! I haven't even tried it, because I didn't think it would work for 50k files, but it works fine...– NickSep 26, 2014 at 8:39
-
I'm sorry, but this doesn't achieve the best possible compression. Not sure why. After
find . -iname '*.gtr' | xargs gzip --best -vkc > dazip.gz
,ls -la dazip.gz
says it's 19945871 bytes, and then afterzcat dazip.gz | gzip --best > darezip.gz
, darezip is just 8641529 bytes, so unmarking as 'the' answer, sorry.– NickSep 26, 2014 at 10:12 -
Before the edit I think you had a pretty good answer (I removed my 'mark as answer', but not the upvote!), so could you bring that back?– NickSep 29, 2014 at 9:29
-
I left my original comment in place. Everything underneath "EDIT" is what I added later.– VincentSep 30, 2014 at 2:03
-
Huurrrr. You are right, haha. Sorry. But yeah, that edit doesn't solve the problem of getting all the contents of those files into a single file.– NickSep 30, 2014 at 9:50