9

Normally I would just do something like:

tar -czf archive.tar.gz *.csv

But when there are too many files in the directory for the shell to expand on a single line this doesn't work.

In these cases I would normally resort to using find. Something like:

find /path -name '*.csv' -exec tar -rf "./archive.tar.gz" {} +;`

But this only seems to work if I don't include the -z option because you can't append to compressed archives, and using -c instead of -r will overwrite the first archive since find runs tar multiple times.

The only other solution I could come up with is to create a .tar file with find (as above) and then use a second command to compress it. Is there a better way to handle cases like this?

I'm using Ubuntu Linux.

3
  • 4
    You can create a tar archive and compress it later. (As used to be the way we did things back when we walked uphill to school, both ways).
    – Hennes
    Sep 6, 2013 at 22:31
  • 3
    Your find command suffers from the same problem with too many arguments since the glob is expanded before find sees it. You should use find -name '*.csv' instead.
    – slhck
    Sep 7, 2013 at 9:27
  • Indeed, I fixed it now. Sep 7, 2013 at 19:11

2 Answers 2

12

As a robust solution, use find to separate filenames by a null character, and then pipe directly to tar, which reads null-delimited input:

find . -maxdepth 1 -name '*.csv' -print0 |
tar -czf archive.tgz --null -T -

This will now handle all file names correctly and is not limited by the number of files either.

Using ls to generate a list of filenames to be parsed by another program is a common antipattern that should be avoided whenever possible. find can generate null-delimited output (-print0) that most utilities can read or parse further. Since the null character is the only character that cannot appear in a filename (and the /, obviously), you'll always be safe with that.

10

No, you cannot append to a compressed tar file without uncompressing it first.

However, tar can accept its list of files to process from a file, so you can just do:

ls *.csv > temp.txt
tar -zcf ball.tgz -T temp.txt

@slhck points out that the above solution will not work if there are spaces (and probably other annoying characters) in your filenames. This version encloses each filename in double quotes:

ls *.csv | sed -e 's/^\(.*\)$/"\1"/' > temp.txt
tar -zcf ball.tgz -T temp.txt

(This will of course break if you have double quotes in your filenames, in which case you get what you deserve. :)

2
  • 2
    If you want to do it in a single command you could use tar -zcf ball.tgz -T <( ls *.csv) or tar -zcf ball.tgz -T <( find -name '*.csv'). See tldp.org/LDP/abs/html/abs-guide.html#PROCESS-SUB
    – Zoredache
    Sep 6, 2013 at 23:23
  • That's not quite the solution I was thinking of. See my answer for a solution that doesn't choke on file names. Don't blame users for creating files with spaces or double quotes in the path – all Linux tools can handle this correctly, you just need to apply the proper options. In general, as a good tip, once you find yourself using ls to enumerate files other than just looking at them, something will probably break.
    – slhck
    Sep 8, 2013 at 11:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .