Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am trying to split a number of huge gz file into N-line compressed gzipped chunks.

To demonstrate, let us consider the following:

seq 100 | gzip > big_file0.gz

I can split this into multiple 10-line compressed files as follows:

zcat big_file0.gz | split -l 10 --filter='gzip > $FILE.gz' - big_file0.

Let us assume we have a number of big files big_file0.gz, big_file1.gz ...

I would now like to split each of these files using GNU parallel. Here's the command I come up with:

parallel "zcat {} | split -l 10 --filter='gzip > $FILE.gz' - {.}." ::: big_file0.gz big_file1.gz

However, the shell substitution for $FILE does not work as expected. $FILE is replaced with an empty string, so all the output is written to a file called .gz.

How can I get the $FILE substitution to work as expected in GNU parallel?

share|improve this question
up vote 2 down vote accepted

Shell expansion of variables is converting $FILE to an empty string. You need to put a backslash in front of $FILE to prevent the shell from doing the expansion.

share|improve this answer

Today you would use GNU Parallel's --pipe option:

parallel --seqreplace // "zcat {} | parallel --pipe -N 10 gzip '>{.}.{#}.gz'" ::: big_file0.gz big_file1.gz

If you are OK with appending big_file0.gz big_file1.gz it is even simpler:

zcat big_file0.gz big_file1.gz | parallel --pipe -N 10 gzip '>{#}.gz'
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.