Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I am trying to split a number of huge gz file into N-line compressed gzipped chunks.

To demonstrate, let us consider the following:

seq 100 | gzip > big_file0.gz

I can split this into multiple 10-line compressed files as follows:

zcat big_file0.gz | split -l 10 --filter='gzip > $FILE.gz' - big_file0.

Let us assume we have a number of big files big_file0.gz, big_file1.gz ...

I would now like to split each of these files using GNU parallel. Here's the command I come up with:

parallel "zcat {} | split -l 10 --filter='gzip > $FILE.gz' - {.}." ::: big_file0.gz big_file1.gz

However, the shell substitution for $FILE does not work as expected. $FILE is replaced with an empty string, so all the output is written to a file called .gz.

How can I get the $FILE substitution to work as expected in GNU parallel?

share|improve this question

1 Answer 1

up vote 2 down vote accepted

Shell expansion of variables is converting $FILE to an empty string. You need to put a backslash in front of $FILE to prevent the shell from doing the expansion.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.