Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a directory full of .gz, I want to expand each archive in parallel with GNU parallel. However I did not achieve anything.

I tried

parallel 'gunzip {}' ::: `ls *.gz`
parallel gunzip `ls *.gz`

with no results, bash tells me:

/bin/bash: archive1.gz: command not found
...

What am I doing wrong?

Thanks

share|improve this question
2  
GNU parallel , +1 –  nsd Jun 8 '13 at 5:20
    
GNU parallel isn't available on my platform. Is there a way to do this with xargs (I don't want to run more gz instances than my number of cores). –  user2284570 Aug 29 at 21:32

2 Answers 2

up vote 4 down vote accepted

I found this, which suggests using the --gnu flag:

parallel --gnu gunzip  ::: *gz

If this works, you should either delete /etc/parallel/config or change its contents to --gnu rather than --tollef (as root):

echo "--gnu" > /etc/parallel/config

Also, never parse the output of ls., use globbing as I have above or find instead:

find . -name "*gz*" -print0 | parallel -q0 gunzip 
share|improve this answer
    
It totally works.. thanks for tips and corrections. –  fbrundu Jun 8 '13 at 15:46
    
Parsing the output of ls is less of an issue when piping into GNU Parallel. As long as you do not have malicious users making filenames containing \n you are safe. In this case, however, ::: *.gz would probably be the best. –  Ole Tange Jun 10 '13 at 20:00
    
Putting \n in filenames is pretty fun indeed, thanks for the tip. However for small admin tasks it shouldn't be an issue so it's not worth taking a habit of using some other tool IMHO. –  Aki Jan 15 at 15:53
    
@Aki In fact it is VERY much worth making a habit of using the correct tool for the job. ls output changes between *nix flavors, and locale settings, it tends to break on any kind of white space, not just newlines, you willavoid a lot of pain if you get used to not parsing ls. Globbing is both simpler and safer. –  terdon Jan 15 at 15:55
1  
@Aki for i in *; do command "$i"; done is easier to write and safer. Why would you want to learn an idiom that is harder and more risky? Try for i in $(ls) in a directory that contains files with spaces in their names. Seriously, parsing ls is almost always not the right or easiest way to do things. If globbing is not an option, use find. If you insist on using ls at least don't use a for loop, this is much safer: ls | while read i; do command "$i"; done at least that won't break on spaces. –  terdon Jan 15 at 18:44

Doing this works:

   ls *.gz | parallel -t gunzip

The -t is optional but is useful as it shows you the commands that are executed on stderr.

I'm not sure you are doing anything wrong ::: should work (it's meant to be equivalent) but not even the examples in the man page work for me.

Update: the --gnu flag makes it work as terdon said.

share|improve this answer
    
I don't know why but it doesn't work : gzip: stdin: not in gzip format .. Thanks anyway –  fbrundu Jun 8 '13 at 15:47
    
Strange, works for me –  parkydr Jun 8 '13 at 15:57
    
Yeah it's weird.. Which platform are you using? –  fbrundu Jun 8 '13 at 16:04
    
Debian jessie/sid (testing). Did your files have spaces in the names? –  parkydr Jun 8 '13 at 16:22
    
Spaces worked as well –  parkydr Jun 8 '13 at 16:25

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.