GNU Parallel is a command-line tool that allows one to run multiple commands in parallel

learn more… | top users | synonyms

2
votes
1answer
40 views

Gnu Parallel hangs as one process is “sleeping”

I am running a command in parallel using Gnu Parallel, which has two parameters as input, a directory and a conf file: parallel --gnu my_command ::: (ls -d dir*test) ::: properties.conf I run it on ...
0
votes
2answers
39 views

Debugging output

I attempted to speed up an import of a large dataset with a folder full of .sql files ls -rS *.sql | parallel -j16 mysql -uusername -ppassword dbname "<" Shortly after running this I get an ...
2
votes
1answer
358 views

Batch download of URLs from command line multithreaded

I have 100,000 URLs of small files to download. Would like to use 10 threads and pipelining is a must. I concatenate the result to one file. Current approach is: cat URLS | xargs -P5 -- curl ...
5
votes
2answers
276 views

How to use GNU parallel with gunzip

I have a directory full of .gz, I want to expand each archive in parallel with GNU parallel. However I did not achieve anything. I tried parallel 'gunzip {}' ::: `ls *.gz` parallel gunzip `ls *.gz` ...
1
vote
2answers
201 views

using awk with parallel

I have about 3,000 files that are each 300MB, and I'd like to search them for a series of substrings as quickly as possible with my 16 core server. This is what I tried but it doesnt seem to ...
1
vote
1answer
526 views

How can I install GNU Parallel alongside Moreutils?

Homebrew has a formula for moreutils and GNU parallel. GNU Parallel conflicts with Moreutils since it also has a binary called parallel, which is just less useful. However I'd still like to install ...
2
votes
1answer
256 views

How do I use GNU split's “filter” option with GNU parallel?

I am trying to split a number of huge gz file into N-line compressed gzipped chunks. To demonstrate, let us consider the following: seq 100 | gzip > big_file0.gz I can split this into multiple ...
2
votes
2answers
711 views

Bash script 'while read' loop causes 'broken pipe' error when run with GNU Parallel

According to the GNU Parallel mailing list this is not a GNU Parallel-specific problem. They suggested that I post my problem here. The error I'm getting is a "broken pipe" error, but I feel I ...