GNU Parallel is a command-line tool that allows one to run multiple commands in parallel

learn more… | top users | synonyms

3
votes
2answers
71 views

execute a command for each line of stdin immediately?

How do I execute a command for each line of stdin immediately? Example: (echo abc; sleep 10; echo def;) | ??? The xargs tool does not do the job because it blocks until it sees all the data (waits ...
1
vote
1answer
37 views

unable to install gnu parallel in ubuntu 12.04

Travis CI is using Ubuntu 12.04. I want to install GNU Parallel on it and used to be able to with sudo apt-get install parallel but now am not able to. What do I need to do? Do I need to compile it ...
1
vote
0answers
48 views

why is my master node idle when running gnu parallel remotely?

I'm using gnu-parallel to run a number of python scripts in parallel across a number of computers in my gigabit network. The files being processed are located on a tmpfs (RAM) folder on my master ...
1
vote
2answers
132 views

zero a large block device with parallelized dd

We have a sixty terabyte block device which we want to zero using dd. Is it possible to parallelize this operation in order to reduce the amount of time it takes? I've tried running the following ...
2
votes
1answer
65 views

gnu-parallel encrypting files with spaces or special characters?

I'm trying to encrypt a bunch of files with the code below: find . -name "*.vi" | sort | parallel --gnu -j 4 --workdir "$PWD" ' echo "Encrypting {/.} ..." gpg -r user@myemail.com -o "/tank/...
2
votes
2answers
72 views

gnu parallel: forward all of stdin to all processes

I'm trying to process input in different ways in parallel, writing out the resulting lines as they come in. My current approach is this: # process_parallel.sh read input parallel --colsep ' ' --...
1
vote
1answer
92 views

How can I force gnuparallel to carry out a command set sequentially?

Gnu parallel is a powerful tool that I use to run many independant BASH commands as a single set in parallel. I would like to be able to run the same commands SEQUENTIALLY without significant changes ...
3
votes
1answer
421 views

Suppressing stderr in GNU Parallel

I'm using GNU Parallel to concurrently run a command several thousand times. To get logs of the execution I'm using --files and --results. To get nice progress bar while it's running I'm using --eta ...
3
votes
1answer
687 views

Using sed with parallel gives empty output when redirecting to file

I'm using the zsh shell. I am trying to use sed to substitute some text in many files, using parallel to speed up the process. When I tested this on one file and let the command output go to stdout I ...
2
votes
3answers
200 views

Gnu parallel and ack not playing nicely due to stdin, pipe

I'm trying to use parallel and ack together to do some searching in parallel. However, ack seems to insist on using stdin if it finds itself in a pipe, even if you give it files to search: $ echo ...
3
votes
1answer
300 views

Gnu Parallel hangs as one process is “sleeping”

I am running a command in parallel using Gnu Parallel, which has two parameters as input, a directory and a conf file: parallel --gnu my_command ::: (ls -d dir*test) ::: properties.conf I run it on ...
0
votes
2answers
52 views

Debugging output

I attempted to speed up an import of a large dataset with a folder full of .sql files ls -rS *.sql | parallel -j16 mysql -uusername -ppassword dbname "<" Shortly after running this I get an ...
3
votes
1answer
2k views

Batch download of URLs from command line multithreaded

I have 100,000 URLs of small files to download. Would like to use 10 threads and pipelining is a must. I concatenate the result to one file. Current approach is: cat URLS | xargs -P5 -- curl >>...
6
votes
2answers
2k views

How to use GNU parallel with gunzip

I have a directory full of .gz, I want to expand each archive in parallel with GNU parallel. However I did not achieve anything. I tried parallel 'gunzip {}' ::: `ls *.gz` parallel gunzip `ls *.gz` ...
1
vote
2answers
679 views

using awk with parallel

I have about 3,000 files that are each 300MB, and I'd like to search them for a series of substrings as quickly as possible with my 16 core server. This is what I tried but it doesnt seem to ...
4
votes
2answers
2k views

How can I install GNU Parallel alongside Moreutils?

Homebrew has a formula for moreutils and GNU parallel. GNU Parallel conflicts with Moreutils since it also has a binary called parallel, which is just less useful. However I'd still like to install ...
2
votes
2answers
883 views

How do I use GNU split's “filter” option with GNU parallel?

I am trying to split a number of huge gz file into N-line compressed gzipped chunks. To demonstrate, let us consider the following: seq 100 | gzip > big_file0.gz I can split this into multiple ...
2
votes
2answers
1k views

Bash script 'while read' loop causes 'broken pipe' error when run with GNU Parallel

According to the GNU Parallel mailing list this is not a GNU Parallel-specific problem. They suggested that I post my problem here. The error I'm getting is a "broken pipe" error, but I feel I ...