GNU Parallel is a command-line tool that allows one to run multiple commands in parallel

learn more… | top users | synonyms

2
votes
2answers
27 views

gnu parallel: forward all of stdin to all processes

I'm trying to process input in different ways in parallel, writing out the resulting lines as they come in. My current approach is this: # process_parallel.sh read input parallel --colsep ' ' ...
1
vote
1answer
67 views

Installing GNU Parallel on Cygwin [closed]

Is this possible? I have googled and searched at length and found no answer to this question. How can I install GNU Parallel on Cygwin? I wish to run a command similar to this : Link Thanks.
0
votes
0answers
13 views

Which Scientific Linux package has GNU parallel?

I would like to use the GNU parallel utility on a Scientific Linux 6.6 distro so I can do cool stuff like: parallel convert '{}' '{.}.jpg' ::: *.png A Google search didn't help me work out which ...
1
vote
1answer
33 views

How can I force gnuparallel to carry out a command set sequentially?

Gnu parallel is a powerful tool that I use to run many independant BASH commands as a single set in parallel. I would like to be able to run the same commands SEQUENTIALLY without significant changes ...
2
votes
1answer
144 views

Suppressing stderr in GNU Parallel

I'm using GNU Parallel to concurrently run a command several thousand times. To get logs of the execution I'm using --files and --results. To get nice progress bar while it's running I'm using --eta ...
3
votes
1answer
384 views

Using sed with parallel gives empty output when redirecting to file

I'm using the zsh shell. I am trying to use sed to substitute some text in many files, using parallel to speed up the process. When I tested this on one file and let the command output go to stdout I ...
2
votes
3answers
136 views

Gnu parallel and ack not playing nicely due to stdin, pipe

I'm trying to use parallel and ack together to do some searching in parallel. However, ack seems to insist on using stdin if it finds itself in a pipe, even if you give it files to search: $ echo ...
2
votes
1answer
139 views

Gnu Parallel hangs as one process is “sleeping”

I am running a command in parallel using Gnu Parallel, which has two parameters as input, a directory and a conf file: parallel --gnu my_command ::: (ls -d dir*test) ::: properties.conf I run it on ...
0
votes
2answers
48 views

Debugging output

I attempted to speed up an import of a large dataset with a folder full of .sql files ls -rS *.sql | parallel -j16 mysql -uusername -ppassword dbname "<" Shortly after running this I get an ...
3
votes
1answer
1k views

Batch download of URLs from command line multithreaded

I have 100,000 URLs of small files to download. Would like to use 10 threads and pipelining is a must. I concatenate the result to one file. Current approach is: cat URLS | xargs -P5 -- curl ...
5
votes
2answers
984 views

How to use GNU parallel with gunzip

I have a directory full of .gz, I want to expand each archive in parallel with GNU parallel. However I did not achieve anything. I tried parallel 'gunzip {}' ::: `ls *.gz` parallel gunzip `ls *.gz` ...
1
vote
2answers
382 views

using awk with parallel

I have about 3,000 files that are each 300MB, and I'd like to search them for a series of substrings as quickly as possible with my 16 core server. This is what I tried but it doesnt seem to ...
1
vote
2answers
1k views

How can I install GNU Parallel alongside Moreutils?

Homebrew has a formula for moreutils and GNU parallel. GNU Parallel conflicts with Moreutils since it also has a binary called parallel, which is just less useful. However I'd still like to install ...
2
votes
2answers
515 views

How do I use GNU split's “filter” option with GNU parallel?

I am trying to split a number of huge gz file into N-line compressed gzipped chunks. To demonstrate, let us consider the following: seq 100 | gzip > big_file0.gz I can split this into multiple ...
2
votes
2answers
1k views

Bash script 'while read' loop causes 'broken pipe' error when run with GNU Parallel

According to the GNU Parallel mailing list this is not a GNU Parallel-specific problem. They suggested that I post my problem here. The error I'm getting is a "broken pipe" error, but I feel I ...