GNU Parallel is a command-line tool that allows one to run multiple commands in parallel

learn more… | top users | synonyms

2
votes
1answer
50 views

Suppressing stderr in GNU Parallel

I'm using GNU Parallel to concurrently run a command several thousand times. To get logs of the execution I'm using --files and --results. To get nice progress bar while it's running I'm using --eta ...
3
votes
1answer
229 views

Using sed with parallel gives empty output when redirecting to file

I'm using the zsh shell. I am trying to use sed to substitute some text in many files, using parallel to speed up the process. When I tested this on one file and let the command output go to stdout I ...
2
votes
3answers
107 views

Gnu parallel and ack not playing nicely due to stdin, pipe

I'm trying to use parallel and ack together to do some searching in parallel. However, ack seems to insist on using stdin if it finds itself in a pipe, even if you give it files to search: $ echo ...
2
votes
1answer
96 views

Gnu Parallel hangs as one process is “sleeping”

I am running a command in parallel using Gnu Parallel, which has two parameters as input, a directory and a conf file: parallel --gnu my_command ::: (ls -d dir*test) ::: properties.conf I run it on ...
0
votes
2answers
47 views

Debugging output

I attempted to speed up an import of a large dataset with a folder full of .sql files ls -rS *.sql | parallel -j16 mysql -uusername -ppassword dbname "<" Shortly after running this I get an ...
3
votes
1answer
1k views

Batch download of URLs from command line multithreaded

I have 100,000 URLs of small files to download. Would like to use 10 threads and pipelining is a must. I concatenate the result to one file. Current approach is: cat URLS | xargs -P5 -- curl ...
5
votes
2answers
648 views

How to use GNU parallel with gunzip

I have a directory full of .gz, I want to expand each archive in parallel with GNU parallel. However I did not achieve anything. I tried parallel 'gunzip {}' ::: `ls *.gz` parallel gunzip `ls *.gz` ...
1
vote
2answers
306 views

using awk with parallel

I have about 3,000 files that are each 300MB, and I'd like to search them for a series of substrings as quickly as possible with my 16 core server. This is what I tried but it doesnt seem to ...
1
vote
2answers
888 views

How can I install GNU Parallel alongside Moreutils?

Homebrew has a formula for moreutils and GNU parallel. GNU Parallel conflicts with Moreutils since it also has a binary called parallel, which is just less useful. However I'd still like to install ...
2
votes
1answer
425 views

How do I use GNU split's “filter” option with GNU parallel?

I am trying to split a number of huge gz file into N-line compressed gzipped chunks. To demonstrate, let us consider the following: seq 100 | gzip > big_file0.gz I can split this into multiple ...
2
votes
2answers
992 views

Bash script 'while read' loop causes 'broken pipe' error when run with GNU Parallel

According to the GNU Parallel mailing list this is not a GNU Parallel-specific problem. They suggested that I post my problem here. The error I'm getting is a "broken pipe" error, but I feel I ...