According to the GNU Parallel mailing list this is not a GNU Parallel-specific problem. They suggested that I post my problem here.

The error I'm getting is a "broken pipe" error, but I feel I should first explain the context of my problem and what causes this error. It happens when trying to use any bash script containing a 'while read' loop in GNU Parallel.

I have a basic bash script like this:

# linkcheck.sh

while read domain
host "$domain"

Assume that I want to pipe in a large list (250mb say).

cat urllist | ./linkcheck.sh

Running host command on 250mb worth of URLs is rather slow. To speed things up I want to break up the input into chunks before piping it and then run multiple jobs in parallel. GNU Parallel is capable of doing this.

cat urllist | parallel --pipe -j0 parallel ./linkcheck.sh {}

{} is substituted by the contents of urllist line-by-line. Assume that my systems default setup is capable of running 500ish jobs per instance of parallel. To get round this limitation we can parallelize Parallel itself:

cat urllist | parallel -j10 --pipe parallel -j0 ./linkcheck.sh {}

This will run 5000'ish jobs. It will also, sadly, cause the error "broken pipe" (bash FAQ). Yet the script starts to work if I remove the while read loop and take input directly from whatever is fed into {} e.g.,

# linkchecker.sh

host "$1"

Why will it not work with a while read loop? Is it safe to just turn off the SIGPIPE signal to stop the "broken pipe" message, or will that have side effects such as data corruption?

Thanks for reading.


So, did

cat urllist | parallel --pipe -j0 parallel ./linkcheck.sh {}

work correctly?  I believe part of your problem may be that you left out the second --pipe, as in

cat urllist | parallel -j10 --pipe parallel -j0 --pipe ./linkcheck.sh {}


BTW, you never need to say

cat one_file | some_command

You can always change this to

some_command < one_file

resulting in one fewer process (and one fewer pipe).  (It may be appropriate/necessary to use cat when you have multiple input files.)

  • Thanks. Missing out the second --pipe was my problem. I could've sworn I'd already tried that, but obviously not. – Joe White Sep 28 '12 at 0:31
  • 1
    @Joe: Also, if you’re using --pipe, I suspect that {} is unnecessary/ignored. – Scott Sep 28 '12 at 14:48

It appears to me that error may be arising because of a bad race condition because of the window between forking a child to run another copy of linkcheck.sh while the pipe is still open and when the child actually tries to read. In that window, another copy has read EOF and the pipe has closed.

  • Yes I think you are correct. Thanks for your input. – Joe White Sep 28 '12 at 0:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.