I have a large number of large files (hundreds of files, hundreds of MB each) which I need to pipe through a number of programs to filter and convert them. I'm taking advantage of multiple CPU cores so I'm running several instances of the same pipe on each file (could be up to a hundred cores, and could use ssh as part of the pipe, in case that makes any difference for the answer). I want to monitor each pipe, and I'm using pv for that. Here's a minimal example of what I have:

$ pv file-001.gz | gunzip | xz > file-001.xz
1.58GB 0:00:02 [ 713MB/s] [=================================>] 100%

In reality, I do several other things in the pipe as well, including passing data to other machines over ssh and piping it through filters on those machines, but the pipe would always end with a redirection into a new file on the main host. Also, no stage in the pipe requires the entire data set; they can operate on a line-by-line or chunk-by-chunk basis.

Currently, I need a separate terminal window for each instance of the pipe. What I would like to do is to start n parallel instances of the pipe in a single terminal/shell, and get the output from each pv instance on a line of its own. Something like this:

1.48GB 0:00:54 [ 704MB/s] [===============================>  ]  97% ETA 00:00:06
1.58GB 0:01:00 [ 713MB/s] [=================================>] 100%
0.75GB 0:00:31 [ 709MB/s] [================>                 ]  50% ETA 00:00:29

The value for n would be the number of lines I can fit in a terminal window, say 3-50 or so. The exact format of the progress report is not important, as long as it includes the speed, percentage done, elapsed time, and estimated remaining time. It's also not important that I use pv, it's ok to use some other program as long as I can easily install it or just plain shell (bash, preferably). What is important, though, is that the method can handle the occasional broken pipe in case a part of the pipe crashes for some reason. I would also like to start new jobs each time a job finishes (successfully or not) and there are still unprocessed files left.

Any ideas on how to do this?

Note that I've already tried GNU Parallel, but its ssh features appear to assume that each input file is first transferred to the remote host, then processed, then the result transferred back, which I want to avoid because of the amount of data involved and the limited amount of space on each processing node.


Did you look at --pipe for GNU Parallel?

cat bigfiles* | pv | parallel --pipe -S server1,server2 'cat | process_pipe'

(cat included for emphasis)

It defaults to 1 MB blocksize, which can be adjusted with --block.

-- edit for 1-1 correspondence --

Based on the above you can get 1-1 correspondence like this:

parallel --eta "cat {} | parallel --pipe -S server1,server2 'cat | process_pipe' > {}.out" ::: bigfiles*

(cat included for emphasis)

It is not quite optimal, as the inner parallel will not know about its siblings and thus may spawn more on server1 than server2. One way to avoid it is -j1 on the outer parallel, but that will not be optimal if the inner only has enough blocks for the first server. In other words: To perfectly balance your workload you may have to fiddle a little with this - maybe even use --load 100% or similar.

--- edit: Deal with crashes ---

If the process_pipe returns with an error, then this should retry the command 2 more times:

parallel --retries 3 --eta "cat {} | parallel --pipe -S server1,server2 'cat | process_pipe' > {}.out" ::: bigfiles*
  • I really like GNU Parallel, but for this particular task, I haven't been able to find how to use it. What I meant in the question was that each pipe can stream-process its input and doesn't need to do read-all-process-all-write-all. However, I can't mix chunks from different files. I need to keep a one-to-one correspondence between an input file, a processing pipe, and the final result file (and run several such instances at once). But +1 for this in any case, because if I didn't have this requirement, this GNU Parallel example would work nicely. – Fabian Fagerholm Sep 20 '12 at 5:28

Any ideas on how to do this?


pv has -c and -N options that should enable you to do what you want

$ pv -cN source access.log | gzip | pv -cN gzip > access.log.gz
source:  760MB 0:00:15 [37.4MB/s] [=>     ] 19% ETA 0:01:02
  gzip: 34.5MB 0:00:15 [1.74MB/s] [  <=>  ]

but I can't see how to apply that feature to multiple pipelines

However, if you look at the man page for pv you'll see this

          (tar cf - . \
           | pv -n -s $(du -sb . | awk '{print $1}') \
           | gzip -9 > out.tgz) 2>&1 \
          | dialog --gauge 'Progress' 7 70

So you could extend this to run a number of tasks in parallel so long as it is acceptable to view progress in a cluster of small windows. I would try Xdialog.

Currently, I need a separate terminal window for each instance of the pipe

My main point is that it isn't necessary for you to interactively open lots of terminal windows, you could have one script open lots of dialog boxes itself.

  • +1 for the suggestion to automate opening dialog boxes. – Fabian Fagerholm Sep 20 '12 at 5:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.