I have a large number of large files (hundreds of files, hundreds of MB each) which I need to pipe through a number of programs to filter and convert them. I'm taking advantage of multiple CPU cores so I'm running several instances of the same pipe on each file (could be up to a hundred cores, and could use ssh as part of the pipe, in case that makes any difference for the answer). I want to monitor each pipe, and I'm using
pv for that. Here's a minimal example of what I have:
$ pv file-001.gz | gunzip | xz > file-001.xz 1.58GB 0:00:02 [ 713MB/s] [=================================>] 100%
In reality, I do several other things in the pipe as well, including passing data to other machines over ssh and piping it through filters on those machines, but the pipe would always end with a redirection into a new file on the main host. Also, no stage in the pipe requires the entire data set; they can operate on a line-by-line or chunk-by-chunk basis.
Currently, I need a separate terminal window for each instance of the pipe. What I would like to do is to start n parallel instances of the pipe in a single terminal/shell, and get the output from each pv instance on a line of its own. Something like this:
1.48GB 0:00:54 [ 704MB/s] [===============================> ] 97% ETA 00:00:06 1.58GB 0:01:00 [ 713MB/s] [=================================>] 100% 0.75GB 0:00:31 [ 709MB/s] [================> ] 50% ETA 00:00:29
The value for n would be the number of lines I can fit in a terminal window, say 3-50 or so. The exact format of the progress report is not important, as long as it includes the speed, percentage done, elapsed time, and estimated remaining time. It's also not important that I use
pv, it's ok to use some other program as long as I can easily install it or just plain shell (bash, preferably). What is important, though, is that the method can handle the occasional broken pipe in case a part of the pipe crashes for some reason. I would also like to start new jobs each time a job finishes (successfully or not) and there are still unprocessed files left.
Any ideas on how to do this?
Note that I've already tried GNU Parallel, but its ssh features appear to assume that each input file is first transferred to the remote host, then processed, then the result transferred back, which I want to avoid because of the amount of data involved and the limited amount of space on each processing node.