3

I often have a long-running job where an application reads through a large file (>2GB) and performs processing on each record.

Is there any UNIX command I can use to see how deep into the file the process might be? The application will have only one file handle to the input file.

3 Answers 3

3

Try pv, Pipe Viewer

2

I've always wondered if lsof's "offset" tells you the current file-handle read position in the file but the documentation does not really spell this out clearly and I've never bothered to test or verify my assumptions. http://www.netadmintools.com/html/lsof.man.html

When writing long-running processes I tend to have them either write info to a log file I can periodically peek at (e.g. tail -f) or have them periodically report progress (e.g. a -v option that makes it report record count once a second).

0

Does this program accept pipes/STDIN/STDOUT? If so, you're in luck. I will demonstrate with md5sum'ing a large file:

$ pv 2GB.zero | md5sum -
1.56GB 0:00:31 [60.2MB/s] [===============================>         ] 80% ETA 0:00:07
f3444be3893c1b9b6fa9dd1acde671a9  -

pv "{filename}" will read the file, pipe it to anything that supports a pipe. You could also do cat "{filename}"|pv| md5sum - but then pv would not know how many bytes the file is, thus it would only know how fast it's going, now how much is left.

The possibilities are endless. Whenever you're using a pipe or redirecting to a file, you just add pv in the pipeline to tell you how fast data is going through the pipe! :)

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .