Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a couple of big files that I would like to compress. I can do this with for example

tar cvfj big-files.tar.bz2 folder-with-big-files

The problem is that I can't see any progress, so I don't have a clue how long it will take or anything like that. Using v I can at least see when each file is completed, but when the files are few and large this isn't the most helpful.

Is there a way I can get tar to show more detailed progress? Like a percentage done or a progress bar or estimated time left or something. Either for each single file or all of them or both.

share|improve this question

You can use pv to achieve this. To report the progress correctly, pvneeds to know how much bytes you are throwing at it. So, the first step is to calculate the size (in kbyte). You can also completely drop the progress bar and just let pv tell you how much bytes it has seen; it would report a 'done that much and that fast'.

% SIZE=`du -sk folder-with-big-files | cut -f 1`

And then:

% tar cvf - folder-with-big-files | pv -p -s ${SIZE}k | \ 
     bzip2 -c > big-files.tar.bz2
share|improve this answer
Cool. pv doesn't seem to come with Mac OS X, but will try this out once I have a computer with MacPorts on it. Could you explain what you are doing there though? Not quite sure what the first line does exactly. – Svish Jul 28 '10 at 12:10
first line: fetch info about how many bytes will be handled. second line: use the size from the first line to allow pv to render 'progress'. since you are piping data, pv does not know how many more bytes will come. – akira Jul 22 '11 at 10:25
One addition: SIZE=$(($SIZE * 1000 / 1024)) - I don't know whether or not this is a quirk on my particular platform, so I'm not adding it to the answer: du returns size where 1 kb = 1024 bytes, while pv seems to be expecting 1 kb = 1000 bytes. (I'm on Ubuntu 10.04) – Izkata Dec 11 '11 at 2:27
@lzkata you could always ask du to use your preferred blocksize, e.g. du -s --block-size=1000, or just work with plain bytes, e.g. drop the k's from the du and pv calls. Nevertheless, I would expect both to use 1024 unless told otherwise, e.g. the --si switch on du, for example. – Legolas Feb 23 '12 at 11:05
or just drop the k-stuff and just use plain bytes (du -sb and pv -s without any modifier). that should end all the confusion. – akira Feb 23 '12 at 11:10

I prefer oneliners like this:

tar cf - /folder-with-big-files -P | pv -s $(du -sb /folder-with-big-files | awk '{print $1}') | gzip > big-files.tar.gz

It will have output like this:

4.69GB 0:04:50 [16.3MB/s] [==========================>        ] 78% ETA 0:01:21

For OSX (from Kenji's answer)

tar cf - /folder-with-big-files -P | pv -s $(($(du -sk /folder-with-big-files | awk '{print $1}') * 1024)) | gzip > big-files.tar.gz
share|improve this answer
On OSX, du does not take -b argument, needed to fallback to : $((du -sk /folder-with | awk '{print $1}') * 1024)) – ıɾuǝʞ Nov 29 '13 at 10:14
@kenji ok, added to the answer. – checksum Dec 20 '13 at 3:52
Nice, a one liner. Can you explain it? Or does it just magically work somehow? – Kissaki Oct 4 '14 at 18:02
Can you write command to extract tar file like above? – Krzysztof Szewczyk Nov 26 '14 at 13:47
Ok, I have it pv $FILE.tgz | tar xzf - -C $DEST_DIR – Krzysztof Szewczyk Nov 28 '14 at 7:57

better progress bar..

apt-get install pv dialog

(pv -n file.tgz | tar xzf - -C target_directory ) \
2>&1 | dialog --gauge "Extracting file..." 6 50

enter image description here

share|improve this answer
This is works for extraction, but you still need to do one of the more complicated commands for creation (which was the original question). It could still be combined with those; it's just more complicated. – Daniel H Aug 9 '14 at 5:05

Inspired by Noah Spurrier’s answer

function tar {
  local bf so
  so=${*: -1}
  case $(file "$so" | awk '{print$2}') in
  XZ) bf=$(xz -lv "$so" |
    perl -MPOSIX -ane '$.==11 && print ceil $F[5]/50688') ;;
  gzip) bf=$(gzip -l "$so" |
    perl -MPOSIX -ane '$.==2 && print ceil $F[1]/50688') ;;
  directory) bf=$(find "$so" -type f | xargs du -B512 --apparent-size |
    perl -MPOSIX -ane '$bk += $F[0]+1; END {print ceil $bk/100}') ;;
  command tar "$@" --blocking-factor=$bf \
    --checkpoint-action='ttyout=%u%\r' --checkpoint=1


share|improve this answer
A little context and explanation maybe? – Kissaki Oct 4 '14 at 18:03

Just noticed the comment about MacOS, and while I think the solution from @akira (and pv) is much neater I thought I'd chase a hunch and a quick playaround in my MacOS box with tar and sending it a SIGINFO signal. Funnily enough, it worked :) if you're on a BSD-like system, this should work, but on a Linux box, you might need to send a SIGUSR1, and/or tar might not work the same way.

The down side is that it will only provide you with an output (on stdout) showing you how far through the current file it is since I'm guessing it has no idea about how big the data stream it's getting is.

So yes, an alternative approach would be to fire up tar and periodically send it SIGINFOs anytime you want to know how far it's gotten. How to do this?

The ad-hoc, manual approach

If you want to be able to check status on an ad-hoc basis, you can hit control-T (as Brian Swift mentioned) in the relevant window which will send the SIGINFO signal across. One issue with that is it will send it to your entire chain I believe, so if you are doing:

% tar cvf - folder-with-big-files | bzip2 -c > big-files.tar.bz2

You will also see bzip2 report it's status along with tar:

a folder-with-big-files/big-file.imgload 0.79  cmd: bzip2 13325 running 
      14 0.27u 1.02s 

      adding folder-with-big-files/big-file.imgload (17760256 / 32311520)

This works nicely if you just want to check if that tar you're running is stuck, or just slow. You probably don't need to worry too much about formatting issues in this case, since it's only a quick check..

The sort of automated approach

If you know it's going to take a while, but want something like a progress indicator, an alternative would be to fire off your tar process and in another terminal work out it's PID and then throw it into a script that just repeatedly sends a signal over. For example, if you have the following scriptlet (and invoke it as say PID-to-signal interval-to-signal-at):


SIGNAL=29      # excuse the voodoo, bash gets the translation of SIGINFO, 
               # sh won't..

kill -0 $PID   # invoke a quick check to see if the PID is present AND that
               # you can access it..

echo "this process is $$, sending signal $SIGNAL to $PID every $INTERVAL s"
while [ $? -eq 0 ]; do
     sleep $INTERVAL;
     kill -$SIGNAL $PID;    # The kill signalling must be the last statement
                            # or else the $? conditional test won't work
echo "PID $PID no longer accessible, tar finished?"

If you invoke it this way, since you're targeting only tar you'll get an output more like this

a folder-with-big-files/tinyfile.1
a folder-with-big-files/tinyfile.2
a folder-with-big-files/tinyfile.3
a folder-with-big-files/bigfile.1
adding folder-with-big-files/bigfile.1 (124612 / 94377241)
adding folder-with-big-files/bigfile.1 (723612 / 94377241)

which I admit, is kinda pretty.

Last but not least - my scripting is kinda rusty, so if anyone wants to go in and clean up/fix/improve the code, go for your life :)

share|improve this answer
If running tar on the command line, typing control-T will send it a SIGINFO. If this was in a script it would be done with kill -INFO pid – Brian Swift Apr 23 '12 at 4:58
Completely forgot about control-T, I clearly have gotten used to spamming too many console windows for my own good.. – tanantish Apr 23 '12 at 20:21
why can't I see -SIGINFO when doing kill -l – Felipe Alvarez Jun 12 '13 at 2:32

Check out the --checkpoint and --checkpoint-action options in the tar info page (as for my distribution, the description for these options is not contained in the man page → RTFI).

With these (and maybe the functionality to write your own checkpoint command), you can calculate a percentage…

Good luck.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .