Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have an issue with tar. I'm tarring about 120GB of small files, and I feel i'm hitting a bottleneck.

My system has 3 HDs, one for the system (sda), one with the files (sdb) and one target HD for the tar files (sdc). The system is dedicated to backups, so it doesn't do anything else.

When the tar is running, I can see that the source HD is writing very often. And my question is why? (see sdb)

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
sda               0.00         0.00         0.00          0          0
sdb             461.00     11304.00      3776.00      11304       3776
sdc               0.00         0.00         0.00          0          0

First I thought it was the atime option, the system accessing the files would require a inode update. So I disabled that in the fstab.

mount: 
/dev/sdb1 on /disk1 type ext4 (rw,noatime)
/dev/sdc1 on /disk2 type ext4 (rw)

But the problem did not go away. The tar still chokes according to dstat: (see the csw)

----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw 
  4   1  85  11   0   0|6301k 4434k|   0     0 |   0     0 | 387  1317 
  0   0  68  31   0   0|  28k   44M|  66B  834B|   0     0 | 223   233 
  0   0  75  25   0   0|   0   408k|  66B  370B|   0     0 | 102    53 
  0   0  75  25   0   0|   0   516k| 126B  396B|   0     0 | 122    66 
  0   0  70  30   0   0|   0  1496k|  66B  354B|   0     0 | 154   109 
  0   0  77  22   0   0|4096B 1484k|  66B  354B|   0     0 | 155    79 
  0   0  77  23   0   0|  40k  552k| 336B  370B|   0     0 |  78   133 
  0   1  72  27   0   0|  40k  640k|  66B  370B|   0     0 | 164   126 
  0   0  75  25   0   0|  16k  580k|  66B  370B|   0     0 | 131    65 
  0   0  75  25   0   0|   0   316k|  66B  370B|   0     0 |  81    53 
  1   0  73  26   0   0| 132k  676k| 366B  396B|   0     0 | 162   143 
  0   0  76  23   0   0| 108k  608k|  66B  370B|   0     0 | 197   217 
  0   0  71  28   0   0| 112k  592k|  66B  370B|   0     0 | 178   155 
  0   0  67  33   0   0|   0   328k|  66B  370B|   0     0 |  81    73 
  8   1  63  28   0   0|5308k 4096B|  66B  354B|   0     0 | 705  2600 
 15   2  62  21   0   1|9452k    0 |  66B  370B|   0     0 |1158  4343 
 17   2  61  20   0   0|  11M    0 |  66B  354B|   0     0 |1256  5027 
 10   1  66  24   0   0|5272k 1308k|  66B  354B|   0     0 | 786  2804 
 14   2  63  20   0   1|9980k    0 |  66B  370B|   0     0 |1257  4766 
 15   3  61  21   0   1|  10M   12k|  66B  354B|   0     0 |1179  468

The exact command i'm using is:

tar -c /disk1/files 2>/dev/null | pigz -p 8 > /disk2/file.tar.gz

The HDs can read/write 100Mb/s easy, still it's slow as hell.

Top shows: 

Tasks: 147 total,   1 running, 146 sleeping,   0 stopped,   0 zombie
Cpu(s):  3.6%us,  0.8%sy,  0.0%ni, 84.9%id, 10.6%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:   7985320k total,  7784168k used,   201152k free,  3311500k buffers
Swap: 23393276k total,        0k used, 23393276k free,  3753780k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                  
19179 johan     20   0  150m 6136  536 S   42  0.1  19:57.48 pigz                                                                                                                     
19178 johan     20   0 19772 1312 1024 D    4  0.0   2:25.52 tar                                                                                                                      
   47 root      20   0     0    0    0 S    2  0.0   2:07.03 kswapd0                                                                                                                  
19317 johan     20   0 93336 1668  732 S    2  0.0   0:00.30 sshd                                                                                                                     
    1 root      20   0 23764 1780 1104 S    0  0.0   0:01.55 init                                                                                                                     
    2 root      20   0     0    0    0 S    0  0.0   0:00.00 kthreadd

What can I do to speed this up?

share|improve this question
    
Do you have a GNU tar ? tar --version ? –  khmarbaise Feb 15 '11 at 14:05
    
have you compared tar with cpio? –  Paulo Scardine Feb 15 '11 at 14:16
    
have you tried to strace the tar execution? –  GrzegorzOledzki Feb 15 '11 at 14:54
1  
what happens if you remove the pipe to pigz? –  fpmurphy1 Feb 15 '11 at 15:52
add comment

migrated from stackoverflow.com Feb 15 '11 at 16:08

This question came from our site for professional and enthusiast programmers.

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.