3

I found some problem with linux gnu tar. wheh I use option

-T -  (for file list from stdin) or
-T named_pipe_file    ,

this desn't work on the fly. for example, simple interactive script:

while read x; do echo $x; done|\
tar cvf tar.tar -T -

tar starts archiving only when I press ^D for marking input EOF the same situation is when I use named pipe:

mkfifo named_pipe
tar cvf tar.tar -T named_pipe
while read x; do echo $x; done >named_pipe

It seems tar makes some buffering. But how long is it? I must repack a lot of files to TAR but have little disk space. Then I must do this on the fly. I waht use tar option --remove-files for this. But without interactivity for -T option it's impossobie. In the plan, "while" part of code should unpack file to file sequently and waiting for TAR for removing, and next file. Thanks for ideas :)

my tar version: tar (GNU tar) 1.26 (C) 2011 FSF

1
  • Are you feeding tar file names or the actual data it should compress? The -T option expects a list of file names and works as advertized in a loop on my system.
    – terdon
    Sep 4, 2013 at 13:44

3 Answers 3

3

tar is able to append to already existing archives, so you could do:

touch tarfile.tar
command_that_produces_file_list | xargs tar rf tarfile.tar

Unfortunately, this doesn't work with on the fly compression. Luckily, the tar format is simple enough we can do some hacking:

command_that_produces_file_list | {
  xargs -i sh -c 'tar c {} | head -c $(( (`stat --printf="%s" {}` + 511) / 512 * 512 + 512))';
  dd if=/dev/zero bs=512 count=2 2>/dev/null;
} | compression_utility

tar output consists of, for each file, a 512-byte header followed by enough 512-byte blocks to hold the file data. It then appends at least 2 512-byte blocks of zeros. What this code does is capture the output of tar and remove the extra blocks of zeros, combine the output from the multiple invocations of tar together, and then sticks on the terminating blocks of zeros. The output is sent down the pipe to the compression utility, which runs concurrently with the tars.

5
  • I know about this. it can but only for uncompressed archive or for tapes with hardvare compression not exposed to software. that tape is shown to software as uncompressed. I made example as simple as it can be. In script I'm using xz compression with flag -9ev and --remove-files for tar.
    – Znik
    Sep 4, 2013 at 13:40
  • You could try compressing files in place and then storing them in a plain tar, otherwise I would find a solution that doesn't use tar. What are you needing to do this for? Sep 4, 2013 at 14:40
  • @user215501 It turns out there is a way to do this with tar, see my newly edited answer. Sep 4, 2013 at 18:15
  • @ingedsubmariner, and (at)user215501 very nice update with tar hack format :) I hope this bug in tar (in my opinion) will be removed soon. I'm not sure who of us do this tricky script
    – Znik
    Sep 5, 2013 at 7:00
  • @wingedsubmariner hat off to you, sir! :) Sep 7, 2013 at 0:35
2

Good news. I get answer for my bug report to [email protected] , cite:

From: Sergey Poznyakoff date:
Thu, 05 Sep 2013 08:40:40 +0300 subject: Re: [Bug-tar] gnu tar, option -T from stdin or named pipe is not interactive

Hi Grzegorz,

This has been fixed in the git HEAD (starting from commit 1fe0c83d).

Regards, Sergey

Then I'm waiting when this will be fixed in linux distros :)

0

Read this explanation (first answer): In what order do piped commands run?

What you see is tar blocking for the completion of the input list before it starts processing. Arguably, doing the processing in parallel with the input, one-by-one could be useful, but I don't think GNU Tar supports that.

I can only guess that waiting for the whole list is done to avoid complexity in the "internal procedures" of handling command line arguments - such as how to deal with "--append and --remove-files". I think most people would prefer to remove all files in bulk after the archive is done, and not on the fly as is desirable in this case.

The GNU people are usually very friendly, you could ask why this is not a feature, how you can do it with other tools and even request this to be a part of Tar in the future;

https://lists.gnu.org/mailman/listinfo/help-tar

3
  • Unfortunately it's not the answer. you can replace "tar" part in my examples by command |sed 's/aa/bb/' or for named pipe sed 's/aa/bb/' <named_pipe (for verivication) . this works perfectly. you put by keyboard strings and 'aa' phrase are replaced immediately if it's found, and print back to the screen. with pipes all is right. tar command is the main problem.
    – Znik
    Sep 5, 2013 at 7:07
  • That's what I said, and user215501 followed my suggestion (I think) with happy news as a result ;) Sep 7, 2013 at 0:37
  • GNU people are really friendly :) But about tar. This has ability for removing source files immediatly. This is done for saving temporary free space, descibed in man. In other side, files in tar archive has the same sequence like given by STDIN for -T - option. In reality tar do some remove buffering. This not delete current archived/done file, but previous one. This is prevention for data loss, not allways this work. When you use strong compression algorithm with high buffering and high dictionary, then is big chance it's no data saved in any file, but is still keeping in the memory.
    – Znik
    Jan 17, 2014 at 14:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .