1

I've got a binary (lets call it displayFiles) which will get several tar files at once and print the tars to stdout. I then pipe the stdout to tar.

This works fine when the binary only downloads a single tar. If I get several tars at once and pipe them to tar is it likely to work? The displayFiles program will not pause between one file and the next.

E.g:

./displayFiles | { tar -xvf -; }
2

The end of a tar archive is marked by two consecutive zero-filled 512-byte records. When reading from stdin, tar should stop after reading the two zero-filled records, so data that follows (if any) can be read by the next tool.

If the data that follows is another tar archive and the next tool is another tar then the tool will work. To extract N concatenated archives you need to call tar N times.

If N is not known in advance, run tar in a loop until it fails:

./displayFiles | while tar -xvf -; do :; done

In the best case expect This does not look like a tar archive from tar that tries to read after displayFiles closes its stdout.

Note a tar archive with extra data (or garbage) after two consecutive zero-filled 512-byte records is still valid (i.e. tar will extract it just fine). If such archive gets to our loop then the next tar will read the extra data. Except one or few edge cases the extra data will make the tool fail and this will end the loop early. But even if we continued the loop, except one or few edge cases the extra data would "desynchronize" the stream and each following tar would start reading the stream not where some concatenated archive begins. Failing early is therefore not bad.

Hopefully tar archives coming from displayFiles don't contain extra data. If they do then there is no easy and reliable way to find individual archives in a stream of concatenated archives.

One way or another some tar in our loop will fail and in general you won't know if all the data was processed. For this reason consider adding some command that will tell you if the stream was depleted. Example:

./displayFiles | (while tar -xvf -; do :; done; exit "$(head -c 1 | wc -c)")

(head -c 1 is not portable. Portable replacement is dd bs=1 count=1 2>/dev/null.)

Exit status 0 means all the data was consumed by tar process(es). It means nothing more (in particular it doesn't mean there were no meaningful errors).

1
  • wow. I had the same problem a while back, and came to the same solution with tar in a while loop, but never figured out how to be sure everything has been extracted. the trick with checking for an empty stream is near genius. thank you very much. – Malik May 5 at 7:33
0

tar -f FILE/tar --file=FILE takes only a single file, so you either have to use a separate command to concatenate all the files together or run a tar command per file using a while read loop or xargs.

1
  • The fact displayFiles | tar … works for a single archive indicates that "gets several tar files at once and prints the tars to stdout" means "prints the contents", not "prints paths". – Kamil Maciorowski May 4 at 23:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.