6

The dilemma: I have a 10GB archive and a 20GB harddrive. When I go to unzip / unarchive the file the harddrive fills up and the process fails.

Question: On ubuntu server 12.04 / linux is there a way that I can unarchive a file while reducing the size of the archive?

14
  • You cannot modify the size of an existing file in this way. Changing the size of the archive would require removing the contents of the archive which already requires more then 10GB of data. You need to increase the overal storage on your server, 10GB is the size of the compressed size, its likely 30-40% larger then that.
    – Ramhound
    Oct 16, 2013 at 17:42
  • Are you positive that deleting files from an archive requires them to be unarchived first? Oct 16, 2013 at 17:44
  • 2
    You can extract each file, one by one, then delete the file within the archive. So I gather increasing the size of the storage is out of the question?
    – Ramhound
    Oct 16, 2013 at 17:47
  • It is out of the question. And what you just said is exactly the kind of solution I'm looking for. The only question is, how? (without doing it manually) Oct 16, 2013 at 17:49
  • The only way I know how to do what you want would be by hand. Are you sure the uncompressed data can even fit on the system. You are looking at 14-15Gb uncompresses of data more then likely plus the compressed data of the largest file. The math just doesn't make sense.
    – Ramhound
    Oct 16, 2013 at 18:05

3 Answers 3

6

Here is another solution. It won't let you extract individual files from an archive and reduce its size, but it does let you extract all files reducing the size of the archive as you go:

#!/bin/sh

# $1, the first paramter, is the .tar.gz file to unarchive

(
    size=$(wc -c $1)
    offset=0
    bs=4096
    while [[ $size > $offset ]]; do
        dd if=$1 bs=$bs count=1 skip=$offset status=none
        fallocate -p -o $offset -l $bs $1
        offset=$(( $offset + $bs ))
    done
) | tar xz

Save this into a file like e.g. untar_and_destroy.sh and execute as:

untar_and_destroy.sh whatever.tar.gz

What this does is give part of the .tar.gz file to tar, asks Linux to deallocate that part of the file, and then repeats for the next part. When you are done, ls -l will say the .tar.gz files has the same size as before, but du will report its size as 0. This is because the .tar.gz has been made into a sparse file, with the same length as before but as all 0s that don't need to be stored onto disk.

Don't use this in production, or anywhere where having that archive deleted would be bad. This makes the archive unreadable as soon as it starts, so if anything goes wrong, e.g. you run out of hard disk space while extracting, you won't get a second chance to run this.

3
  • This is the kind of answer I was looking for. If it were possible to not corrupt the archive that would be even better, but I think that might just not be a possibility with my constraints. Oct 17, 2013 at 14:22
  • You need count=1 option in dd, otherwise it will output the whole file on first iteration of the loop, which doesn't free the disk space while running the command. Also, I would rephrase tar xfz - to tar xz.
    – Zouppen
    Aug 17, 2016 at 11:29
  • @Zouppen Good points, I've edited those in. Aug 18, 2016 at 3:10
2

I don't know of any tools that can do this, and I don't think any of the common archiving formats support this.

One possible solution to your problem would be to keep the archive on a different machine and pipe it over to the machine you want to decompress it onto. For example, you could run this command on the machine with the archive:

cat archive.tar.gz | ssh YOUR_SERVER tar xfz -

The archive will be streamed to the tar process running on the server, which will decompress it without needing the archive to ever be present on the server.

3
  • This suggestion, and all the others like it, is very cool. However, this doesnt address the issue of needing to do it on the machine itself. The network speed of the machine makes it impractical to move such a large amount of data in a timely manner. Oct 16, 2013 at 20:31
  • so possibly an archival format that would allow for files to be removed from the archive without needing to make a second copy of the archive would be necessary? Oct 16, 2013 at 20:31
  • The only way you could make this work would be by extracting files from the end of the archive, and then shrinking the length of the archive. You can hack the tar format to let you do this, but not after it has been compressed. It really wouldn't make much sense to even use an archive in this situation, rather than just the files themselves. Oct 16, 2013 at 23:21
2

Although it may be impractical to expand your primary storage, perhaps you could extract the file contents to an external storage device.

Alternatively, generate a list of files in the archive, then write a script which extracts some of those. Move those files to the cloud, select another batch to extract, lather, rinse, repeat.

But, every archival app I know of has to have the original archive file intact while it creates a new archive file without what you don't want, so external storage is going to be very, very useful.

1
  • The second solution you offer is more of what I'm looking for. However, moving it to the cloud and back is impractical. This needs to be done in place. Oct 16, 2013 at 20:29

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .