Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

The dilemma: I have a 10GB archive and a 20GB harddrive. When I go to unzip / unarchive the file the harddrive fills up and the process fails.

Question: On ubuntu server 12.04 / linux is there a way that I can unarchive a file while reducing the size of the archive?

share|improve this question
    
You cannot modify the size of an existing file in this way. Changing the size of the archive would require removing the contents of the archive which already requires more then 10GB of data. You need to increase the overal storage on your server, 10GB is the size of the compressed size, its likely 30-40% larger then that. –  Ramhound Oct 16 '13 at 17:42
    
Are you positive that deleting files from an archive requires them to be unarchived first? –  Ethan Willis Oct 16 '13 at 17:44
2  
You can extract each file, one by one, then delete the file within the archive. So I gather increasing the size of the storage is out of the question? –  Ramhound Oct 16 '13 at 17:47
    
It is out of the question. And what you just said is exactly the kind of solution I'm looking for. The only question is, how? (without doing it manually) –  Ethan Willis Oct 16 '13 at 17:49
    
The only way I know how to do what you want would be by hand. Are you sure the uncompressed data can even fit on the system. You are looking at 14-15Gb uncompresses of data more then likely plus the compressed data of the largest file. The math just doesn't make sense. –  Ramhound Oct 16 '13 at 18:05

3 Answers 3

up vote 4 down vote accepted

Here is another solution. It won't let you extract individual files from an archive and reduce its size, but it does let you extract all files reducing the size of the archive as you go:

#!/bin/sh

# $1, the first paramter, is the .tar.gz file to unarchive

(
    size=$(wc -c $1)
    offset=0
    bs=4096
    while [[ $size > $offset ]]; do
        dd if=$1 bs=$bs skip=$offset status=none
        fallocate -p -o $offset -l $bs $1
        offset=$(( $offset + $bs ))
    done
) | tar xfz -

Save this into a file like e.g. untar_and_destroy.sh and execute as:

untar_and_destroy.sh whatever.tar.gz

What this does is give part of the .tar.gz file to tar, asks Linux to deallocate that part of the file, and then repeats for the next part. When you are done, ls -l will say the .tar.gz files has the same size as before, but du will report its size as 0. This is because the .tar.gz has been made into a sparse file, with the same length as before but as all 0s that don't need to be stored onto disk.

Don't use this in production, or anywhere where having that archive deleted would be bad. This makes the archive unreadable as soon as it starts, so if anything goes wrong, e.g. you run out of hard disk space while extracting, you won't get a second chance to run this.

share|improve this answer
    
This is the kind of answer I was looking for. If it were possible to not corrupt the archive that would be even better, but I think that might just not be a possibility with my constraints. –  Ethan Willis Oct 17 '13 at 14:22

I don't know of any tools that can do this, and I don't think any of the common archiving formats support this.

One possible solution to your problem would be to keep the archive on a different machine and pipe it over to the machine you want to decompress it onto. For example, you could run this command on the machine with the archive:

cat archive.tar.gz | ssh YOUR_SERVER tar xfz -

The archive will be streamed to the tar process running on the server, which will decompress it without needing the archive to ever be present on the server.

share|improve this answer
    
This suggestion, and all the others like it, is very cool. However, this doesnt address the issue of needing to do it on the machine itself. The network speed of the machine makes it impractical to move such a large amount of data in a timely manner. –  Ethan Willis Oct 16 '13 at 20:31
    
so possibly an archival format that would allow for files to be removed from the archive without needing to make a second copy of the archive would be necessary? –  Ethan Willis Oct 16 '13 at 20:31
    
The only way you could make this work would be by extracting files from the end of the archive, and then shrinking the length of the archive. You can hack the tar format to let you do this, but not after it has been compressed. It really wouldn't make much sense to even use an archive in this situation, rather than just the files themselves. –  wingedsubmariner Oct 16 '13 at 23:21

Although it may be impractical to expand your primary storage, perhaps you could extract the file contents to an external storage device.

Alternatively, generate a list of files in the archive, then write a script which extracts some of those. Move those files to the cloud, select another batch to extract, lather, rinse, repeat.

But, every archival app I know of has to have the original archive file intact while it creates a new archive file without what you don't want, so external storage is going to be very, very useful.

share|improve this answer
    
The second solution you offer is more of what I'm looking for. However, moving it to the cloud and back is impractical. This needs to be done in place. –  Ethan Willis Oct 16 '13 at 20:29

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.