Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I've got a 100GB drive that has a 95GB file. I need to free up some space on the drive (and right now transferring the file off the drive is not an option). The file would compress well with gzip or bz2 or whatever, but all these programs write the compressed file to a separate file. I don't have enough free space for this.

Is there a way using standard compression tools or other Unix utilities to compress the file without using any additional disk space (or at least a minimal amount of additional disk space)? I'm picturing something that compresses part of the file at a time and writes the results directly over the file. I realize this would be risky, as the file would be corrupted if the compression was interrupted, but I don't think I have a choice.

share|improve this question
add comment

4 Answers

This is a proof of concept bash one-liner, but it should get you started. Use at your own risk.

truncate -s `gzip -c file | dd of=file conv=notrunc 2>&1 | sed -n '$ s/ .*$// p'` file
mv file file.gz

This works by piping gz data to a dd process that writes it back to the same file. Upon completion, the file is truncated to the size of the gz output.

This assumes that the last line of dd's output matches:

4307 bytes (4.3 kB) copied, 2.5855e-05 s, 167 MB/s

Where the first field is an integer of bytes written. This is the size the file will need to be truncated to. I'm not 100% sure that the output format is always the same.

share|improve this answer
    
Nifty trick. Could you explain why conv=notrunc is necessary? –  sleske Jan 17 '12 at 15:35
    
Maybe it's not. gzip -c file | dd of=file appears to work just as well. –  user710307 Jan 17 '12 at 20:56
    
user710307: Good point. It does seem to work (though I'm not quite sure why). Care to edit your answer? –  sleske Jan 18 '12 at 8:17
    
BTW, to educate myself, I asked a separate question about how this works behind the scenes: superuser.com/questions/379718/… –  sleske Jan 18 '12 at 8:23
2  
Yep. So conv=notrunc is necessary. –  user710307 Jan 18 '12 at 15:51
show 2 more comments

There is no tool that works this way, for precisely the reason you give. Few people are willing to write a tool that deliberately implements risky behavior.

share|improve this answer
    
I was hoping that it would be an unsafe, non-default option to a utility. Could you think of an alternative? Is there a way to truncate a file in place to, e.g. remove the first 2 GB? That would let me use my limited free space to compress one chunk at a time, shrinking the source file as i went. –  Lee Jan 14 '12 at 3:03
    
There's really no sane way to remove data from the beginning of a file on any filesystem, with any tool. –  Ignacio Vazquez-Abrams Jan 14 '12 at 3:12
    
But you can remove data from the end of the file. It can be done in principle. You slice data off the end of the file to put in separate files, truncating the original files as you go. Then you compress the files in forward order, deleting them as you go. It would be a pain to implement and if anything went wrong you'd be screwed. But it's possible. –  David Schwartz Jan 14 '12 at 6:07
add comment

It's not so much that gzip and bzip2 overwrite the original. Rather, they write the compressed data to disk as a new file, and if that operation succeeds, they unlink the original uncompressed file.

If you have sufficient RAM, you could write a script to temporarily compress the files in atmpfs filesystem, then remove the original on disk and replace it with the compressed version. Maybe something like this:

# some distributions mount /dev/shm as tmpfs; replace with bzip2 if you prefer
if gzip -q9c /full/disk/somefile > /dev/shm/somefile.gz
then
    rm -f /full/disk/somefile && mv -i /dev/shm/somefile.gz /full/disk
fi

Just be mindful of your memory usage, since tmpfs is essentially a RAM disk. A large output file could easily starve the system and cause other problems for you.

share|improve this answer
1  
That's just crazy enough to work –  Amazed Jan 14 '12 at 5:38
    
I like to push the envelope. –  James Sneeringer Jan 15 '12 at 5:21
add comment

The split and csplit commands could be used to split the large file up into smaller parts, and then compress them individualy. Reassembling would be rather time consuming though.

share|improve this answer
    
Another good option. One could probably write some script to do this. However, this yields many separately compressed files, which will need to be re-concatenated after uncompressing, which is not so nice. –  sleske Jan 17 '12 at 15:38
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.