22

I've got a 100GB drive that has a 95GB file. I need to free up some space on the drive (and right now transferring the file off the drive is not an option). The file would compress well with gzip or bz2 or whatever, but all these programs write the compressed file to a separate file. I don't have enough free space for this.

Is there a way using standard compression tools or other Unix utilities to compress the file without using any additional disk space (or at least a minimal amount of additional disk space)? I'm picturing something that compresses part of the file at a time and writes the results directly over the file. I realize this would be risky, as the file would be corrupted if the compression was interrupted, but I don't think I have a choice.

  • 1
    One last option we used to use at my old place was to have a dir somewhere which contained a whole bunch of 1G files filled with garbage. Then, if you got into a pinch, you could remove some of them to give you a bit of emergency space. – user449989 May 20 '15 at 11:55
14

This is a proof of concept bash one-liner, but it should get you started. Use at your own risk.

truncate -s `gzip -c file | dd of=file conv=notrunc 2>&1 | sed -n '$ s/ .*$// p'` file
mv file file.gz

This works by piping gz data to a dd process that writes it back to the same file. Upon completion, the file is truncated to the size of the gz output.

This assumes that the last line of dd's output matches:

4307 bytes (4.3 kB) copied, 2.5855e-05 s, 167 MB/s

Where the first field is an integer of bytes written. This is the size the file will need to be truncated to. I'm not 100% sure that the output format is always the same.

| improve this answer | |
  • Nifty trick. Could you explain why conv=notrunc is necessary? – sleske Jan 17 '12 at 15:35
  • Maybe it's not. gzip -c file | dd of=file appears to work just as well. – user710307 Jan 17 '12 at 20:56
  • 1
    People at the linked question tried it (and I tried it, too); it does not work in general. Seems it only works for very small files - maybe because gzip will read a small file into RAM before compressing it. For large files (a few MB), it does not work, even if they are compressible. – sleske Jan 18 '12 at 15:17
  • 3
    Yep. So conv=notrunc is necessary. – user710307 Jan 18 '12 at 15:51
  • 1
    Isn't it possible that at any time the compression program (e.g. gzip) writes more header and data bytes than the original data bytes, thus overwriting some parts of the file? I guess this depends on the chosen compression program. Has anyone an idea how to prevent this from happening or how (im)probable it is? – Daniel Böhmer Jan 18 '16 at 15:01
8

It's not so much that gzip and bzip2 overwrite the original. Rather, they write the compressed data to disk as a new file, and if that operation succeeds, they unlink the original uncompressed file.

If you have sufficient RAM, you could write a script to temporarily compress the files in atmpfs filesystem, then remove the original on disk and replace it with the compressed version. Maybe something like this:

# some distributions mount /dev/shm as tmpfs; replace with bzip2 if you prefer
if gzip -q9c /full/disk/somefile > /dev/shm/somefile.gz
then
    rm -f /full/disk/somefile && mv -i /dev/shm/somefile.gz /full/disk
fi

Just be mindful of your memory usage, since tmpfs is essentially a RAM disk. A large output file could easily starve the system and cause other problems for you.

| improve this answer | |
3

There is no tool that works this way, for precisely the reason you give. Few people are willing to write a tool that deliberately implements risky behavior.

| improve this answer | |
  • I was hoping that it would be an unsafe, non-default option to a utility. Could you think of an alternative? Is there a way to truncate a file in place to, e.g. remove the first 2 GB? That would let me use my limited free space to compress one chunk at a time, shrinking the source file as i went. – Lee Jan 14 '12 at 3:03
  • 1
    There's really no sane way to remove data from the beginning of a file on any filesystem, with any tool. – Ignacio Vazquez-Abrams Jan 14 '12 at 3:12
  • 2
    But you can remove data from the end of the file. It can be done in principle. You slice data off the end of the file to put in separate files, truncating the original files as you go. Then you compress the files in forward order, deleting them as you go. It would be a pain to implement and if anything went wrong you'd be screwed. But it's possible. – David Schwartz Jan 14 '12 at 6:07
1

The split and csplit commands could be used to split the large file up into smaller parts, and then compress them individualy. Reassembling would be rather time consuming though.

| improve this answer | |
  • Another good option. One could probably write some script to do this. However, this yields many separately compressed files, which will need to be re-concatenated after uncompressing, which is not so nice. – sleske Jan 17 '12 at 15:38
  • Splitting the file with split or csplit has the same problem as the original question, namely that you'd have to have enough free disk space to hold the original file, plus all the split files. – Brian Minton Feb 10 at 23:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.