27

I've got a 100GB drive that has a 95GB file. I need to free up some space on the drive (and right now transferring the file off the drive is not an option). The file would compress well with gzip or bz2 or whatever, but all these programs write the compressed file to a separate file. I don't have enough free space for this.

Is there a way using standard compression tools or other Unix utilities to compress the file without using any additional disk space (or at least a minimal amount of additional disk space)? I'm picturing something that compresses part of the file at a time and writes the results directly over the file. I realize this would be risky, as the file would be corrupted if the compression was interrupted, but I don't think I have a choice.

1
  • 1
    One last option we used to use at my old place was to have a dir somewhere which contained a whole bunch of 1G files filled with garbage. Then, if you got into a pinch, you could remove some of them to give you a bit of emergency space.
    – user449989
    May 20, 2015 at 11:55

5 Answers 5

17

This is a proof of concept bash one-liner, but it should get you started. Use at your own risk.

truncate -s `gzip -c file | dd of=file conv=notrunc 2>&1 | sed -n '$ s/ .*$// p'` file
mv file file.gz

This works by piping gz data to a dd process that writes it back to the same file. Upon completion, the file is truncated to the size of the gz output.

This assumes that the last line of dd's output matches:

4307 bytes (4.3 kB) copied, 2.5855e-05 s, 167 MB/s

Where the first field is an integer of bytes written. This is the size the file will need to be truncated to. I'm not 100% sure that the output format is always the same.

10
  • Nifty trick. Could you explain why conv=notrunc is necessary?
    – sleske
    Jan 17, 2012 at 15:35
  • Maybe it's not. gzip -c file | dd of=file appears to work just as well.
    – user710307
    Jan 17, 2012 at 20:56
  • 1
    People at the linked question tried it (and I tried it, too); it does not work in general. Seems it only works for very small files - maybe because gzip will read a small file into RAM before compressing it. For large files (a few MB), it does not work, even if they are compressible.
    – sleske
    Jan 18, 2012 at 15:17
  • 3
    Yep. So conv=notrunc is necessary.
    – user710307
    Jan 18, 2012 at 15:51
  • 1
    Isn't it possible that at any time the compression program (e.g. gzip) writes more header and data bytes than the original data bytes, thus overwriting some parts of the file? I guess this depends on the chosen compression program. Has anyone an idea how to prevent this from happening or how (im)probable it is? Jan 18, 2016 at 15:01
11

It's not so much that gzip and bzip2 overwrite the original. Rather, they write the compressed data to disk as a new file, and if that operation succeeds, they unlink the original uncompressed file.

If you have sufficient RAM, you could write a script to temporarily compress the files in atmpfs filesystem, then remove the original on disk and replace it with the compressed version. Maybe something like this:

# some distributions mount /dev/shm as tmpfs; replace with bzip2 if you prefer
if gzip -q9c /full/disk/somefile > /dev/shm/somefile.gz
then
    rm -f /full/disk/somefile && mv -i /dev/shm/somefile.gz /full/disk
fi

Just be mindful of your memory usage, since tmpfs is essentially a RAM disk. A large output file could easily starve the system and cause other problems for you.

2
4

There is no tool that works this way, for precisely the reason you give. Few people are willing to write a tool that deliberately implements risky behavior.

3
  • I was hoping that it would be an unsafe, non-default option to a utility. Could you think of an alternative? Is there a way to truncate a file in place to, e.g. remove the first 2 GB? That would let me use my limited free space to compress one chunk at a time, shrinking the source file as i went.
    – Lee
    Jan 14, 2012 at 3:03
  • 1
    There's really no sane way to remove data from the beginning of a file on any filesystem, with any tool. Jan 14, 2012 at 3:12
  • 2
    But you can remove data from the end of the file. It can be done in principle. You slice data off the end of the file to put in separate files, truncating the original files as you go. Then you compress the files in forward order, deleting them as you go. It would be a pain to implement and if anything went wrong you'd be screwed. But it's possible. Jan 14, 2012 at 6:07
1

The split and csplit commands could be used to split the large file up into smaller parts, and then compress them individualy. Reassembling would be rather time consuming though.

3
  • Another good option. One could probably write some script to do this. However, this yields many separately compressed files, which will need to be re-concatenated after uncompressing, which is not so nice.
    – sleske
    Jan 17, 2012 at 15:38
  • Splitting the file with split or csplit has the same problem as the original question, namely that you'd have to have enough free disk space to hold the original file, plus all the split files. Feb 10, 2020 at 23:03
  • Doesn't that run into the same problem with disk space? You would have to generate parts starting at the end of the file and truncate it in every step. Mar 26, 2021 at 13:18
0

If each file fits in memory, you could use sponge

gzip -c somefile | sponge somefile
1
  • Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.
    – Community Bot
    Dec 6, 2023 at 17:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .