Weird GZip / Disk space issue

Question

I'm working on a server which is using PCI Compliant partitioning rules, which means that each of the following directories has its own partition: /, /var, /boot, /usr, /tmp and /home. The server is hosting a website at the moment with an Apache / MySQL backend.

I received a warning that the /var partition (which is sized at 2 GB) was running out of space. I checked, and the Apache access log that was located at /var/log/access.log was generating a ridiculous amount of text: roughly 1 GB in about 10 days with no log rotate setup.

I reconfigured the Apache setup to put the logs on another partition with more space, and I setup log rotate with gzip compression.

Here's the weird part. The old Apache access log was located at /var/log/access.log and was around 920 MB uncompressed. The /var partition was about 80% full, so it was using 1.6 GB of the 2.0 GB total.

I ran the following command to compress the log file:

cd /var/log && gzip -9 access.log > access-2011-12-19.log.gz

I realize now that I was a little rusty with my gzip usage because it created two files: access-2011-12-19.log.gz, which was empty, and access.log.gz, which was now 68 MB and contained the original 920 MB log file. I deleted the access-2011-12-19.log.gz file, and the old log file was now gone.

The problem: I then checked the space on the /var partition, and to my confusion, now read as bring 78% full, or around 1.4 GB.

It seems compressing a 920 MB log file to 68 MB should save a lot of disk space, but it seems it did not. What could be the problem? The server is running CentOS 6 if that helps any.

Answer 1

How long after the gzip finished did you check the disk usage? The last step of gzipping is deleting the original file. After you delete a huge file, it can take quite a while for the file-system to track down and release all the associated inodes. I've sometimes seen it take a few hours, though that was on a much bigger file than yours.

Edited to add: Another possibility is that some process was still holding access.log open; maybe even that Apache was still writing to it? Even after you unlink ("delete") a file from the filesystem, it won't truly be deleted as long as there are open file-descriptors pointing to it.

Also, I should mention — if you want gzip to write a compressed file (or decompressed file) to standard output, and not remove the uncompressed original (or compressed original), you can specify the -c option. Alternatively, you can send in the original file on standard input rather than passing it its name as an argument. So, either of these:

gzip -c -9 access.log > access-2011-12-19.log.gz
gzip -9 < access.log > access-2011-12-19.log.gz

(I tend to use the latter approach, personally.)