10

To save some disk space I figured it might be a good idea to compress the VMware vSphere Client I have installed because I almost never use it.

I was surprised to find that it had the exact opposite effect on free disk space. I tracked it down further and found that it is related to compressing the Help folder. The loss of disk space is not reflected in the folder size.

I repeated the compress/decompress cylce three times to make sure another program did not coincidentally take up the disk space. It might be notable that the folder contains a large number of small files (≈ 30k).

Why is that so and can I somehow find other folders that I should decompress to save disk space?


Without compression:

folder size without compression free space without compression

With compression:

folder size with compression free space with compression

5
  • 3
    So you're compressing a ~300MB folder and you lose 5GB of disk space? Jun 8, 2013 at 22:29
  • 1
    Yes, that's exactly what happens - repeatably.
    – AndreKR
    Jun 8, 2013 at 23:12
  • Further investigation showed that this also happens with an artificial test set of 30k files with each 10kB size. Furthermore, it does only happen on my C and D drive, not on a freshly formatted NTFS Truecrypt volume.
    – AndreKR
    Jun 9, 2013 at 0:36
  • @AndreKR, any idea why this is not happening on the Truecrypt volume? Also, have you found any way to release the disk space (without rebooting)? Jan 2, 2015 at 20:58
  • I left a more detailed comment under Vesper's answer below, but running chkdsk seems to release the disk space.
    – s.co.tt
    Feb 2, 2021 at 17:18

4 Answers 4

10

A bit of background knowledge regarding the folder size screenshots you provided:

Uncompressed

As expected with many small files there is a lot of overhead. Your hard disk is partitioned with a certain block size - 4KB by default for NTFS.

Each file has to allocate a multiple of 4KB, meaning no matter if you have a 1KB or a 3.5KB file, both will take 4KB of space. If you have a 13KB file it will use 16KB on your drive. The difference between "Size" and "Size on disk" is the overhead introduced by unused space in blocks, the so called cluster tips.

Compressed

After compression "Size" is still the same as the amount of net data has not changed at all. However compression was able to reduce the total size about 130MB. In fact even more because the overhead here also applies. So compression did actually save some space in that folder and that is also displayed in the folder size.

Now regarding the behavior you see with the reduced free disk space on drive C: This can have multiple reasons. One thing to understand is that free disk space will always be less than

<Disk size> - <total size of all files>

This is because there is a lot of metadata which also eats up space (VSS Snapshots, System Restore points, MFT and so on).

During compression of single files NTFS will temporarily keep the original file until compression has completed. This is to ensure that you still would be left with a valid version of the file in case your computer crashed. This however should only be temporary. Nevertheless everything points to NTFS metadata to cause this.

To more accurately verify results you can do the following:

  • Start with an uncompressed folder
  • Disable System Protection for each volume (Computer properties/System Protection)
  • Delete the restore points for each volume in the same dialog
  • Use "Disk Cleanup" in the properties of your volume C: to remove temporary files
  • Note the free disk space
  • Compress the folder
  • Restart your computer
  • Use disk cleanup again
  • Check the free space on your disk

In theory you should be able to see an increase in free space

3

Having recently researched a similar problem, I can also say that a compressed file takes at least 4 kilobytes of space per file, and a temporary space of 64 kilobytes, which is the size of one "Compression Unit" for NTFS with 4kb cluster size. The article on blogs.msdn.com also mentions that when the file is compressed, the disk space is allocated to hold one full CU, and is released at undetermined time. This should be the reason of why you are experiencing the loss of 5GB, albeit temporary (a reboot should definitely fix that loss, some other means should do that too, but not defrag - have tried and failed). Apparently what's allocated seems to be a lot bigger (64kb*(31048+582) = 2072903680 or 1.93 GB), but this is explainable as NTFS has transactions which take time and processor units to be committed into raw data, and when that process will be finished, you will get all 5GB plus 150 MB space released due to compression back.

To summarize, you only lose space temporarily if you compress a whole lot of files. But, if those files are frequently modified, your disk space gets allocated to hold uncompressed data for those files in case their altered content won't be able to be compressed to fit in whatever space that cluster had occupied prior to write action.

1
  • Regarding the temporary space used during compression: I've just been dealing with this issue and it seems that running chkdsk [x:] frees up that temporary space during step 1 of 3. You can monitor free space, and break out of chkdsk as soon as you see the space released. From my non-scientific observations, it seems that once it scans the portion of the disk with the temporary storage, it releases it. (Note that you do not need to run chkdsk /f or use any arguments other than the drive letter, and the drive can therefore be online the entire time.)
    – s.co.tt
    Feb 2, 2021 at 17:09
2

I had the same phenomena:

Server migration, I copied the data folders of a drive of the old Windows Server 2012R2 (with 2 compressed folders) on a newer Windows Server 2016 Datacenter drive of the same size where I created the folder structure and set the compressed flags on these two folders previously to the copy process. During copying I run out of diskspace, and anywhere I looked only 3GB of 20GB are used, but the drive itself tells me 19.x GB are used. A colleague told me to remove the compression flag, and miraculously the lost 17GB reappear.

Then I read your article and decided to reapply the flag and try a reboot, but funnily the used disk space didn't increase this time.

I guess there could be an issue in Windows Server 2016 (maybe since ever) that internally generated temporary files are not cleaned up properly when files are copied to a compressed folder (as opposed to when the compression flag is applied to already existing files).

0

Adjunct: In addition to the explanations above, an application folder can contain files that simply do not compress well - already strongly data-reduced media formats (jpeg images, video files, mp3 audio...), already compressed files (a lot of data formats these days are built as ZIP archives!), files with actually random data or something resembling random data in them, encrypted files - this might result in there being more data being used for bookkeeping of what's been compressed than was saved by compression....

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .