Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a Hex-core Xeon X5650 running Windows 7 x64.

I'm currently doing simultaneous compression of three NTFS hard drives, 2TB each, with the commands:

compact e: /i /c /s
compact f: /i /c /s
compact g: /i /c /s

Its getting reasonable compression: up to 40:1, and typically 2:1.

The resulting CPU load is only about 3%

Does anyone know why is the CPU load so low?

Does anyone know if the Win7 realtime compression/decompression kernel is using SSE3 or SSSE3 or SSE4.2 instructions behind the scenes?

share|improve this question
    
I should also add: VMware runs like a dog if the files its using for its virtual hard drive are NTFS compressed. It seems to be fine in most other places on the system. –  Contango Jan 31 '12 at 15:37

3 Answers 3

up vote 2 down vote accepted

I would expect that your cpu usage is so low because a number of factors but firstly keep in mind that the compression algorithm is designed for speed rather than high compression rates.

The fundamental problem here is whether you are compressing thousands upon thousands of small files or several GB of humongous files.

If you are compressing files under 2MB I would expect that most of the time taken to compress the files is take up in the operations needed to find a file in the filesystem, get the hard drive to give you the data, and then writing the compressed data back, then repeat with the next tiny file. The compression algorithm can probably handle around 40-50MB/s on a modern machine (I would expect but am not certain of this) and so the hard drives seek and data transfer times are truly the limiting factors for those files, compression times are likely to be nearly nil.

For larger files you should almost certainly be seeing a higher CPU usage. I certainly see at least 1 core pop up to about 50% CPU usage (roughly 8-10% total CPU on my 8-core processor) when I tell it to compress a large directory on my system...

I really think that hard drive seek times are your problem here, otherwise you need to specifically look at which cores are being worked while compressing, it may be that the compression routine is only able to be used by one thread at a time, though I would doubt this to be the case.

share|improve this answer
    
Thanks for the answer. I think that the bottleneck is almost entirely in the HD, which means that compression will probably speed things up (not slow things down). The key is having a CPU that supports something like SSE4.2 for hardware accelerated compression. –  Contango Oct 29 '11 at 16:27

Yes, the new SSE4.2 instruction set in the Intel i7 and Intel Xeon X5650 can be used to speed up compression. Given that my CPU usage is 3% while simultaneously compressing three 2TB hard drives, Win7 is almost certainly using SSE4.2 behind the scenes for NTFS sector level compression.

See Power Savings with Hardware based GZIP Compression, Tony Summers et. al., AHA Products Group, Comtech EF Data Corporation.

share|improve this answer

No no no, there is noSSE4 involved. None of SSE4 instructions can assist LZ(w/77) method used by NTFS. Your disk is too slow to saturate CPU with compression which in general is good - you can live with compressed disk files. Problems start when you do small changes in huge files - it will go and re-read and re-compress all file

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.