19

Under Windows 7 (64 bit), I'm reading through 9000 moderately sized files. In total, there is more than 200 MB of data. Using Java (JDK 1.6.21) I'm iterating over the files. The first 1400 or so go at full speed but then speed drops off to 4ms per file. It turns out that the main cost is incurred simply by opening the files.

I'm opening the files using new FileInputStream (and of course closing them in time to avoid file leaks). After some investigating, I see that Windows' disk cache is using only 100 MB or so of RAM although I have 8 GiB available. I've tried increasing the cache size using the CacheSet tool but any values I provide are considered out of range.

I've also tried enabling the LargeSystemCache registry key but (after rebooting) the CacheSet tool still indicates I'm using 100 MB of cache (and doesn't increase during the test run). Does anybody have any suggestions to "encourage" Windows 7 to cache my 9000 files?

2
  • 3
    Where do you got the information from that only 100MB are used for disk cache? From what I know Windows uses by default all free memory for disk caching. A value of only 100MB seems to me like a minimum value.
    – Robert
    Jul 18, 2011 at 11:03
  • 1
    Part of the problem could be Java and it's JVM. There are tons of options to tweak the JVM - publib.boulder.ibm.com/infocenter/javasdk/v6r0/… . I think your best bet is to just write it in C++ or FreeBASIC (both which compile to native code and only limited by what the OS allows). Feb 1, 2013 at 20:05

3 Answers 3

5

I've tried increasing the cache size using the CacheSet tool but any values I provide are considered out of range.

  1. Make sure that you explicitly run CacheSet "As Administrator". CacheSet will say that the values are out of range when anything goes wrong.
  2. Make sure that the cache size is divisible by 4 KB

Also, I noticed that CacheSet has some trouble with large values, possibly because it is a 32-bit application from 2006. I went and wrote a 64-bit program around the same API, which allows setting cache sizes over 4GB: http://blog.thecybershadow.net/2012/12/14/64-bit-cacheset/

Note: The CacheSet article mentions CacheMan, however the latest version seems to use the same API as CacheSet - it's just packaged into an UI with a bunch of presets and other functionality.

4

Superfetch doesn't cache all of a large file, just the parts that are accessed. So accessing that file requires disk read still. A disk cache at 210mb like mine can cache everything from all those small files in Windows. And the performance is noticeable. I doubt though, that it would have impact on faster systems.

1
  • 2
    What is superfetch, and how does it relate to this question?
    – jpaugh
    Feb 7, 2017 at 19:31
1

I know this is old, but for others who run into this problem: You need to run CacheSet with administrative privileges manually. If you don't, you will get the OutOfRange error; it will NOT automatically ask for privileges.

You must log in to answer this question.