I have to index a rather large amount of data with Windows Search, so I wanted to have an estimation of index size. I've indexed a subset of data (about 60GB; 13e6 pages of text data each about 4-5KB).

The index has grown to about 78GB, so it now occupies more space than the data itself. These results contradict what I've read here:

The average size of an index is about 10% of the size of all the content that is being indexed.

Is this the expected size of the index? What can be done to reduce it?

migrated from stackoverflow.com Aug 16 '11 at 4:47

This question came from our site for professional and enthusiast programmers.

Average index size is not indicative of what yours would be. This is probably because in addition to all of your files getting indexed (and that would be large already as a size percentage as your files are all small) the content of text files also gets indexed, further increasing the size of the index.

  • "The average size of an index is about 10% of the size of all the content that is being indexed." --> "the content of text files also gets indexed" --> ? – Mehrdad Aug 16 '11 at 6:31
  • Yes. For some file types. This allows you to search for text in the files. – soandos Aug 16 '11 at 6:32
  • Content is indexed. I think that estimate about 10% of original content assumes that content is indexed. Otherwise if index depended only on file properties it could not depend on content size. – Vladimir Nesterovsky Aug 16 '11 at 8:05

The 10% rule relies on a number of assumptions:

  1. The corpus contains files with prose from a human language
  2. The files on average are reasonably large compared to their metadata
  3. The corpus does not contain code files (.cpp, .cs, ect)

Number 3 is because the index is compressed on disk assuming that words are going to be repeated often. This is true for most text (check out how many times 'the' appears in your index), but for code there are so many unique 'words' from variable names that it breaks this assumption.

In your case though I would suspect the issue is number 2, a lot of number of small files. There is a fixed amount of metadata overhead that comes with each file that needs to be stored in the index. For example, the index has to store the full path, date modified, date created, ect for every file on top of the contents. It adds up to about 3K per file, and if the average file size is 4-5K that is most likely the issue.

If search is important and you can't combine the file I recommend turning off indexing the contents of those file types that you have a lot of. It should reduce the size somewhat while still having the metadata of the file searchable.

Indexing only the contents using Indexing Options control panel page

I disabled my search service for windows . I use alternative search papplications. "Everything" with only 900 KB size for extra fast file search and "FileSeek Pro" for search throgh content of files.

  • Are you saying that your search index was only 900KB? It's not super clear from the wording of the post. Also, what is your total data size, and how much is documents and profile things that are indexed? – music2myear Jun 16 '17 at 23:46
  • no no, I'm not speaking about windows search index because I disabled all services related to windows search, instead of it I use two alternative softwares which can search more and more faster than Windows Search. You can download it here https://www.voidtools.com/downloads/ and File seek pro for searching content of files. I think Windows search is slow and uses extra sources, by disabling it you can experience a faster computer – Nihan J Co. Jun 17 '17 at 7:33
  • Ok, well your answer is still unclear. There is no indication what the 900KB refers to, whether "everything" is an application or a setting. Please clean up and clarify your answer so that it will not require more questions to understand. – music2myear Jun 17 '17 at 15:30

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.