1

I wrote a horribly quick and dirty caching system for a video search website that gzips and caches bing search results inside a hidden "/cache/" directory.

Everything was going smoothly for the first few months, until I started noticing very popular search results not displaying any videos.

I looked into the cache folder, and sure enough, it was chalk-full of about 30,000 cached files, many of them now being created with no content inside of them... even for very popular search terms.

I deleted about 10,000 cache files (that were older than 1 month, or empty in terms of results) and now everything seems to be working smoothly again.

Obviously I will have to write a proper MySQL cacheing system in the near future, but could this large number of files inside one directory be causing CentOS to hiccup at all?

Maybe pulling a cached file and unzipping it is simply too much?

I have a mechanism in place that warns me whenever a download doesn't go smoothly. The bing server is not blocking me, I am indeed being served results, I simply fail to cache them (on occasion) when the number of files in the cache folder gets "too big".

All ideas/comments are welcome!

3 Answers 3

2

It depends on what kind of file system you're using. I believe ext2 and ext3, for example, are limited to 32000 subfolders (you can have that many or more files, but performance will suffer...); ext4 twice that, and some others allow more or an unlimited number. See this question on Server Fault for a discussion and an answer covering various linux file system types.

2
  • Thank you frabjous, yes I believe there were around 32,000 files in that particular cache folder so that just might of been the culprit. I will vote you up once I have 15 reps! Sep 30, 2010 at 5:50
  • you can find the limit here
    – phuclv
    Aug 29, 2014 at 4:45
0

I have seen the opposite on FC7 and on Ubuntu, where directories handle over 100K files with no problem. Instead, there is a problem when the number of sub-directories is 32K or more - but not merely files.

Since you say this solution was "horribly quick and dirty", perhaps the problem is not in CentOS but in your code? Or even in the language you are using? Is it possible that your code is attemping to hold all these files open at once, so that it runs out of file handles or some such resource?

2
  • I guess it's more of a "keep it simple stupid" approach. The code is written in PHP. It checks for a filetime on a cache file, if none is found or file is too old, it retrieves new information from bing. I think that this 32,000 number seems about right. I cannot say how many files I deleted, probably around 12,000 - and there are 20,000 left inside the cache folder now and everything is working fine so.. my best guess so far is the magic number 32. Sep 30, 2010 at 5:30
  • @darkAsPitch: two things: [1] both frabjous and I said that the 32K limit is with subdirectories, not files. Are you using a subdir per cache entry? [2] I pointed this out twenty minutes before frabjous. I am curious as to why are you willing to vote up that answer instead of this one?
    – Beel
    Sep 30, 2010 at 6:05
0

The real answer has to do with Bing's shoddy API - see this thread on the official bing API 2.0 forum: http://www.bing.com/community/Developer/f/12254/t/662869.aspx

Basically they randomly hide results for random search queries - forcing you to hit their servers twice as hard as required to get the information from them. Since many times a "no results" response indeed has results, you have to check again.

Thanks everybody for your input!

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .