I wrote a horribly quick and dirty caching system for a video search website that gzips and caches bing search results inside a hidden "/cache/" directory.
Everything was going smoothly for the first few months, until I started noticing very popular search results not displaying any videos.
I looked into the cache folder, and sure enough, it was chalk-full of about 30,000 cached files, many of them now being created with no content inside of them... even for very popular search terms.
I deleted about 10,000 cache files (that were older than 1 month, or empty in terms of results) and now everything seems to be working smoothly again.
Obviously I will have to write a proper MySQL cacheing system in the near future, but could this large number of files inside one directory be causing CentOS to hiccup at all?
Maybe pulling a cached file and unzipping it is simply too much?
I have a mechanism in place that warns me whenever a download doesn't go smoothly. The bing server is not blocking me, I am indeed being served results, I simply fail to cache them (on occasion) when the number of files in the cache folder gets "too big".
All ideas/comments are welcome!