Running on a LAMP stack, CentOS 6, Apache 2.2.3, PHP 5.4.25, MySQL 5.5.31
Have a very complex web application. It's been running pretty smoothly for about 5 years, now. Of a sudden we've run into some connection lags. We're getting poor performance over streaming downloads and XHR requests and image retrievals are touch and go. Sometimes they work flawlessly, and swiftly, sometimes they just sit and spin until they time-out or eventually load.
The XHRs will sometimes time out, but the images will always eventually load. I am having a very hard time figuring out what is going on, behind the scenes. It's affecting two different computers with slightly different configurations running slightly different versions of the same software platform. The only thing they have in common is the database server which sits on a third machine in the same rack.
But a lot of times there is lag, for instance, on retrieving cached photos, for which there is no database interaction, at all.
We've contacted the data center and we're trying to determine whether the cause is software or hardware because our speeds are hypothetically 100Mbps, but continuous downloads are averaging 430KByte/s downloads.
I have checked all of my throttling policies, and we don't have any ... pfifo_fast is how it's configured, right now. Even if I'm the only user in the system it still hiccups like that, especially if I try to move rapidly between folders. The lazy-load on the folder tree sometimes seems to spin. Likewise, sometimes, the gallery will load instantly, and sometimes it will take several seconds before the first images begin to load.
Is it possible that all of these problems are caused by slow connection speeds to (and between) the servers?
We're still working with the data center, I've been pouring over my own code, and I don't want to rule anything out, but I have no idea what to do, at this point.
I'm pretty handy as a software developer, but I'm not so hot at Linux maintenance. I know just enough to get by. These really perplexing problems are beyond my realm of expertise and I was hoping someone on here could help me.
Thanks in advance for any helpful replies.