1

I had an incident happen not too long ago with a server running mysql. Someone had enabled slow query logging on the server but never bothered to implement a log rotation, so the log files slowly filled up the hard drive over time until the disk became completely full and mysql no longer worked properly. When I went to go investigate, I discovered what the problem was and so I deleted all the slow query log files and disabled slow query logging. This resolved the immediate problem, but about a month later the filesystem on the server spontaneously went read-only. The disk was only at about 18% usage when this happened, so the disk was nowhere near full. I contacted the datacenter to investigate for any possible hardware problems, but they told me the hardware was fine and that it was just filesystem errors.

So my question is, what might have caused this to happen? Could the disk being filled up a month before have caused some kind of filesystem errors that slowly grew worse over time until the filesystem eventually went read-only a month later?

P.S. I am aware that I probably should been paying more attention to the disk usage on the server. I know I could have prevented the disk from ever getting filled up in the first place if I did, which was my mistake. However, I am still curious to know what lead up the filesystem spontenously going read-only a month later. I'd appreciate you guys' feedback on this.

Thanks!

4
  • 1
    It isn't likely that these two events are connected. Did you have an outage?
    – Paul
    Oct 29, 2014 at 23:19
  • No, there was no outage. The server was running fine between the first incident and the second incident. I just logged in one day and discovered that the filesystem had gone read-only. My guess is that when the disk filled up a month before, it might have caused some sort of filesystem errors which slowly grew worse over time until it eventually lead to the OS putting the filesystem in read-only mode to prevent further damage. I'm not sure if this is actually what happened though, which is why I am curious to know what you guys think. Oct 29, 2014 at 23:29
  • If nothing else happened, then that may be it - the filesystem errors were there since the log filled the space. Or even prior to that, and were always there, but only just became a problem when whatever files were affected were accessed. When you do the repair it will be clearer.
    – Paul
    Oct 29, 2014 at 23:32
  • Did all filesystems on the server become read-only, or just the one? Is the filesystem SAN-based, or local disk? If you know approximately when the problem started, you may be able to find a reason in /var/log/messages (won't be applicable if all filesystems went read only since it wouldn't be able to write to this file). Oct 30, 2014 at 16:01

0

You must log in to answer this question.

Browse other questions tagged .