1

We built around 12 machines a few months ago to run Ubuntu. They each have the following specs:

ASUS Z8NA-D6 motherboard Dual quad core Intel(R) Xeon(R) CPU E5520 @ 2.27GHz OCZ Mod Extreme Pro 500W power supply 12 GB Kingston RAM Nvidia GeForce 9800 GT graphics card

My machine ran well for awhile. However, it started experiencing random lockups. These lockups are not X lockups, they are complete system freezes. The nic stops responding, the magic sysrq buttons won't work. The machine is dead.

I first suspected RAM. Memtest86 didn't find anything, but I replaced the RAM anyway. Still, lockups. So I replaced the graphics card. Still, more lockups. They became more and more frequent and started to happen 2-3 times a day.

So I replaced the motherboard and power supply in one fell swoop. Suddenly, no more lockups! Woohoo!

Except, a week later, in the morning, the machine wouldn't wake up. I reset it, started it up, and the log files showed the last entry at around 11 pm the evening before. This has started occurring with more frequency...now just about every morning I come in, the machine is locked up, and has been since the night before.

Yesterday, in the 3 weeks since I replaced the motherboard and power supply, the machine actually locked up on in in mid-work. This is the first time since replacing the two (MB and PS) that this happened while I was using it. All others occurred while I was away.

I'm at a loss. Nothing is in syslog or message that would indicate a problem around the time of the lockup. Temps are good...I use lmsensors to monitor and have a script that writes the output to file every minute. They never get that high.

The only thing I haven't replaced at this point is the case and the hard drives. I doubt either could be the cause.

What would you do if you were in my shoes? Is there a troubleshooting approach I'm missing?

For the record, all of the other machines, all eleven of them, don't have any problems. They're all running the same version of Ubuntu (Lucid) that I am.

Improve this question
4
  • Is your Kingston RAM ECC? If not, why not? Non-ECC Gaming memory is not a good idea in a big workstation, especially 12GB of it.
    – kmarsh
    Jun 2, 2010 at 17:26
  • 1
    Also, have you compared motherboard rev and BIOS revision levels across systems? Just because you ordered them all at the same time, doesn't mean you got 12 identical boards.
    – kmarsh
    Jun 2, 2010 at 17:36
  • I believe it is ECC...but something in the back of my mind makes me believe ECC is turned off in the bios. Still, could the lack of ECC really cause this?
    – jbwiv
    Jun 2, 2010 at 20:29
  • Lack of ECC means you have no idea whether memory is erratic or not. Turning ECC on means you either have good memory, or you know that you don't.
    – kmarsh
    Jun 3, 2010 at 12:08

1 Answer 1

Reset to default
2

Since you have some identical systems, you have a good candidate for a binary parts swap.

Take a working system and swap the, say, CPUs. See if the problem migrates to the other machine. Then all the RAM. (I know you have replaced all the RAM; however with non-ECC RAM, it is quite possible you replaced bad RAM with bad RAM.) Continue with all other parts until the problem moves over. You can spare the motherboards because eventually you'll have all the other parts swapped, which is the same.

If you can't spare a machine, you can try halving the memory, or if you are running two CPU's, remove one.

Make one change at a time so you can identify the issue immediately. You may encounter a situation where both machines become unreliable. This can be because the original system has more than one issue, and/or there is some borderline issue, such as a very marginal power supply.

EDIT: After doing a little research, I am of the opinion that your $39.99 after rebate power supply is extremely marginal, in both quality and size. You have invested a lot in these system. Please Do not replace this cheap 500W power supply with a cheap 1000W power supply. Go to jonnyguru.com and get some recommendations on quality power supplies, list your components and let them choose the wattage. I'm thinking 650W or so, but the quality is equally important as the wattage. They recently reviewed a 1000W P/S that was actually more of a 650W, so you have an idea that the wattage ratings of many power supplies cannot be trusted.

Even if you find the issue is with another component, I still recommend upgrading the power supplies on these systems.

Improve this answer
12
  • Yeah, I'm with you. But I at least thought replacing the PS would eliminate it as a source, even if I did replace it with the same model.
    – jbwiv
    Jun 2, 2010 at 20:29
  • I see the 500W P/S (really 423W) has two 18A 12V rails. If you assume one rail is for the video card and hard disks, and most modern systems run almost entirely off of 12V, that leaves: RAM 12 * 5W = 60/12V = 5A, M/B ??? , CPU 2*80W/12V = 13.3A, Total, something more than 18.3A on an 18A capable rail. So, your system spec is extremely marginal for this power supply.
    – kmarsh
    Jun 3, 2010 at 12:08
  • I think you were right about the power supply. I picked up the best I could find at the local shop (Thermaltake 850W Black Widow). Since putting it in yesterday, no freezes. When I came in this morning, machine was fine as well. Thank you for your help! I will update this if something changes!
    – jbwiv
    Jun 4, 2010 at 12:20
  • Glad to hear it. After suffering from a spate of power supply failures a few years ago, I became an aficionado. At that time there were NO serious P/S reviews on the web, just unbox, plug it in, take a volt meter to the 12V rail under load and call it good. The only trusted brands were Antec and PCP&C. Now, it is still the Wild West as to the levels of quality available on the market, but several web review sites now have serious equipment for testing power supplies, including jonnyguru, hardocp, hardware secrets.
    – kmarsh
    Jun 4, 2010 at 13:44
  • Sadly, I came in this morning after a long weekend and it was locked up again. It appears from my heartbeat log I set up that it locked up Friday evening at around 11:30. Nothing in the logs again. Same symptoms...machine is unpingable, and won't respond to keyboard input. Monitors are dark. I'm at a complete loss. At this point, the only things I haven't replaced are the CPUs, the harddrives, and the case itself. Could any of these be the possible cause? I've never seen a bad CPU myself.
    – jbwiv
    Jun 8, 2010 at 19:16

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .