Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I've been experiencing some weird errors from some hard drives, and I'm having trouble narrowing down the problem. It's part of a RAID array (Linux software RAID, budget hardware), so when it dropped out of the array my first reaction was to simply swap it out with a spare. But the rebuild kept failing.

The drives are in 5-drive external SATA enclosures (4 enclosures, 5 drives each) and I noticed that this enclosure was also showing odd symptoms. Normally when a drive is added, the lights blink in a certain way as the enclosure's SATA port replicator detects the new drive. And that kept happening during the re-build, which implied to me that maybe the port replicator's hardware was the actual culprit.

I have a spare enclosure for such a problem, so I swapped it out and put the drives in the new one. But at this point mdadm couldn't rebuild the drive at all. It always failed immediately after starting it. I tried a couple of different spare drives, same symptom. Surely I haven't had multiple drives fail the exact same way at the exact same time?

The enclosures connect to 2 SATA controller cards installed in the host, 2 ports each. Maybe it was one of the cards? So I moved around the SATA cables so that the "offending" enclosure was on the other card. But the same problem with the same drive bay in the same enclosure persisted.

At this point I'm running out of things to test. I have 2 drives, 2 enclosures, and 2 controller cards in the host. Any combination of them produces the same problems.

At this point mdadm is currently trying to re-build onto the drive (not sure why it didn't fail this time), but at a significantly reduced speed. It's fluctuating between 1/2 and 1/4 normal speed. The replacement enclosure is also re-detecting the drive like the previous one was.

Now, I don't know much about diagnosing hardware. With commodity hardware it's usually a replacement cycle. But all replacement parts are behaving the same way in this case, so I'm not sure what the problem is. I've been Googling some things I'm seeing in /var/log/syslog but so far haven't understood much of it. I can tell you that...

When the enclosure "re-detects" the drive, this shows up in syslog:

Oct  3 17:43:52 gibson kernel: [ 1478.755088] ata5: controller in dubious state, performing PORT_RST
Oct  3 17:43:54 gibson kernel: [ 1480.909415] ata5.05: limiting SATA link speed to 1.5 Gbps

Other troubling and often-repeated messages in syslog look like:

Oct  3 17:46:05 gibson kernel: [ 1612.163891] ata5.00: exception Emask 0x0 SAct 0x0 SErr 0x400000 action 0x6
Oct  3 17:46:05 gibson kernel: [ 1612.163894] ata5.00: irq_stat 0x00060002, device error via D2H FIS
Oct  3 17:46:05 gibson kernel: [ 1612.163897] ata5.00: SError: { Handshk }
Oct  3 17:46:05 gibson kernel: [ 1612.163899] ata5.00: failed command: WRITE DMA
Oct  3 17:46:05 gibson kernel: [ 1612.163904] ata5.00: cmd ca/00:00:00:29:87/00:00:00:00:00/e0 tag 0 dma 131072 out
Oct  3 17:46:05 gibson kernel: [ 1612.163905]          res 51/84:90:70:29:87/00:00:00:00:00/e0 Emask 0x10 (ATA bus error)
Oct  3 17:46:05 gibson kernel: [ 1612.163907] ata5.00: status: { DRDY ERR }
Oct  3 17:46:05 gibson kernel: [ 1612.163909] ata5.00: error: { ICRC ABRT }

and sometimes this:

Oct  3 18:07:10 gibson kernel: [ 2877.073010] ata5.00: failed to read SCR 1 (Emask=0x40)
Oct  3 18:07:10 gibson kernel: [ 2877.073015] ata5.00: failed to read SCR 0 (Emask=0x40)
Oct  3 18:07:10 gibson kernel: [ 2877.073017] ata5.01: failed to read SCR 1 (Emask=0x40)
Oct  3 18:07:10 gibson kernel: [ 2877.073020] ata5.01: failed to read SCR 0 (Emask=0x40)
Oct  3 18:07:10 gibson kernel: [ 2877.073022] ata5.02: failed to read SCR 1 (Emask=0x40)
Oct  3 18:07:10 gibson kernel: [ 2877.073024] ata5.02: failed to read SCR 0 (Emask=0x40)
Oct  3 18:07:10 gibson kernel: [ 2877.073026] ata5.03: failed to read SCR 1 (Emask=0x40)
Oct  3 18:07:10 gibson kernel: [ 2877.073028] ata5.03: failed to read SCR 0 (Emask=0x40)
Oct  3 18:07:10 gibson kernel: [ 2877.073030] ata5.04: failed to read SCR 1 (Emask=0x40)
Oct  3 18:07:10 gibson kernel: [ 2877.073032] ata5.04: failed to read SCR 0 (Emask=0x40)
Oct  3 18:07:10 gibson kernel: [ 2877.073034] ata5.05: failed to read SCR 1 (Emask=0x40)

Are there other tests I can run? Other things I can try? What could be wrong which presents symptoms regardless of the hardware being used?

The array has run fine for years, with the occasional drive replacement. No software changes have been made on the host (unless I've been compromised and don't know it, which is certainly possible). But a drive failure a few months back has resulted in constant re-building and ultimately left me in the current state.

Edit: Another pattern which I just realized, not sure if it means anything. The drives in the array are sdb1 through sdu1. They're not always seen by the booting OS in the same order, so any given drive could change its letter on reboot. When the drive was consistently failing immediately upon an mdadm rebuild, it was /dev/sdq1. But the throughout the current symptoms (slow re-build, enclosure "re-detecting" it over and over, and basically the errors logged above), it's been /dev/sdu1.

Edit: I downloaded Knoppix 7.2 to see if I could start the array and add the drive there. Just to see if maybe it's a software issue. Same exact symptoms. So... the hardware was replaced, the software was replaced, but the problem persists. This leaves me at kind of a dead end at the moment.

Edit: I've also tried just zero-ing out the drive with this:

dd if=/dev/zero of=/dev/sdu bs=1M

But the same symptoms persist. At this point I'm suspecting that maybe something is wrong with the controller, but in a way I don't understand. Each of the 2 cards works, but for whatever reason both together can't manage this many drives? Might not make any sense, I'm just trying to identify patterns here.

Edit: Output of smartctl -a -d ata /dev/sdu:

smartctl 5.40 2010-10-16 r3189 [x86_64-slackware-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Caviar Green family
Device Model:     WDC WD20EADS-00S2B0
Serial Number:    WD-WCAVY0536607
Firmware Version: 01.00A01
User Capacity:    2,000,398,934,016 bytes
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Sat Oct  5 07:04:21 2013 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x84) Offline data collection activity
                                        was suspended by an interrupting command from host.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                 (42900) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 255) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x303f) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   179   179   051    Pre-fail  Always       -       90370
  3 Spin_Up_Time            0x0027   149   149   021    Pre-fail  Always       -       9525
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       22
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   092   092   000    Old_age   Always       -       6170
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       18
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       9
193 Load_Cycle_Count        0x0032   001   001   000    Old_age   Always       -       610025
194 Temperature_Celsius     0x0022   117   073   000    Old_age   Always       -       35
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       125
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       62927
200 Multi_Zone_Error_Rate   0x0008   090   090   000    Old_age   Offline      -       22176

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Update: I may have my first promising break in this caper. After spending the past 6 or so hours (on and off) swapping out drives/enclosures/cables/cards and systematically trying to find a pattern, I finally found one. And one I'm able to consistently reproduce.

It doesn't like hot-swapping.

It says it does. It purports to. And it makes every effort to keep this promise. But it's a lie. I won't pretend to know anything about the low-level hardware architecture and/or kernel architecture which plays a role in this, but I can at least logically determine a reproducible pattern.

And I bet I started the hot-swap debacle this summer without even realizing it and likely caused this whole mess as a result. syslog shows that the errors I've been seeing started in June, which was just before the major failure occurred and just before I went to SF for the summer and couldn't work on it.

But if I reboot any time I need to move a disk, so far I'm not seeing the errors. The 20th disk has been re-added and is re-building at the correct speed (which still takes approximately 36 hours for a 2TB disk), and I'm watching (tail -f) syslog which thus far has been quiet.

It's going to be a few days before I can report on this with confidence. But so far it's looking promising. The answer to this may end up being "don't hot-swap with a Rosewill RSV-S5 on Linux", fingers crossed.

share|improve this question
    
Ok I'm an amateur when it comes to hardware administration, but have you checked the sata data or sata power cables? Can you use new (preferably buy new) ones just to see if that is the problem? Similar case: askubuntu.com/questions/133946/are-these-sata-errors-dangerous -- Your problem really sounds like a cable issue. Have you checked the SMART output? The other thing I could think of is bad power supply? –  medigeek Oct 3 '13 at 22:36
    
@medigeek: It's definitely worth a try. But swapping out the enclosure and switching cables back to the controller should have found that. If it was the power supply or the drive's connector to the enclosure, the replacement enclosure should have fixed it. If it was the SATA cable from the enclosure to the controller, then when I swapped it with the other controller's cable the symptoms should have moved to the other enclosure. I really feel like a software issue is the only remaining option given the experimentation so far. –  David Oct 3 '13 at 23:43
    
@medigeek: I've tried mucking with smartctl but haven't found out how to get meaningful information out of it yet. –  David Oct 3 '13 at 23:43
    
If it's software raid as you mention it should be ok.. can you show the output of cat /proc/mdstats and ls /dev/sd[a-z]? Then try smartctl -a /dev/sdX or smartctl -a -d ata /dev/sdX (where X is substituted by a device letter from /dev/sdX listed with ls command) –  medigeek Oct 5 '13 at 7:18
    
@medigeek: mdstat shows nothing out of the ordinary (19/20 drives, rebuilding the 20th), and all 20 drives are in the /dev/sd* list (as well as in /proc/partitions). I've added the output of smartctl -a -d ata /dev/sdu to the question, hopefully there's something helpful there. A colleague has suggested that it could be a timing issue when the controller cards are communicating with the drives. Maybe 20 drives is pushing a boundary I don't know about? –  David Oct 5 '13 at 10:41

1 Answer 1

up vote 0 down vote accepted

While I still don't really know much about the low-level architecture around the hardware or the kernel drivers which control it, it would appear that the "solution" in this case is to not hot-swap the drives.

So, when using a Rosewill RSV-S5 (or any of its ilk, I imagine) on Linux (maybe on other systems too, I don't know), never hot-swap. And if syslog ever says that a PORT_RST was performed, reboot just to be safe.

Hot-swapping may seem like it works at first. There's no FS mounted on the drive, it's just a bare partition, the hardware is set up for it with drive rails and whatnot, and when you put in the new drive the system recognizes it and assigns it a device with which you can interact. But keep an eye on syslog just to be safe.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.