After nouveau freezing everything with dual monitor for the millionth time I had to cut power on my macbook pro (mid 2010, fedora 24, SAMSUNG HN-M500MBB hard drive). Wasn't doing anything IO heavy just viewing slides with evince.
At reboot it starts spitting out errors about a bad sector and hanging with errors like:
blk_update_request: I/O error, dev sda, sector 969158669
ata1: EH complete
ata1.00: exception Emask 0x0 SAct 0x3c000000 SErr 0x0 action 0x6 frozen
ata1.00: failed command: READ FPDMA QUEUED
ata1.00: cmd 60/08:d0:08:30:c4/00:00:39:00:00/40 tag 26 ncq dma 4096 in
res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
ata1.00: status: { DRDY }
ata1.00: failed command: READ FPDMA QUEUED
ata1.00: cmd 60/28:d8:c8:2f:c4/00:00:39:00:00/40 tag 27 ncq dma 20480 in
res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
ata1.00: status: { DRDY }
ata1.00: failed command: READ FPDMA QUEUED
ata1.00: cmd 60/38:e0:88:2f:c4/00:00:39:00:00/40 tag 28 ncq dma 28672 in
res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
ata1.00: status: { DRDY }
ata1.00: failed command: READ FPDMA QUEUED
ata1.00: cmd 60/78:e8:08:2f:c4/00:00:39:00:00/40 tag 29 ncq dma 61440 in
res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
ata1.00: status: { DRDY }
ata1: hard resetting link
ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata1.00: configured for UDMA/133
ata1.00: device reported invalid CHS sector 0
with the occasional
sd 0:0:0:0: [sda] tag#19 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 0:0:0:0: [sda] tag#19 Sense Key : Medium Error [current]
sd 0:0:0:0: [sda] tag#19 Add. Sense: Unrecovered read error - auto reallocate failed
sd 0:0:0:0: [sda] tag#19 CDB: Read(10) 28 00 39 c4 30 08 00 00 08 00
blk_update_request: I/O error, dev sda, sector 969158669
Buffer I/O error on dev dm-2, logical block 1, async page read
and
ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
ata1.00: irq_stat 0x40000001
ata1.00: failed command: READ SECTOR(S) EXT
ata1.00: cmd 24/00:01:0d:30:c4/00:00:39:00:00/e0 tag 6 pio 512 in
res 51/40:01:0d:30:c4/00:00:39:00:00/e0 Emask 0x9 (media error)
ata1.00: status: { DRDY ERR }
ata1.00: error: { UNC }
Here's smartctl output after trying to read a couple of sectors after the bad one with hdparm:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 100 100 051 Pre-fail Always - 469
2 Throughput_Performance 0x0026 252 252 000 Old_age Always - 0
3 Spin_Up_Time 0x0023 086 086 025 Pre-fail Always - 4463
4 Start_Stop_Count 0x0032 092 092 000 Old_age Always - 8099
5 Reallocated_Sector_Ct 0x0033 252 252 010 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 252 252 051 Old_age Always - 0
8 Seek_Time_Performance 0x0024 252 252 015 Old_age Offline - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 19382
10 Spin_Retry_Count 0x0032 252 252 051 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 980
12 Power_Cycle_Count 0x0032 092 092 000 Old_age Always - 8214
181 Program_Fail_Cnt_Total 0x0022 097 097 000 Old_age Always - 66246139
191 G-Sense_Error_Rate 0x0022 100 100 000 Old_age Always - 3820
192 Power-Off_Retract_Count 0x0022 100 100 000 Old_age Always - 20
194 Temperature_Celsius 0x0002 064 051 000 Old_age Always - 32 (Min/Max 15/49)
195 Hardware_ECC_Recovered 0x003a 100 100 000 Old_age Always - 0
196 Reallocated_Event_Count 0x0032 252 252 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 100 100 000 Old_age Always - 15
198 Offline_Uncorrectable 0x0030 252 252 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0036 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x002a 100 100 000 Old_age Always - 255
223 Load_Retry_Count 0x0032 100 100 000 Old_age Always - 980
225 Load_Cycle_Count 0x0032 001 001 000 Old_age Always - 1583719
Note the pending sectors... Both short and long self tests report the same bad sector as the kernel.
Hdparm oddly manages to read everything successfully but it(see the edit below) kind of hangs and says
reading sector 969158769: SG_IO: bad/missing sense data, sb[]: 70 00 03 00 00 00 00 0a 00 51 e0 01 11 04 00 00 a0 71 00 00 00 00 00 00 00 00 00 00 00 00 00 00
succeeded
And it says it for something like 200 sectors after the first bad one. I rewrote a couple of ones with hdparm --write-sector and they stopped complaining. Now I'm doing a backup and ordered a new drive, but meanwhile I'd like to understand what happened and maybe try to fix this one.
Note the reallocated sector count is not being increased after I rewrote a couple of bad ones, which adds up to the weirdness of the whole thing. After a rewrite they read and write fine like nothing happened but the firmware doesn't seem to remap them as bad sectors.
Any idea? Should I just ditch the drive?
PS. OSX in another partition still works pretty fine.
EDIT: aftermath
After a backup I started experimenting a bit with the hard drive.
After the first bad sector there were about another 150 with the same issues. I tried reading them with dd and dd_rescue and they failed.
hdparm --read-sector worked (with the sense error above) but returned inconsistent data (different at each read).
hdparm --write-sector seemed to fix them so I just rewrote all the failing sectors.
Now smartctl reports 0 pending sectors and 0 reallocations, both short and long self tests complete without errors. Linux boots fine, all the errors vanished.
I'm a bit worried about those ~70kb I killed, it's a bit tricky with LVM to understand what they really contained. I dumped a couple of MBs around that area and it's all zeros, so I'm positive it's either empty space or swap.
Too soon to celebrate yet, but the outcome looks promising, will update the question if anything new happens.
ddordd_rescuefor that instead ofhdparm(which AFAIK can only write zeroes).