2

Has my Solid State Disk become a Super Slim Doorstopper?

I know this is a long question, but I tried to make it as elaborative and informative as possible. For a tl;dr simply skip the first half of the question, though I think the information in there might be relevant to the issue.

What happened

First of all: I live in an area that is suffering from a massive heatwave at the moment. The inside air temperature of my room has never been below 30°C in 2-3 weeks. Since days it's never been below 34°C, not even in the middle of the night. I don't have AC and my fan does practically nothing. My SSD's temp. sensor seems broken (always reports 5°C), my HDDs were at 48°C, 54°C and 54°C pretty much always. GPU around 60°C and CPU around 52°C. That's not good, but still sounds tolerable to me.

Last night I was using my PC, arch linux on a 64GB SSD, when everything would freeze. I couldn't even SSH into the machine anymore. So after waiting half an hour in hopes of getting at least an SSH connection, I had to turn off the power. I'd like to mention too that sometimes my PC would become really slow when I use audacity (writes temp. data to SSD as audacity seems to not support NTFS filesystems and my SSD is the only non-NTFS filesystem I have) and that recently I came across this question talking about SSDs getting slower when they fill. I can say that my SSD goes to +95% used space multiple times a week, if not daily, due to a lot of audacity recordings.

So after turning the PC off I tried to turn it on again, at the BIOS screen it went through all the disks and the SSD said S.M.A.R.T. error. After starting grub (on another drive) and trying to boot into arch (boot partition on another drive too) I got the message Device /dev/mapper/mydisk-root not found, or something similar. mydisk-root should be the root partition within the volume group of my LUKS encrypted SSD. So I tried rebooting a few times but always got the same result, when I eventually gave up, turned off the PC (on the PSU) and went to sleep.

Next actions I performed

After I woke up I wanted to boot a live linux USB to perform a SMART scan, look at dmesg, whatever there is. Suddenly the BIOS said S.M.A.R.T. ok again. I kept going with the live USB though, where I was able to unlock and mount the SSD just like usual. I was able to perform a full backup without any issues too.

Then I went to take a SMART test. A long test failed twice at 50%, details below. A short test completed and I can't see anything bad in the results. The last SMART test I took was just 2 weeks ago, it was a long test (see test log) and everything was fine.

Question 1: How toast is my SSD?

This is the output of the SMART attribute table before I tried any tests, so I think these should be the results from the long test I made two weeks ago still:

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   100   100   050    Pre-fail  Always       -       0
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   100   100   001    Old_age   Always       -       23891
 12 Power_Cycle_Count       0x0032   100   100   001    Old_age   Always       -       1063
170 Grown_Failing_Block_Ct  0x0033   100   100   010    Pre-fail  Always       -       0
171 Program_Fail_Count      0x0032   100   100   001    Old_age   Always       -       10
172 Erase_Fail_Count        0x0032   100   100   001    Old_age   Always       -       0
173 Wear_Leveling_Count     0x0033   080   080   010    Pre-fail  Always       -       611
174 Unexpect_Power_Loss_Ct  0x0032   100   100   001    Old_age   Always       -       244
181 Non4k_Aligned_Access    0x0022   100   100   001    Old_age   Always       -       302 89 212
183 SATA_Iface_Downshift    0x0032   100   100   001    Old_age   Always       -       0
184 End-to-End_Error        0x0033   100   100   050    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   001    Old_age   Always       -       2
188 Command_Timeout         0x0032   100   100   001    Old_age   Always       -       0
189 Factory_Bad_Block_Ct    0x000e   100   100   001    Old_age   Always       -       58
194 Temperature_Celsius     0x0022   100   100   000    Old_age   Always       -       0
195 Hardware_ECC_Recovered  0x003a   100   100   001    Old_age   Always       -       0
196 Reallocated_Event_Count 0x0032   100   100   001    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   100   100   001    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   100   001    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   100   100   001    Old_age   Always       -       1
202 Perc_Rated_Life_Used    0x0018   080   080   001    Old_age   Offline      -       20
206 Write_Error_Rate        0x000e   100   100   001    Old_age   Always       -       10

This is the full -a result after the attempted long test today, which failed (see test log):

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x80) Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      ( 117) The previous self-test completed having
                    the read element of the test failed.
Total time to complete Offline 
data collection:        (  295) seconds.
Offline data collection
capabilities:            (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:    (   2) minutes.
Extended self-test routine
recommended polling time:    (   4) minutes.
Conveyance self-test routine
recommended polling time:    (   3) minutes.
SCT capabilities:          (0x003d) SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   100   100   050    Pre-fail  Always       -       0
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   100   100   001    Old_age   Always       -       23891
 12 Power_Cycle_Count       0x0032   100   100   001    Old_age   Always       -       1063
170 Grown_Failing_Block_Ct  0x0033   100   100   010    Pre-fail  Always       -       0
171 Program_Fail_Count      0x0032   100   100   001    Old_age   Always       -       10
172 Erase_Fail_Count        0x0032   100   100   001    Old_age   Always       -       0
173 Wear_Leveling_Count     0x0033   080   080   010    Pre-fail  Always       -       611
174 Unexpect_Power_Loss_Ct  0x0032   100   100   001    Old_age   Always       -       244
181 Non4k_Aligned_Access    0x0022   100   100   001    Old_age   Always       -       302 89 212
183 SATA_Iface_Downshift    0x0032   100   100   001    Old_age   Always       -       0
184 End-to-End_Error        0x0033   100   100   050    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   001    Old_age   Always       -       2
188 Command_Timeout         0x0032   100   100   001    Old_age   Always       -       0
189 Factory_Bad_Block_Ct    0x000e   100   100   001    Old_age   Always       -       58
194 Temperature_Celsius     0x0022   100   100   000    Old_age   Always       -       0
195 Hardware_ECC_Recovered  0x003a   100   100   001    Old_age   Always       -       0
196 Reallocated_Event_Count 0x0032   100   100   001    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   100   100   001    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   100   001    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   100   100   001    Old_age   Always       -       1
202 Perc_Rated_Life_Used    0x0018   080   080   001    Old_age   Offline      -       20
206 Write_Error_Rate        0x000e   100   100   001    Old_age   Always       -       10

SMART Error Log Version: 1
Warning: ATA error count 0 inconsistent with error log pointer 2

ATA Error Count: 0
    CR = Command Register [HEX]
    FR = Features Register [HEX]
    SC = Sector Count Register [HEX]
    SN = Sector Number Register [HEX]
    CL = Cylinder Low Register [HEX]
    CH = Cylinder High Register [HEX]
    DH = Device/Head Register [HEX]
    DC = Device Command Register [HEX]
    ER = Error register [HEX]
    ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 0 occurred at disk power-on lifetime: 23890 hours (995 days + 10 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  00 50 00 d0 14 d1 40   at LBA = 0x00d114d0 = 13702352

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 08 d0 14 d1 40 00   1d+05:22:14.080  READ FPDMA QUEUED
  60 00 08 c8 14 d1 40 00   1d+05:22:14.080  READ FPDMA QUEUED
  60 03 08 c0 14 d1 40 00   1d+05:22:14.080  READ FPDMA QUEUED
  60 10 08 b8 14 d1 40 00   1d+05:22:14.080  READ FPDMA QUEUED
  60 00 08 b0 14 d1 40 00   1d+05:22:14.080  READ FPDMA QUEUED

Error -1 occurred at disk power-on lifetime: 23890 hours (995 days + 10 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  00 50 00 d0 14 d1 40   at LBA = 0x00d114d0 = 13702352

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 d5 00 d8 13 d1 40 00   1d+05:22:14.080  READ FPDMA QUEUED
  60 00 00 d8 12 d1 40 00   1d+05:22:14.080  READ FPDMA QUEUED
  60 da 00 d8 11 d1 40 00   1d+05:22:14.080  READ FPDMA QUEUED
  60 d0 00 d8 10 d1 40 00   1d+05:22:14.080  READ FPDMA QUEUED
  60 d1 80 58 10 d1 40 00   1d+05:22:14.080  READ FPDMA QUEUED

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       50%     23891         66387896
# 2  Extended offline    Completed: read failure       50%     23889         66387896
# 3  Extended offline    Completed without error       00%     23437         -
# 4  Short offline       Completed without error       00%       564         -
# 5  Vendor (0xff)       Completed without error       00%       558         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

This is the full -a result after the attempted short test today, which succeeded:

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x80) Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                    without error or no self-test has ever 
                    been run.
Total time to complete Offline 
data collection:        (  295) seconds.
Offline data collection
capabilities:            (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:    (   2) minutes.
Extended self-test routine
recommended polling time:    (   4) minutes.
Conveyance self-test routine
recommended polling time:    (   3) minutes.
SCT capabilities:          (0x003d) SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   100   100   050    Pre-fail  Always       -       0
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   100   100   001    Old_age   Always       -       23891
 12 Power_Cycle_Count       0x0032   100   100   001    Old_age   Always       -       1063
170 Grown_Failing_Block_Ct  0x0033   100   100   010    Pre-fail  Always       -       0
171 Program_Fail_Count      0x0032   100   100   001    Old_age   Always       -       10
172 Erase_Fail_Count        0x0032   100   100   001    Old_age   Always       -       0
173 Wear_Leveling_Count     0x0033   080   080   010    Pre-fail  Always       -       611
174 Unexpect_Power_Loss_Ct  0x0032   100   100   001    Old_age   Always       -       244
181 Non4k_Aligned_Access    0x0022   100   100   001    Old_age   Always       -       302 89 212
183 SATA_Iface_Downshift    0x0032   100   100   001    Old_age   Always       -       0
184 End-to-End_Error        0x0033   100   100   050    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   001    Old_age   Always       -       2
188 Command_Timeout         0x0032   100   100   001    Old_age   Always       -       0
189 Factory_Bad_Block_Ct    0x000e   100   100   001    Old_age   Always       -       58
194 Temperature_Celsius     0x0022   100   100   000    Old_age   Always       -       0
195 Hardware_ECC_Recovered  0x003a   100   100   001    Old_age   Always       -       0
196 Reallocated_Event_Count 0x0032   100   100   001    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   100   100   001    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   100   001    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   100   100   001    Old_age   Always       -       1
202 Perc_Rated_Life_Used    0x0018   080   080   001    Old_age   Offline      -       20
206 Write_Error_Rate        0x000e   100   100   001    Old_age   Always       -       10

SMART Error Log Version: 1
Warning: ATA error count 0 inconsistent with error log pointer 2

ATA Error Count: 0
    CR = Command Register [HEX]
    FR = Features Register [HEX]
    SC = Sector Count Register [HEX]
    SN = Sector Number Register [HEX]
    CL = Cylinder Low Register [HEX]
    CH = Cylinder High Register [HEX]
    DH = Device/Head Register [HEX]
    DC = Device Command Register [HEX]
    ER = Error register [HEX]
    ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 0 occurred at disk power-on lifetime: 23890 hours (995 days + 10 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  00 50 00 d0 14 d1 40   at LBA = 0x00d114d0 = 13702352

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 00 08 d0 14 d1 40 00   1d+05:22:14.080  READ FPDMA QUEUED
  60 00 08 c8 14 d1 40 00   1d+05:22:14.080  READ FPDMA QUEUED
  60 03 08 c0 14 d1 40 00   1d+05:22:14.080  READ FPDMA QUEUED
  60 10 08 b8 14 d1 40 00   1d+05:22:14.080  READ FPDMA QUEUED
  60 00 08 b0 14 d1 40 00   1d+05:22:14.080  READ FPDMA QUEUED

Error -1 occurred at disk power-on lifetime: 23890 hours (995 days + 10 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  00 50 00 d0 14 d1 40   at LBA = 0x00d114d0 = 13702352

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  60 d5 00 d8 13 d1 40 00   1d+05:22:14.080  READ FPDMA QUEUED
  60 00 00 d8 12 d1 40 00   1d+05:22:14.080  READ FPDMA QUEUED
  60 da 00 d8 11 d1 40 00   1d+05:22:14.080  READ FPDMA QUEUED
  60 d0 00 d8 10 d1 40 00   1d+05:22:14.080  READ FPDMA QUEUED
  60 d1 80 58 10 d1 40 00   1d+05:22:14.080  READ FPDMA QUEUED

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     23891         -
# 2  Extended offline    Completed: read failure       50%     23891         66387896
# 3  Extended offline    Completed: read failure       50%     23889         66387896
# 4  Extended offline    Completed without error       00%     23437         -
# 5  Short offline       Completed without error       00%       564         -
# 6  Vendor (0xff)       Completed without error       00%       558         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

I find it very funny how all three attribute tables are the same. Or am I missing something here? I'm not an expert on SMART but according to my knowledge these are all three perfect results. (?) I didn't try yet, but since mounting and getting the files worked and the BIOS reports it as ok again I assume I could boot into it again too. Should I though?

Question 2: Why did this happen?

Is this simply an aging thing or did my constant use of audacity on the SSD cause this?

Has it something to do with the SSD constantly reaching 90-100% used space?

How can it go from everything is fine to I can't even perform a SMART test anymore within just two weeks?

What do these smart test results say? The attribute table after the test today still looks great to me, or am I wrong?

Question 3: Is this contagious?

If this SSD were to be broken and I were to buy a new one, could I simply dd if=/old/ssd of=/new/ssd and be fine or would that cause trouble? What would be the best approach to move to a new disk? Please note that I am using LUKS on the entire device in RAW mode with a detached header, and I'd like to just "clone" all of that onto the new disk.


Edit: I just booted into that SSD again and it seems to work. I'll get a new SSD asap though since I assume using this one is a bad idea. The following are the latest entries in syslos before the crash:

syslos

  • 1
    Dudes I've spent an hour writing that trying to make it as well-formatted as possible adding as much information as I have and another 15 minutes just proof-reading it. Explain your downvotes so I can edit/add whatever you consider bad/missing. – confetti Aug 4 '18 at 23:40
  • @DrMoishePippik I'd rather not use wine, but does your audacity on ubuntu write to NTFS as a directory for temporary recording files? Can't find much about this anywhere, but seems like a driver issue or something to me. – confetti Aug 5 '18 at 16:35
3

The SMART status shows a lot of old or dying indicators, but nothing particularly screams " this killed it!".

Your log shows a power on lifetime of 995 days and 10 hours, suggesting that you leave your machine on permanently which isn't a bad thing of itself, it just means that the drive has seen a lot of hours of small writes as the operating system does book-keeping and general use.

To me it looks like the SSD is just old and worn out. The Perc_Rated_Life_Used is surprisingly low, as is Erase_Fail_Count

The worrying thing for me is your "regular" hitting of 95%+ full, which will reduce the pool of empty blocks available for the wear levelling algorithm to do its job. You will effectively end up stressing a small amount of blocks harder during the times when you are starved of space resulting in a small cluster of blocks with a massive level of writes, while the average across the drive is pretty low. By doing it repeatedly the wear leveller will probably choose the "best" (least written) blocks to write to first, but as you get to 100% full you are left with the "worst" blocks. Combine that with general programs and the operating system running its tasks means that you are going to be wearing out the worst blocks a lot faster. It's a perfect way to stress the worst parts of the drive and send it to an early grave.

You effectively force the key filesystem and SSD bookkeeping functions into the worst cells as they are likely to be written regularly to the drive, particularly when the SSD is mostly full, and sooner or later something bad is going to happen. If you've run out of reallocatable blocks and a key structure cannot be moved then the drive could deadlock itself.

This is why people say that you should always try to keep some anecdotal amount of space free on your drive, because the less space you have free the harder you are working the area that is free.

It's possible that old age and heavy writes to small groups of blocks have worn out parts of the drive.

Chances are copying what you need to a new drive will be fine, hardware failures like this tend not to be contagious.

  • Thank you a lot for all the information. I got just one question: How do you get the "bad indicators" from the SMART log? As per my understanding, a value of 100 means the best value possible and the worst column shows the worst value the device has ever seen of that attribute. Some websites say 255 is the best value, but all the drives I've owned so far had 100 as a maximum, even new ones. – confetti Aug 5 '18 at 16:32
  • 1
    Apologies... after testing, I could not get Audacity working in wine; I'd confused it with another AV app, perhaps MPC-HC, which does. Another thought: create an ext4 ramdisk for Audacity temp files? See unix.stackexchange.com/questions/401295/… – DrMoishe Pippik Aug 5 '18 at 18:11
  • That's a great idea. I'll try it out soon. I should probably move from NTFS to ext anyway... – confetti Aug 5 '18 at 18:23
  • @confetti the "raw value" for wear levelling shows 611, which isn't particularly high, but 1000 power cycles over 23000. As I said, nothing stands out as being bad from the SMART information, but 23000 hours is quite a long working life and if it spent any real time being mostly full then that's not good for an SSD. – Mokubai Aug 5 '18 at 19:02
  • How does one interpret the raw value? The article I've read before said that it's just an encoded vendor-specific form that when decoded gives the same as value, worst and threshold. The SSD has been used for four years now, around 18/7 on-time. It's been above 75% often but "just" since around a months it's been above 95% a lot. So I assume it's time to buy a new SSD? – confetti Aug 5 '18 at 19:09

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.