S.M.A.R.T. on Windows Server 2012 R2

Question

One of hard drives on my Windows Server 2012 R2 starts to fail and I detected this too late - only when I was not able to read one file and got some strange errors on read.

After checking the drive with a GUI tool it showed that everything was Ok.

Then I checked it with chkdsk:

chkdsk d: /f /r
.................................
Stage 4: Looking for bad clusters in user file data ...
A disk read error occurredc0000010
The disk does not have enough space to replace bad clusters
detected in file 191666 of name \RECEIV~1\Langs\E141~1\New\202004~1.mp4.
  235248 files processed.
File data verification completed

So, I conclude that the hard drive was silently relocating sectors until there are no more spare sectors left.

I think it is time for some S.M.A.R.T. reporting tool that runs in background on the server. Which would you take? Or maybe there are some built-in capabilities in Windows Server 2012 R2 ?

Update

Here is the S.M.A.R.T. info of the drive

*** DiskCheckup V3.5 Build: 1002 Report ***

SysInfo DLL Version:                                                  SysInfo v2.3 Build: 1000
Time of export:                                                       15:27:38 03-Nov-2022

Device information:                                                   
   Device ID:                                                         1
   Interface:                                                         USB
   Device Capacity:                                                   3815445 MB
   Serial Number:                                                     XXXXXXXXX
   Model Number:                                                      TOSHIBA MQ04UBB400
   Firmware Revision:                                                 JS000U
   Partitions:                                                        
      D:         3815317 MB

ATA information:                                                      

   Disk geometry:                                                     
      Cylinders:                                                      486401
      Tracks/Cylinder:                                                255
      Sectors/Track:                                                  63
      Bytes/Sector:                                                   512
      Total disk sectors:                                             7814037168
      Logical sector size:                                            512
      Physical sector size:                                           4096
      Media rotation rate:                                            5400 RPM
      Buffer size:                                                    N/A
      ECC size:                                                       N/A

   Standards compliance:                                              
      ATA8-ACS Supported:                                             Yes
      ATA/ATAPI-7 Supported:                                          Yes
      ATA/ATAPI-6 Supported:                                          Yes
      ATA/ATAPI-5 Supported:                                          Yes
      ATA/ATAPI-4 Supported:                                          Yes
      SATA Compliance:                                                Yes
         SATA Gen3 Signaling Speed (6.0Gb/s) Supported:               No
         SATA Gen2 Signaling Speed (3.0Gb/s) supported:               Yes
         SATA Gen1 Signaling Speed (1.5Gb/s) supported:               Yes
      Transport Type:                                                 Serial
         SATA 3.0 Transport Compilance:                               Yes
         SATA 2.6 Transport Compilance:                               Yes
         SATA 2.5 Transport Compilance:                               Yes
         SATA II: Ext Transport Compilance:                           Yes
         SATA 1.0a Transport Compilance:                              Yes
         ATA8-AST Transport Compilance:                               Yes
   World Wide ID:                                                     0000000000000000

   Feature support:                                                   
      SMART supported:                                                Yes
         SMART enabled:                                               Yes
      SMART self-test supported:                                      Yes
      SMART error log supported:                                      Yes
      LBA supported:                                                  Yes
      IORDY supported:                                                Yes
      CFast supported:                                                No
      DMA supported:                                                  Yes
         Maximum Multiword DMA mode supported:                        2
         Multiword DMA selected:                                      None
         Maximum UltraDMA mode supported:                             5
         UltraDMA selected:                                           5
      Maximum PIO mode supported:                                     4
      SATA Compliance:                                                Yes
         NCQ priority information supported:                          No
         Unload while NCQ commands are outstanding supported:         Yes
         Phy Event Counters supported:                                Yes
         Receipt of power management requests supported:              Yes
         NCQ feature set supported:                                   Yes
         Software Settings Preservation:                              Supported, Enabled
         In-order data delivery:                                      Not supported
         Initiating power management:                                 Supported, Disabled
         DMA Setup auto-activation:                                   Supported, Disabled
         Non-zero buffer offsets:                                     Not supported
      Trusted Computing supported:                                    No
      Host Protected Area (HPA) supported:                            Yes
      Read look-ahead supported:                                      Yes
         Read look-ahead enabled:                                     Yes
      Write cache supported:                                          Yes
         Write cache enabled:                                         Yes
      Power management supported:                                     Yes
      Security mode supported:                                        Yes
         Security mode enabled:                                       No
      Device Configuration Overlay (DCO) supported:                   Yes
      48bit Addressing supported:                                     Yes
      Auto Acoustic Managment (AAM) supported:                        No
      Power-up in Standby (PUIS) supported:                           No
      Advanced Power Management (APM) supported:                      Yes
         Advanced Power Management (APM) enabled:                     Yes
         Current APM level:                                           Minimum power consumption without Standby
      CompactFlash Association (CFA) supported:                       No
      General Purpose Logging (GPL) supported:                        Yes
      Streaming supported:                                            No
      Media card pass through supported:                              No
      Extended power conditions supported:                            No
      Extended status reporting supported:                            Yes
         Extended status reporting enabled:                           No
      Write-read-verify supported:                                    Yes
         Write-read-verify enabled:                                   No
      Free-fall control supported:                                    No
      TRIM command supported:                                         No
      SCT command transport supported:                                Yes
      NV Cache enabled:                                               No
      NV Cache Power Management supported:                            No

SMART ATTRIBUTES:
 ID    Description                             Status        Value         Worst         Threshold     Raw Value     TEC                 
---------------------------------------------------------------------------------------------------------------------------------------------
  1    Raw Read Error Rate                     OK            100           100           50            0             N/A                 
  2    Throughput Performance                  OK            100           100           50            0             N/A                 
  3    Spin Up Time                            OK            100           100           1             5359          N/A                 
  4    Start/Stop Count                        OK            100           100           0             7453          N/A                 
  5    Reallocated Sector Count                OK            100           100           50            0             N/A                 
  7    Seek Error Rate                         OK            100           100           50            0             N/A                 
  8    Seek Time Performance                   OK            100           100           50            0             N/A                 
  9    Power On Time                           OK            71            71            0             11938         N/A                 
 10    Spin Retry Count                        OK            249           100           30            0             N/A                 
 12    Power Cycle Count                       OK            100           100           0             14            N/A                 
191    G-sense Error Rate                      OK            100           100           0             0             N/A                 
192    Power off Retract Count                 OK            100           100           0             1             N/A                 
193    Load Cycle Count                        OK            97            97            0             30040         N/A                 
194    Temperature                             OK            100           100           0             31 C          N/A                 
196    Reallocation Event Count                OK            100           100           0             0             N/A                 
197    Current Pending Sector Count            OK            100           100           0             0             N/A                 
198    Uncorrectable Sector Count              OK            100           100           0             0             N/A                 
199    UltraDMA CRC Error Count                OK            200           253           0             0             N/A                 
220    Disk shift                              OK            100           100           0             0             N/A                 
222    Loaded hours                            OK            99            99            0             701           N/A                 
223    Load/Unload retry count                 OK            100           100           0             0             N/A                 
224    Load friction                           OK            100           100           0             0             N/A                 
226    Load-in Time                            OK            100           100           0             275           N/A                 
240    Head flying hours                       OK            100           100           1             0             N/A                 

Answer 1

I think it is time for some S.M.A.R.T. reporting tool that runs in background on the server. Which would you take?

My personal favorite is HD Sentinel. It's actively maintained and supported and it's easy on system resources.

So, I conclude that the hard drive was silently relocating sectors until there are no more spare sectors left.

This may be so, but I have seen examples of chkdsk throwing this message while there were plenty of sectors to spare. I doubt the accuracy of the error message. First of all drives reallocate sectors, not clusters. Clusters are only a 'thing' from a file system perspective.

But silent reallocations are possible if the drive itself detects a problem with a sector while it was still able to recover the data from the sector. So IOW, a drive will only 'silently reallocate' IF the data from a sector is recoverable after error recovery procedures (ERP) which are performed by the drive itself.

In your case chkdsk reports a sector is bad, so this means the drive was not able to read the sector. In SMART you should then be able to see one or more 'pending' sectors' (Raw value > 0). Such pending sectors are only reallocated on write. IOW if you'd write zeros to the sector, this signals the drive you are willing to give up on the data in the bad or pending sector.

Again a tool like HD Sentinel can try this: The surface scanner can be configured to scan for read errors, and upon error can be instructed to write zeros to the problematic sector.

But then again, I'd recommend to first look at overall SMART values. And in general I'd advise to replace a drive that has started reallocating sectors and continues to do so.

Summarizing, in general the conditions for sector reallocation are:

Non-recovered write errors

When a write operation can not be completed after the Error Recovery Procedure (ERP) is fully carried out, the sector(s) are reallocated to the spare location. An error is reported to the host system only when the write cache is disabled and the auto reallocation fails.

You can observe such a sector reallocation happened using a S.M.A.R.T. utility, each reallocated sector increments ‘reallocated sector count’ attribute by one (in RAW value).

Note: If the number of available spare sectors reaches a certain threshold (like 16 sectors) , the write cache function may be disabled automatically.

Non-recovered read errors

When a read operation fails after a defined ERP (error recovery procedure) is fully carried out, a hard error is reported to the host system. This location is registered internally as a candidate for the reallocation.

You can observe such an event using a S.M.A.R.T. utility, in ‘pending sectors’ attribute.

When a registered location is specified as a target of a write operation, a sequence of media verifications is performed automatically. When the result of this verification meets the criteria, this sector is reallocated.

As a result if this event the SMART RAW value of ‘pending sectors’ decrements while ‘reallocated sector count’ increments.

Recovered read errors

When a read operation for a sector fails once and then is recovered at the specific ERP step, this sector of data is reallocated automatically. A media verification sequence may be run prior to the relocation according to the predefined conditions.

You can observe such a sector reallocation happened using a S.M.A.R.T. utility, each reallocated sector increments ‘reallocated sector count’ attribute by one (in RAW value).