1

We have a customer system where a disk is completely inaccessible for a long time after deleting files. Typically, the disk is inaccessible for more than 45 minutes.

The system is a production system with about 55 TB, configured with 11 disks in RAID 5. The drives are all SAS drives with 7200rpm. The system also has a backup system which have the exact same problem, so a hardware fault is unlikely. 100GB of the RAID array is dedicated to the OS partiton running Windows 2019 server, while the rest is for the data partition that is hanging.

The number of deleted files are close to 5000 in number, with an average of 47 fragmentations. They are all in the same folder. The total number of files on the partition is around 500k spread over 90 folders. File size ranges from 1MB to 1GB.

The steps we do to reproduce it is simply to delete a folder. Using our own software, the delete calls are quickly processed; takes about 1.5sec. However, after that, the entire drives hangs completely for typically more than 45minutes. No calls (read/write) seem to complete during that time. The disk is shown in File Explorer but not even showing free space and total size. The OS disk is accessible during this time.

Deleting the same number of defragmented files (after a manual defrag or just copying the files) takes 1-2minutes which is also a fairly long time. This points strongly to fragmentation being the issue, but we don't know why. Deleting files should be fairly straight forward on even a fragmented system. Or are we missing something? Deleting using File Explorer (shitft+delete) causes same error, but the delete window hangs at about 50%. It completes after about the same amount of time.

The RAID controller reports no errors.

We can fix this by having our software write less fragmented files, which is doable. We are first and foremost interested in an explanation, so this doesn't happen for other customers.

The most relevant post we can find is this:
https://stackoverflow.com/questions/39161241/why-does-file-operations-hangs-after-deleting-folder-on-large-ntfs-volume
It is 6 years old, and no real information about what exactly is happening.

Tried using xperf for collecting information during the hang. Ran this from the OS disk so it is able to write, but this also hangs, so we are unable to get a report.

contig64 -f shows
Free cluster space: 35559636795392
Number of free space fragments: 1699 (this number might be interesting, it seems very low)
Largest free space block: 13484514672640
Defrag.exe shows 99% which is obviously too high.

Does anyone have any insights? Perhaps know some details about NTFS which would explain why it would spend alot of time after delete.

Edit after further investigation.
Using xperf to collect debug data and WPA for analysis, we have confirmed that our problem is exactly the same as the link above. It's the same stack trace and very similar symptoms. This certainly points to an ntfs bug (or at least vulnerability) where a single thread blocks the rest of the threads if they are accessing the same drive. It's no true explanation, but at least it narrows the problem down quite a bit.
There have been a fix that fixes something similar in windows 11, but that might be hard for us to verify as upgrading might be out of the question. The fixed problem is described here; https://github.com/microsoft/Windows-Dev-Performance/issues/64

5
  • Could it be that drives are SMR? Could it be that the array has undergone rebuilds in the past? (Wears out disks grossly, would rather switch to RAID10 despite losing capacity). Also hardware errors in the controller itself are sometimes difficult to detect.
    – jf1
    May 4, 2023 at 20:34
  • Drives are not SMR. And since it happens on two separate systems, a hardware fault is very unlikely. May 6, 2023 at 18:24
  • 1699 (this number might be interesting, it seems very low) - How so, this is number of contiguous blocks of free clusters I assume. May 6, 2023 at 21:07
  • Is this hardware RAID? If so do both RAID arrays use same hardware? May 6, 2023 at 21:08
  • Hardware RAID and same hardware yes. May 8, 2023 at 9:18

1 Answer 1

0

First I would use a program like ultra defrag for its ability to defragment the MFT of the NTFS file system.

Then I would get the windows version of smartctl or similar and check the SMART health report on each drive.

I would suspect the drives are starting to wear out, but at this point its only a guess.

I have experience bad sectors freezing drives and even the whole computer for minutes at a time per bad sector. Even 9 bad sectors could easily freeze your system for 45 mins or longer.

Also chkdsk /r on the NTFS to handle and bad sectors that would cause the drive to freeze up. The bad news is it needs to run while the volume is unmount.

You could try

chkdsk /r /scan which provides an online version of the chkdsk scan. Since its online it may/may not fix them but it will detect them.

/scan NTFS only: Runs an online scan on the volume
/forceofflinefix NTFS only: (Must be used with "/scan") Bypass all online repair; all defects found are queued for offline repair (i.e. "chkdsk /spotfix").

3
  • There are no reports of bad sectors, although we haven't tried to unmount it. That would probably be hard to get the customer to agree to. May 6, 2023 at 18:26
  • Event log might reveal bad sectors, but so would read-only scan for which unmount isn't required, right? Can be done for example using HD Sentinel which would also allow you to examine SMART values. May 6, 2023 at 21:05
  • chkdsk /forceofflinefix /scan /r n: I was able to start an online scan with this command without taking the volume offline.
    – cybernard
    May 8, 2023 at 3:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .