We have a customer system where a disk is completely inaccessible for a long time after deleting files. Typically, the disk is inaccessible for more than 45 minutes.
The system is a production system with about 55 TB, configured with 11 disks in RAID 5. The drives are all SAS drives with 7200rpm. The system also has a backup system which have the exact same problem, so a hardware fault is unlikely. 100GB of the RAID array is dedicated to the OS partiton running Windows 2019 server, while the rest is for the data partition that is hanging.
The number of deleted files are close to 5000 in number, with an average of 47 fragmentations. They are all in the same folder. The total number of files on the partition is around 500k spread over 90 folders. File size ranges from 1MB to 1GB.
The steps we do to reproduce it is simply to delete a folder. Using our own software, the delete calls are quickly processed; takes about 1.5sec. However, after that, the entire drives hangs completely for typically more than 45minutes. No calls (read/write) seem to complete during that time. The disk is shown in File Explorer but not even showing free space and total size. The OS disk is accessible during this time.
Deleting the same number of defragmented files (after a manual defrag or just copying the files) takes 1-2minutes which is also a fairly long time. This points strongly to fragmentation being the issue, but we don't know why. Deleting files should be fairly straight forward on even a fragmented system. Or are we missing something? Deleting using File Explorer (shitft+delete) causes same error, but the delete window hangs at about 50%. It completes after about the same amount of time.
The RAID controller reports no errors.
We can fix this by having our software write less fragmented files, which is doable. We are first and foremost interested in an explanation, so this doesn't happen for other customers.
The most relevant post we can find is this:
It is 6 years old, and no real information about what exactly is happening.
Tried using xperf for collecting information during the hang. Ran this from the OS disk so it is able to write, but this also hangs, so we are unable to get a report.
contig64 -f shows
Free cluster space: 35559636795392
Number of free space fragments: 1699 (this number might be interesting, it seems very low)
Largest free space block: 13484514672640
Defrag.exe shows 99% which is obviously too high.
Does anyone have any insights? Perhaps know some details about NTFS which would explain why it would spend alot of time after delete.
Edit after further investigation.
Using xperf to collect debug data and WPA for analysis, we have confirmed that our problem is exactly the same as the link above. It's the same stack trace and very similar symptoms. This certainly points to an ntfs bug (or at least vulnerability) where a single thread blocks the rest of the threads if they are accessing the same drive. It's no true explanation, but at least it narrows the problem down quite a bit.
There have been a fix that fixes something similar in windows 11, but that might be hard for us to verify as upgrading might be out of the question. The fixed problem is described here; https://github.com/microsoft/Windows-Dev-Performance/issues/64