1

I have a directory with

42048 File(s) 32,442,230,574 bytes

Running defrag tells me everything is fine

defrag E: /A /V
Microsoft Drive Optimizer
Copyright (c) Microsoft Corp.

Invoking analysis on New Volume (E:)...


The operation completed successfully.

Post Defragmentation Report:

        Volume Information:
                Volume size                 = 931.50 GB
                Cluster size                = 4 KB
                Used space                  = 814.77 GB
                Free space                  = 116.73 GB

        Fragmentation:
                Total fragmented space      = 1%
                Average fragments per file  = 1.00

                Movable files and folders   = 516620
                Unmovable files and folders = 4

        Files:
                Fragmented files            = 160
                Total file fragments        = 2273

        Folders:
                Total folders               = 25503
                Fragmented folders          = 2
                Total folder fragments      = 8

        Free space:
                Free space count            = 78165
                Average free space size     = 1.52 MB
                Largest free space size     = 2.97 GB

        Master File Table (MFT):
                MFT size                    = 2.50 GB
                MFT record count            = 2631423
                MFT usage                   = 100%
                Total MFT fragments         = 14

        Note: File fragments larger than 64MB are not included in the fragmentation statistics.

        You do not need to defragment this volume.

So I should not have any fragmentation issues. But when I try to delete some files in that directory I see in WPA that the deletion of one file needs 600ms and after defrag did run over night it need 2400ms for a single file! enter image description here

Since Defrag says everything is ok what is so fragmented here? I have tried contig.exe to check $MFT which has 7 fragments but that does not look too bad:

E:\> contig -a $MFT

Contig v1.8 - Contig
Copyright (C) 2001-2016 Mark Russinovich
Sysinternals

E:\$Mft is in 13 fragments
E:\$Mft::$BITMAP is defragmented

Summary:
     Number of files processed:      2
     Number unsuccessfully procesed: 0
     Average fragmentation       : 7 frags/file

The slow methods seems to be NtfsInsertCachedLcnAtIndex which is called by NtfsScanEntireBitmap. I suspect that the NTFS bitmap is degraded to a very long list. The problem is that I cannot access $BITMAP with contig.exe. It says Access is denied. Has anyone had similar issues and how were they solved except to format the hard drive? enter image description here Would changing the NTFS cluster size from 4 KB to 32 KB make things better or would this make no difference for a volume where many 1-2 MB files are created and deleted all the time?

Update 1 Added Smart Data

enter image description here

Update 2 Added File Deletion Graph over n File Deletions - No Clear Trend Visible

enter image description here

Update 3 Creating 100 500K files in an empty folder with contig.exe and deleting then file by file shows this pattern

    for( $i=0;$i -le 100;$i++){​​​​​​​​ contig -n "$i.json" 500000  }​​​​​​​​
    Get-ChildItem *.json | foreach {​​​​​​​​ (Measure-Command {​​​​​​​​ Remove-Item $_ }​​​​​​​​ ).TotalMilliSeconds }​​​​​​​​

5.2938
2.7115
1.2143
1.297
1.1552
1.3336
1.8351
1.1386
2230.3417
0.862
0.7522
0.7832
0.7061
0.711
0.7167
0.713
0.7137
2627.1645

Contig creates unfragmented files so it cannot be file fragmentation. But for some files the deletion is very inefficient. This more looks like an NTFS file system issue. I will check with MS.

Update 4 This is an actual limit inside NTFS. I have written some tests to create e.g. several hundred K files in a folder to check NTFS file creation query performance. That works well and despite the rumors it works up to several hundred K files with no big degradation even if the file names are nearly identical which I did suspect could cause some tree balancing issues. After doing a sustained write performance test on another machine with a Samsung SSD I was hit by the file deletion issue again. This time it takes 2-6s to delete a single file via a del . from cmd.exe in a directory with 20K files inside them.

enter image description here

Some NTFS tables are degraded to huge linked lists which should have O(1) lookup time but not in this case anymore. I wonder how many people were hit by this and believed some expert answer that you do not need to defrag your drive on SSDs... You do not need to move the clusters around but you still need to defrag NTFS tables which certainly need some cleanup as my Samsung SSD 860 EVO 1TB shows. The original HDD issue did get better from day to day after defrag did run several days. I need to "defrag" some things inside NTFS but I have no idea what. It should not be the volume bitmap as this will influence only seek times which are fast with a SSD.

4
  • Is the drive healthy? Check SMART.
    – gronostaj
    Mar 2, 2021 at 10:49
  • Smart data is ok according to HWInfo. Chckdsk also does not report anyhting. Mar 2, 2021 at 10:59
  • Did you run chkdsk?
    – Moab
    Mar 2, 2021 at 13:50
  • @Moab: Yes chkdsk is fine with the drive Mar 3, 2021 at 6:56

2 Answers 2

2

A larger sector size might improve the performance a bit, but I suspect not by much.

I would say that your problem is that of having 42048 files in one folder.

Windows folder functions are sequential in nature, so the more entries in the folder, the larger is its entry on the disk that needs to be read and written on each operation, and longer the required search time.

My advice would be to break up the folder into smaller sub-folders of no more than a few thousand files each.

10
  • That is not the entire story. I have reduced file count from 40k down to 20K and I am trying deletion of files. The time did go up from 600ms , 2400ms and now 3500 ms per file deletion. There is more behind this than simply moving files to a different folder. Something is truly degrading here. Mar 2, 2021 at 12:19
  • If this is also the case for a folder with only 1000 files, then the disk itself is degrading. I would have liked to see the SMART data, just in case. Is the disk almost full?
    – harrymc
    Mar 2, 2021 at 13:59
  • I have added Smart Data Mar 2, 2021 at 14:18
  • 1
    It seems like you have Reallocated Sectors. I hope that you have good backups to the disk.
    – harrymc
    Mar 2, 2021 at 14:28
  • Mostly it is a scratch drive. I will check to get a new one. Mar 2, 2021 at 14:48
1

The problem I had was that opening a folder containing roughly 70 000 files with a length of 3-5MBytes on a 1 or 2TB hdd with NTFS lasted 60-90 seconds in WinXP SP3 32bit. When I sorted out double recordings the number dropped to maybe 4000 files and explorer behaviour was was OK then.

I have difficulties in checking your SMART data. I consider a smartmontools log file (parameter "-a") as easier to read.

You said:

The slow methods seems to be NtfsInsertCachedLcnAtIndex which is called by NtfsScanEntireBitmap. I suspect that the NTFS bitmap is degraded to a very long list.

No, not a list. A list is very time consuming to access. The NTFS bitmap is really a bitmap stored into a file. Given a cluster number x you would check roughly the byte at file position rounddown(x/8) and examine the bit equivalent to x MOD 8 to check if the cluster is free or not. The length is determined when formating a drive. To verify I just checked my D-drive: length is 107 734 683 648 bytes, bitmap file length is 3 287 808. Cluster size is 8x512 bytes. 14 Gbytes are unused. 107 734 683 648/4096/8 gives roughly 3 287 808. Deleting a file is cheap by setting the MFT entry to deleted and deleting the clusters used by this file. But I guess you may have to rebalance the directory list which is stored as a tree in case of deletion and maybe some other structures that keep track of free contiguous cluster areas.

Every deletion will affect the lists described here:

https://link.springer.com/article/10.1007/s11036-019-01441-1

https://www.researchgate.net/publication/338057560_Disk_Cluster_Allocation_Behavior_in_Windows_and_NTFS

Upon deletion of each file you will have to rework this lists if you use them to avoid defragmentation.

This is what I guess the time needed comes from:

  1. Rebalancing the directory tree
  2. Rebalancing the free cluster lists.

P.S: Would like to learn which nice little debugging tools you used to gather that information you presented, p.e. the time log with all the function calls listed.

1
  • 2
    I have used Windows Performance Toolkit which is part of the Windows SDK. The UI is WPA (Windows Performance Analyzer). To record the data you can use the wpr.exe which is part of Windows 10. Start recording with wpr -start CPU -start DiskIO ... repro issue wpr -stop c:\temp\Issue.etl Then open in WPA and look at the data. Thats it. The rest is years of experience where to look at. Mar 4, 2021 at 7:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .