25

I have a 10M folders. Each folder contains 13 files.

All these folders I would like to put in one main folder (root).

Is there any limitation in Windows Server for that?

15
  • 22
    Do you actually have 10,000,000 folders? Questions are expected to be practical questions.
    – Ramhound
    Feb 7, 2021 at 16:06
  • 22
    I'm fascinated. Why do you need 10M folders of 13 files each? Feb 7, 2021 at 22:18
  • 39
    I find that somewhere in the 100,000 file or folder range, directory access starts to slow down quite a bit. Make a hierarchy - 100 folders of 100 folders of 1,000 folders of 13 files and you'll have no problem at all. Or something similar. If the names are pseudo-random hexadecimal then just use the first 3 characters as one directory, the next three as another, the next 3 as another. Or something like that. But 10,000,000 (or even 1,000,000) in one directory? That's asking for some serious access time problems. Feb 8, 2021 at 0:20
  • 34
    Consider a database. This is not the correct solution.
    – J...
    Feb 8, 2021 at 15:55
  • 26
    The proliferation of backseat drivers amuses me. Let's assume the OP knows what they are doing, and just answer their question. I have worked on several systems that were designed to work for 10 years (not Windows systems). 30 years later, they are still running, and they have literally trillions of files on them all stored in a flat system (no hierarchy at all). No one ever imagined those systems would still be operating after 30 years, or designed them for such use. Sometimes you just have to work with what you have, and don't have the luxury of redesigning everything from scratch. Feb 9, 2021 at 12:24

3 Answers 3

46

As far the theoretical capacities of NTFS are concerned, there is no problem.

The Microsoft article on Maximum Sizes on an NTFS Volume specifies that the maximum of files per volume is 4,294,967,295, and that should also be the maximum on folders. However, you would need an extremely fast computer with lots of RAM to be able to even view that folder in Explorer.

From my own experience, on a good computer of several years ago, viewing a folder with thousands of sub-folders took some dozen of seconds just to show the folder. I have no idea what would happen with 10 million sub-folders, but surely you would need a lot of patience even if the computer could handle it. Eventually.

I really suggest to rethink again your folder architecture.

11
  • 3
    In particular, even though NTFS stores data in a B+-tree (or B*-tree?), Windows only provides sequential mechanisms to list directories. So you probably want to assume linear searches will occur...
    – user541686
    Feb 8, 2021 at 9:08
  • 4
    In practice insane amount of folders are usually in temp, cache, upload and other programmatically managed folders which are not intended to be viewed using Explorer. And usually the folder contents is never listed, the files are accessed using hashes or paths stored in a database.
    – Džuris
    Feb 8, 2021 at 10:18
  • 2
    Seconding the point about usability issues. This is a case where the OP probably wants a database, but if not they almost certainly need to do directory hashing of some sort to generate a more deeply nested tree structure with a much lower fan-out at each level to make this even remotely usable. Feb 8, 2021 at 15:16
  • 8
    If anyone's curious where 4,294,967,295 comes from, it's 2^32-1. Feb 9, 2021 at 0:48
  • 10
    The OP never mentioned that he wanted to view the files on explorer. Feb 9, 2021 at 5:56
47

This may be an X/Y problem. Perhaps what you are doing is better suited for a database rather than a filesystem. With a database, you can easily store and access many millions of records quickly and efficiently. The accepted answer is correct in saying NTFS is theoretically able to store this many records, but it won't be very fast. This is true for essentially all filesystems (e.g. NTFS, exFAT, ext4, HFS...). They simply aren't designed to be sufficiently scalable for what you're trying to do.

One of the main reasons for this is that most operating systems' filesystem API can only return the entire list of directory entries at once. There is no way to retrieve only directories that match a certain pattern in typical filesystems, for example. It would have to retrieve them all and then parse the (massive) output for the names you want. The same is true with other file/directory attributes in addition to name like size, creation and modification time, etc. This isn't the case with databases.

17
  • 3
    Small note - ext4 evolved from ext3 which evolved from ext2 which evolved from BFS/UFS which predates DOS. HFS similarly have evolution unrelated to DOS FAT. Feb 8, 2021 at 16:34
  • 6
    Even NTFS more closely derived from OS/2's HPFS than DOS FAT (though it was designed from scratch in any case, refer Helen Custer's Inside the Windows NT File System if you can find it anywhere. I think exFAT might be the only system in that list that actually has origins in DOS.
    – throx
    Feb 8, 2021 at 22:46
  • 1
    @throx I meant the general filesystem concept that is often followed today, not that they're all derived from the same implementation. I've removed that sentence since it was poorly worded.
    – forest
    Feb 8, 2021 at 22:50
  • 1
    @forest I got that, but saying that the ideas originated in DOS is wildly off base. DOS added directory structures in 1982, whereas MULTICS invented them a mere 13 years prior in 1969, which UNIX v1 through v7 and its various branches copied widely before DOS came anywhere near. All of the file systems other than exFAT were derived from MULTICS through UNIX and VMS well prior to DOS.
    – throx
    Feb 9, 2021 at 2:52
  • 5
    "filesystem API can only return the entire list of directory entries at once". Is that really true? The Win32 API allows you to only returns files of a specific pattern and also allow you to query one file after another (FindFirstFile/FindNextFile and so on). Which would seem to support the intended scenario just fine. I'd surprised if *Nix didn't have similar APIs. It's probably true that many applications would handle the situation badly though, but the APIs themselves seem to support the scenario.
    – Voo
    Feb 9, 2021 at 15:09
26

The number of files inside a folder has nothing to do with the OS. It's a feature of the file system although the system you use may in turn has lower limitations. Some file systems limit the number of files in a folder but some others just limit the total number of files in a volume, and some don't have any limits at all. See file systems' limits. Note that basically a directory is just a file whose content is a list of other files

If you use exFAT, the maximum number is 2 796 202 files per folder. In NTFS the limit is 232-1 files per volume. And if you use FAT then the limit depends on the FAT version

  • FAT12: 4 068 for 8 KiB clusters
  • FAT16: 65 460 for 32 KiB clusters
  • FAT32: 268 173 300 for 32 KiB clusters

Windows also natively support a few other file systems like ReFS, or you can install drivers for other non-native file systems. They may in turn have different limits

But in any case having a huge number of files in a folder is a very bad idea. The listing and operating speed depends on how the file system stores its metadata, for example in FAT it's a linear list so it's very slow. But even with an efficient way to list files like a B+tree in NTFS it's still slow. In general I avoid having more than 2000 files in a folder

The better solution in your case should be some kind of database. However if you really have to store the files directly in a drive then you need to distribute the files evenly to multiple smaller folders. The common way is to hash the file name or content and split into folders having part of that name. For example if the hash is 0xabcdef12 (32 bits) then store the file in ab/cd/ef/12, ab/cde/f12 or 2af/0de/f12 (each path component represents 8/8/8/8, 8/12/12 and 10/10/12 bits of the original value respectively). This way no folder should have too many or too few files. See

This method is commonly used in git or docker

See also

8
  • Probably (I didn't DV, but the thought crossed my mind) because you have too much on the technical limits before getting to the "But in any case..." warning. It would be better (IMHO) to start with the "in practice, it's a bad idea" paragraph (and the possible remedies) and only then delve into the technical details). On the plus side, your "The better way" section is very helpful.
    – TripeHound
    Feb 8, 2021 at 6:48
  • 12
    "The number of files inside a folder has nothing to do with the OS. It's a feature of the file system." – If the file system supports 100000 files but the OS requires each file to have a unique number and requires the type of that number to be a 16 bit integer, then there will be a OS-imposed limitations. Correct would be to say that the limits are the minimum of the filesystem limits, OS limits, and the limits of the tools you use to access the files. Especially on Windows, there are tons of weird limits that do not exist in NTFS, but exist in different APIs and in the Explorer. Feb 8, 2021 at 8:34
  • 2
    In particular, there are several different APIs supported by Windows for backwards compatibility, and they may have different limits, and the tools the OP uses may use one or the other of those APIs. Feb 8, 2021 at 8:35
  • 2
    @NotThatGuy no one traverses through millions of files and folders manually. And even if you distribute the file to some other rules then some folders may have millions of files and others have only a few files which not an improvement at all. And as I said, a proper database is highly recommended
    – phuclv
    Feb 8, 2021 at 12:54
  • 2
    @NotThatGuy "traversing" in the comment means going into subfolders manually. Only in that case you care about the meanings of the folder names. Automation code never cares about that so a hash is sufficient and better. Besides having a too big folder is also slower to look for a specific file by the automation code
    – phuclv
    Feb 8, 2021 at 14:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .