9

I have 100,000+ files in a directory in my MacOS X and looks it is slow for my script to read a file in them.

Is there any limitation or recommendation to have that many files? Should I split them to some directories?

The limitation I found was that I can't mv * foo for all 100,000 files. It shows an error, saying "too long argument." It works with approximately less than 20,000 files.

  • Currently I have 380,000 files in a directory and realize that even opening a file simply takes 10+ seconds. I have decided to separate them to some directories. – Daisuki Honey Nov 28 '14 at 7:20
  • 1
    The HFS+ file system should be able to store and access large numbers of files in a directory by their full name without too much trouble. But you need to watch out with wildcards. When you use * or ? as part of an argument to a command, the operating system searches the whole directory for matching files (slow), and then it replaces your argument with a list of every matching file (long), which it then passes to the command. You might do better with a loop or with several mv commands, e.g., mv a* foo && mv b* foo. – Matthias Fripp May 4 '16 at 23:08
1

According to this Stack Overflow answer and specific details on Apple’s site, an individual folder can contain up to 2.1 billion items.

That said, just because it can hold up to 2.1 billion items doesn’t mean it can maintain performance at that level. According to Wikipedia; emphasis is mine:

The Catalog File, which stores all the file and directory records in a single data structure, results in performance problems when the system allows multitasking, as only one program can write to this structure at a time, meaning that many programs may be waiting in queue due to one program "hogging" the system. It is also a serious reliability concern, as damage to this file can destroy the entire file system.

So performance is naturally degraded thanks to the fact the catalog file can only be used by one program at a time. And if the directory grows in size, the risk/degradation caused by that issue will only escalate; more files means more of a chance for programs to access files in that one directory. Further confirmation of that idea here; again emphasis is mine:

The catalog file is a complicated structure. Because it keeps all file and directory information, it forces serialization of the file system—not an ideal situation when there are a large number of threads wanting to perform file I/O. In HFS, any operation that creates a file or modifies a file in any way has to lock the catalog file, which prevents other threads from even read-only access to the catalog file. Access to the catalog file must be single- writer/multireader.

  • Thanks so much. I understand the access to the catalog file will be the bottleneck and it can cause serious performance problem especially for multitasking. – Daisuki Honey Nov 26 '14 at 12:43
  • @DaisukiHoney You’re welcome! So if you found my answer helpful, please remember to up vote it. And if it was the answer that solved your issue please remember to check it off as such. – JakeGould Nov 27 '14 at 4:12
  • Yes, definitely I am voting your answer and check it off. Again, thanks so much. – Daisuki Honey Nov 27 '14 at 8:44
  • The Wikipedia sections you cite are talking about scalability limits per filesystem, not per directory: there is only one Catalog File per filesystem and all access must serialize on that. It's fairly irrelevant to the question. – poolie Dec 18 '16 at 5:00
  • @poolie The question is about per directory that exists on a file system. The catalog file exists per file system but the directory itself exists on the same file system as well. It is relevant to a question dealing with 10,000+ files in a directory that exists on a single file system. But this question is 2+ years old, so thank you for the Wiki link. I have updated my answer to include the new wording as well as a direct link to the section in question. – JakeGould Dec 18 '16 at 19:00
4

Short Answer: Well, if you're reading 100,000 files, I might expect the script to be slow.

Long Answer: To answer this question more thoroughly, you have to look at the file system on a Mac. Macs use the HFS+ (Hierarchical File System Plus), which is a modern file system that has limitations, but only in extreme situations.

From my experience, it’s a lot like a Linux EXT journaling file system. It supports mounting directories, UNIX-like permissions, etc. It addressed files in a 32-bit format, making the maximum number of files that can be stored in a volume 4,294,967,295, according to this source.

The file system starts to break with files bigger than 8 EB on modern systems and up to 2.1 billion files and folders in one location as outlined here.

Given the way the HFS+—or really any file system is setup for that matter—having a lot of files in a folder should not do anything 'weird'.

Honestly, I don’t think there would be a performance improvement distributing the files across a more complex folder hierarchy. Actually, this technique might be less efficient because your script would have to make calls to change directories mid-process.

  • Right. I thought about changing directory hierarchy but it causes more complicated algorithm and I suspect that much performance improvement. Thanks for the answer. I currently have 200,000 files in the directory and might have 1,000,000 at the end. I hope it works fine without that bad performance. – Daisuki Honey Nov 26 '14 at 12:48
  • @DaisukiHoney If you are working with that many files, it might be worth it to see if you can subdivide things into directories. Might be difficult to do at this stage, but might make things a bit more stable moving forward. – JakeGould Nov 27 '14 at 4:12
  • @JakeGould Thanks for the advice. I have been thinking about restructuring because I might add some more files. Thanks. – Daisuki Honey Nov 27 '14 at 8:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.