Can storing 300k files in one folder cause problems?

Question

I'm crawling a large website (over 200k pages) using wget (is there a better tool btw?). Wget is saving all the files to one directory.

The partition is HFS (I think), will it cause problems if I have all the files in one dir? Assuming I will access all of them only from the console (I know Finder has problems with dirs>5k files).

Or is there perhaps a way to create a micro-partition that would be compressed and would allow for a fast, optimized access to this amount of files?

Answer 1

Despite the feasibility of the underlying file-system, you REALLY should NEVER store that many files in one directory. When it comes time to browse the contents of that directory, you'll quickly discover that there is a HUGE amount of lag while the OS tries to build the file listing and such. It really puts a significant amount of strain on the system.

Most tools out there that do any sort of "web archiving" will usually build a directory structure similar to the website's layout. Nearly all websites do not base all their contents off of the root directory... i.e. mydomain.com/document-1 ... they will have some logistics behind it all that split it up into several paths (for a variety of reasons) i.e. images go in mydomain.com/images and stuff about goldfish are in mydomain.com/goldfish/ etc...

There are several tools out there that can & will build this sort of directory structure for you. even wget has options to download an entire site. Personally, I've used "httrack" in the past, and it worked quite well. There are also command-line options for wget to download an entire site as well. Look at the -r (recursive) option. Just make sure you setup your domain list so you don't download links infinitely across multiple sites. Best do some reading-up on the wget man page.

Answer 2

Wikipedia states that HFS has a file limit of 65535. So if your partition is indeed HFS, you'll hit that.

---

From Wikipedia:

Additionally, the limit of 65,535 allocation blocks resulted in files having a "minimum" size equivalent 1/65,535th the size of the disk. Thus, any given volume, no matter its size, could only store a maximum of 65,535 files. Moreover, any file would be allocated more space than it actually needed, up to the allocation block size. When disks were small, this was of little consequence, because the individual allocation block size was trivial, but as disks started to approach the 1 GB mark, the smallest amount of space that any file could occupy (a single allocation block) became excessively large, wasting significant amounts of disk space. For example, on a 1 GB disk, the allocation block size under HFS is 16 KB, so even a 1 byte file would take up 16 KB of disk space. This situation was less of a problem for users having large files (such as pictures, databases or audio) because these larger files wasted less space as a percentage of their file size. Users with many small files, on the other hand, could lose a copious amount of space due to large allocation block size. This made partitioning disks into smaller logical volumes very appealing for Mac users, because small documents stored on a smaller volume would take up much less space than if they resided on a large partition. The same problem existed in the FAT16 file system.