I'm about to write a program that will do a LOT of writes to create a bunch of big files, so I was wondering which filesystem I should go with, and why the workings of such filesystems make it so?
-
2This would probably more suited to serverfault.com– CaffGeekOct 27, 2011 at 18:39
-
4Premature optimization much? How many files or directories are we talking here? What makes you think NTFS/FAT/EXT, the default systems on whatever platform you're running would not be sufficient?– FoscoOct 27, 2011 at 19:12
-
7How large is a LARGE file (is it larger than a Large file)? How many is a LOT? Do you have a particular operating system in mind? What kind of platform are you going to use this on (big iron, PC, smart dust, …, and running what operating system)? What operations are you going to use (write all at once vs. keep appending vs. more complex patterns)? Do you need features such as permissions, quotas or integrity protection? What are your reliability requirements? Answer these questions, and we might come up with recommendations that are actually helpful.– Gilles 'SO- stop being evil'Oct 27, 2011 at 19:54
-
2+1 to the comment asking for more detail. We need a rough estimate on what "LOT" and "big" are. For coding a solution, I'm betting any modern filesystem (NTFS, ext4) will do. Are you running into specific problems?– kbyrdOct 27, 2011 at 19:56
-
@kbyrd, I would never consider NTFS for large directories and deep hierarchies requiring fast lookup. NTFS is dead slow in these cases.– SK-logicOct 28, 2011 at 9:15
1 Answer
If you want a file system that can address unbelievably large address spaces, use ZFS.
Its size limitations are larger than the # of atoms on earth, thus it is physically impossible to run out of addressable memory when using ZFS.
Keep in mind that using ZFS is more theoretical than practical. It is very difficult to obtain enough memory to exceed the capacity of most modern file systems, so I don't necessarily recommend ZFS. Use what makes sense based on other factors.
-
Could you expand on that a little? What features allow zfs to have these unbelievably large address spaces? Oct 27, 2011 at 19:32
-
1@KaitlynMcmordie, from the Wikipedia article:
ZFS is a 128-bit file system, so it can address 1.84 × 1019 times more data than 64-bit systems such as NTFS.
– riwalkOct 27, 2011 at 19:56 -
2It uses 128-bit pointers for data blocks, and it merges the block, filesystem and logical volume layers to permit filesystems to span multiple physical devices. Basically, you create "pools" of space on one or more drives, and ZFS takes care of spreading the data across them. You can add devices to a pool dynamically, so you (theoretically) won't ever run out of space.– TMNOct 27, 2011 at 19:58