0

I'm about to write a program that will do a LOT of writes to create a bunch of big files, so I was wondering which filesystem I should go with, and why the workings of such filesystems make it so?

5
  • 2
    This would probably more suited to serverfault.com
    – CaffGeek
    Oct 27, 2011 at 18:39
  • 4
    Premature optimization much? How many files or directories are we talking here? What makes you think NTFS/FAT/EXT, the default systems on whatever platform you're running would not be sufficient?
    – Fosco
    Oct 27, 2011 at 19:12
  • 7
    How large is a LARGE file (is it larger than a Large file)? How many is a LOT? Do you have a particular operating system in mind? What kind of platform are you going to use this on (big iron, PC, smart dust, …, and running what operating system)? What operations are you going to use (write all at once vs. keep appending vs. more complex patterns)? Do you need features such as permissions, quotas or integrity protection? What are your reliability requirements? Answer these questions, and we might come up with recommendations that are actually helpful. Oct 27, 2011 at 19:54
  • 2
    +1 to the comment asking for more detail. We need a rough estimate on what "LOT" and "big" are. For coding a solution, I'm betting any modern filesystem (NTFS, ext4) will do. Are you running into specific problems?
    – kbyrd
    Oct 27, 2011 at 19:56
  • @kbyrd, I would never consider NTFS for large directories and deep hierarchies requiring fast lookup. NTFS is dead slow in these cases.
    – SK-logic
    Oct 28, 2011 at 9:15

1 Answer 1

5

If you want a file system that can address unbelievably large address spaces, use ZFS.

Its size limitations are larger than the # of atoms on earth, thus it is physically impossible to run out of addressable memory when using ZFS.

Keep in mind that using ZFS is more theoretical than practical. It is very difficult to obtain enough memory to exceed the capacity of most modern file systems, so I don't necessarily recommend ZFS. Use what makes sense based on other factors.

3
  • Could you expand on that a little? What features allow zfs to have these unbelievably large address spaces? Oct 27, 2011 at 19:32
  • 1
    @KaitlynMcmordie, from the Wikipedia article: ZFS is a 128-bit file system, so it can address 1.84 × 1019 times more data than 64-bit systems such as NTFS.
    – riwalk
    Oct 27, 2011 at 19:56
  • 2
    It uses 128-bit pointers for data blocks, and it merges the block, filesystem and logical volume layers to permit filesystems to span multiple physical devices. Basically, you create "pools" of space on one or more drives, and ZFS takes care of spreading the data across them. You can add devices to a pool dynamically, so you (theoretically) won't ever run out of space.
    – TMN
    Oct 27, 2011 at 19:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .