up vote 0 down vote favorite
share [g+] share [fb]

I'm about to write a program that will do a LOT of writes to create a bunch of big files, so I was wondering which filesystem I should go with, and why the workings of such filesystems make it so?

link|improve this question
100% accept rate
1  
This would probably more suited to serverfault.com – Chad Oct 27 '11 at 18:39
3  
Premature optimization much? How many files or directories are we talking here? What makes you think NTFS/FAT/EXT, the default systems on whatever platform you're running would not be sufficient? – Fosco Oct 27 '11 at 19:12
6  
How large is a LARGE file (is it larger than a Large file)? How many is a LOT? Do you have a particular operating system in mind? What kind of platform are you going to use this on (big iron, PC, smart dust, …, and running what operating system)? What operations are you going to use (write all at once vs. keep appending vs. more complex patterns)? Do you need features such as permissions, quotas or integrity protection? What are your reliability requirements? Answer these questions, and we might come up with recommendations that are actually helpful. – Gilles Oct 27 '11 at 19:54
2  
+1 to the comment asking for more detail. We need a rough estimate on what "LOT" and "big" are. For coding a solution, I'm betting any modern filesystem (NTFS, ext4) will do. Are you running into specific problems? – kbyrd Oct 27 '11 at 19:56
@kbyrd, I would never consider NTFS for large directories and deep hierarchies requiring fast lookup. NTFS is dead slow in these cases. – SK-logic Oct 28 '11 at 9:15
feedback

migrated from programmers.stackexchange.com Oct 28 '11 at 20:49

This question came from our site for professional programmers interested in conceptual questions about software development.

1 Answer

up vote 4 down vote accepted

If you want a file system that can address unbelievably large address spaces, use ZFS.

Its size limitations are larger than the # of atoms on earth, thus it is physically impossible to run out of addressable memory when using ZFS.

Keep in mind that using ZFS is more theoretical than practical. It is very difficult to obtain enough memory to exceed the capacity of most modern file systems, so I don't necessarily recommend ZFS. Use what makes sense based on other factors.

link|improve this answer
Could you expand on that a little? What features allow zfs to have these unbelievably large address spaces? – Kaitlyn Mcmordie Oct 27 '11 at 19:32
1  
@KaitlynMcmordie, from the Wikipedia article: ZFS is a 128-bit file system, so it can address 1.84 × 1019 times more data than 64-bit systems such as NTFS. – Stargazer712 Oct 27 '11 at 19:56
2  
It uses 128-bit pointers for data blocks, and it merges the block, filesystem and logical volume layers to permit filesystems to span multiple physical devices. Basically, you create "pools" of space on one or more drives, and ZFS takes care of spreading the data across them. You can add devices to a pool dynamically, so you (theoretically) won't ever run out of space. – TMN Oct 27 '11 at 19:58
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.