I would like to know, from more experienced people, what would be the best choice of file system to use for a file server having more than 20TB of hard disks. Personally I always used EXT3 (back in the days) and EXT4 (since available) [and once ReiserFS 3 though it caused many data corruption] on my personal computers and on the "little servers" BOOT and ROOT disks.

However as EXT4 tools (though not EXT4 itself) is limited to 16TB partitions this may not be my best bet. Distribution will be Debian 6.0 (Squeeze) and / or Gentoo (latest version), so kernel should be pretty recent (on Debian at least with backports), meaning linux kernel >= 2.6.32.

File server will be used for maily three purposes (and separated partitions as well, because the purpose is to keep data "safe" and don't really care much of overhead). All disks are though to be encrypted using LUKS:

  1. Media, downloads and local debian repository [I have at least 6 machines running Debian] >20TB (maybe further separation between Media, Downloads and Debian repository)
  2. Data (Documents, Photos, ...) ~ 4TB SAFE (meaning raid1 or raid6 + backup disk)
  3. Backups >= 20 TB for backups of other computers in my gigabit lan (can you suggest a software that backups the entire OS even if it's windows, BackupPC says it does that, any alternatives ?)

Fast speeds are not really necessary (concurrent accesses: maximum to 2 or 3 large files, say videos), even if it's "just" 200MB/s reads from a 10 HDD Raid6 I can live with that.

In summary I look for a reliable, scalable (i.e. easily expandable) filesystem that supports more than 20TB / partition. The safer and reliable the FS is, the better. Hardware employed will be at least a quad core (amd x4 630 or intel i5-2500k) and plenty of RAM (>8GB, maybe >16GB) so hardware requirements should be met.

My PCs / Server will be connected to an UPS (Uninterrupted Power Supply) in case of power outage Might do media and backups on separate machines as well (i.e. two servers).

  • 7
    At this scale, you really need to be seriously evaluating ZFS. Rebuild times and error rates become serious concerns with as many discs as you're talking about and zfs is the only stable fs available now with robust error checking and correction all the way through the stack. – afrazier Dec 25 '11 at 14:00
  • 1
    ZFS is not linux natively supported (only with FUSE) or is natively supported in an early pre-alpha state. I don't consider using solaris an option. Never tried FreeBSD once and may be interested, however I don't now if its software raid support (and general hardware support) is as good as linux's – user51166 Dec 25 '11 at 16:12
  • 1
    I know, but ZFS is running natively on other platforms. While hardware support isn't the same as Linux, that should be the least of your worries. ZFS is a full storage stack, so software raid is out of the equation. Evaluate how you're going to store, manage, protect, and backup your data first before choosing your OS or storage system. Don't discount ZFS just because it's not native on Linux, it's probably the most advanced storage solution available for free right now. – afrazier Dec 25 '11 at 18:36
  • Thank you for your replies, but I do not understand: even if I could use FreeBSD without problems (not so sure about it), is there anything like software raid implemented ? Anything like LUKS (Linux Encryption) for FreeBSD ? Thanks. I'm familiar with Gentoo and Debian GNU/Linux mostly. The server is a home server. – user51166 Dec 25 '11 at 19:10
  • Or are you suggesting another operating system than FreeBSD ? – user51166 Dec 25 '11 at 19:20

Alot of people are suggesting ZFS. But ZFS is not available natively under Linux except through fuse. I wouldn't recommend this for your situation where performance is likely to be important.

Unfortunately, ZFS will never be available as a native kernel module unless licencing issues are sorted out somehow.

XFS is good, but some people have reported corruption issues and I can't really comment on that. I've played with small XFS partitions and not had these problems but not in production.

ZFS has too many advantages & useful features that cannot be ignored though. In summary they are (see ZFS Wiki for a full description of what they mean):

  • Data integrity
  • Storage pools
  • L2ARC
  • High capacity
  • Copy on write
  • Snapshots & clones
  • Dynamic striping
  • Variable block sizes
  • Lightweight filesystem creation
  • Cache management
  • Adaptive endianness
  • Deduplication
  • Encrypion

So how do we get around it? My suggested alternative which may suit your situation is to consider nexenta. This is an Open Solaris kernel with GNU userland tools running on top. Having an Open Solaris kernel means having ZFS available natively.

  • From their site "Community Edition: Unlimited, FREE version for up to 18TB of storage". Seems like I'd get another limitation like EXT4's – user51166 Dec 28 '11 at 7:21
  • And I understood that ZFS is simply the "best" as almost all of you are saying. Just trying to figure out the "best" operating system / solaris distribution able to run it. – user51166 Dec 28 '11 at 7:26
  • Debian GNU/kFreeBSD seems to support ZFS and I like the Debian way, not sure I can use this however since there seems to be a small supporting community and still has some major bugs. – user51166 Dec 28 '11 at 7:31
  • @user51166 - You should also consider FreeBSD or FreeNAS if your server is purely for storage. Both have ZFS support. – Matt H Jan 17 '12 at 8:11

You should give a try to XFS, fit well in your requirements :

XFS is a 64-bit file system. It supports a maximum file system size of 8 exbibytes minus one byte, though this is subject to block limits imposed by the host operating system. On 32-bit Linux systems, this limits the file and file system sizes to 16 tebibytes.

  • I heard it could cause data losses when facing power outages and its journaling doesn't cover data (only journal on metadata). Had used it once on my desktop but generated a lot of fsck errors, therefore I prefererred not to use it anymore. I repeat: performance is not the (main) scope of this choice: stability is. – user51166 Dec 25 '11 at 20:43
  • I don't think that XFS is unstable, i use it in several file servers and i I didn't have any problems ... – aleroot Dec 25 '11 at 20:46
  • No, it's stable in the sense of having a stable version. Does that imply that I won't have data losses through the years ? XFS is surely good if you're looking for performance, though I remember reading on the net there were data loss issues (though not as many as with reiser 3 fortunately). Forgot to specify, but I'm looking at a LUKS setup, therefore LVM will be used, if that can help. – user51166 Dec 25 '11 at 20:48
  • Doesn't not exist a perfect FileSystem... I think that for your requirements XFS is the best fit . There should not be any problem using LVM on XFS. – aleroot Dec 25 '11 at 20:59
  • Surely not. Any special "restrictions" / "characteristics" ? Online fsck and / or defragmentation issues like ext4's issues ? – user51166 Dec 25 '11 at 21:01

Your easiest option is to use XFS. A lot of the bad experiences around XFS are based on old versions and desktop hardware problems that I don't think are really relevant for new deployments onto standard quality server hardware. I wrote a blog post about this subject that may help you sort out the current situation. There are multiple busy XFS database installations with hundreds of users and terabytes of data I help manage. They're all on the Debian Lenny kernel (2.6.26) or later and I haven't heard a hint of trouble with them in years. I wouldn't use XFS with any earlier kernel than that. I have heard some direct reports of people seeing strange XFS behavior still when the system runs out of memory or disk space; I haven't seen that myself yet though.

The only other reasonable option is to use ext4 with some hacking to support larger filesystems. I wouldn't expect that to have a very different reliability level. I've had to recover data from multiple broken ext4 systems that ran into kernel bugs, so far ones all fixed upstream but not in the distributor's kernel at that time. ext4 has its own set of metadata issues like Delayed allocation data loss, things that were less likely to happen on ext3. I would estimate the odds of you hitting an ext4 bug would be even higher than normal if you're forcing it over the normal size limit, simply because it seems more likely you'll be hitting a less well tested new code path at some point.

Alternative idea is to just use safer and boring ext3, accept the 16TB limit, and partition things better so no single filesystem has to be that large.

One loose end related to journal issues. You didn't talk about how all these drives are going to be connected. Make sure you understand the implication of any write caching that's in your storage chain here. Either disable it or make sure the filesystem is flushing the cache out. I've stashed some resources about that at Reliable Writes if that's not something you're checking yet.

Drives suck. RAID arrays suck. Filesystems suck. Multiple failures happen. I'm glad to see you're already thinking about backups; going from good to great reliability on storage requires more than just RAID and some spare drives. Redundancy costs something at every level, and the money for hardware vs. software complexity one is tricky to navigate. And watch your performance expectations. While a RAID array like you're considering will easily do hundreds of MB/s, all it takes is two concurrent readers seeking the disk around constantly to drop that to only a few MB/s instead. I can easily crush a 24 disk RAID10 array such that it only delivers <5MB/s against a benchmark workload. One thing that helps there is to make sure you tweak readahead upward if multiple streaming readers are possible.

  • I'm going to use this at home, therefore I plan on using mainstream hardware. Might as well use server Hardware but I still have to see yet what SB-E is going to offer next year in the xeon department (I would've also liked to play a bit with virtualisation). If it's not too much expensive I plan going with cheap server hardware and ECC memory (lots of it). Performance-wise I don't want anything exceptional. – user51166 Dec 27 '11 at 22:02
  • And yes, I'm thinking about backups, but still not have implemented them yet. I'm still looking for a backup solution capable of creating a system image and / or easily tar/zip/... required folders with a managament interface which allow automatic restores. BackupPC seems just a bittle limited and not sure I'd trust Crashplan (use it to backup data to remote location, would like redundancy therefore another system). Do I have to write a web GUI myself or something like this already exist (open source software, or at least free of charge to use) – user51166 Dec 27 '11 at 22:05

Deploying on ZFS using FreeBSD could happen here using gbde for encryption. ZFS itself would be the software RAID provider, via RAIDZ. The storage management complexity of building zpools is not significantly different than what Linux is going to put you through with mdadm, and in some cases will actually be easier. My first ZFS install (on Solaris 10 about 3 years ago) had a 17TB filesystem over 48 drives. I survived multiple failures there without a problem, learning ZFS management as I went.

The main upside is that ZFS's checksumming provides better error detection than Linux can, which is a defense against bad hardware worth considering. Main downsides are around FreeBSD being less popular. You don't know how to administer it yet, hardware support is a bit weaker than Linux, and since it's a less popular platform there aren't as many people to ask for help if you do run into issues.

A many terabyte storage array does really highlight what ZFS is good at though. It's worth considering seriously if you are willing to take a plunge into something new. If you want to explore true backup paranoia, build Linux and FreeBSD backup servers, to reduce the odds of OS bugs as a single point of failure cause.

  • Already have Linux as file server and I still have to really begin use it (I only began some tests of software RAID-6 + LUKS on 6 disks). Only problem is that I have very little time to play with it. I like your answer. So what do you suggest as operating system if going the ZFS way ? FreeBSD and opensolaris (not maintained anymore), OpenIndiana (OpenSolaris opensource as I've seen it) and Solaris Express 11 (Oracle :S) seem to be the only choices. I don't use this at work, its my hobby at home, but I would like something stable anyway and which is easy to use as well. – user51166 Dec 27 '11 at 21:54
  • I've really grown used to use aptitude and emerge. My understanding in how you compile / manage / update ports in freebsd (cd /usr/... && make install) is that it just doesn't seem "right" (I hope the purist will forgive me for the use of this term, but it seems strange to me that if you want to update a package you have to do that, no automatic dependency resolving [I just took a small look at the FreeBSD's manual]). Or is there a simple package management system like debian's or gentoo's ? – user51166 Dec 27 '11 at 21:55
  • I already know I could simply rsync what I want or tar/diff etc but I would like to know if anything more practical already exist. Thanks – user51166 Dec 27 '11 at 22:08

According to wikipedia's file system comparison page, there are many notable file systems that suit your needs such as JFS, XFS, UDF, ZFS, GPFS and Btrfs. Just click the maximum file size to sort and select the most appropropriate one

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.