What easy to manage file system to choose on Linux for homeserver?

Question

I have a homeserver running on a HP Proliant Microserver Gen7 with 10GB of ECC Memory running Debian 8 (jessie). Right now i have one small system disk (backed up using rsnapshot) and 2x 3TB storage disks setup as a Raid 1 using mdadm, on top i have LVM groups for different purposes, one of those - is additionally dm-crypt / luks encrypted to hold private data.

I am planning to buy two additional, same size HD for the storage pool and probably setting up the server from scratch, definitely using debian again but probably under xenserver virtualization this time. So the storage pool would be 4x 3TB, with Raid1 leaving a usable 6TB, with Raid5 (or raidz1) 9TB.

While the combination of mdadm / lvm / dm-crypt works nicely, i find it incredibly complex, escpecially to plan an easy disaster recovery strategy. I need to research for each of the layers what the best metadata backup and restore process is etc.

In a perfect world i would make the whole storage an encrypted raidz1 pool, BUT encryption didnt make it into the Zfs on Linux yet and from my research it is totally unclear WHEN it can be expected to be implemented

Btrfs has a bit of the same situation: at least it seems production stable - but alas - encryption will also only arrive in an unforseeable future.

So from the filesystems i'd prefer to use, both don't natively support encryption on linux (yet). Well, there is a bunch of howtos and tutorials on how to use LUKS encrypted LVM's in conjunction with either ZFS or BtrFS. There is the approach to use LVM on top of ZFS or ZFS on top of LVM - to me that sounds like a horrible mess.

I don't like the concept of EncFS at all, so that is no option either.

Hopefully there are some options i havent heard about, thus my question here: What else is out there to achieve this an "easy (as in extendable etc. like zfs, error correcting) to manage file system to choose on Linux for homeserver, which can be completely or in parts encrypted?

UPDATE Dec. 2017: ZFS on Linux featuring Encryption coming soon: https://blog.heckel.xyz/2017/01/08/zfs-encryption-openzfs-zfs-on-linux/

Answer 1

mdadm / lvm / dm-crypt is probably your best bet - and its not that complex at all - you just handle each layer appropriately - or, depending on your needs, mdadm /dm-crypt/lvm (if you want all the LV's to share a single device with 1 passphrase)

You are right not to use encfs - its insecure.

Answer 2

if you choose the ZFS setup,i would suggest:

• do not use LVM,create ZVOLs on top of pool

• for OS - GPT partition for unencrypted ZFS pool with mirror(or raidz)

• for private data - use LUKS on GPT partition , then on top build a ZFS pool mirror(or raidz)

after boot you login via ssh ,enter password for encrypted partitons,and import the pool which is on top of LUKS,then start virtual machines (best write a shell script)

so the suggested schema is GPT / LUKS / ZFS / data or ZVOL for VM

do not forget to make a BIOS partition (size 1M) if you do not use UEFI boot also do not forget to make a partition for GRUB ,like 150M

for heavy random io i suggest disabling ZFS prefetch (echo 1 > /sys/module/zfs/parameters/zfs_prefetch_disable)

Answer 3

This is an advise for all that wants some data encrypted, allways encrypt 100% of all possible storage that is connected to the computer, no matter if it is the Grub partition, system partition, home partition, data partition, swap, etc. ALLWAYS ENCRYPT ALL.

Why? Because you do not control where data is saved by the applications / system.

Disarter sample events secuence:

1. Boot normal
2. Use an app
3. Open a 'private' document
4. Sysmtem crash (software or hardware)
5. Next boot does not work
6. One device end its life (do not work again)

You can not delete what is on such device (it is not working), you can not read it, so, what was saved there? maybe something 'private'? You can not even know what neither where.

But someone else (maybe with some millions of money on the pocket) ca red it, so you can not put in trash that device, not until you confirm 'private' data is not there... but you can not know unless you give the device to a company, but then again 'private' is in risk, maybe they 'copy' it (bad person working there, etc).

Ask you this: Do you know each source line of all code you use (boot code, system code, applications code, etc), source code that you compile by your self... and never use any kind of 'binary'... so you must had compiled with a pencil and a paper the compiler.

Else you use code that you do not know what it does.

Side note: If you need to use 'private' data, never ever again connect that computer to Internet... maybe some part of the app you had used, did save some data and in a long future send it to internet servers.

Extreme sample: you use a write app to read a TXT fole, it has a hidden code to save it hidden somewhere... then after some years of updates that app (or another one) has hidden code to read that hidden saved data and sent it to an internet server or 3rd party computer, etc.

Cases are extreme? I learn them the hard way... on my Linux enter a troyan (inside an app i want to use) apparently does not do nothing (has no sockect connection, etc), but it was loging some kind of 'writes' to the disk and making a double write... after near two weeks another toyan took that info and tried to send it to a IP.. goodness i had another computer in the 'middle' of my internet connection (yes, of course i am and i was a total paranoid, that saved me) to 'discover' that connections and 'block' them... not to mention i only use 'Linux' since with Windows that can not be under control. Such troyans were there without knon of te app author, they where injected when app was stored on the repository. Hope understand i preffer not to say which app was since now has been 'fixed'... who knows since when!

Do you think i am only thinking on troyan, hidden things, etc., not at all, remember swap, temp files, etc... some apps saves the document you are using / creating to help you not to loose what you had done, where on the hell tey save that data? Not necesary on you encrypted partition, maybe on SWAP, temp folder or in general where the author of the app wanted,,so you can not allways know where it is saved.... again encrypt all.

Oh, yes, grub.cfg can reside inside a RAID0, LUKS over LUKS ... over LUKS, etc. You only need a very small partition for Stage 1.5 or 2 of 8MiB or less in non formatted (bulk dump) to let Grub2 boot... seach for BIOS GRUB partition on internet.

Sorry, i not yet tried ZFS and not sure if Grub files (grub.cfg, etc) can reside inside a ZFS... but i had tested having them over: Ext4 over LVM over LUKS over LVM over LUKS ... so on some levels ... over LUKS over a real GPT (olso in a MBR) without EFI partition and it boots ok... only needed to add extensions 'crypto' and 'lvm', when installing grub2... i prefer to type my own grub.cfg so i never use anything else than grub2-install related to grub2 (again i am paranoid).

I allways use as main bootloader my own grub.cfg that calls other bootloaders, that way i let system to manage its own bootloader without touching the main one that i edit manually with a text editor.

Sorry, i am a total paranoid.

What must you think? If there is any 'place' where non-encrypted data can be written / read, any kind of 'private' data is a total miss, it can somtime be written on that unencrypted part. To be safe, never ever have any non-encrypted part.

P.D.: If you use EFI partition have in mind any code can write to it beacuse it has no protection at all and it is a non encryptable FAT32 partition.

P.P.D.: Where can be data stored on a FAT32? Anywhere, also on free space. And in Ext4? Exactly the same... allways encrypt all the storage.