Just a point to make some light:
- If one disk can fail without prior knowing when will it fail... why two can not fail at the same time? or three or four or five, etc.?
But, think on this other situation:
- One disk has failed
- You put a new one and let it 're-build' to be safe
- While such 're-build' is been done (maybe some hours) another disk fails
You lost all!!! That is the main reason to tolerate a simultaneous two disk fail.
But if you are really paranoid (like me) you will use only Stripping, no other RAID level.
Am i mad? No, of course... i do not mind one disk fail and loose all data! I will really not loose anything just because i am really paranoid.
My paranoid scheme:
- ZFS with stripping (2x, 3x, 4x ... or 10x disks), just to fit the max bandwidth of the bus (sata III, usb 3.1 gen 2, etc)
- And other similar five ZFS (without power, so can not get damaged)
How to use all of such ZFS:
- Have only two of them at the same time (just to sync changes)... like a master with five slaves
Paranoid point of view:
- If one ZFS fails... like been hit by a storm ligth, burned, stolen, etc... i still have five copies more inside different places (bank, bunker, internet cloud, friends house, vacation house, etc)
To sync some of them... i must carry with me one ZFS and do the sync on the other physical place / country / etc... so i use one of thoose slaves.
Think when i talk about HDDs on one ZFS, on another ZFS can be 3.1 Gen2 64GiB USB sticks! No need to be identical size, type, etc.
Really paranoid point of view.
For thoose not so paranoid, and if they can support to loose all... there is RaidZ levels... i prefer to 'mirror' by my self.
Another advantage of do manual 'syncs' is your own fails:
- You have a hypotetic perfect never ever fail ZFS system, but you decide to delete a file... some days after you discovered you missed the file you wanted to delete and delete one that you did not want to (by human error)... you are out of lucky unless you have a BackUp
So... again... if you need a BackUp, why not speed up the ZFS at max by only striping?... you will still be needing to do BackUp after all.
Well, the answer can be:
- System partition for a24h online system
For data partitions i do not see the need for RaidZ levels versus having a external USB ZFS clone ready to be connected as soon as main ZFS in use fails... just unmount ZFS main point and mount the other good ZFS in place... then with another machine re-create the off-line ZFS that got damaged.
If you have two computers, you have what you need to 7x24h with just only a few seconds (boot time) of down-time.
For home users that want their photos stay safe... use more than two ZFS and have at least one or two of them never ever connected at the same time as the rest. Paranoid? Not at all.
It is just i learn the hard way how things can go really wrong!
Just to clarify... what happen...
- I was using the PC normally
- Data was getting to fail
- I took out the HDDs and place the ones that was OK
- Data on them also got damaged in less than ten seconds
- I lost all my data (not fully since i was a little paranoid and have BackUp on DVD media)
The problem was really in the the controller (integrated on the mainboard), not the disks... it was altering data on read/write operations.
Since then... my data is not on internal sata disks anymore... only on external USB 3.1 Gen 2 enclosures i can plug and un-plug (each disk in its own enclosure)... and when one fails... i test all things prior to connect to that PC anything with real data... RAM, processor, USB port, etc... with a dummy USB enclosure to test patterns... if that is OK... then i test the failed disk... if it works on another enclosure or not, on another PC or not, etc... to really know who is causing the fail... really paranoid, i know!
But learning from experience:
- Never ever trust a disk fail is caused by the disk it self... test it... it could be the controller, the cables, the power suppy, etc.
- Or worst, it could be the RAM
Talking about RAM, i had an not so old PC (BIOS based 64Bits processor) that if (think on three identical RAM Modules, call them A, B and C):
- i put A on the PC works perfect (no matter on what of the four slots)
- i put B on the PC works perfect (no matter on what of the four slots)
- i put C on the PC works perfect (but only on one of the four slots, on the other tree it causes a lot of fails, BSODs, etc)
- i put A & B it causes a lot of fails, BSODs, etc
- i put A & C on the PC works perfect (no matter on what of the four slots)
- i put B & C on the PC works perfect (no matter on what of the four slots)
- i put A & B & C on the PC it causes a lot of fails, BSODs, etc
If i test such modules on another PC, all of them works really great, no problem, no defect, etc.
Things can go really mad... and if you do not test all combinations you can not know what is failing... yes really paranoid to test all combinations of three ram modules on four slots... but what i had discoverd let me blind... the failing alone module works perfect if another one is present but not if another two... yes, i also tested with another three and fail!
Oh, yes, all same brand, kind, etc... identical... only way to difference them is a sticker i placed on them for such! And yes, buyed at the same time, they came on the same manufacture locked plastic case.
Sometimes a ZFS fail can be caused by RAM... then no matter how many levels of RaidZ you have... you can loose all.
Oh, near forget... use ECC memory, else ZFS can not be trusted! And do not use any HDD with internal cache, internat RAID, etc. unless it is designed to do writes on the order sent (or do all of them in a power cut).
I only use HDDs with enought internal energy capacity to write all its cache to the plates after an abrupt power lost!
All this points / weakness let me with one conlcusion:
- Best speed up things and do your own BackUPs
- Have some 100% copies Off-Line and without power
- Have something ready to recieve power and replace the failed part
That is why i have >2 PCs with >10 USB HDDs (5 ZFS copies *3 HDDs each in stripped)... i got the best of both... safe and speed! But i get the worst part... neeed to sync all of them periodically... but data safety is the most important part! also i do not want to wait a lot for such syncs (aka, striping comes in work).
And if a 3*HDDs stripe fail... i can test which one had fail of such three for some hours/days/weeks... while i have another ZFS copie On-Line and working... after be sure the other two are OK i buy a new one (maybe 30 to 45 days to arrive) and re-create such 3*HDDs ZFS and put data on it to have that fifth copy again.
Not cheap at all! But not as a main-data center price, of course... and no electricity bill (most HDDs are without power, only 3 or 6 at the same time).
Hope some one finds the idea useful (i use it daily).