What is the safest and least expensive way to store 10 terabytes of data?

Question

I'm a member of a production company and we're preparing for our first feature film. We've been discussing methods of data storage to keep all of our original content safe (for as long as possible). While we understand data is never 100% safe, we'd like to find the safest solution for us.

We've considered:

• 16TB NAS for on-site storage

• 4-5 2TB hard drives (cheap, but not redundant), copy original footage to drives then seal in static free bag

• Burn data to Blu-Ray disks (time consuming and expensive: 200 disks == $5000)

• Tape drive(s)?

I know the least about tape drives, except the fact that they're more reliable than disks. Any experience/knowledge with this amount of data is hugely appreciated.

Answer 1

This assumes you're going for an archive rather than regular backup or live data.

Go for a set of SATA hard drives (1 or 2 TB), plus a few extras. Copy your data onto the disks. Use QuickPar (or an archiver which support parity volumes) to create additional parity files. Distribute parity files among your hard disks.

The parity info will allow you to reconstruct files if you have enough parity files left. So if one disk dies but other disks with parity info still work, you can use the parity files to reconstruct the original.

For more redundancy, do it all twice and store at separate locations.

If you're really paranoid, store a PCI and PCI Express to SATA controller at each site too!

Edit: Heck, if paranoia is what it's all about, go the whole hog and store a PC with gigabit ethernet at each site capable of reading the disks!

Answer 2

The LTO4 drives we use can cram in 800GB per tape. The downside is the cost of the drives themselves. If you balked at the $5K for Blu-Ray disks, you're not going to like the cost of LTO4. On the other hand, the cartages themselves are pretty cheap on a per-GB basis so you can keep a lot of copies. These tapes have a good shelf-life, but you do need to plan to move them to new media as you replace your tape drive. From how you describe how you're going to use this storage, I don't think tape is a good fit. Not unless this is going to be a backup solution instead of an archive solution. Different problems.

You might want to consider some kind of cloud based backup vendor for this, if your daily net-change is small enough. Of course, this depends on how beefy your Internet connection is, and restoring from it could be equally painful. These services are pretty new, so its unknown how they handle the bankruptcy problem for your data; what happens to it if they go out of business?

Local disk storage is definitely a way to go. A drive enclosure with easily swapped drives is probably your best bet considering your cost constraints. If you can live with that data offline, then go for it. Disks are cheap, and disk-enclosures are cheaper than tape drives. Fewer moving parts.

---

I'm assuming you're needing this for archive, not active usage in a video role. If it is, that's a different storage problem.

Answer 3

In your NAS research, have you looked at the DroboPro or DroboElite self-managed storage devices?

Consider what the cost would be if you lost some/all of your footage and budget accordingly ;)

Answer 4

Have you considered hosted services? I am not sure how the price weighs up, but you may want to consider a hosting service that sells bulk storage. Maybe AWS?

Answer 5

Eight to ten 2TB hard drives; make two sets. Store as already mentioned, sepatately ...

I would stay away from tape as it's much more sensitive to the conditions it's stored in than disks; also, five years from now, odds are good you'll still be able to plug a SATA drive into something, but you may not be able to access a tape drive that can read your tapes.

Opticals would be the best in terms of long-term stability, but, as you mention, the costs and logistics make them somewhat impractical.

Answer 6

I would recommend having some sort of NAS/RAID for live files that you're frequently working with. If you're going to use large capacity drives, you might want to go with a RAID-6 or similar which like a RAID-5 + hot spare allows you to fail two disks before your RAID is degraded, but unlike the hot spare configuration your 2nd extra drive will also have parity on it so that the raid doesn't need to rebuild parity to the spare on failure. With drives getting larger, there's a greater chance of failing a drive during a rebuild, so having as much live parity as possible is good.

In addition to this, I would recommend using hard disks like tape drives. These days one can even purchase hard drive cases (also these) that allow you to file away drives for archival. You can dump files to the RAID and then make extra archival copies to drives to be filed away.

Some video production houses like the Pixel Corps use similar setups.

Also, I would highly recommend storing files in more than one location (not under the same roof). In addition, definitely follow the mantra of "if it's not written in at least two places, you haven't saved it yet". Also, RAID does not count as a backup in and of itself.

Answer 7

I'm going to assume this is largely for offline backup type storage: easily created and restored, but not heavily used in day-to-day operations.

2TB drives can be had for about $140 these days. So, for 10TB, 5 drives, $700. Figure about $400 to build a semi-low-powered server around that running Debian or Ubuntu or some other Linux flavour. If you get a system with multiple NICs you can bond the network interfaces together to (potentially, I believe you need a managed switch that supports trunking) increase the available bandwidth the server has, and protect against NIC failures. You can then expose the storage via samba, NFS, or some other method to the workstations that need to be using it. For this large a data volume, you'll probably want some sort of redundancy so that when (not if) one of these drives bites it, you wont be without your data. RAID5 is not a good idea here, for several reasons, but RAID10 fits the bill. Since the server is running Linux, you can use mdadm to manage the RAID array. You'll need to double the number of physical devices to do this, so you're up to 10 drives. Total cost of this storage server: ~$1800 plus the sweat equity in building it and administering it.

If you really love your data, (if this is truly important production data, you should!), you could build a duplicate server, and backup the backup. You can do this with an rsync crontab, which might be horribly slow, a netcat/tar pipe solution, which would be faster to be sure, or set up the raid devices as the base of a drbd device. With the storage duplicated across both machines automatically, you've (essentially) guaranteed to have redundant backups of the data. You can get even fancier and set up a heartbeat failover system to guarantee availability of your data volume.

Answer 8

Buffalo TeraStation III 8TB Network Attached Storage - £1,085.56

amazon.co.uk link

I got a terastation 1tb version which I have upgraded to 4tb, it works flawlessly in RAID5 for me. You could look at buying a smaller version and upgrading it yourself to save some money - or buy multiple terastations etc..

Answer 9

I don't know much about tape drives nor the semantics of storing data on tape, so I can't help you there. Given the prohibitive cost of using bluray as a mass scale backup, I would stick with a NAS or individual drives.

One thing to be wary about with NAS/Raid is that, while they can store a considerable amount of information in a single virtual drive...you have to keep all the drives together, and RAID is generally not a great long-term "backup" solution. You run the potential of losing more than one drive, losing the NAS device itself (and there are rarely guarantees that a replacement device will be able to read data off of the drives, even if it is an "identical" device.)

Given how radically cheap hard drives are these days, and their general stability, I would opt for individual drives. You could easily grab 10 2Tb drives for a small cost (perhaps $1200 - $1500), and generate two sets of backups. You get both an offline and redundant backup of your data, without the risks involved with RAID or a NAS device. Individual drives also offer a greater range of compatibility, as SATA is unlikely to go anywhere anytime soon.

Answer 10

If you really want to make sure you won't lose it, then you want at least two archive options. Probably tape and hard drives. Making just one copy without some form of redundancy is just another way of rolling the data dice.

Answer 11

Depending on storage needs, if you are looking at backing up individual workstations, look at a Thermaltake BlacX external storage dock for removable SATA drives. They support both external USB and SATA connections. Just drop the drive into the dock, copy the data over and you have a backup.

I have docks at home and work. I make backups at each location and bring them to the other location for off site storage. I also have a scratch drive for moving in-progress work between locations.

The docks are about $60.00 each and you can get 1.5TB drives for under $100.