19

I'm a member of a production company and we're preparing for our first feature film. We've been discussing methods of data storage to keep all of our original content safe (for as long as possible). While we understand data is never 100% safe, we'd like to find the safest solution for us.

We've considered:

  • 16TB NAS for on-site storage

  • 4-5 2TB hard drives (cheap, but not redundant), copy original footage to drives then seal in static free bag

  • Burn data to Blu-Ray disks (time consuming and expensive: 200 disks == $5000)

  • Tape drive(s)?

I know the least about tape drives, except the fact that they're more reliable than disks. Any experience/knowledge with this amount of data is hugely appreciated.

3
  • 1
    Don't forget to tell them not to try and hang them with super-strong magnets. Bad for floppy drives, REALLY BAD for hard drives...
    – Hello71
    Jul 13, 2010 at 23:31
  • 2
    This is not an answer, but maybe it is interesting for you: blog.backblaze.com/2009/09/01/…
    – Bobby
    Jul 14, 2010 at 10:16
  • How about using DiamonDisc DVD's tomsguide.com/us/… I know, a bit pricey.
    – Moab
    Jul 14, 2010 at 19:25

12 Answers 12

15

This assumes you're going for an archive rather than regular backup or live data.

Go for a set of SATA hard drives (1 or 2 TB), plus a few extras. Copy your data onto the disks. Use QuickPar (or an archiver which support parity volumes) to create additional parity files. Distribute parity files among your hard disks.

The parity info will allow you to reconstruct files if you have enough parity files left. So if one disk dies but other disks with parity info still work, you can use the parity files to reconstruct the original.

For more redundancy, do it all twice and store at separate locations.

If you're really paranoid, store a PCI and PCI Express to SATA controller at each site too!

Edit: Heck, if paranoia is what it's all about, go the whole hog and store a PC with gigabit ethernet at each site capable of reading the disks!

3
  • 11
    +1 I think you have effectively defined and coined a new syndrome: dataparanoidism. ;P
    – jrista
    Jul 14, 2010 at 5:20
  • Definitely buying double the hard drives and copying all data twice for redundancy, thanks!
    – Josh T
    Jul 14, 2010 at 17:57
  • on the "really-paranoid" aspect, I think a usb sata adapter will likely be usable more years into the future than a motherboard technology specific connector (e.g. PCI, PCI-E), albeit slower. Jul 14, 2010 at 18:05
9

The LTO4 drives we use can cram in 800GB per tape. The downside is the cost of the drives themselves. If you balked at the $5K for Blu-Ray disks, you're not going to like the cost of LTO4. On the other hand, the cartages themselves are pretty cheap on a per-GB basis so you can keep a lot of copies. These tapes have a good shelf-life, but you do need to plan to move them to new media as you replace your tape drive. From how you describe how you're going to use this storage, I don't think tape is a good fit. Not unless this is going to be a backup solution instead of an archive solution. Different problems.

You might want to consider some kind of cloud based backup vendor for this, if your daily net-change is small enough. Of course, this depends on how beefy your Internet connection is, and restoring from it could be equally painful. These services are pretty new, so its unknown how they handle the bankruptcy problem for your data; what happens to it if they go out of business?

Local disk storage is definitely a way to go. A drive enclosure with easily swapped drives is probably your best bet considering your cost constraints. If you can live with that data offline, then go for it. Disks are cheap, and disk-enclosures are cheaper than tape drives. Fewer moving parts.


I'm assuming you're needing this for archive, not active usage in a video role. If it is, that's a different storage problem.

4
  • LTO5 is now 1.5TO. Not sure compressed or not but for sure, 1.5!
    – r0ca
    Jul 14, 2010 at 0:54
  • 2
    I would strongly recommend going this route. The cost is a little high for one-time ($5,500-ish), but if you're in it for the long term, 90% of that cost is for the drive; the tapes are $100 each and likely to go down in cost over time. Plus you can transport with little risk off damage, and keep them in a safety deposit box or similar, also with low risk. Also it is 1.5T uncompressed, 3T compressed at 2:1. Jul 14, 2010 at 2:55
  • 1
    FWIW, LTO4 presently runs about ~$40/piece CAD/USD. LTO5 runs ~$120/piece CAD/USD.(Source: CDW)
    – Chealion
    Jul 14, 2010 at 3:02
  • Yes, but the tape drives themselves run in the mid thousands. I see one on NewEgg for $3700. Jul 14, 2010 at 3:17
3

In your NAS research, have you looked at the DroboPro or DroboElite self-managed storage devices?

Consider what the cost would be if you lost some/all of your footage and budget accordingly ;)

2

Have you considered hosted services? I am not sure how the price weighs up, but you may want to consider a hosting service that sells bulk storage. Maybe AWS?

2
  • 2
    Just to provide some numbers, 10TB on Amazon S3 is $1500/mo. There's also a "value" option at $1000/mo called reduced redundancy.
    – hyperslug
    Jul 14, 2010 at 3:01
  • But there is also Amazon Glacier, which is a lot cheaper than S3 storage (for 10TB $70/month). Retrieval takes a long time and is expensive if you need to do that a lot, though.
    – Chris
    Apr 14, 2016 at 2:43
2

Eight to ten 2TB hard drives; make two sets. Store as already mentioned, sepatately ...

I would stay away from tape as it's much more sensitive to the conditions it's stored in than disks; also, five years from now, odds are good you'll still be able to plug a SATA drive into something, but you may not be able to access a tape drive that can read your tapes.

Opticals would be the best in terms of long-term stability, but, as you mention, the costs and logistics make them somewhat impractical.

2

I would recommend having some sort of NAS/RAID for live files that you're frequently working with. If you're going to use large capacity drives, you might want to go with a RAID-6 or similar which like a RAID-5 + hot spare allows you to fail two disks before your RAID is degraded, but unlike the hot spare configuration your 2nd extra drive will also have parity on it so that the raid doesn't need to rebuild parity to the spare on failure. With drives getting larger, there's a greater chance of failing a drive during a rebuild, so having as much live parity as possible is good.

In addition to this, I would recommend using hard disks like tape drives. These days one can even purchase hard drive cases (also these) that allow you to file away drives for archival. You can dump files to the RAID and then make extra archival copies to drives to be filed away.

Some video production houses like the Pixel Corps use similar setups.

Also, I would highly recommend storing files in more than one location (not under the same roof). In addition, definitely follow the mantra of "if it's not written in at least two places, you haven't saved it yet". Also, RAID does not count as a backup in and of itself.

2

I'm going to assume this is largely for offline backup type storage: easily created and restored, but not heavily used in day-to-day operations.

2TB drives can be had for about $140 these days. So, for 10TB, 5 drives, $700. Figure about $400 to build a semi-low-powered server around that running Debian or Ubuntu or some other Linux flavour. If you get a system with multiple NICs you can bond the network interfaces together to (potentially, I believe you need a managed switch that supports trunking) increase the available bandwidth the server has, and protect against NIC failures. You can then expose the storage via samba, NFS, or some other method to the workstations that need to be using it. For this large a data volume, you'll probably want some sort of redundancy so that when (not if) one of these drives bites it, you wont be without your data. RAID5 is not a good idea here, for several reasons, but RAID10 fits the bill. Since the server is running Linux, you can use mdadm to manage the RAID array. You'll need to double the number of physical devices to do this, so you're up to 10 drives. Total cost of this storage server: ~$1800 plus the sweat equity in building it and administering it.

If you really love your data, (if this is truly important production data, you should!), you could build a duplicate server, and backup the backup. You can do this with an rsync crontab, which might be horribly slow, a netcat/tar pipe solution, which would be faster to be sure, or set up the raid devices as the base of a drbd device. With the storage duplicated across both machines automatically, you've (essentially) guaranteed to have redundant backups of the data. You can get even fancier and set up a heartbeat failover system to guarantee availability of your data volume.

1
  • rsync != backup. Be careful as this can sync corrupted files just as easily. That is, unless you use a more sophisticated system with symbolic links or something to create historical archives automatically (files that change are left in the older version; files that don't just have symbolic links). Jul 14, 2010 at 7:02
2

Buffalo TeraStation III 8TB Network Attached Storage - £1,085.56

amazon.co.uk link

I got a terastation 1tb version which I have upgraded to 4tb, it works flawlessly in RAID5 for me. You could look at buying a smaller version and upgrading it yourself to save some money - or buy multiple terastations etc..

2

Josh, you haven't made it clear whether this is working data or archive data. If the data is valuable there is no real difference between backup and archive, only where you store i.e. on your working system or archive system. Again if the data is valuable, your working system and archive system will be identical if you want to protect the data and observe the rules of backup.

Archive more applies to the type of media than what is stored on it. Archive implies that the data will never change i.e. Written Once, and Read Many times, commonly called WORM. WORM drives are typically tape but things such as the RDX Quikstor and RDX Quikstation (8 bay RDX station) have the WORM option for specific drives so that they cannot be over written. I suspect that the RDX Quikstation may suit your archiving.

Remember that an archive system is part of a backup plan. You should never consider having only one copy of any data.

Some rules of backup are:

3-2-1 Rule.

Three (3) copies of your data. One (1) source and two (2) backups on two different media, one (1) of which is maintained offsite.

Examples: Onsite server backed up to onsite NAS, and backed up offsite to tape or RDX. Onsite server backed up to onsite NAS, and backed up offsite NAS. Onsite server backed up to onsite NAS, and real-time replicated to offsite NAS. Your onsite backup should be disk.

Media:

Your offsite backup depends on time. Backing up offsite to portable media automatically introduces time delays. i.e. Monday’s overnight backup gets taken offsite Tuesday night. If you have a disaster before the close of business Tuesday, you lose two days worth. How much you can afford to lose depends entirely on your business and frequency of change of data.

Media storage:

Fully observe the rules for media storage. Some tapes require that you store them vertical between narrow temperature and humidity ranges, well away from sources of magnetic fields. Dropping a tape will reduce its reliability. Sending tapes home with the office girl in her handbag next her mobile phone is a disaster waiting to happen. Tapes also require anything up to two hours in the backup environment before being inserted into the drive, equalise temperature and humidity between tape and tape drive. Tapes also need to be discarded and replaced at the required time/write intervals.

Personally, for these reasons, I do not like tape. The cost is not small and neither is the cost of having someone manage it. Investing in tape but not following the rules is pointless. You can do it on the cheap but reliability drops.

Eliminate Single points of failure:

Don’t spread backup data across multiple media. E.g. full backup on one tape and differentials across 6 other tapes has multiplied your risk of data loss 7 times. If I have a backup I like it to contain all the files necessary to restore the data. Better to have 7 full backups.

In a way, raid counts as multiple media. Wherever you use RAID, make sure that it has hot swap auto failover. Have at least one spare drive in a drive bay as a dedicated hot swap spare, and the systems will automatically swap out dead drive with hot swap spare. If you have terabytes of data, consider having two or more hot swap spares in your RAID.

Observe Offsite Distances:

Your offsite distance is important. There is little point in having offsite backup if it only guards against data loss in some circumstances. Anyone who has been through an earthquake, flood, tornado, hurricane etc will tell you that. Recent earthquakes in our city saw lots of large companies flounder with no offsite backups (tapes still next to servers) or unreadable backups. Generally 60 miles / 90 km is a good distance.

I don’t know your budget so giving advice is difficult. Budget does impact on choice of backup and storage. I have worked for a film producer providing I.T. support, so I have some understanding of the moving parts of your industry. There are a lot of unanswered questions:

What servers do you have, if any? How many staff do you have? Is this your one and only feature film? What size is the largest file or file set? What is the time between this film and the next? E.g. if you are full time producing you have the cash-flow to cover monthly expenses such as remote backup through fibre. What sort of cash-flow will the film generate? What requirements do you have moving forward? These and other questions set your requirements, methodology and budget.

For example: If you were producing 3 feature films a year and commercials every month employing a dozen staff, you could afford to have: Servers with RAID backed up to RAID NAS onsite. Dedicated fibre link (not that expensive) with 100mbit un-contended (not throttled by the ISP) link replicated to a data centre 60m/90k away. Important archive data should be written to tape or RDX and stored in data centre or tape vault. Budget is your sole determining factor as to the degree of redundancy and backup you are able to afford.

Or if it's low budget, and income is from sales and the future is unknown, you may be considering TAPE, RDX or USB drives (not recommended) stored at home.

Present and future requirements, cashflow and budget, weighed against loss of data are your defining factors. I suggest that you employ a seasoned backup specialist and/or business analyst to work through the storage and backup process for you.

1

I don't know much about tape drives nor the semantics of storing data on tape, so I can't help you there. Given the prohibitive cost of using bluray as a mass scale backup, I would stick with a NAS or individual drives.

One thing to be wary about with NAS/Raid is that, while they can store a considerable amount of information in a single virtual drive...you have to keep all the drives together, and RAID is generally not a great long-term "backup" solution. You run the potential of losing more than one drive, losing the NAS device itself (and there are rarely guarantees that a replacement device will be able to read data off of the drives, even if it is an "identical" device.)

Given how radically cheap hard drives are these days, and their general stability, I would opt for individual drives. You could easily grab 10 2Tb drives for a small cost (perhaps $1200 - $1500), and generate two sets of backups. You get both an offline and redundant backup of your data, without the risks involved with RAID or a NAS device. Individual drives also offer a greater range of compatibility, as SATA is unlikely to go anywhere anytime soon.

1

If you really want to make sure you won't lose it, then you want at least two archive options. Probably tape and hard drives. Making just one copy without some form of redundancy is just another way of rolling the data dice.

1

Depending on storage needs, if you are looking at backing up individual workstations, look at a Thermaltake BlacX external storage dock for removable SATA drives. They support both external USB and SATA connections. Just drop the drive into the dock, copy the data over and you have a backup.

I have docks at home and work. I make backups at each location and bring them to the other location for off site storage. I also have a scratch drive for moving in-progress work between locations.

The docks are about $60.00 each and you can get 1.5TB drives for under $100.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .