51

I have many DVDs with movies and games, they are getting old and dusty, some of them are damaged. And I have more than 300 GB of general files in my PC. I want to store and preserve all of my data and keep them for a long long time.

I want to know what is the best and more reliable storage media that I should use and... I can buy.
I will copy all of my DVDs to a new media and throw all of my DVDs away. It will take a very smaller physical space.

I think that everything that I need to store is currently 800 GB.

I do not like optical discs, because they are fragile and I have lost many CDs and DVDs.
Hard drives would be good, but they can suddenly stop working, and it is very expensive to recover data from a broken hard drive.

Other questions:
How often will I have to copy the files to a new storage media?
Is it good idea to have another copy of each file stored in another storage media?

Edit:
If I put all of the movies and games in a HDD, if the HDD stops working, I will lose everything.
This is how valuable they are for me: If I lose 10% of them, I will not cry, because I can get new ones, but if I lose 90%, it will be a disaster.

Edit 2:
I do not think that that online storage is reliable even if I pay for it, because they may be closed or may get bankrupt. They are not from my country and it would be difficult to get the files back.

6
  • 2
    As a side note I have recovered a number of previously unreadable CDs and DVDs using a SkipDR amazon.co.uk/Digital-Innovations-1018300-SkipDr-Classic/dp/…
    – Wudang
    Jan 4, 2012 at 8:06
  • 21
    Whatever you do, have multiple copies in different locations.
    – pjc50
    Jan 4, 2012 at 10:05
  • @user606723: Circular link...
    – Joanis
    Jan 6, 2012 at 4:37
  • Paper printouts. Or punched cards. (Paper tape unfortunately has a tendency to disintegrate after about 20 years.) Jan 7, 2012 at 3:12
  • pjc50 has the most important factor pinned down. Jan 10, 2012 at 18:48

15 Answers 15

29

The simple answer is: multiple copies. Whatever else you do, don't trust any single media, location or service.

Personally, I currently use external (USB-connected) hard disks for backup purposes. A 2-3 TB drive can be sourced quite cheaply and will provide you with plenty of storage capacity not just for the time being but also account for any reasonable future needs. At the physical size of roughly a large paperback book, it will easily hold the content of 200-300 movie DVDs without further compression. Buy two, or three if you are paranoid, preferably one of which from a completely different manufacturer (might want to mix Seagate and Western Digital, for example, since they are unlikely to use disks with the exact same design or manufacturing defects), and keep at least one of them in a physically separate location - a bank safety deposit box is a relatively cheap alternative that will give you physical security as well, but even just keeping one copy at work or at a friend's home will almost always work just as well. If you can arrange to be able to refresh an off-site copy without bringing it to your own location, that is even better. If any of the content is privacy-sensitive, keep that in mind when planning how to handle off-site copies.

Also keep in mind that the amount of data you are talking about (300 GB counting as more or less "irrecoverable", another 500 GB "nice to keep" but which in a pinch you could probably get from other sources such as second-hand movie DVDs) is not really all that much. I currently have a grand total of about 100 GB of digital photos alone, and it's not hard for me to add during a single day some 10-15 GB to that - and I have done that on a few occasions going to events where I have had reason to take lots of photos. Many of those photos are of in various ways questionable quality, many are mundane (nice to have, but in a pinch there's nothing truly special about them), but some of them actually are irreplacable from a content point of view as well as actually of good quality. For backup purposes, though, I treat them all the same way: multiple copies. I've had a few hard drives fail on me and while a few times I've lost data I would really have liked to keep, overall this strategy has meant that I can restore the most recent backup to a new drive and be on my merry way. If the live copy fails restore the backup to a replacement primary drive; if the backup drive fails, get a replacement backup drive and make a new backup.

If you do go the multiple storage media route, too, remember to keep checking each for signs of degredation. It's fairly quick and easy to do a SHA1 hash run over all the files on a drive and compare the results, as well as storing the list of hashes itself in multiple locations. That way, even if you get read errors at some point, you can determine which copy is "good".

6
  • I second the redundancy principle, this is the only way to ensure long term file fixity. Another way to achieve that is by using error correction codes like Reed-Solomon (see PAR, DVDisaster, pyFileFixity), but multiple copies are always a good and necessary precaution.
    – gaborous
    Dec 28, 2016 at 13:34
  • @gaborous Why is RAID not a backup? on Server Fault is relevant.
    – user
    Dec 28, 2016 at 13:36
  • Yes that's why error correction codes are a valuable addition, which people often confuse with RAID but they are two very different technologies for different purposes. I also did a more technically extensive explanation of the differences here.
    – gaborous
    Dec 28, 2016 at 13:39
  • @gaborous Actually, some RAID levels (basically everything but RAID 0 and RAID 1, and their combined variants) use what you could call error correction codes. I'd rather have two pieces of media with full, known good copies, than five media of which three must be successfully read for me to be able to reconstruct a copy, let alone the issues with online vs offline backups.
    – user
    Dec 28, 2016 at 13:42
  • I disagree, RAID5/6 are indeed similar but lesser variants of error correction codes because they are limited to respectively 1 failing disk or k failing disks (where k is usually only 2 currently in most RAID6 implementations...). In other words, if more than 2 copies of one byte are corrupted, the byte is irrecoverable. Error correction codes on the other hand allow to have a window ("block") of up to 255 bytes, and is way more efficient against bursting errors (which is the most common type according to several studies). Anyway, both ecc and duplication are good complementary strategies.
    – gaborous
    Dec 28, 2016 at 17:57
16

You have to consider how valuable the data you're backing up is to you. I would split it into at least 2 categories:

  1. Irreplaceable
  2. Would like to keep but won't be heartbroken if the data is lost.

Strategy

For category 1: I would suggest one of the popular online storage systems (Amazon S3, Dropbox, etc). Here you're paying for someone to help you manage the backup, and ensure longer term access and easier access. Otherwise follow the suggestions for category 2 and ensure correct redundancy of data exists and care is taken. Assuming it's a smaller percentage of the 800 Gb total.

For category 2: it's your decision on how much to invest storage fees and time to upload data online. For that reason I would even suggest you use a large HDD to backup all the data, and store that drive disconnected from a PC, and just be aware it has a lifetime linked to a PC that supports current tech eg SATA. Then you can port the data to a new drive / new tech in the future. A 1 or 2TB HDD is reasonably well priced and will cover your data requirements now and into the short term.

Redundancy

Multiple drives with the same data would be your redundancy, which again can even be stored 'off site' if you're truly concerned about data safety.

Security

As an added feature if you're trying to protect the data from unauthorized access encrypt it locally before uploading online, and/or storing to HDD. Something like TrueCrypt will be ideal

6
  • 9
    I do exactly the opposite of this, putting cat 1 data on disconnected hard drives (and cloud) and cat 2 data trusted to some third party. I'd never trust any third party with things that are irreplaceable.
    – Sirex
    Jan 4, 2012 at 7:48
  • @Sirex good point Jan 4, 2012 at 8:33
  • 2
    This is a horrible idea. If it's irreplaceable and not family photos, chances are you don't want others to have access (e.g. business or insurance documents).
    – Daniel Beck
    Jan 4, 2012 at 13:44
  • 2
    @Daniel Beck that's why I put in the security section, and state to encrypt locally before uploading online. If your data is strongly encrypted before uploading, then it will be fine to store online. Jan 4, 2012 at 22:34
  • NickJosevski - are you admitting that your answer is incorrect and the best course of action would be that suggested by @Sirex? Feb 29, 2012 at 20:01
11

First, think about the ways you might lose your data, and decide which you want to protect against. Some examples:

  • accidentally delete something
  • hard disk dies
  • a software or hardware bug
  • malware
  • theft
  • fire
  • natural disaster (fire, flood, quake, volcano, lightning, etc.)
  • government seizure

In my life, I've lost plenty of data, but only to the first two causes on that list.

Hard disks are compact, hold a lot of data, get bigger all the time, readily available, don't require special equipment to use, cheap, and getting cheaper (flooding in Thailand not withstanding).

I keep all my data one one drive; a second USB drive holds regular automated backups; a third identical USB drive sits offsite (at work, or in a safe deposit box is good). Monthly I carry the current backup drive to the offsite location, and bring the other drive back.

All storage media decay; the only way to be sure that your data is good is to use it. As part of that monthly routine, I pick an arbitrary file and restore if from my backup.

11

I had the same problem as you and looked hard and long for a solution. Many NAS drives have 2 hdd bays.. i wanted more.. but that always had a big price tag on it..

but then I found this! demonstrated model has been superseded sine original post

enter image description here

  • 4 SATA-2 BAYS (not hot swap) max of 8TB (modding can provide more space)
  • 1 SATA ontop for CDROM/SYSTEM HDD running on DMA33
  • ESATA on the BACK
  • 4 USB on the FRONT! 2 ON THE BACK
  • USB INSIDE (where the door is- so you boot of USB drive and not have it sticking out)
  • 8GB MAX DDR3!
  • Hardware Embedded SATA RAID 0/1 But you should use ZFS instead

Product Spec Sheet

I bought mine for £100 (there was a HP deal on) but normal price is £200 - Still half price of many other NAS.. AND! Its a full fledged processor! 1.6Ghz Dual Core AMD (special type that uses low power)

I could not believe my freaking eye balls!

Then what? This is what

I went over to FreeNAS and installed freenas7 - I know FreeNAS 8 is out.. but it seems to chew down on memory.. but the ZFS driver is allot more stable and faster on it.. So the choice is yours.

I installed 4x1TB and used RAIDZ similar to RAID-5 on ZFS (the HP server does have hardware raid(0,1) too.. but the driver does not work on freenas7 :( -- but does work on Windows Essentials! )

so I got 3TB of storage with 1 redundant hard drive(RAID-5)

Did i forgot to mention it has a full PCI-E slot in it? (low profile) Here is a sneak peak at the MB and the nice usb plug in directly on mobo

enter image description here

MODDING

-EDIT AUGUST 2012

I found this post that has a section how to hack the BIOS and enable that extra CD-ROM port to run at 1.5GBs SATA! - YES! I did it and works well.

-EDIT OCT 2017

We have since had the Gen9 and Gen10 of the microservers - I am still using the original configuration from the time of this post.. yea.. still going strong and so happy with it.

It is high time to probably invest in a newer MicroServer and get 4 3TB drives in there using ZFS RaidZ-2 (2 Redundant drives using 4 drives total) Currently just just using RaidZ-1 which is not fully optimised for 4 drives but it has been fine for me for over 5 years :D

11
  • Pity the HP microserver is about half again as large as the 4x3.5" drive WHS boxes HP used to sell. It's still better than the PC-Q08 which is about twice the volume. Does the HP use a standard miniITX board or a proprietary size? Jan 4, 2012 at 16:45
  • I've just built my folks a Mini-ITX PC using a Lian Li PC-Q08 case, which could happily house six 3.5" and a 2.5" drive, maxing out at around 25GB, if you so desired. *8')
    – Mark Booth
    Jan 4, 2012 at 16:47
  • @DanNeely - Thanks - Corrected my comment...
    – Mark Booth
    Jan 4, 2012 at 16:47
  • Hot Swap is actually important. When a drive fails, the last thing you want to do is subject the other drives to a full power cycle. Jan 4, 2012 at 17:12
  • 1
    RAID-Z1 is also software-based, and offers the same redundancy as software RAID 5. I'm not sure there's a huge difference between these two options in practice, but RAID-Z1 avoids partial stripe writes, so there's no painful read-modify-write cycle. Also, RAID-Z can heal itself after silent data corruption, but ZFS on RAID 5 cannot - it would just know that something was wrong.
    – sblair
    Jan 5, 2012 at 16:00
9

Well, if you are serious about saving your data, I would suggest you build two NAS units from inexpensive computer parts. The hardware does not have to be bleeding edge, just capable of accessing Terrabyte SATA drives. You could use one of the many UNIX/Linux flavours or an old Windows version as an operating system. Then load your DVDs and disk files onto this unit and then copy it to the other. You will have to keep the two systems synced with each other in some fashion once a day, week or month depending on your paranoia level.

This is better than tapes or Blu-ray as the data is readily available and you will have two copies floating around. Once in a while, you will have to scan the NAS hard disks and decide if you need to replace them as hardware does fail.

3
  • 5
    This fails miserably when you have an accident that deletes data on the first copy and the second one is automagically synced to mirror it.
    – Blrfl
    Jan 4, 2012 at 15:47
  • 1
    Well. Never trust automation. Things will go wrong so one always has to be sensible about things. If your NASs are big enough you could do generations of backups. YOu can also plug USB or eSata drives in and copy for off site external storage.
    – kingchris
    Jan 4, 2012 at 17:36
  • @Blrfl One of the funniest stories I read was where the guy manually synched over garbage onto his secondary system. 1.5T gone in a self inflicted SNAFU. Funny because he wrote very humorously about the situation, but as he said, a really good lesson in having offline backup copies so he ended up recovering about 95%. Multiple backups and diversity in storage mostly save the day. Offsite helps as well. Feb 28, 2012 at 18:19
6

It's been said a couple of times already: nothing really exists if it's not in three copies in two different formats and one of the copies should be off site.

In this way you can be sure to have at least one readable copy. Don't rely on hardware based NAS etc if they have proprietary software (most do). When the hardware crashes you probably will not find the same model hardware again...

I suggest you go with simple USB disks, but several of them. Take a copy to work and be protected against theft or fire at home (don't all eggs in the same basket). If paranoid take double copies off site.

Let the disks spin up and check them every three to six months. You want to know if they stop working. Normally disks are not good at warning in advance before they go bad. S.M.A.R.T really doesn't work that well.

I would also suggest using md5sum or some other program that can compute checksums for each and every file you have. Check these hash sums every now and then. If a file has been corrupted the checksums will not match. Then you know and can act accordingly. Without checksums the files will get corrupt without you knowing about it. You think the files are OK, but they are not.

If really paranoid don't rely on only NTFS, FAT32, HFS+ or other filesystem format. Use several. Who knows maybe you will no longer have a Windows PC when you want to find old files in the future. In this case you should also think of having your movies in several formats.

Redundancy, redundancy, redundancy and detect when errors happen!

6

I have gone over the same idea of what to do, many times, and the end results are always the same.

CD-Rs never failed me over time, a good CD even going bad, the data is mostly there after ages. Low density. The DVD-Rs though not so, even very good burns, the density of the data is high, and the ability of it to fail is higher because of that. less quality optical media is plastic (like) and all plastic gets more brittle, and whatever stuff they hold them together with eventually fails. Even great quality opticals begin to show signs of delamination after much time. . . . CD would pull it off, but it is too dang small, and DVD is too pickey (in time), for me it is not an option. Think about putting your most valuable pictures on 2X on some very good quality CDs with a very slow good burn. They will last longer than your InkJet prints :-)

Blu-ray-R, Burned blu-rays :-) I don't even have to consider, the data density is way higher, some store bought "Glass mastered" blu-rays have had failures in the short time of their existence. A mere single fingerprint. Burned, many users found that their blu-ray burns, didn't last time at all, let alone 10 years.

Tape, got a whole lot of tape, stored properly the stuff is indeed good for 10-20 years. If anyone would like to show me where I can get the Frilling tape Unit that these tapes were for, I might even be able to access their data, assuming it used the same read patterns as 15 years ago. Tape changes, tape size change, and tape machines change. The media I stored on it didn't :-) If I had continued to painstakingly move this data to anything it would be useful, right now it is as useful as a wire recording :-)

The Cloud? Not an option for me. While they will redundant my redundant. It is not their data, make them care :-) Often you also agree that they now have the data, how far you agree they have your data can include they own it. Today my data is there, tomorrow it is bought out by someone or a dead link :-) Just keeping up with the changes to the TOU, would be harder than transferring it myself. Add bandwidth caps, quantity of data, and lack of speed. Not going to happen for me.

Hard Drive.

1) Nothing is forever, if I have to keep messing with it to keep it alive, and I don't pay myself by the hour :-) to deal with it. Then it would be nice if it moves 50MB/s instead of 3.

2) Working drive I myself have had very good luck with hard drives, what else has been bit x bit accurate through so much transfer and quantity and use. As long as it has not been spun-up for 4 years, or treated poorly.

3) Storing a drive, I myself have had zero problems properly storing drives the same as tape and other media. I can wake it up after having been parked for 10 years, and it doesn't act any different.

But people who store way more drives than I do, and might not store properly, can get a hard lock on the motors. The motor won't spin from being parked for so long. Humidity and oxidation could be large factors. Say a person lived in a rainforest next to a salty ocean, the drive is not 100% sealed. Minor dehumidification, or major sealing with silica and even O2 removers, would help. Moisture, road films, smoke, kitchen grease, and all the other things that can slip into tiny holes, should be considered.

I have the best luck, speed, and compatibility and time with the hard drive, even as the other stuff failed. It has its own built-in reader :-) . It is just a matter of having (at least) 2.

My theory of 2 HD: (which didn't work for 2 DVDs)

A) One working, one that stays in the system, is allowed to live 3-4 years, is monitored, subjected to possible virus, is "known" it is watched and used, motor can't lock-up, but it can croak over time. It is maintained, updated, I know if it starts to fail. I can only hope it doesn't fail hard, because they do. I try not to let them rot on the vine.

B) One in a box, one that stays spun-down most of the time, it doesn't suffer from spin-up time, or wear, but it isn't parked for years either. This second hard drive, can be brought out of storage, updated, spun-up a bit so it doesn't lock up, and checked to see how it is doing. Be that some convenient external box, or very well stored resealed in the static bag, with adsorbers, for longer terms.

Every once in a while, be that 1 week or 1 year, the working drive and the storage drive can be updated/synced.

That just leaves C) storing it ANY other way also. And only small parts of my system have that kind of value.

That is my plan so far, and without any plan (before), that has been what has worked, when the others were failing. If my other data was SOOO valuable on the other medias, I would have used more methods for it, and kept shifting it around.

Whatever method, the one thing that is true in all cases, if you love it, it's going to be a labor of love to keep it forever. When they come out with crystal media storage, make sure you sling a copy onto it too :-)

2
  • I'm a little unclear which method you prefer here: CD-Rs, 2HDs, or a combo of methods? Feb 29, 2012 at 18:35
  • 1
    CD-Rs have held up "ok" the data density is not as high as other optical media, so they usually survive time, but they do not hold a lot of data. the 2 HDs one active and one inactive, Is the way that i have preserved tons of data. Also many hidden tips in there about storage, i actually have a storage room for my own creations, that attempts to model itself (cheaply) with the storage methods of the library of congress. De-humidification, temperature control, and dust control, well beyond normal. O2 and moisture are the enemies :-) Dust was the enemy of tape.
    – Psycogeek
    Mar 2, 2012 at 10:29
4

Besides the question of physical storage media, there is a body of research on which digital formats are preferable for long, long storage. The US Library of Congress Sustainability of Digital Formats page is a good place to start.

You may also want to review the thinking that underlies the Archivematica technologies.

2
  • While this is an interesting aspect for long-term archival storage and retrieval, the OP's question does not read to me as to take such aspects into account.
    – user
    Jan 12, 2012 at 14:58
  • 1
    In the past I have answered the exact question asked, then someone came up with a more general and robust answer. And they were right to do so.
    – W_Whalley
    Jan 16, 2012 at 15:51
3

This question asks for opinions, so mine is that one should constantly use the best, fastest available media, make multiple backups and continue to upgrade storage as is necessary. Every medium, including stone tablet can be suddenly damaged or destroyed.

6
  • 2
    It does not. Please cite something that shows why this approach is good (cost effective and reliable).
    – soandos
    Jan 4, 2012 at 4:16
  • 1
    What does not what, please?
    – Kris
    Jan 4, 2012 at 4:22
  • 1
    The question do does not ask for opinions. If it does, it does not deserve to be a question here.
    – soandos
    Jan 4, 2012 at 5:50
  • @soandos, > The question do does not ask for opinions. Sure it does. > If it does, it does not deserve to be a question here. Why not? A lot of questions here are like this. Not every question has a single, definitive answer. Even questions on SO don’t always (usually?) have just one way to do things.
    – Synetech
    Jan 4, 2012 at 6:03
  • 2
    @Synetech inc.: If you read the site FAQ, you will see that questions asking for a subjective response are prohibited. When I re-read the question, I saw that I had misunderstood the implications. Regards,
    – Xavierjazz
    Jan 4, 2012 at 6:09
3

I don't have too much to add to what's already been said. One thing though - I would not trash your optical disks unless you really have to; I'd put them in a box and store them in a closet or something. After all, it's a form of backup already, and if you make copies from it, then until something goes wrong with them, they are a known good backup. The only optical disks I've had go bad were when I used to make labels for them. For some reason, the labels seemed to screw them up.

Personally, I prefer multiple hard drives for backup. They are quick, easy, and reliable (assuming you have more than one copy). I don't like tapes for person use. Businesses use tapes because they often store multi generations of all their backups in a safe deposit box, and it's pretty hard to fit a lot of hard drives in one. But for personal use, I've found tapes to be a lot of trouble and not always reliable.

I'd consider storing really important, irreplaceable stuff on either USB thumb drives and/or optical. I mean things like photos, not music or video (unless it's irreplaceable).

2

Seriously, consider paper for really critical stuff. Consider that an inexpensive laser printer can print at 1200 DPI, and inexpensive scanners can scan at 2-4 times that resolution. If you scale down to 600x600 resolution for reliability you can put about 2.8MB on a sheet of paper (with substantial margins).

Use good quality acid-free paper and laser (vs inkjet) printing and the medium is good for easily 200 years if reasonable care is taken in storage.

And extracting the data is so simple that one could write a scanner program from scratch, if somehow the original got lost. So long as there are computers and scanners/digital cameras the data would be accessible.

1
  • And another option is film. Good quality safety film, as 35mm, microfilm, or microfiche, is good for 100 years or so, and a reader could always be cobbled up fairly easily. Jan 29, 2012 at 21:55
1

Well, Google uses tape to back up data (and so do others), so I think it's probably a safe bet.

3
  • How often do they swap tapes? Is that fact that they store everything in triplicate (or more) a factor in this?
    – soandos
    Jan 4, 2012 at 4:17
  • 1
    Most large enterprises use tape; but the costs are prohibitive for typical consumer/prosumer data-sets. Until you're in the dozens of TB range multiple external hard drives/redundant NASes are more cost effective. Jan 4, 2012 at 18:14
  • 2
    Not only that, but while it works quite well (cost notwithstanding) as a backup medium, tape is inappropriate for random access storage because of its sequential access nature.
    – user
    Jan 17, 2012 at 10:26
1

This is hard to answer because it depends how important your data is to you and how much you are willing to spend. I'll tell you my setup, which for me works well.

First thing I would do is get a NAS. I currently own a Synology NAS and the software is great on it. With the software it currently has on there, you can backup to other devices as well.

Second thing is you can store data online in the cloud. You have to be careful about what you store though. I personally only store stuff I want to keep, but isn't anything too personal. Problem with this too, is upload speed. It took me about 30 days to upload about 230GB of stuff. You can always send a hard drive for them to upload your data though for a cost of course. I currently use Crashplan, which is pretty cheap and exactly what I need.

Third, as is stated before, you can get a Synology NAS and attached an external device on some of the models they sell. This really wouldn't matter unless you take that hard drive off site, to say a safety deposit box. Maybe backup a months worth of data and just keep switching out the hard drives some where off site.

As for my current setup, I have a Synology NAS with crashplan installed on there and it automatically backs up the data with the folders I selected to crashplan. This is a perfect setup for me, but it can be costly. The NAS itself is RAID 5, providing with redundancy.

I paid about $1200 or so for the NAS and hard drives. But that's cheap compared to trying to recover data. I find that CDs/DVDs are kind of unreliable and shouldn't be used for long term backup. A flash drive would probably be better than a CD or DVD.

If you do all of this, you should be ok in case of hardware failure or some kind of natural disaster like a fire.

1

My recommendation is to put everything onto a hard drive, then back that drive up using cloud software. My personal recommendation would be CrashPlanbecause that's what I use at home, it's affordable, lets you back up any file you want, anywhere you want, and even lets you backup to local media too. It does periodic checks to verify the integrity of your backup and has some other nice features. You can set it up with a local encryption key, so even if CrashPlan received (and complied with) a subpoena, no one would be able to read your data. I have nearly 3TB backed up with them and so far have loved paying the $5 a month charge for this rather nice service. I in no way benefit from "selling" them here, I'm just a very satisfied customer. Maybe you should take a look at their service.

0

I store my files in a HD and in the cloud. I'm currently using Google Docs because is cheaper than Amazon and DropBox.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .