up vote 1 down vote favorite
share [g+] share [fb]

For the sake of data integrity, is it wiser to archive your files or just leave them unarchived. No compression is being used. My thinking is that if you leave your files unarchived, if there is some form of corruption it will only hurt a smaller number of files. Though if you archive, lets say all of your documents, if there is even the slightest corruption, the entire archive is unrecoverable.

So whats the best way to keep a clean file system, but not be subject to data corruption.

link|improve this question
25% accept rate
feedback

migrated from serverfault.com May 24 '10 at 9:12

This question came from our site for system administrators and desktop support professionals.

2 Answers

up vote 1 down vote

For archived you mean, for example, put together with tar or similar, right?
Why don't you use a file system with native compression and deduplication like ZFS? I find it almost perfect when used for storage system.

link|improve this answer
I dont think thats what the op meant. i think the op wants to keep his disk clean and his filesystem simple. like moving seldomly used files to separate locations. the problem is to balance the following scenarios - lots of files and folders but corruption on file level or one siple archive but all data dies if that file gets corrupted. This is why i suggested winrar, assuming he were on win. zfs deduplication doesnt simplyfy anything, it just saves disk space. the folders dont change for the user. unneeded files are still listed. – deploymonkey May 23 '10 at 11:42
I thought later he could mean that, but it's not an easy task. There're some file systems that can do tiered storage (moving files around based on policy you define) or some products, but as far as i remember they are all commercial. Doint things manually, you could do a script that archives files based on date and making an archive not too big and split it. Of course this file must be backed up as soon as you can, so in the case it'll be lost, you can always recover it. As i think anyway i won't make any archive, but just move on some other disks old files. – Pier May 23 '10 at 17:06
EDIT: if you want to experiment, you could try castor.web.cern.ch/castor – Pier May 23 '10 at 17:07
feedback
up vote 1 down vote

So whats the best way to keep a clean file system, but not be subject to data corruption

As I understand it, You want to move unused files which may be needed later on.

I don't know of current open source software, but, assuming you are on Windows, winrar will do what you want. The trial version is not time limited. There might be open source software that does the same, but I don't have that at hand.

The problem you mentioned does apply to simple yip archives. They are dead when corrupted. Winrar on the other hand usually lets you get the rest of the files - don't click on solid archive because that would deactivate this capability.

Also you can specify up to 10% parity information, so up to 10% of the archive may be corrupted and it still can be reconstructed. Lastly winrar supports split archives with recovery volumes, which means that you can create additional part files that are just for data recovery if you lose one of the part files.

Under those circumstances I switched to archiving because it is safer than not archiving.

If you need input on other measures like RAID, *nix solutions like RAID, special filesystems like ZFS, just comment back and I will provide info to my best knowledge.

Sorry for the typos, I'm on a machine with wrong codepage.

link|improve this answer
Agree ZFS (hard to corrupt, supports snapshots, supports parity, supports specifying multiple copies of a file to be stored), using archives with parity support, or if your favorite archive doesn't support parity use something like PAR2 to add external parity. Not to mention backup locally, backup offsite, backup to hard drive, backup to DVD (use Nero's SecurDisc or DVDisaster). – Scott McClenning Jun 17 '10 at 6:29
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.