up vote 3 down vote favorite
share [g+] share [fb]

I'm looking for a utility (either for Linux or Windows) which can verify that file contents seem to match the extension. This is for catching badly corrupted files.

I'm on the verge of writing a little bash script which will just run "file" on everything, but I don't want to figure out what "file" returns for a all of the properly-formatted file types that I'm going to encounter.

Background: A friend recently accidentally erased their Windows user folder (yup... Desktop, registry, My Docs, the whole thing...). I used an aggressive unerase utility to recover as much as I could. This particular utility recovers multiple copies of the same file for some reason (ie, "Letter.doc", "Letter(1).doc", "Letter(2).doc", etc...). Some of these files will contain random junk, some will be a valid document, and may of them will be the same document.

So, I've used fdupes and a perl script to nuke all of the duplicates, but now I want to separate the ones which look like they could be the original file from the ones which are probably not. I can't believe I'm the first person who's needed to do this...

link|improve this question
I won't put this as an answer since I am prepared to be wrong, but I don't even think that this would be possible. There are so many file formats out there (many of the proprietary) that a tool such as what I think you are looking for would simply have no way of keeping track of all of them. I would say that having your friend go through every file and try to open each one in the appropriate program will provide some much needed negative reinforcement and training on why to be careful with what you delete. – EBGreen Aug 17 '11 at 14:02
feedback

migrated from stackoverflow.com Aug 17 '11 at 13:55

This question came from our site for professional and enthusiast programmers.

2 Answers

up vote 1 down vote

You can try the TrIDNet program (there also a CLI and an online version). It identifies files based on binary signatures, based on a built-in (and extensible) library. Here is a list of the file extensions that it can identify by default, although it is possible to add a definition for a new type manually.

Do note that if the binary signature of the file is not intact corrupted, the software will not know that the file is corrupted. It's up to the implementation of the application that uses the file to check for corruption (since there's no way to "just know" if a file is corrupt).

This is just the way files work, it's up to the program that uses said binary files to check if they are corrupted or not (some file types store checksums inside of them).

link|improve this answer
While this is an interesting concept, in reality it simply demonstrates the issue that I discuss above. For example there are 30 different .Dat formats that it recognizes. Since the way it works is by scanning several known good files and looking for patterns that they all have in common. It isn't really going to tell you if the .dat file that you are checking is valid for the program you want to open it in or not. At best it will tell you "Of all the programs that I know of that open DAT files, these are the ones that might open this file". – EBGreen Aug 17 '11 at 14:22
Well, the Linux "file" program already does what TrIDNet seems to do. What I'm looking for is something which can separate files where the extension matches what the initial file signature appears to be. I realize that, in order to be completely sure, each file needs to be opened by the application that it is native to. I'm just looking for a head start by being able to have a directory of "possibly intact" files and another directory of "probably junk" files. – Jemenake Aug 17 '11 at 15:15
You hit the nail on the head there "Initial File Signature". There is no standard that says that a specific signature must be associated with a specific extension. For instance, I have seen .DAT files that are really just an XML file with a .DAT extension. But that is that the program using them expected. – EBGreen Aug 17 '11 at 15:34
@Jemenake then why not use some of the signatures from TrIDNet to determine what the file's content is supposed to be, and then just separate the ones where the content matches the extension? – Breakthrough Aug 17 '11 at 18:15
That's exactly what I'm trying to do. But, before I write a script to do it for 10,000+ files, I was hoping that someone already had written a program to do something like this. But, looks like they haven't. – Jemenake Aug 17 '11 at 18:47
feedback
up vote 0 down vote

For office apps, there's an Open and Repair... option in the open dialog.

For images, IrfanView can reconstruct file headers.

For videos, VLC Player will do the same.

Like EBGreen said above though, it's highly unlike there's a consumer level product that'll actually do this for many files types, let alone all file types.

link|improve this answer
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.