up vote 5 down vote favorite
1
share [g+] share [fb]

Are there any tools that allow to "diff" text files of terabyte size?

link|improve this question
98% accept rate
What kind of differences are you anticipating? Small differences on otherwise similar lines, or large chunks added or removed? – Tom Anderson Apr 7 '11 at 11:51
Voting to migrate to SuperUser. – Tom Anderson Apr 7 '11 at 11:53
feedback

migrated from stackoverflow.com Apr 7 '11 at 11:53

This question came from our site for professional and enthusiast programmers.

2 Answers

up vote 1 down vote accepted

You can use "bdiff". It is used to compare files that are too large to be loaded in memory.

http://manpages.unixforum.co.uk/man-pages/unix/solaris-10-11_06/1/bdiff-man-page.html

link|improve this answer
1  
That will work well for scattered small changes. Because it works by splitting the input into chunks and diffing corresponding chunks, i think it will do a pretty bad job on inputs which differ with large insertions or deletions. Still the best suggestion so far, though! – Tom Anderson Apr 7 '11 at 11:52
feedback
up vote 0 down vote

Are the files sortable? (e.g. via "sort" command)

If they are, and depending on the kind of difference you are looking for and if this is a once-off exercise, why don't you sort them each, split into the letters of the alphabet (or whatever, as long as you get multiple files) and then compare each letter..

Primitive, but might work neat and easily for you?

link|improve this answer
yes, this is a good poin that might work, thanks! – flow Apr 26 '11 at 22:01
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.