Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Are there any tools that allow to "diff" text files of terabyte size?

share|improve this question

migrated from stackoverflow.com Apr 7 '11 at 11:53

This question came from our site for professional and enthusiast programmers.

    
What kind of differences are you anticipating? Small differences on otherwise similar lines, or large chunks added or removed? –  Tom Anderson Apr 7 '11 at 11:51
    
Voting to migrate to SuperUser. –  Tom Anderson Apr 7 '11 at 11:53

2 Answers 2

up vote 1 down vote accepted

You can use "bdiff". It is used to compare files that are too large to be loaded in memory.

http://manpages.unixforum.co.uk/man-pages/unix/solaris-10-11_06/1/bdiff-man-page.html

share|improve this answer
1  
That will work well for scattered small changes. Because it works by splitting the input into chunks and diffing corresponding chunks, i think it will do a pretty bad job on inputs which differ with large insertions or deletions. Still the best suggestion so far, though! –  Tom Anderson Apr 7 '11 at 11:52

Are the files sortable? (e.g. via "sort" command)

If they are, and depending on the kind of difference you are looking for and if this is a once-off exercise, why don't you sort them each, split into the letters of the alphabet (or whatever, as long as you get multiple files) and then compare each letter..

Primitive, but might work neat and easily for you?

share|improve this answer
    
yes, this is a good poin that might work, thanks! –  flow Apr 26 '11 at 22:01

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.