Are there any tools that allow to "diff" text files of terabyte size?

  • What kind of differences are you anticipating? Small differences on otherwise similar lines, or large chunks added or removed? – Tom Anderson Apr 7 '11 at 11:51
  • Voting to migrate to SuperUser. – Tom Anderson Apr 7 '11 at 11:53

You can use bdiff. It is used to compare files that are too large to be loaded in memory.

     bdiff - big diff

     bdiff filename1 filename2 [n] [-s]

     bdiff is used in a manner analogous to diff  to  find  which
     lines  in  filename1  and filename2 must be changed to bring
     the files into agreement. Its purpose is to allow processing
     of  files too large for diff. If filename1 (filename2) is -,
     the standard input is read.
  • 1
    That will work well for scattered small changes. Because it works by splitting the input into chunks and diffing corresponding chunks, i think it will do a pretty bad job on inputs which differ with large insertions or deletions. Still the best suggestion so far, though! – Tom Anderson Apr 7 '11 at 11:52
  • the link has been dead. I've updated it for you. Or use alternate man pages like unix.com/man-page/ultrix/1/bdiff, docs.oracle.com/cd/E36784_01/html/E36870/bdiff-1.html – phuclv Aug 31 '19 at 1:23

Are the files sortable? (e.g. via "sort" command)

If they are, and depending on the kind of difference you are looking for and if this is a once-off exercise, why don't you sort them each, split into the letters of the alphabet (or whatever, as long as you get multiple files) and then compare each letter..

Primitive, but might work neat and easily for you?

  • yes, this is a good poin that might work, thanks! – Open the way Apr 26 '11 at 22:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.