up vote 2 down vote favorite
share [g+] share [fb]

I am looking for an open source (possibly 64 bit) windows text editor that will allow me to remove duplicate lines from an extremely large (4GB+) text file.

What do you use to remove duplicate lines from your large text files?

link|improve this question
92% accept rate
1  
duplicates of .. what? words? lines of words? provide a sample (considerably shorter than 4gb) – akira Dec 8 '10 at 20:14
Added the Windows tag, since this is a Windows-specific question. – Sasha Chedygov Dec 8 '10 at 22:28
Thank you both for your clarifications. – darkAsPitch Dec 8 '10 at 22:29
How did you end up with a 4GB text file? That kind of sucks... – Ciaran Dec 8 '10 at 22:37
feedback

3 Answers

up vote 3 down vote accepted

sort -u file > outfile

A handy Win32 native port of sort is in UnxUtils

For more complicated meanings of "remove duplicates" there is Perl (et al).

link|improve this answer
Thank you for the reply, but UnxUtils was unavailable for download when I attempted it. – darkAsPitch Dec 8 '10 at 23:03
1  
You can download UnxUtils from sourceforge.net/projects/unxutils/files/unxutils/current/… – RedGrittyBrick Dec 9 '10 at 8:41
feedback
up vote 2 down vote

If you have Cygwin or MinGW you could probably accomplish this with

cat file | sort | uniq >> outfile

assuming you want unique lines. I know not how this will perform, since sorting a dataset that large will probably take a long time (or if it is already sorted you can just leave that part out) or how, exactly, these commands function (if they will consume 4GB of ram or not).

link|improve this answer
Thank you! CygWin and the sort command was exactly what I needed! – darkAsPitch Dec 8 '10 at 23:02
feedback
up vote 0 down vote

You can remove duplicate lines in a huge file with PilotEdit.

link|improve this answer
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.