I am looking for an open source (possibly 64 bit) windows text editor that will allow me to remove duplicate lines from an extremely large (4GB+) text file.

What do you use to remove duplicate lines from your large text files?

  • 1
    duplicates of .. what? words? lines of words? provide a sample (considerably shorter than 4gb) – akira Dec 8 '10 at 20:14
  • Added the Windows tag, since this is a Windows-specific question. – Sasha Chedygov Dec 8 '10 at 22:28

sort -u file > outfile

A handy Win32 native port of sort is in UnxUtils

For more complicated meanings of "remove duplicates" there is Perl (et al).

  • Thank you for the reply, but UnxUtils was unavailable for download when I attempted it. – darkAsPitch Dec 8 '10 at 23:03
  • 1
    You can download UnxUtils from sourceforge.net/projects/unxutils/files/unxutils/current/… – RedGrittyBrick Dec 9 '10 at 8:41
  • It doesn't seem to work for large files unfortunately, there is a bug (i think) in the UnxUtils and it complains about not being able to read from /tmp/<temp_file>... – Gordon Carpenter-Thompson Sep 2 '13 at 12:33
  • @Gordon: Interesting. How large is "large" in MB or GBytes? and what O/S and filesystem, how much free space? – RedGrittyBrick Sep 2 '13 at 14:39
  • The OS was windows 2008 R2 Datacenter running on Amazon Web Services. The file was about 2Gb. It's only a smallish instance so it might have been down to limited RAM/diskspace. Maybe the error message is misleading. I gave up and got it sorted using a cygwin port on the same instance. – Gordon Carpenter-Thompson Sep 2 '13 at 15:06

If you have Cygwin or MinGW you could probably accomplish this with

cat file | sort | uniq >> outfile

assuming you want unique lines. I know not how this will perform, since sorting a dataset that large will probably take a long time (or if it is already sorted you can just leave that part out) or how, exactly, these commands function (if they will consume 4GB of ram or not).

  • Thank you! CygWin and the sort command was exactly what I needed! – darkAsPitch Dec 8 '10 at 23:02

You can remove duplicate lines in a huge file with PilotEdit.


I found a tool called PilotEdit which was able to do it.

  • Was already posted by @Dracoder – Rishi Dua Jun 16 '14 at 7:14

I also posted this answer on a duplicate question about >50GB files

Assuming all lines are shorter than 7kB, and that you have bash, dd, tail, head, sed and sort installed from cygwin/unix:

  while LANG= dd 2>/dev/null bs=1024 skip=${i}000 if=large_text_file count=1021 \
  | LANG= sed -e '1d' -e '$d'  | LANG= sort -u ;
  LANG= dd 2>/dev/null bs=1024 skip=${i}000 if=large_text_file count=1021 \
  | LANG= tail -n 1
  LANG= head -n 1 large_text_file
} | LANG= sort -u > your_result

This divides the file in chunks of 1024000 bytes, and adds also 3*7*1024 bytes ("21" in 1021) from next chunk. As the divisions may cut a line, first (1d) and last ($d) lines of each cunks are destroyed (sed).

So to compensate, something containing last chunk is extracted again and only its last line is kept (tail -n 1), and the first line is also extracted again (head -n 1).

When the loop fails, the last chunk has been extracted.

sort -u may be viewed as a compressor, but it only sorts its input then skip duplicates. The first "sort" compresses all chunks. The second sort compresses again the concatenations of all these chunks (and that second sort has been missing from above code since third edit, sorry).

You said text file, but I assume binary anyway, hence the LANG= (gets all faster also).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.