6

I am looking for an open source (possibly 64 bit) windows text editor that will allow me to remove duplicate lines from an extremely large (4GB+) text file.

What do you use to remove duplicate lines from your large text files?

2
  • 1
    duplicates of .. what? words? lines of words? provide a sample (considerably shorter than 4gb)
    – akira
    Dec 8, 2010 at 20:14
  • Added the Windows tag, since this is a Windows-specific question. Dec 8, 2010 at 22:28

5 Answers 5

5

sort -u file > outfile

A handy Win32 native port of sort is in UnxUtils

For more complicated meanings of "remove duplicates" there is Perl (et al).

5
  • Thank you for the reply, but UnxUtils was unavailable for download when I attempted it. Dec 8, 2010 at 23:03
  • 1
    You can download UnxUtils from sourceforge.net/projects/unxutils/files/unxutils/current/… Dec 9, 2010 at 8:41
  • It doesn't seem to work for large files unfortunately, there is a bug (i think) in the UnxUtils and it complains about not being able to read from /tmp/<temp_file>... Sep 2, 2013 at 12:33
  • @Gordon: Interesting. How large is "large" in MB or GBytes? and what O/S and filesystem, how much free space? Sep 2, 2013 at 14:39
  • The OS was windows 2008 R2 Datacenter running on Amazon Web Services. The file was about 2Gb. It's only a smallish instance so it might have been down to limited RAM/diskspace. Maybe the error message is misleading. I gave up and got it sorted using a cygwin port on the same instance. Sep 2, 2013 at 15:06
2

If you have Cygwin or MinGW you could probably accomplish this with

cat file | sort | uniq >> outfile

assuming you want unique lines. I know not how this will perform, since sorting a dataset that large will probably take a long time (or if it is already sorted you can just leave that part out) or how, exactly, these commands function (if they will consume 4GB of ram or not).

1
  • Thank you! CygWin and the sort command was exactly what I needed! Dec 8, 2010 at 23:02
1

You can remove duplicate lines in a huge file with PilotEdit.

1

I found a tool called PilotEdit which was able to do it.

1
  • Was already posted by @Dracoder
    – Rishi Dua
    Jun 16, 2014 at 7:14
0

I also posted this answer on a duplicate question about >50GB files

Assuming all lines are shorter than 7kB, and that you have bash, dd, tail, head, sed and sort installed from cygwin/unix:

{
  i=0
  while LANG= dd 2>/dev/null bs=1024 skip=${i}000 if=large_text_file count=1021 \
  | LANG= sed -e '1d' -e '$d'  | LANG= sort -u ;
  do
    i=$((1+$i))
  done
  LANG= dd 2>/dev/null bs=1024 skip=${i}000 if=large_text_file count=1021 \
  | LANG= tail -n 1
  LANG= head -n 1 large_text_file
} | LANG= sort -u > your_result

This divides the file in chunks of 1024000 bytes, and adds also 3*7*1024 bytes ("21" in 1021) from next chunk. As the divisions may cut a line, first (1d) and last ($d) lines of each cunks are destroyed (sed).

So to compensate, something containing last chunk is extracted again and only its last line is kept (tail -n 1), and the first line is also extracted again (head -n 1).

When the loop fails, the last chunk has been extracted.

sort -u may be viewed as a compressor, but it only sorts its input then skip duplicates. The first "sort" compresses all chunks. The second sort compresses again the concatenations of all these chunks (and that second sort has been missing from above code since third edit, sorry).

You said text file, but I assume binary anyway, hence the LANG= (gets all faster also).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .