Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I am looking for an open source (possibly 64 bit) windows text editor that will allow me to remove duplicate lines from an extremely large (4GB+) text file.

What do you use to remove duplicate lines from your large text files?

share|improve this question
1  
duplicates of .. what? words? lines of words? provide a sample (considerably shorter than 4gb) –  akira Dec 8 '10 at 20:14
    
Added the Windows tag, since this is a Windows-specific question. –  Sasha Chedygov Dec 8 '10 at 22:28
    
Thank you both for your clarifications. –  darkAsPitch Dec 8 '10 at 22:29
add comment

5 Answers

up vote 3 down vote accepted

sort -u file > outfile

A handy Win32 native port of sort is in UnxUtils

For more complicated meanings of "remove duplicates" there is Perl (et al).

share|improve this answer
    
Thank you for the reply, but UnxUtils was unavailable for download when I attempted it. –  darkAsPitch Dec 8 '10 at 23:03
1  
You can download UnxUtils from sourceforge.net/projects/unxutils/files/unxutils/current/… –  RedGrittyBrick Dec 9 '10 at 8:41
    
It doesn't seem to work for large files unfortunately, there is a bug (i think) in the UnxUtils and it complains about not being able to read from /tmp/<temp_file>... –  Gordon Carpenter-Thompson Sep 2 '13 at 12:33
    
@Gordon: Interesting. How large is "large" in MB or GBytes? and what O/S and filesystem, how much free space? –  RedGrittyBrick Sep 2 '13 at 14:39
    
The OS was windows 2008 R2 Datacenter running on Amazon Web Services. The file was about 2Gb. It's only a smallish instance so it might have been down to limited RAM/diskspace. Maybe the error message is misleading. I gave up and got it sorted using a cygwin port on the same instance. –  Gordon Carpenter-Thompson Sep 2 '13 at 15:06
add comment

If you have Cygwin or MinGW you could probably accomplish this with

cat file | sort | uniq >> outfile

assuming you want unique lines. I know not how this will perform, since sorting a dataset that large will probably take a long time (or if it is already sorted you can just leave that part out) or how, exactly, these commands function (if they will consume 4GB of ram or not).

share|improve this answer
    
Thank you! CygWin and the sort command was exactly what I needed! –  darkAsPitch Dec 8 '10 at 23:02
add comment

You can remove duplicate lines in a huge file with PilotEdit.

share|improve this answer
add comment

I also posted this answer on a duplicate question about >50GB files

Assuming all lines are shorter than 7kB, and that you have bash, dd, tail, head, sed and sort installed from cygwin/unix:

{
  i=0
  while LANG= dd 2>/dev/null bs=1024 skip=${i}000 if=large_text_file count=1021 \
  | LANG= sed -e '1d' -e '$d'  | LANG= sort -u ;
  do
    i=$((1+$i))
  done
  LANG= dd 2>/dev/null bs=1024 skip=${i}000 if=large_text_file count=1021 \
  | LANG= tail -n 1
  LANG= head -n 1 large_text_file
} | LANG= sort -u > your_result

This divides the file in chunks of 1024000 bytes, and adds also 3*7*1024 bytes ("21" in 1021) from next chunk. As the divisions may cut a line, first (1d) and last ($d) lines of each cunks are destroyed (sed).

So to compensate, something containing last chunk is extracted again and only its last line is kept (tail -n 1), and the first line is also extracted again (head -n 1).

When the loop fails, the last chunk has been extracted.

sort -u may be viewed as a compressor, but it only sorts its input then skip duplicates. The first "sort" compresses all chunks. The second sort compresses again the concatenations of all these chunks (and that second sort has been missing from above code since third edit, sorry).

You said text file, but I assume binary anyway, hence the LANG= (gets all faster also).

share|improve this answer
add comment

I found a tool called PilotEdit which was able to do it.

share|improve this answer
    
Was already posted by @Dracoder –  Rishi Dua Jun 16 at 7:14
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.