I am looking for an open source (possibly 64 bit) windows text editor that will allow me to remove duplicate lines from an extremely large (4GB+) text file.
What do you use to remove duplicate lines from your large text files?
I am looking for an open source (possibly 64 bit) windows text editor that will allow me to remove duplicate lines from an extremely large (4GB+) text file.
What do you use to remove duplicate lines from your large text files?
A handy Win32 native port of sort
is in UnxUtils
For more complicated meanings of "remove duplicates" there is Perl (et al).
If you have Cygwin or MinGW you could probably accomplish this with
cat file | sort | uniq >> outfile
assuming you want unique lines. I know not how this will perform, since sorting a dataset that large will probably take a long time (or if it is already sorted you can just leave that part out) or how, exactly, these commands function (if they will consume 4GB of ram or not).
I also posted this answer on a duplicate question about >50GB files
Assuming all lines are shorter than 7kB, and that you have bash, dd, tail, head, sed and sort installed from cygwin/unix:
{
i=0
while LANG= dd 2>/dev/null bs=1024 skip=${i}000 if=large_text_file count=1021 \
| LANG= sed -e '1d' -e '$d' | LANG= sort -u ;
do
i=$((1+$i))
done
LANG= dd 2>/dev/null bs=1024 skip=${i}000 if=large_text_file count=1021 \
| LANG= tail -n 1
LANG= head -n 1 large_text_file
} | LANG= sort -u > your_result
This divides the file in chunks of 1024000 bytes, and adds also 3*7*1024 bytes ("21" in 1021) from next chunk. As the divisions may cut a line, first (1d) and last ($d) lines of each cunks are destroyed (sed).
So to compensate, something containing last chunk is extracted again and only its last line is kept (tail -n 1), and the first line is also extracted again (head -n 1).
When the loop fails, the last chunk has been extracted.
sort -u may be viewed as a compressor, but it only sorts its input then skip duplicates.
The first "sort" compresses all chunks. The second sort
compresses again the concatenations of all these chunks (and that second sort
has been missing from above code since third edit, sorry).
You said text file, but I assume binary anyway, hence the LANG= (gets all faster also).