I need to remove duplicate lines from a huge txt. The file is about 150 mb big. When I try PSPad, I get memory error (despite I have 8 GB RAM).
Have you any idea or advice how to remove these duplicates?
Gawk: pattern scanning and processing language Download->Binaries->Zip
Copy "awk.exe" (gawk-3.1.6-1-bin\bin\awk.exe) to your directory. Create bat file:
awk "!x[$0]++" huge.txt>output.txt
You can download $Notepad++ and use the TextFX plugin. Install Text FX by going to Plugins -> Plugin Manager -> Show Plugin Manager -> Available tab -> TextFX -> Install
. After you have it installed, there will be a new menu called TextFX
Select the portion of your document with duplicated (or just select the whole document). Go to TextFX -> TextFX Tools
, select +Sort outputs only UNIQUE...
and either sort lines case sensitive
or sort lines case insensitive
.
www.cygwin.com
and download the basic install (i.e. you get a very basic "bash" in a terminal - given that you do NOT ADD anything). Then open a terminal, cd to where you have the file and typesort THEFILE.txt | uniq -ui >UNIQUE.txt
... typeuniq --help | less
andsort --help | less
for short info on how these two commands work.