simple my question is different because i need to merge the files into one too then remove the duplicates lines from the that file which will be over 50GB txt i have large .txt from 10GB+ files
i want to merge them into 1 .txt file
then remove all the duplicates lines from that 1 large .txt file combined which will be around 50GB txt file or 100GB txt file
so what can handle that kind of large file and remove the duplicates from it smoothly ?
i need the fastest way because i tryied bouth notepad++ and emeditor they work super heavy with them for merge or duplicates removing and take forever
i have 12GB RAM
cd
to move to the folder with your text files and then typecopy file1.txt + file2.txt combined_file.txt
. That will join both files and will take about 3 seconds per GB if you are working on an SSD. It will be slower on a hard disk.sort -u
command mentioned in the other thread is also very fast and can handle 0.1 GB per second.