This question already has an answer here:

simple my question is different because i need to merge the files into one too then remove the duplicates lines from the that file which will be over 50GB txt i have large .txt from 10GB+ files

i want to merge them into 1 .txt file

then remove all the duplicates lines from that 1 large .txt file combined which will be around 50GB txt file or 100GB txt file

so what can handle that kind of large file and remove the duplicates from it smoothly ?

i need the fastest way because i tryied bouth notepad++ and emeditor they work super heavy with them for merge or duplicates removing and take forever

i have 12GB RAM

marked as duplicate by Ramhound, Pimp Juice IT, Dave M, music2myear, Kamil Maciorowski Sep 29 '17 at 7:53

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • Scripting is probably going to be the fastest, but do note, working with files this large means this is going to take forever regardless. Therefor, getting the fastest method is really a matter of opinion. Its going to take more time to find the fastest than just to get it done. – LPChip Sep 15 '17 at 21:08
  • that didn't help me and i didn't understand anything of it and my question typecially different since i look for more large files its 10GB minimum and can go to 100GB and i have free space for work already over 300GB – DeathRival Sep 15 '17 at 21:09
  • ok i have found a way from How Does One Remove Duplicate Text Lines From Files Larger Than 4GB you can delete my question if you want what i found is: pilotedit.com/index.html thanks for who post it – DeathRival Sep 15 '17 at 21:44
  • Ramhound already pointed you to a good answer. But let me add a few things. You can join several text files together using the copy command. Open a command prompt, use cd to move to the folder with your text files and then type copy file1.txt + file2.txt combined_file.txt. That will join both files and will take about 3 seconds per GB if you are working on an SSD. It will be slower on a hard disk. – SpiderPig Sep 15 '17 at 21:47
  • The sort -u command mentioned in the other thread is also very fast and can handle 0.1 GB per second. – SpiderPig Sep 15 '17 at 21:55

If you are using Linux, you could do it like that:

cat aa.txt bb.txt | sort -u > newfile.txt

Here aa.txt is the first text file and bb.txt the second one.

sort -u sorts the file alphabetically and with -u (see also here https://stackoverflow.com/a/9377125/7311363) you're eliminating duplicates. With > newfile.txt you're writing that to newfile.txt.

  • 2
    And if you aren't using Linux, boot with a LiveCD/DVD and mount your drive and then do it. – ivanivan Sep 16 '17 at 0:19

Not the answer you're looking for? Browse other questions tagged or ask your own question.