-1

simple my question is different because i need to merge the files into one too then remove the duplicates lines from the that file which will be over 50GB txt i have large .txt from 10GB+ files

i want to merge them into 1 .txt file

then remove all the duplicates lines from that 1 large .txt file combined which will be around 50GB txt file or 100GB txt file

so what can handle that kind of large file and remove the duplicates from it smoothly ?

i need the fastest way because i tryied bouth notepad++ and emeditor they work super heavy with them for merge or duplicates removing and take forever

i have 12GB RAM

5
  • Scripting is probably going to be the fastest, but do note, working with files this large means this is going to take forever regardless. Therefor, getting the fastest method is really a matter of opinion. Its going to take more time to find the fastest than just to get it done.
    – LPChip
    Sep 15, 2017 at 21:08
  • that didn't help me and i didn't understand anything of it and my question typecially different since i look for more large files its 10GB minimum and can go to 100GB and i have free space for work already over 300GB
    – user677589
    Sep 15, 2017 at 21:09
  • ok i have found a way from How Does One Remove Duplicate Text Lines From Files Larger Than 4GB you can delete my question if you want what i found is: pilotedit.com/index.html thanks for who post it
    – user677589
    Sep 15, 2017 at 21:44
  • Ramhound already pointed you to a good answer. But let me add a few things. You can join several text files together using the copy command. Open a command prompt, use cd to move to the folder with your text files and then type copy file1.txt + file2.txt combined_file.txt. That will join both files and will take about 3 seconds per GB if you are working on an SSD. It will be slower on a hard disk.
    – Tesseract
    Sep 15, 2017 at 21:47
  • The sort -u command mentioned in the other thread is also very fast and can handle 0.1 GB per second.
    – Tesseract
    Sep 15, 2017 at 21:55

1 Answer 1

2

If you are using Linux, you could do it like that:

cat aa.txt bb.txt | sort -u > newfile.txt

Here aa.txt is the first text file and bb.txt the second one.

sort -u sorts the file alphabetically and with -u (see also here https://stackoverflow.com/a/9377125/7311363) you're eliminating duplicates. With > newfile.txt you're writing that to newfile.txt.

1
  • 2
    And if you aren't using Linux, boot with a LiveCD/DVD and mount your drive and then do it.
    – ivanivan
    Sep 16, 2017 at 0:19

Not the answer you're looking for? Browse other questions tagged .