Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

What is going to be my best way to search and remove duplicate's from over 50GBs worth of text files and merge them into one? I figure a GUI app will just hang so I need a CLI style with threading support or a way to use linux.

share|improve this question
Do you want each collection of duplicates merged together, or did you mean you want to merge all the remaining text files into one file at the end of this procedure? – iglvzx May 5 '12 at 20:55
I would like to merge every text file into one without duplicates – Rachel Nark May 6 '12 at 2:05
up vote 0 down vote accepted

Here is a post may help you but in small scale you can write your own script that way to handle huge number of files Also this post may help using the sort command

share|improve this answer

Auslogics Duplicate File Finder is free and should have no problems with 50GBs of text files. Make sure you download direct from the site to avoid the CNET downloader and don't accept the Auslogics toolbar. It's fairly intuitive and easy to get rid of duplicates.

To concatenate your text files in windows you could try this in a cmd window as admin:

copy *.txt bigfile.txt

Which will copy all your text files into one big file. No idea if this will work with the volume of files you have.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .