Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

What is going to be my best way to search and remove duplicate's from over 50GBs worth of text files and merge them into one? I figure a GUI app will just hang so I need a CLI style with threading support or a way to use linux.

share|improve this question
    
Do you want each collection of duplicates merged together, or did you mean you want to merge all the remaining text files into one file at the end of this procedure? –  iglvzx May 5 '12 at 20:55
    
I would like to merge every text file into one without duplicates –  Rachel Nark May 6 '12 at 2:05
add comment

2 Answers

up vote 0 down vote accepted

Here is a post may help you but in small scale you can write your own script that way to handle huge number of files Also this post may help using the sort command

share|improve this answer
add comment

Auslogics Duplicate File Finder is free and should have no problems with 50GBs of text files. Make sure you download direct from the site to avoid the CNET downloader and don't accept the Auslogics toolbar. It's fairly intuitive and easy to get rid of duplicates.

To concatenate your text files in windows you could try this in a cmd window as admin:

copy *.txt bigfile.txt

Which will copy all your text files into one big file. No idea if this will work with the volume of files you have.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.