Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I've a wordlist which contains, username,phonenumber and email. It's gathered from various sources therefore in variable sizes. I need to filter the duplicate username and all things in a row.

So far I've used the method using Notepad++. But the limitations are it fill filter only one file at a time. And it can't handle files like 500MB.

so if a file has username-useremail@example.com-someIntegerPhoneNumber here means the same should not appear in another file.

In simple I need to achieve the result above by Notepad++ for multiple files which are HIGHER than 500 MB.

Any tools or programs? Or any efficient Java or C# snippet?

share|improve this question
1  
I could give you a C# or even PowerShell snippet (though 500MB may cause you to run out of memory), but do you want to combine the files into one, or do you have some way to determine which file will keep the duplicate? There may also be an existing program. Once again, 500MB may cause issues. –  Bob May 19 '12 at 7:28
    
@Bob - If its possible to process 100 MB file means, I can split the bigger ones to 100 and process them. The first loaded file is master file and the second file should be clear of the first one's content. –  Jones May 19 '12 at 7:42

1 Answer 1

Here's a C# program that does.. something like what you asked. I'm actually not 100% sure what you want.

Usage is: program.exe "outputfolder" "file1.txt" "file2.txt" "file3.txt"

It rewrites the listed files in the output folder, processed in the order specified. If a username has been encountered in any line or file before, it will skip the line. It doesn't check email or phone number in any way.

using System;
using System.Collections.Generic;
using System.IO;

namespace CreateUniqueFile
{
    class Program
    {
        static void Main(string[] args)
        {
            string fullpath;
            string outpath;
            List<string> files = new List<string>();

            for (int i = 1; i < args.Length; i++)
            {
                fullpath = Path.GetFullPath(args[i]);
                if (!File.Exists(fullpath))
                    Console.WriteLine("File not found: {0}", fullpath);
                else
                {
                    files.Add(fullpath);
                    Console.WriteLine(fullpath);
                }
            }

            if (files.Count == 0)
            {
                Console.WriteLine("No files provided!");
                return;
            }
            else
            {
                outpath = Path.GetFullPath(args[0]);
                Console.WriteLine("Output will go to folder: \"{0}\"", outpath);
                Console.WriteLine("Process files in the above order? (Y/N)");
                bool yes = false;
                while (!yes)
                {
                    switch (Console.ReadKey().Key)
                    {
                        case ConsoleKey.Y:
                            yes = true;
                            break;
                        case ConsoleKey.N:
                            return;
                        default:
                            break;
                    }
                }
            }

            if (!Directory.Exists(outpath))
                Directory.CreateDirectory(outpath);

            HashSet<string> seennames = new HashSet<string>();

            string line, username;

            foreach (string path in files)
            {
                string writepath = outpath + '\\' + Path.GetFileName(path);
                if (File.Exists(writepath))
                {
                    writepath = outpath + '\\' + Path.GetFileNameWithoutExtension(path) + " (2)" + Path.GetExtension(path);
                    // Dodgy thing to increment the number, don't touch!
                    while (File.Exists(writepath))
                    {
                        writepath = writepath.Substring(0, writepath.LastIndexOf('(') + 1) +
                            (Int32.Parse(writepath.Substring(writepath.LastIndexOf('(') + 1, writepath.LastIndexOf(')') - writepath.LastIndexOf('(') - 1)) + 1) +
                            writepath.Substring(writepath.LastIndexOf(')'));
                    }
                }

                using (StreamWriter writer = new StreamWriter(writepath))
                {
                    using (StreamReader reader = new StreamReader(path))
                    {
                        while (!reader.EndOfStream)
                        {
                            line = reader.ReadLine();
                            username = line.Split('-')[0];
                            if (!seennames.Contains(username))
                            {
                                writer.WriteLine(line);
                                seennames.Add(username);
                            }
                        }

                        Console.WriteLine("{0} processed, output to {1}", path, writepath);
                    }
                }
            }
        }
    }
}
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.