Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I am trying to cleanup a text and for some reason every line duplicated 3 times am i able to get ride of duplicates with regex or tricks or do you know a software which could do that , text file is like this

Party Started 10:17 (89/1/2)
Party Started 10:17 (89/1/2)
Party Started 10:17 (89/1/2)
Jessica At Dinner 17:54 (89/1/2)
Jessica At Dinner 17:54 (89/1/2)
Jessica At Dinner 17:54 (89/1/2)

How can i clean it up , and get ride of duplicated lines , it's about 69,587 lines

share|improve this question

3 Answers 3

up vote 8 down vote accepted

You could use uniq, standard with bash. Just type:

uniq filewithdup.txt > filenew.txt
share|improve this answer
    
Thanks works great :D –  echolab Jun 24 '12 at 11:03
    
You can get uniq in windows through gnuwin32, unxutils, or GOW - I prefer Gow, but gnuwin seems to have a more comprehensive toolset. –  Journeyman Geek Jun 24 '12 at 11:20

Since you mention MS Office, I'll give you a native Windows solution.

If you are using Windows Vista or later, there's Windows PowerShell built in. You can use the command Get-Unique:

The Get-Unique cmdlet compares each item in a sorted list to the next item, eliminates duplicates, and returns only one instance of each item. The list must be sorted for the cmdlet to work properly.

Get-Content input.txt | Get-Unique | Set-Content output.txt

If it's not sorted, you can use Sort-Object -Unique (it also works on already sorted input, but do not use if you do not wish to remove duplicates with other lines between them).

Get-Content input.txt | Sort-Object -Unique | Set-Content output.txt
share|improve this answer
    
I have both linux and windows , unix solution just worked but man i learned something , this powershell seems powerful , Thanks sir –  echolab Jun 24 '12 at 11:08
1  
@echolab PowerShell aliases many standard Linux and Windows commands, so you can do things like ls or dir and have it work. The arguments may be different, so man <command>, help <command> or Get-Help <command> are very useful. You can also use anything from the .NET Framework, so it is very powerful, but can be a little confusing. In this case, cat input.txt | unique > output.txt would also work. –  Bob Jun 24 '12 at 11:13
1  
You can also download PowerShell for XP –  Joe the Person Jun 24 '12 at 17:44

Regex was tagged, so:

/(.+)\n\1/g
share|improve this answer
    
nope, gets false positives. s/^(.*\n)\1+/\1/mg on the whole thing, perhaps? –  sreservoir Jun 24 '12 at 21:22
    
@sreservoir, Ah, definitely a good idea to look for multiple repeats. If you include the line break in the main match, then it won't match a repeat at the end of the file. Also, .* was giving me trouble and matching empty strings or word barriers, or something of the sort. Also, what do you mean about false positives? Can I have an example? –  mowwwalker Jun 24 '12 at 21:39

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.