Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am trying to cleanup a text and for some reason every line duplicated 3 times am i able to get ride of duplicates with regex or tricks or do you know a software which could do that , text file is like this

Party Started 10:17 (89/1/2)
Party Started 10:17 (89/1/2)
Party Started 10:17 (89/1/2)
Jessica At Dinner 17:54 (89/1/2)
Jessica At Dinner 17:54 (89/1/2)
Jessica At Dinner 17:54 (89/1/2)

How can i clean it up , and get ride of duplicated lines , it's about 69,587 lines

share|improve this question
up vote 8 down vote accepted

You could use uniq, standard with bash. Just type:

uniq filewithdup.txt > filenew.txt
share|improve this answer
Thanks works great :D – echolab Jun 24 '12 at 11:03
You can get uniq in windows through gnuwin32, unxutils, or GOW - I prefer Gow, but gnuwin seems to have a more comprehensive toolset. – Journeyman Geek Jun 24 '12 at 11:20

Since you mention MS Office, I'll give you a native Windows solution.

If you are using Windows Vista or later, there's Windows PowerShell built in. You can use the command Get-Unique:

The Get-Unique cmdlet compares each item in a sorted list to the next item, eliminates duplicates, and returns only one instance of each item. The list must be sorted for the cmdlet to work properly.

Get-Content input.txt | Get-Unique | Set-Content output.txt

If it's not sorted, you can use Sort-Object -Unique (it also works on already sorted input, but do not use if you do not wish to remove duplicates with other lines between them).

Get-Content input.txt | Sort-Object -Unique | Set-Content output.txt
share|improve this answer
I have both linux and windows , unix solution just worked but man i learned something , this powershell seems powerful , Thanks sir – echolab Jun 24 '12 at 11:08
@echolab PowerShell aliases many standard Linux and Windows commands, so you can do things like ls or dir and have it work. The arguments may be different, so man <command>, help <command> or Get-Help <command> are very useful. You can also use anything from the .NET Framework, so it is very powerful, but can be a little confusing. In this case, cat input.txt | unique > output.txt would also work. – Bob Jun 24 '12 at 11:13
You can also download PowerShell for XP – Joe the Person Jun 24 '12 at 17:44

Regex was tagged, so:

share|improve this answer
nope, gets false positives. s/^(.*\n)\1+/\1/mg on the whole thing, perhaps? – muhmuhten Jun 24 '12 at 21:22
@sreservoir, Ah, definitely a good idea to look for multiple repeats. If you include the line break in the main match, then it won't match a repeat at the end of the file. Also, .* was giving me trouble and matching empty strings or word barriers, or something of the sort. Also, what do you mean about false positives? Can I have an example? – mowwwalker Jun 24 '12 at 21:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .