Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

i'm using notepad++ and sed on Windows.
I have a file text 2000+ lines but sometimes i have this rows:

http://www.shragle.com/files/9baa908b/Bvdub-The_First_Day-%2528HN031%2529-2012.rar
http://www.shragle.com/files/9baa908b/Bvdu...1%2529-2012.rar
http://rapidshare.com/files/152133956/2005_-_Candlemass.part1.rar
http://rapidshare.com/files/152133956/2005...emass.part1.rar
http://www.filefactory.com/file/a181d18/n/...nimal_2010_.rar
http://rapidshare.com/files/152136400/2005_-_Candlemass.part2.rar
http://rapidshare.com/files/152136400/2005...emass.part2.rar
http://www.shragle.com/files/3e0f3488/Cesa...testHits2CD.rar

How can i remove this rows?

http://www.shragle.com/files/9baa908b/Bvdu...1%2529-2012.rar
http://rapidshare.com/files/152133956/2005...emass.part1.rar
http://rapidshare.com/files/152136400/2005...emass.part2.rar

I don't want remove this rows:

http://www.shragle.com/files/3e0f3488/Cesa...testHits2CD.rar
http://www.filefactory.com/file/a181d18/n/...nimal_2010_.rar

because this rows are unique and valid for me, i want remove only similar rows

Similar rows, i want delete the ... line

http://rapidshare.com/files/152133956/2005_-_Candlemass.part1.rar
http://rapidshare.com/files/152133956/2005...emass.part1.rar

unique row because in my text file is presented only one time:

http://www.filefactory.com/file/a181d18/n/...nimal_2010_.rar
share|improve this question
    
possible duplicate of Deleting Duplicated Lines In TEXT File? –  Bob Jul 9 '12 at 17:33
    
There's no NPP solution in the possible duplicate, but there's some regex that may work in sed. There's also a native Linux (with Windows ports) and native Windows solution that are easier. –  Bob Jul 9 '12 at 17:35
3  
If you only want to remove similar rows as opposed to exact duplicates, could you describe your criteria for removal? What qualifies a row for removal? –  Bob Jul 9 '12 at 17:36
    
mmm..if u look if you look closely i want remove not duplicate lines, but a type of lines. I want to delete the lines that are not really unique. shragle.com/files/9baa908b/Bvdu is present on two lines then the 2nd row continues with ... I want eliminate ... rows –  user143822 Jul 9 '12 at 17:42
    
How many characters at the start of two lines have to be equal so you consider them similar? –  Dennis Jul 9 '12 at 17:45
show 3 more comments

1 Answer

up vote 0 down vote accepted

The following will delete a line if its first 40 characters are identical to those of a former:

Find what:         ((?<=\n)|^)([^\n]{40})([^\n]*\n)(.*\n)?\2[^\n]*($|\n)
Replace with:      \2\3\4
Search Mode:       Regular expression
. matches newline: checked

You might have to click Replace All more than once.

How it works

  • ((?<=\n)|^) checks if the rest of the regular expression is preceded by a newline or the beginning of the file.

  • ([^\n]{40})([^\n]*\n) matches the first 40 characters of a line (stored in \2) and the rest of it (stored in \3).

  • (.*\n)? is any number of lines (stored in \4).

  • \2[^\n]* is a line beginning with the same 40 characters.

  • ($|\n) checks if the rest of the regular expression is followed by a newline or the end of the file.

If this pattern is encountered, there's a line similar to \2\3.

By replacing all by \2\3\4, that similar line gets deleted.

share|improve this answer
    
But i write this regex ((?<=\n)|^)([^\n]{40})([^\n]*\n)(.*\n)\2[^\n]*($|\n) on Notepad++ or sed? On notepad++ doesn't work –  user143822 Jul 9 '12 at 18:11
    
I wrote it for Notepad++ with the new regex engine. The expression was missing a ?. It should work now. –  Dennis Jul 9 '12 at 18:21
    
thank you!, now it works..but it works only if the similar rows are one below the other. Is a regex to detect this type of string in random order? –  user143822 Jul 9 '12 at 18:28
    
yeah,the regex works well, but How I find these similar lines when these lines are in scattered order ?. How do I detect them? Which regex must i use for similar strings in random order? –  user143822 Jul 9 '12 at 19:08
    
Dennis, Look here please: img507.imageshack.us/img507/8539/regexproblem.png –  user143822 Jul 9 '12 at 19:33
show 3 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.