5

One of most common typos is to repeat the same word twice, as as here. I need an automatic procedure to remove all the repeated words in a text file. This should not be a strange feature for a modern editor or spell-checker, for example I remember that MS Word introduced this feature several years ago! Apparently, the default spell-check on my OS (hun-spell) can't do this, as it only finds words not in the dictionary.

It would be OK to have a solution valid for a specific text editor editor for linux (pluma/gedit2 or Sublime-text) and a solution based on a bash script.

  • 1
    Is perl an acceptable alternative to bash? Because that'd be my first port of call. – Sobrique Nov 22 '14 at 23:01
  • @Sobrique Please, feel free to add it! I would favor bash-based answers though – altroware Nov 23 '14 at 1:13
10

With GNU grep:

echo 'Hi! Hi, same word twice twice, as as here here! ! ,123 123 need' |  grep -Eo '(\b.+) \1\b'

Output:

twice twice
as as
here here
123 123

Options:

-E: Interpret (\b.+) \1\b as an extended regular expression.

-o: Print only the matched (non-empty) parts of a matching line, with each such part on a separate output line.

Regex:

\b: Is a zero-width word boundary.

.+: Matches one or more characters.

\1: The parentheses () mark a capturing group and \1 means use here the value from first capturing group.


Reference: The Stack Overflow Regular Expressions FAQ

  • Your grep command fails for the following type of example: echo "the thesis" | grep -Eo '(\b.+\b) \1' outputs: the the. grep -Eo '(\b.+) \1\b' seems to work though. Any idea why? – el_tenedor Mar 24 '15 at 14:38
  • @el_tenedor: Thank you for this hint. I've added a \b after second string to fix this substring bug. – Cyrus Mar 24 '15 at 18:21
  • Still, the second \b in the parenthesis seems to be redundant. Why do we need it? – el_tenedor Mar 27 '15 at 16:16
  • Yes, it's redundant and may be omitted. – Cyrus Mar 27 '15 at 17:42
  • I've updated my answer. – Cyrus May 9 '15 at 7:05
1

Perlishly, I'd be thinking:

use strict;
use warnings;

local $/;

my $slurp = <DATA>;
$slurp =~ s/\b(\w+)\W\1/$1/go;
print $slurp;

__DATA__
Hi! Hi, same same? word twice twice, as as here here! ! ,123 123 need
need as here 

Bear in mind though - a lot of pattern matching is line oriented, so you've got to be careful if you cross line boundaries. If you can exclude that case, then you've got an easier job because you can parse one line at a time. I'm not doing that, so you'll end up reading the whole file into memory.

  • That's great, I preferred bash-based answer, but this is OK as well. – altroware Dec 15 '14 at 20:32
  • Perl is in nearly as many places as bash, and is more fully featured as a programming language. – Sobrique Dec 15 '14 at 20:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.