I've written myself a linux program program that needs a regular expression as input.

I want to call the program in the bash shell and pass that regular expression as a command line argument to the program (there are also other command line arguments). A typical regular expression looks like

[abc]\_[x|y]

Unfortunately the characters [, ], and | are special characters in bash. Thus, calling

program [abc]\_[x|y] anotheragument

doesn't work. Is there a way to pass the expression by using some sort of escape characters or quotation marks etc.?

(Calling program "[abc]\_[x|y] anotheragument" isn't working either, because it interprets the two arguments as one.)

up vote 22 down vote accepted

You can either

  1. Escape each single special symbol with a backslash (as in \[abc\]_\[x\|y\]) or
  2. Doublequote the entire argument (as in "[abc]_[x|y]").

EDIT: As some have pointed out, dobleqouting does not prevent variable expansion nor command substitution. Therefore if your regex contains something that can be interpreted by bash as one of those, use single quotes instead.

  • 4
    In bash, double-quoting does not bypass expanding variables "$HOME" or parameters "${USER:-root}", command substitution in either form "$(date)" or "`date`", arithmetic expansion "$((1 + 2))", history expansion "!!" or backslash escaping "\\". Use single quotes instead. See the man page of the bash manual, the section titled "Quoting". – Flimm Dec 3 '13 at 11:29

Use single quotes. Single quotes ensure that none of the characters are interpreted.

$ printf %s 'spaces  are  not  interpreted away
neither are new lines
nor variable names $TESTING
nor square brackets [TESTING]
nor pipe characters or redirection symbols | > <
nor the semicolon ;
nor backslashes \a \b \c \\
the only thing that does not work is the single quote itself
'

There are two solutions if you need to embed a single quote:

$ printf '%s\n' '[ Don'"'"'t worry, be happy! ]'
[ Don't worry, be happy! ]
$ printf '%s\n' '[ Don'\''t worry, be happy! ]'
[ Don't worry, be happy! ]
  • Right you are. +1 – U-D13 Dec 8 '13 at 11:03

Per man bash

There are three quoting mechanisms: the escape character, single quotes, and double quotes.

A non-quoted backslash (\) is the escape character. It preserves the literal value of the next character that follows, with the exception of <newline>. If a \<newline> pair appears, and the backslash is not itself quoted, the \<newline> is treated as a line continuation (that is, it is removed from the input stream and effectively ignored).

Enclosing characters in single quotes preserves the literal value of each character within the quotes. A single quote may not occur between single quotes, even when preceded by a backslash.

Enclosing characters in double quotes preserves the literal value of all characters within the quotes, with the exception of $, `, \, and, when history expansion is enabled, !. The characters $ and ` retain their special meaning within double quotes. The backslash retains its special meaning only when followed by one of the following characters: $, `, ", \, or <newline>. A double quote may be quoted within double quotes by preceding it with a backslash. If enabled, history expansion will be performed unless an ! appearing in double quotes is escaped using a backslash. The backslash preceding the ! is not removed.

The special parameters * and @ have special meaning when in double quotes (see PARAMETERS below).

Words of the form $'string' are treated specially. The word expands to string, with backslash-escaped characters replaced as specified by the ANSI C standard. Backslash escape sequences, if present, are decoded as follows:

       \a     alert (bell)
       \b     backspace
       \e
       \E     an escape character
       \f     form feed
       \n     new line
       \r     carriage return
       \t     horizontal tab
       \v     vertical tab
       \\     backslash
       \'     single quote
       \"     double quote
       \nnn   the eight-bit character whose value is the octal value nnn
              (one to three digits)
       \xHH   the eight-bit character whose value is the hexadecimal value HH
              (one or two hex digits)
       \uHHHH the Unicode (ISO/IEC 10646) character whose value is
              the hexadecimal value HHHH (one to four hex digits)
       \UHHHHHHHH
              the Unicode (ISO/IEC 10646) character whose value is
              the hexadecimal value HHHHHHHH (one to eight hex digits)
       \cx    a control-x character

The expanded result is single-quoted, as if the dollar sign had not been present.

A double-quoted string preceded by a dollar sign ($"string") will cause the string to be translated according to the current locale. If the current locale is C or POSIX, the dollar sign is ignored. If the string is translated and replaced, the replacement is double-quoted.

You can use a backslash ( \ ) in front of special characters to escape them like so:

john@awesome:~ # echo \&
&

Although it might not be useful as a regex, some character sequences may be interpreted as Bash variable names. To prevent this from occurring and avoid having them expanded, use single quotes instead of double quotes:

program '[abc]_[x|y]' anotherargument

Quote each argument separately (if they need quoting) so they are interpreted as independent arguments. You can also use arrays in some cases:

param_array=('[abc]_[x|y]' anotherargument)    # create an array
param_array+=(yetanother)     # append another element to the array
program "${param_array[@]}"   # use the array elements as arguments to program
program "[abc]_[x|y]"
program "[abc]_[x|y]" anotherargument

Escaping them should work fine:

  programm \[abc\]_\[x\|y\]

Where does the pattern come from? Is it fixed or from a user? Is it the user who is invoking the script on the local system, or someone remote?

You use quotes to wrap data to keep the shell from interpreting it. There are two options:

  1. Double-quotes, which still permit some interpretation ($expand and `backticks`)
  2. Single-quotes, which pass everything through literally

Because $ is a valid character in regexps (end-of-line/buffer) you probably want to use single-quotes to hold the regexp, unless you're storing in in a variable. If you're taking arbitrary data from someone untrusted, you'll need to replace ' with '"'"' and then wrap in single-quotes.

Note that [abc]_[x|y] looks like you want to match x or y, while it's actually matching one of the three characters xy|. The square brackets match the characters within and only - for ranges and a ^ at the start for negation. So, [abc]_(x|y) might be what you meant, and the parentheses are the characters which are special to shell. Square-brackets are not special to shell, it just looks like they are. Double-square brackets [[ ... ]] are special.

  • This is one of the most correct answers here (I appreciate especially the instruction for replacing ' with '"'"'), however, it is still not correct. [ IS a special character to shell, it is used in wildcards when doing path-expansion (which shell does for everything unquoted). – jpalecek Oct 21 '11 at 12:32
  • It's special in some contexts, such as variable subscripting or for globbing, but you can still type foo=a[b] and then echo $foo and see that the string did not need quoting. You're right, I was too brief. – Phil P Oct 22 '11 at 4:38
  • If you are unlucky, there is a file ab in the current directory, and then foo will contain ab rather than a[b]. Quote your square brackets, people. – clacke Jul 20 '16 at 5:41
  • (For clarity: I do quote (as the original answer made clear, where I was pushing for quoting), and this is a side-derailment which I'm addressing). This assertion surprised me, so I tested it. It's not true in zsh or bash, but is true in BSD /bin/sh. This is against POSIX and is non-standard behavior, so you'll need to quote to handle it. In zsh, you can setopt glob_assign to enable this behavior too, so quoting is the safest answer. – Phil P Apr 20 at 16:07

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.