15

I had this strange behavior this morning in a bash terminal :

user@home:/home/user$ [ -f /etc/openvpn/client.conf ] && echo true
bash: [: missing «]»
user@home:/home/user$ [ -f /etc/openvpn/client.conf ] && echo true
true
  • The first command was pasted from a script edited with gedit.
  • The second was typed directly in the terminal.

After some digging, I find out that removing the 30th character (the space between client.conf and "]") and replacing it with a space made the command work again.

My assumption was right : an unknown blank character has slipped into the command, but the question is :

  1. How can I reveal those characters in the terminal so that I can debug the command? And more important:
  2. How can I prevent this from happening again?

BTW, I running Ubuntu 18.04 / French language, the script that I paste the command from is in a USB drive and may have been edited on Windows too.


Thank you for your very good answers. The bad character is a c2 a0 non-breaking space UTF-8 character. The question How to remove special 'M-BM-' character with sed has interesting fact about that character.

The strange thing is that the script is free of this character. So I don't know where it came from.

  • 3
    Use an editor that highlights such characters. Syntax highlighting helps a lot, too. Never paste directly from the web to the terminal, always go through the aforementioned editor. – choroba May 16 '18 at 9:15
  • 2
    Yo might want to find the problem command in your history list, then pipe the output through a hex display program. So that you don't need to wade through a long listing, either rerun the command to place it at the bottom of the history list and run history 2|xxd (because the history command itself is always the last in the list), or type history|grep "CommandWithProblem"|xxd. You can use any other hex display program instead of xxd, but this defaults to a format I like. – AFH May 16 '18 at 10:12
  • @Gabriel Glenn, please mark the best / most helpful / whatever answer as "accepted" using the tick - rather than commenting on each that the answer helped. info – Attie May 16 '18 at 12:44
  • 1
    @Attie, Yes I will, I just usually wait 24 hours before accepting the best answers, as suggested in : meta.stackexchange.com/questions/5234/… – Gabriel Glenn May 16 '18 at 14:22
  • 1
    Personally I'd use set -x. This would show you the command & how it's split. It wouldn't necessarily say "bad character here", but it would show you that bash wasn't splitting on that character. – Patrick May 17 '18 at 2:04
11

One option is to look at the characters you're trying to use with a hex viewer or editor. hexdump is a good option if you are limited to the terminal.

$ hexdump -Cv <<"EOF"
> [ -f /etc/openvpn/client.conf ] && echo true
> EOF
00000000  5b 20 2d 66 20 2f 65 74  63 2f 6f 70 65 6e 76 70  |[ -f /etc/openvp|
00000010  6e 2f 63 6c 69 65 6e 74  2e 63 6f 6e 66 20 5d 20  |n/client.conf ] |
00000020  26 26 20 65 63 68 6f 20  74 72 75 65 0a           |&& echo true.|
0000002d

You can see here that the space, close-square-brace, space are correct - 0x20, 0x5D, 0x20.

These values are ASCII codes, displayed in hexadecimal. Any value outside the range 0x20 - 0x7E is not a "printable character" as far as ASCII is concerned, and most likely won't play well with command line interfaces.

Note: I copied your first "broken" line for use in the hexdump example above, so something has replaced the not-an-ASCII-space with an ASCII space between your original source and your rendered question.


To repeat this, take the following steps:

  1. Type hexdump -Cv <<"EOF" and press Enter
  2. Paste the text you would like to use
  3. Type EOF on a line of its own, and press Enter

Terminals and Command Line Interfaces don't handle special characters well - as you have discovered. If you aren't very careful with formatting documents, you will also have problems with Microsoft Word (and others) using "smart quotes", em-dashes, the list goes on...

Spot the difference: (the top is "smart quotes", the bottom is "straight quotes")

example of smart quotes vs straight quotes

$ hexdump -Cv <<"EOF"
> “quoted string”
> EOF
00000000  e2 80 9c 71 75 6f 74 65  64 20 73 74 72 69 6e 67  |...quoted string|
00000010  e2 80 9d 0a                                       |....|
00000014

Here, the open quotes are not a simple ASCII quote ("), but are a Unicode / UTF-8 series - 0xE2, 0x80, 0x9C, or U+201C - which the terminal will not handle as you might expect.

Kiwy's suggestion of cat -A also does the job:

$ cat -A <<"EOF"
> “quoted string”
> EOF
M-bM-^@M-^\quoted stringM-bM-^@M-^]$

Note: when using echo "..." | hd, you stand a chance that bash will substitute parts of the string you are trying to inspect. This is particularly of concern when trying to inspect components of a script.

For example try:

$ echo "${USER}"
attie

$ echo "`whoami`"
attie

$ echo "$(whoami)"
attie

$ cat <<EOF
> ${USER}
> EOF
attie

These methods are replacing components with the relevant text. To avoid this, use one of the following approaches. Note the use of single quotes ('), and a "quoted heredoc" ("EOF").

$ echo '${USER}'
${USER}

$ echo '`whoami`'
`whoami`

$ echo '$(whoami)'
$(whoami)

$ cat <<"EOF"
> ${USER}
> EOF
${USER}
| improve this answer | |
  • This solution works : echo "[ -f /etc/openvpn.ovpn ]" | hd returns [...] c2 a0 [...]. We can see the c2 a0 UT-8 character non-breaking space – Gabriel Glenn May 16 '18 at 11:38
18

You could use cat with the -A option: from the manual:

   -A, --show-all
          equivalent to -vET
   -E, --show-ends
          display $ at end of each line
   -T, --show-tabs
          display TAB characters as ^I
   -v, --show-nonprinting
          use ^ and M- notation, except for LFD and TAB

So cat -A yourscrip.sh will show you invisible and strange characters.

| improve this answer | |
  • 7
    This solution works : echo "[ -f /etc/openvpn.ovpn ]" | cat -A returns [ -f /etc/openvpn/client.ovpnM-BM- ]$. We can see the M-BM- UT-8 character non-breaking space – Gabriel Glenn May 16 '18 at 11:36
  • @GabrielGlenn glad this helped you. – Kiwy May 16 '18 at 11:37
9

echo "<your command>" | hd should work. Look for backspace (0x08) or characters with codes >=80. echo "<your command>" | wc -b and checking that the count matches what you see is also a good idea.

Copying stuff from files produced by anything with "Office" in its name is dangerous, because such software often takes the liberty to replace characters: in French, look out for double quotes replaced by "guillemets", in English for plain quotes replaced by their open/close equivalents. The hardest one I ever found was a 0-width non-breaking space in the middle of a file name (3 days of server downtime...).

| improve this answer | |
  • 2
    It's worth mentioning hd is short for hexdump which is also mentioned in Attie's answer. – Mikael Kjær May 16 '18 at 9:41
  • @MikaelKjær - On Ubuntu, hd is equivalent to hexdump -C. – AFH May 16 '18 at 10:06
  • 1
    @xenoid : I said 'edited on Windows', not edited with a Office Writer, we are not crazy ;). If it was edited, it was with Notepad++. – Gabriel Glenn May 16 '18 at 10:37
  • 1
    This solution works : echo "[ -f /etc/openvpn.ovpn ]" | hd returns [...] c2 a0 [...]. We can see the c2 a0 UT-8 character non-breaking space – Gabriel Glenn May 16 '18 at 11:39
2

Bash, and other shells like zsh, can open the current command line in an editor. The default shortcut for bash is C-x C-e (CtrlX CtrlE), and it opens in the first available of $VISUAL, $EDITOR and emacs. In practice this is invaluable for debugging and modifying complex commands. Depending on how you look at it, zsh is more friendly than bash here: when the editor exits, bash runs the command immediately, whereas zsh waits for you to press Enter (giving you more chances to edit the command).

After opening the command in an editor, you can configure your editors to show non-ASCII characters differently.

For example, with Vim, using these settings:

set encoding=latin1
set isprint=
set display+=uhex

enter image description here

Or, adapting the other answers' methods:

bash-4.4$ f() { cat -A "$@"; false; }   # exit false to prevent bash from running the command
bash-4.4$ VISUAL=f
bash-4.4$ [ -f /etc/openvpn/client.conf ] && echo true  # C-x C-e here
[ -f /etc/openvpn/client.confM-BM- ] && echo true$
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.