0

Using test2.txt

$ cat /tmp/test2.txt | hexdump -C
00000000  47 61 6c 6c 6f 20 63 65  6c 74 69 63 6f 0a 47 65  |Gallo celtico.Ge|
00000010  73 97 20 42 61 6d 62 69  6e 6f 0a                 |s. Bambino.|

Here is what happened to awk:

$ cat /tmp/test2.txt | awk '/\x97/{print}'
Ges Bambino

$ cat /tmp/test2.txt | awk '/[\x7F-\xFF]/{next;}; 1'
Gallo celtico
Ges Bambino

I.e., the line Ges Bambino line contains a char \x97, and awk confirms that in the first command. However, \x97 is in the range of \x7F-\xFF so the line should be skipped in the second command, but it wasn't.

This really looks like an awk bug to me.
Any comments?

PS. making it more evident:

$ cat /tmp/test2.txt | awk '/\x97/{next}; 1'
Gallo celtico

$ cat /tmp/test2.txt | awk '/[\x97]/{next}; 1'
Gallo celtico
Ges Bambino

UPDATE:

As @KamilMaciorowski pointed out, it is local related. I.e. the above happens when under

$ set | egrep '^LANG|^LC'
LANG=zh_CN.UTF-8

whereas both

cat /tmp/test2.txt | LC_ALL=C awk '/[\x97]/{next}; 1'
cat /tmp/test2.txt | LC_ALL=C awk '/[\x7F-\xFF]/{next;}; 1'

are giving correct results.

Still, this is an issue, is it?

  • Is it the same when you run LC_ALL=C awk …? Also you don't need cat here, I think. – Kamil Maciorowski Jul 22 '17 at 15:59
  • I cannot replicate the problem; it may be related to your locale. What's the output of set | egrep '^LANG|^LC'? – Kamil Maciorowski Jul 22 '17 at 16:08
  • @KamilMaciorowski: A true follower of Unix philosophy would instead praise one for usage of pipes, no? – grawity Jul 22 '17 at 16:17
  • @KamilMaciorowski, indeed, it is locale related. OP updated. – xpt Jul 22 '17 at 20:07
2

Bytes and characters are only identical in the default C (aka POSIX) locale, but in all other locales they're different. So if your system uses an UTF-8 locale (e.g. en_US.UTF-8) by default, then bracketed "character classes" in gawk regexes are based on characters, not individual bytes.

For example, /[eęė]/ is equivalent to /[e\xC4\x99\xC4\x97]/ assuming an *.UTF-8 locale; however, it will match the letter ę but not č, despite both having a C4 byte. (For some reason, this is different from a plain /\xC4/ outside a character class, which does match the literal byte C4.)

Anyway, the same applies to ranges inside character classes, and since byte FF doesn't make a valid UTF-8 sequence, the regex library might just declare the whole range invalid or something.

gawk has the -b, --characters-as-bytes option to disable this.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.