In utf-8 collation, why 11- is less then 1-?

Question

I found that the sort result in ASCII:

Source file test:

1-
11-
1-a
11-a

Sort using ASCII:

$ LANG=en_US.ascii sort test
1-
1-a
11-
11-a

And using UTF-8:

$ LANG=en_US.utf8 sort test
1-
11-
11-a
1-a

I feel it's so counter-intuitive, and it's not dictionary order.

Isn't the character '-' (002d) is always less then [0-9] (0030-0039)? What's the general rule in UTF-8 collation?

And how to bypass it, just make - be less then [0-9] while keep other characters unchanged for UTF-8, in Linux? (So it can affects the result of ls --sort, sort, etc. )

Answer 1

The minus sign is ignored in the first pass. So the first pass sorts 1, 11, 1a, 11a. Since 1 < a, you get 11a < 1a and thus 11-a < 1-a.

- is a variable collation element, meaning that you/the implementor can choose to ignore it. The glibc implementation apparently does so. In practice, most punctuation is affected by this behavior.

You can read up on the gory details in the Unicode Collation Algorithm, modulo how glibc implements it.

Answer 2

As explained by Peter Eisentraut, this is because the sorting algorithm for Unicode ignores - when sorting.

The only way around this is to define your own locale, with different collation (sorting rules). This is however rather non-trivial. Also, it would give you a system with unusual sorting rules, which may cause problems with other software.

So realistically, you'll either have to switch your locale to ASCII (if you don't need Unicode character), or sort using a program where you can configure the sorting rules directly.

Answer 3

The Unicode Collation Algorithm is thoroughly broken. The committee that designed it totally ignored the usual dictionary rules. For example "abc def" will sort after "abca", not before. It also fails basics like distinguishing between non-identical characters -- I'd be fine with considering Å (A-ring) and Å (angström sign), or A (basic letter) and Ａ (CJK "fullwidth") identical, but "#" and "&"??? -- come on.

And glibc folks have chosen to follow the committee rather than common sense.

If your language is English, you're lucky -- you may set LC_COLLATE=C to get correct collation, although you might be unhappy with it being case sensitive. It will also sort all UTF-8 strings according to their Unicode code points.

For other languages, though, national characters will not be sorted correctly.