Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I found that the sort result in ASCII:

Source file test:

1-
11-
1-a
11-a

Sort using ASCII:

$ LANG=en_US.ascii sort test
1-
1-a
11-
11-a

And using UTF-8:

$ LANG=en_US.utf8 sort test
1-
11-
11-a
1-a

I feel it's so counter-intuitive, and it's not dictionary order.

Isn't the character '-' (002d) is always less then [0-9] (0030-0039)? What's the general rule in UTF-8 collation?

And how to bypass it, just make - be less then [0-9] while keep other characters unchanged for UTF-8, in Linux? (So it can affects the result of ls --sort, sort, etc. )

share|improve this question
3  
Where precisely are you seeing this? With sort 8.5 from GNU coreutils, "1-" always comes before "11-", with any locale. –  grawity Jan 1 '11 at 15:17
    
It's my mistake. I have truncated the strings. I changed the example please try again. –  Xiè Jìléi Jan 1 '11 at 18:39
    
@grawity I see this on gmail when I open zip files. I see this in Win7 with images: 11, 12, 13, ..., 19, 1. –  Wolfpack'08 Jul 23 at 17:23

2 Answers 2

up vote 6 down vote accepted

The minus sign is ignored in the first pass. So the first pass sorts 1, 11, 1a, 11a. Since 1 < a, you get 11a < 1a and thus 11-a < 1-a.

- is a variable collation element, meaning that you/the implementor can choose to ignore it. The glibc implementation apparently does so. In practice, most punctuation is affected by this behavior.

You can read up on the gory details in the Unicode Collation Algorithm, modulo how glibc implements it.

share|improve this answer
    
Then, is there any configuration to glibc to suppress this ignorance? –  Xiè Jìléi Jan 2 '11 at 17:53
    
Not that I'm aware of. –  Peter Eisentraut Jan 2 '11 at 19:05

As explained by Peter Eisentraut, this is because the sorting algorithm for Unicode ignores - when sorting.

The only way around this is to define your own locale, with different collation (sorting rules). This is however rather non-trivial. Also, it would give you a system with unusual sorting rules, which may cause problems with other software.

So realistically, you'll either have to switch your locale to ASCII (if you don't need Unicode character), or sort using a program where you can configure the sorting rules directly.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.