0

I've always assumed that sort without any additional arguments, will sort a file in lexical order. However today I ran across following test case:

sort test2.txt
a/a
a/c
a//c
a/d

As you can see the third line has two slashes, so the second slash is in a place where neighboring lines have letters 'c' and 'd' respectively. I doubt that '/' comes between 'c' and 'd' in any code-page, so I guess that the default comparison algorithm is not strictly lexical.

Is there some pre-processing (like removing non-letters?) or special cases (like "a sequence of one or more symbols is equal to any other sequence of symbols"?) for comparing symbols?

I've read man page for sort but I found no enlightenment there.

I use

sort --version
sort (GNU coreutils) 8.5
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Mike Haertel and Paul Eggert.

and I really need to sort millions of URLs for further analysis which assumes lexical ordering - are there any options which I could pass to sort to achieve this?

1
  • I've performed the same test on my Windows+Cygiwn sort (GNU coreutils) 8.15 Packaged by Cygwin (8.15-1), and it sorts lexically as expected
    – qbolec
    Sep 11, 2014 at 9:03

1 Answer 1

3

It's your locale, prefix your sort command with a locale setting that specifies the collation order you desire

$ cat test2.txt
a/d
a/a
a/c
a//c

$ sort test2.txt
a/a
a/c
a//c
a/d

$ LANG=C sort test2.txt
a//c
a/a
a/c
a/d

man sort says

   *** WARNING *** The locale specified by the  environment  affects  sort
   order.  Set LC_ALL=C to get the traditional sort order that uses native
   byte values.

GNU say

Most of the language specific locales have tables that specify the sort behavior to ignore punctuation and to fold case. This is counter intuitive to most long time computer users!


NOTE

If your millions of URLs contain any non-ASCII characters (which would make them actually IRIs) you may get unwanted results using a byte-value sort. You can use URL-encoding to avoid this issue at the risk of making the URLs hard for humans to read.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .