sort utility on cyrillic text

Question

I have to sort some lines of cyrillic characters and I want to use the sort utility (on MAC OS X 10.6). The problem is that result is incorrect. I take the text into clipboard, then run pbpaste | sort This is plaintext data, and I also tried passing a file to the sort command.

My source data is

Биатлон
Арбуз
Фото
Пума
Перина
Новость
Карандаш
Перина с двумя пододеяльниками
Карикатура
Куст
Фигура

And after sorting I get

Куст
Пума
Фото
Арбуз
Перина
Фигура
Перина с двумя пододеяльниками
Биатлон
Новость
Карандаш
Карикатура

Theese lines aren’t even grouped by first letter. I tried option -d, but then I get an error

sort: string comparison failed: Illegal byte sequence sort: Set LC_ALL='C' to work around the problem. sort: The strings compared were \320\321\321\321' and\320\320\320\321\321\320’.

Exporting the variable as recommended doesn’t solve the problem. What can I do to use the sort utility for such a task? Any additional info is necessary?

Answer 1

I believe you need to set the locale. I think you can do this:

LC_ALL="ru" pbpaste | sort

(The first part sets the LC_ALL environment variable for the duration of the command.)

I'm running Leopard. When I issue a pbpaste, all characters are substituted for question marks. I did save the text to a UTF-8-encoded file called sort_temp.txt, and running:

sort sort_temp.txt 

gives the same improperly sorted list that you posted, but running

LC_ALL="ru" sort sort_temp.txt

or

LANG="ru" sort sort_temp.txt

gives a properly sorted list.

Answer 2

Try to save the data in unicode encoded (or UTF8) file, and if sort utility is unicode enabled, it should work. I guess, the problem is that pbpaste does not set the encoding properly. I'm not Mac user, so its just a guess.