{"id":83055,"date":"2020-05-28T01:42:15","date_gmt":"2020-05-27T23:42:15","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/kak-linuxovskij-sort-sortiruet-stroki"},"modified":"2020-05-28T01:42:15","modified_gmt":"2020-05-27T23:42:15","slug":"kak-linuxovskij-sort-sortiruet-stroki","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/kak-linuxovskij-sort-sortiruet-stroki","title":{"rendered":"How Linux's sort command sorts strings","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

Introduction<\/h1>\n

<\/p>\n

It all started with a short script meant to combine information about addresses e-mail<\/em> of employees, obtained from a mailing list of users, with the positions of employees obtained from the HR database. Both lists were exported to text files in Unicode encoding UTF-8<\/em> and saved with Unix line endings.<\/p>\n

<\/p>\n

Contents mail.txt<\/em><\/p>\n

<\/p>\n

Ivanov Andrey;ia@example.com<\/code><\/pre>\n
<\/p>\n
Contents buhg.txt<\/em><\/p>\n
<\/p>\n
Ivanova Alla;painter\nYolkina Ella;crane operator\nIvanov Andrey;locksmith\nAbakanov Mikhail;painter<\/code><\/pre>\n
<\/p>\n
To merge, the files were sorted using a Unix command sort<\/em> and fed into a Unix program join<\/em>, which unexpectedly terminated with an error: <\/p>\n
<\/p>\n
$> sort buhg.txt > buhg.srt\n$> sort mail.txt > mail.srt\n$> join buhg.srt mail.srt > result\njoin: buhg.srt:4: is not sorted: Ivanov Andrey;locksmith<\/code><\/pre>\n
<\/p>\n
Viewing the sorting result visually showed that overall, the sorting was correct, but in the case of matching male and female surnames, females were placed before males:<\/p>\n
<\/p>\n
$> sort buhg.txt\nAbakanov Mikhail;painter\nYolkina Ella;crane operator\nIvanova Alla;painter\nIvanov Andrey;locksmith<\/code><\/pre>\n
<\/p>\n
Looks like a glitch in Unicode sorting or a manifestation of feminism in the sorting algorithm. The former, of course, is more plausible.<\/p>\n
<\/a><\/noindex><\/p>\n
Let's put that aside for now join<\/em> and focus on sort<\/em>. Let's try to solve the problem by trial and error. To start, we will change the locale from en_US<\/em> to ru_RU<\/em>. For sorting, it would be sufficient to set the environment variable LC_COLLATE<\/em>, but we won't be petty:<\/p>\n
<\/p>\n
$> LANG=ru_RU.UTF-8 sort buhg.txt\nAbakanov Mikhail;painter\nYolkina Ella;crane operator\nIvanova Alla;painter\nIvanov Andrey;locksmith<\/code><\/pre>\n
<\/p>\n
Nothing changed.<\/p>\n
<\/p>\n
Let's try to re-encode the files into a single-byte encoding: <\/p>\n
<\/p>\n
$> iconv -f UTF-8 -t KOI8-R buhg.txt \n | LANG=ru_RU.KOI8-R sort \n | iconv -f KOI8-R -t UTF8<\/code><\/pre>\n
<\/p>\n
Again, nothing changed.<\/p>\n
<\/p>\n
Nothing can be done, we'll have to search for a solution on the internet. There\u2019s nothing specific about Russian surnames, but there are questions about other sorting oddities. Here\u2019s an example of such a problem: Unix sort treats '-' (dash) characters as invisible<\/a><\/noindex>. In short, the strings \"a-b\", \"aa\", \"ac\" are sorted as \"aa\", \"a-b\", \"ac\".<\/p>\n
<\/p>\n
The standard answer everywhere is: use the programmer's locale \"C\"<\/em> and you will be happy. Let's try it:<\/p>\n
<\/p>\n
$> LANG=C sort buhg.txt\nYolkina Ella;crane operator\nAbakanov Mikhail;painter\nIvanov Andrey;locksmith\nIvanova Alla;lawyer<\/code><\/pre>\n
<\/p>\n
Something has changed. The Ivanovs lined up in the correct order, but Yolkina slid somewhere. Let's return to the original task:<\/p>\n
<\/p>\n
$> LANG=C sort buhg.txt > buhg.srt\n$> LANG=C sort mail.txt > mail.srt\n$> LANG=C join buhg.srt mail.srt > result<\/code><\/pre>\n
<\/p>\n
It worked without errors, just as the internet promised. Despite the inclusion of Elkin in the first line.<\/p>\n
<\/p>\n
The problem seems to be resolved, but just in case, let's try another Russian encoding \u2014 Windows encoding. CP1251<\/em>:<\/p>\n
<\/p>\n
$> iconv -f UTF-8 -t CP1251 buhg.txt \n | LANG=ru_RU.CP1251 sort \n | iconv -f CP1251 -t UTF8 <\/code><\/pre>\n
<\/p>\n
The sorting result, surprisingly, will match the locale. \"C\"<\/em>, and the entire example, accordingly, passes without errors. It's some kind of mysticism.<\/p>\n
<\/p>\n
I don't like mysticism in programming, as it usually hides errors. I will have to seriously address how it works sort<\/em> and what it affects. LC_COLLATE<\/em> .<\/p>\n
<\/p>\n
In the end, I will try to answer questions:<\/p>\n
<\/p>\n