{"id":95911,"date":"2020-10-05T01:42:09","date_gmt":"2020-10-04T23:42:09","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/eshhyo-odin-velosiped-hranim-yunikodnye-stroki-na-30-60-kompaktnee-chem-utf-8"},"modified":"2020-10-05T01:42:09","modified_gmt":"2020-10-04T23:42:09","slug":"eshhyo-odin-velosiped-hranim-yunikodnye-stroki-na-30-60-kompaktnee-chem-utf-8","status":"publish","type":"post","link":"https:\/\/prohoster.info\/en\/blog\/administrirovanie\/eshhyo-odin-velosiped-hranim-yunikodnye-stroki-na-30-60-kompaktnee-chem-utf-8","title":{"rendered":"Another round of applause: we store Unicode strings 30-60% more compactly than UTF-8","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"Another
\n
\nIf you are a developer and faced with the task of choosing a character encoding, Unicode is almost always the correct choice. The specific representation method depends on the context, but more often than not, there is a universal answer \u2014 UTF-8. It is advantageous as it allows the use of all Unicode characters without wasting too<\/em> many bytes in most cases. However, for languages that use more than just the Latin alphabet, 'not too many' means at least two bytes per character<\/strong>. Can we do better without reverting to prehistoric encodings that limit us to only 256 available characters?<\/p>\n

Below, I invite you to explore my attempt to provide an answer to this question and the implementation of a relatively simple algorithm that allows storing strings in most languages of the world without adding the overhead present in UTF-8.<\/a><\/noindex><\/p>\n

Disclaimer.<\/i> I will make a few important clarifications right away: the solution described is not proposed as a universal replacement for UTF-8<\/strong>, it is suitable only in a narrow list of cases (which will be outlined below), and it should never be used for interaction with third-party APIs (which are not familiar with it). Most often, for compact storage of large volumes of text data, general-purpose compression algorithms (such as deflate) will suffice. Moreover, during the creation of my solution, I found an existing standard within Unicode that addresses the same problem \u2014 it is somewhat more complex (and often worse), but it is still an accepted standard, rather than something hastily put together. I will also discuss it.<\/p>\n

About Unicode and UTF-8<\/h2>\n

\nTo start \u2014 a few words about what exactly Unicode<\/strong> and UTF-8<\/strong>.<\/p>\n

As is known, 8-bit encodings were popular in the past. They were simple: 256 characters can be numbered from 0 to 255, and numbers from 0 to 255 can obviously be represented in one byte. Going back to the origins, ASCII encoding is limited to 7 bits, which is why the highest bit in its byte representation equals zero, and most 8-bit encodings are compatible with it (they only differ in the 'upper' part, where the highest bit equals one).<\/p>\n

How does Unicode differ from those encodings, and why are there so many specific representations associated with it \u2014 UTF-8, UTF-16 (BE and LE), UTF-32? Let's break it down in order.<\/p>\n

The main Unicode standard only describes the correspondence between characters (and in some cases, the individual components of characters) and their numbers. And there are many possible numbers in this standard \u2014 ranging from 0x00<\/b><\/code> up to 0x10FFFF<\/b><\/code> (1,114,112). If we wanted to store a number within such a range in a variable, neither 1 nor 2 bytes would be enough. And since our processors are not very well suited for working with three-byte numbers, we would have to use a full 4 bytes for one character! This is UTF-32, but due to this \u201cwastefulness,\u201d this format is not popular.<\/p>\n

Fortunately, the characters within Unicode are not arranged randomly. Their multitude is divided into 17 \u201cplanes<\/em>,\u201d each containing 65,536 (0x10000<\/b><\/code>) \u00abcode points<\/em>). The concept of a \u201ccode point\u201d here is simply a number assigned to a character<\/em>, which Unicode has assigned to it. However, as mentioned earlier, not only individual characters are numbered in Unicode, but also their components and control marks (and sometimes, a number does not correspond to anything at all \u2014 perhaps for the time being, but that is not too important for us), so it is more accurate to always refer to the number of actual numbers, rather than characters. However, for brevity, I will often use the term \u201ccharacter,\u201d implying the term \u201ccode point.\u201d<\/p>\n

\"Another
\nUnicode planes. As can be seen, most of them (planes 4 to 13) are still unused.<\/i><\/p>\n

The most remarkable thing is that all the main content lies in the zero plane, which is called "Basic Multilingual Plane<\/em>". If the line contains text in one of the modern languages (including Chinese), you won't go beyond this plane. However, you also can't exclude the rest of Unicode \u2014 for example, emojis are mainly located at the end of the next plane, "Supplementary Multilingual Plane<\/em>" (which extends from 0x10000<\/b><\/code> up to 0x1FFFF<\/b><\/code>). Therefore, UTF-16 operates as follows: all characters that fall into Basic Multilingual Plane<\/em>are encoded \"as is,\" with their corresponding two-byte number. However, some numbers in this range do not represent specific characters at all, but indicate that the next pair of bytes needs to be considered \u2014 combining the values of these four bytes together, we get a number that encompasses the entire valid range of Unicode. This representation is called \u201csurrogate pairs\u201d \u2014 you may have heard of them.<\/p>\n

Thus, UTF-16 requires two or (in very rare cases) four bytes for a single 'code point'. This is better than using four bytes all the time, but the Latin script (and other ASCII characters) in this encoding consumes half of its space in zeros. UTF-8 aims to rectify this: ASCII takes just one byte again; codes from 0x80<\/b><\/code> up to 0x7FF<\/b><\/code> \u2014 two bytes; from 0x800<\/b><\/code> up to 0xFFFF<\/b><\/code> \u2014 three, and from 0x10000<\/b><\/code> up to 0x10FFFF<\/b><\/code> \u2014 four. On one hand, the Latin script has improved: compatibility with ASCII has returned, and the distribution is more evenly 'spread out' from 1 to 4 bytes. However, alphabets other than Latin do not gain any advantage compared to UTF-16, and many now even require three bytes instead of two \u2014 the range covered by two-byte encoding has shrunk 32 times, from 0xFFFF<\/b><\/code> up to 0x7FF<\/b><\/code>, and neither Chinese nor, for example, Georgian falls into it. Cyrillic and five other alphabets \u2014 hooray \u2014 are lucky, 2 bytes per character.<\/p>\n

Why is this the case? Let's look at how UTF-8 represents character codes:
\n\"Another
\nDirectly for representing numbers, bits marked by the symbol are used here. x<\/b><\/code>It can be seen that in two-byte encoding, there are only 11 such bits (out of 16). The leading bits here serve only a functional purpose. In the case of four-byte encoding, a full 21 bits out of 32 are allocated for the code point number \u2014 it seems like three bytes (which give a total of 24 bits) would be enough, but control markers consume too much.<\/p>\n

Is this bad? In fact, not really. On one hand \u2014 if we are very concerned about the space used, we have compression algorithms that can easily eliminate all unnecessary entropy and redundancy. On the other \u2014 Unicode's goal was to provide the most universal encoding possible. For instance, a string encoded in UTF-8 can be handed off to code that previously only worked with ASCII, without fear that it will see a character from the ASCII range that isn't actually there (since in UTF-8, all bytes starting with a zero bit are indeed ASCII). And if suddenly we want to trim a small tail off a large string without decoding it from the beginning (or restore part of the information after a damaged section) \u2014 it\u2019s not difficult for us to find the offset where a certain character begins (just skip the bytes that have a bit prefix) 10<\/b><\/code>).<\/p>\n

So why invent something new?<\/h2>\n

\nAt the same time, there are rare situations when compression algorithms like deflate are poorly applicable, yet there is a desire to achieve compact storage of strings. Personally, I encountered such a task while contemplating the construction of a compressed prefix tree<\/a><\/noindex> for a large dictionary that includes words from arbitrary languages. On one hand, each word is very short, so compressing it would be ineffective. On the other hand, the tree implementation I was considering was designed for the condition that each byte of the stored string would create a separate tree node, making it very useful to minimize their number. In my library Az.js<\/a><\/noindex> (as in pymorphy2<\/a><\/noindex>, on which it is based) this problem is solved simply\u2014strings packed in a DAWG<\/a><\/noindex>-dictionary are stored there in the good old CP1251<\/a><\/noindex>. But, as one can easily understand, this works well only for a limited alphabet\u2014a string in Chinese can't be accommodated in such a dictionary.<\/p>\n

I would also like to point out another unpleasant nuance that arises when using UTF-8 in such a data structure. In the picture above, it can be seen that when a character is written in two bytes, the bits related to its number are not consecutive but are interrupted by a couple of bits 10<\/b><\/code> in the middle: 110xxxxx 10xxxxxx<\/b><\/code>. Because of this, when the lower 6 bits of the second byte overflow in the character code (i.e., a transition occurs 10111111<\/b><\/code> \u2192 10000000<\/b><\/code>), the first byte also changes. As a result, the letter '\u043f' is represented by the bytes 0xD0 0xBF<\/b><\/code>, while the following letter '\u0440' is already 0xD1 0x80<\/b><\/code>. In the prefix tree, this leads to splitting the parent node into two\u2014one for the prefix 0xD0<\/b><\/code>, and the other for 0xD1<\/b><\/code> (even though all Cyrillic could be encoded using only the second byte).<\/p>\n

What I came up with<\/h2>\n

\nFacing this task, I decided to practice bit manipulation and at the same time get to know the structure of Unicode a bit better. The result was a coding format called UTF-C (the 'C' stands for compact<\/em>), which uses no more than 3 bytes for one code point and often allows just one extra byte for the entire encoded string<\/strong>. This results in such encoding being 30-60% more compact than UTF-8 for many non-ASCII alphabets.<\/strong>.<\/p>\n

I documented examples of the encoding and decoding algorithms in the form of libraries in JavaScript and Go.<\/a><\/noindex>, you can freely use them in your code. However, I would like to emphasize that in some sense, this format remains a 'bicycle', and I do not recommend using it without understanding why you need it<\/strong>. After all, it\u2019s more of an experiment than a serious 'improvement of UTF-8'. Nevertheless, the code is written neatly, concisely, with a lot of comments and test coverage.<\/p>\n

\"Another
\nTest results and comparison with UTF-8<\/i><\/p>\n

I also created a demo page<\/a><\/noindex>, where you can evaluate the algorithm's performance, and I will discuss its principles and development process in more detail.<\/p>\n

Eliminating redundant bits<\/h2>\n

\nI based it on UTF-8, of course. The first and most obvious thing to change is to reduce the number of control bits in each byte. For example, the first byte in UTF-8 always starts with either 0<\/b><\/code>, or 11<\/b><\/code> \u2014 and the prefix 10<\/b><\/code> only exists for subsequent bytes. Let's replace the prefix 11<\/b><\/code> to 1<\/b><\/code>, and for subsequent bytes, we will completely remove the prefixes. What do we get?<\/p>\n

0xxxxxxx<\/b><\/code> \u2014 1 byte
\n10xxxxxx xxxxxxxx<\/b><\/code> \u2014 2 bytes
\n110xxxxx xxxxxxxx xxxxxxxx<\/b><\/code> \u2014 3 bytes<\/p>\n

Wait, where is the four-byte representation? It has become unnecessary \u2014 with three-byte encoding, we now have access to 21 bits, which is more than enough for all numbers up to 0x10FFFF<\/b><\/code>.<\/p>\n

What have we sacrificed here? The most important thing \u2014 detecting the boundaries of characters from arbitrary positions in the buffer. We cannot point to an arbitrary byte and find the start of the next character from it. This is a limitation of our format, but in practice, the need for this rarely arises. Usually, we can run through the buffer from the beginning (especially when dealing with short strings).<\/p>\n

The situation with covering languages with 2 bytes has also improved: now the two-byte format provides a range of 14 bits, which corresponds to codes up to 0x3FFF<\/b><\/code>. The Chinese are unfortunate (their characters mainly fall in the range from 0x4E00<\/b><\/code> up to 0x9FFF<\/b><\/code>), but for Georgians and many other nations, it has become better \u2014 their languages also fit into 2 bytes per character.<\/p>\n

Introducing the encoder state<\/h2>\n

\nNow let's think about the properties of the strings themselves. In a dictionary, words are most often composed of characters from one alphabet, and this is also true for many other texts. It would be good to specify this alphabet once and then only indicate the letter number within it. Let's see if the arrangement of characters in the Unicode table helps us.<\/p>\n

As mentioned above, Unicode is divided into planes<\/em> up to 65536 codes each. However, this is not a very useful division (as already mentioned, we are most often in the zero plane). A more interesting division is into blocks.<\/em> These ranges no longer have a fixed length and carry more meaning \u2014 as a rule, each combines characters of one alphabet.<\/p>\n

\"Another
\nA block containing characters of the Bengali alphabet. Unfortunately, for historical reasons, this is an example of not very dense packing \u2014 96 characters are scattered across 128 code points of the block.<\/i><\/p>\n

The beginnings of blocks and their sizes are always multiples of 16 \u2014 this is simply done for convenience. Additionally, many blocks start and end at values that are multiples of 128 or even 256 \u2014 for example, the main Cyrillic occupies 256 bytes from 0x0400<\/b><\/code> up to 0x04FF<\/b><\/code>. This is quite convenient: if we save the prefix once 0x04<\/b><\/code>, then any Cyrillic character can be written in one byte afterwards. However, this way we lose the ability to revert to ASCII (and to any other characters at all). Therefore, we do it this way:<\/p>\n

    \n
  1. Two bytes 10yyyyyy yxxxxxxx<\/b><\/code> not only denote the character with number yyyyyy yxxxxxxx<\/b><\/code>, but also change the current alphabet<\/em> to yyyyyy y0000000<\/b><\/code> i.e. we remember all bits except the least significant 7 bits<\/strong>);<\/li>\n
  2. One byte 0xxxxxxx<\/b><\/code> is a character of the current alphabet. It simply needs to be added to the offset we memorized in step 1. As long as we haven't changed the alphabet, the offset is zero, so we have maintained compatibility with ASCII.<\/li>\n<\/ol>\n

    \nSimilarly for codes that require 3 bytes:<\/p>\n

      \n
    1. Three bytes 110yyyyy yxxxxxxx xxxxxxxx<\/b><\/code> denote the character with number yyyyyy yxxxxxxx xxxxxxxx<\/b><\/code>, change the current alphabet<\/em> to yyyyyy y0000000 00000000<\/b><\/code> we remember everything except the least significant 15 bits<\/strong>), and set a flag that we are now in long<\/em> mode (when switching the alphabet back to two-byte, we will reset this flag);<\/li>\n
    2. Two bytes 0xxxxxxx xxxxxxxx<\/b><\/code> in long mode, this is the current alphabet character. Similarly, we add it to the offset from step 1. The only difference is that now we read two bytes (because we've switched to that mode).<\/li>\n<\/ol>\n

      \nSounds good: now as long as we need to encode characters from the same 7-bit Unicode range, we spend 1 extra byte at the beginning and only one byte per character.<\/p>\n

      \"Another
      \nThe work of one of the early versions. It already often surpasses UTF-8, but there is still room for improvement.<\/i><\/p>\n

      What became worse? Firstly, we have a state, namely the offset of the current alphabet<\/em> and a flag for long mode.<\/em>This additionally limits us: now the same characters can be encoded differently in different contexts. For instance, substring searches will have to consider this, rather than simply comparing bytes. Secondly, as soon as we switched alphabets, encoding ASCII characters became problematic (this includes not only Latin letters but also basic punctuation, including spaces) \u2014 they require a re-switching of the alphabet to 0, which introduces an extra byte (and another one to revert to our main alphabet).<\/p>\n

      One alphabet is good, two are better.<\/h2>\n

      \nLet's try to slightly modify our bit prefixes, adding another one to the three described above:<\/p>\n

      0xxxxxxx<\/b><\/code> \u2014 1 byte in normal mode, 2 in long mode.
      \n11xxxxxx<\/b><\/code> \u2014 1 byte
      \n100xxxxx xxxxxxxx<\/b><\/code> \u2014 2 bytes
      \n101xxxxx xxxxxxxx xxxxxxxx<\/b><\/code> \u2014 3 bytes<\/p>\n

      \"Another
      \n
      \nNow, in the two-byte encoding, one available bit has become less \u2014 code points can fit up to 0x1FFF<\/b><\/code>, not 0x3FFF<\/b><\/code>. However, it\u2019s still noticeably more than in two-byte UTF-8 codes, most of the common languages still fit, with the most notable loss being       hiragana.<\/a><\/noindex> and katakana.<\/a><\/noindex>, the Japanese are sad.<\/p>\n

      What is this new code 11xxxxxx<\/b><\/code>? \u042d\u0442\u043e \u043d\u0435\u0431\u043e\u043b\u044c\u0448\u043e\u0439 \u00ab\u0437\u0430\u0433\u0430\u0448\u043d\u0438\u043a\u00bb \u0440\u0430\u0437\u043c\u0435\u0440\u043e\u043c \u0432 64 \u0441\u0438\u043c\u0432\u043e\u043b\u0430, \u043e\u043d \u0434\u043e\u043f\u043e\u043b\u043d\u044f\u0435\u0442 \u043d\u0430\u0448 \u043e\u0441\u043d\u043e\u0432\u043d\u043e\u0439 \u0430\u043b\u0444\u0430\u0432\u0438\u0442, \u043f\u043e\u044d\u0442\u043e\u043c\u0443 \u044f \u043d\u0430\u0437\u0432\u0430\u043b \u0435\u0433\u043e \u0432\u0441\u043f\u043e\u043c\u043e\u0433\u0430\u0442\u0435\u043b\u044c\u043d\u044b\u043c (auxiliary<\/em>) alphabet. When we switch the current alphabet, a piece of the old alphabet becomes auxiliary. For example, switching from ASCII to Cyrillic \u2014 now there are 64 characters in the 'backup', containing Latin letters, digits, a space, and a comma<\/strong> (the most frequent inserts in non-ASCII texts). Switching back to ASCII \u2014 and the main part of Cyrillic will become the auxiliary alphabet.<\/p>\n

      Thanks to access to two alphabets, we can handle a large number of texts with minimal costs for switching alphabets (punctuation will most often lead us back to ASCII, but afterwards many non-ASCII characters will be drawn from the additional alphabet without needing to switch again).<\/p>\n

      Bonus: by defining the additional alphabet with a prefix 11xxxxxx<\/b><\/code> and setting its initial offset to 0xC0<\/b><\/code>, we achieve partial compatibility with CP1252. In other words, many (but not all) Western European texts encoded in CP1252 will appear the same in UTF-C.<\/p>\n

      However, there is a challenge: how to obtain an auxiliary alphabet from the main one? You can keep the same offset, but \u2014 alas \u2014 here the Unicode structure already plays against us. Very often, the main part of the alphabet is not at the beginning of the block (for example, the Russian capital '\u0410' has the code 0x0410<\/b><\/code>, while the Cyrillic block starts with 0x0400<\/b><\/code>). Therefore, by taking the first 64 characters into the 'stash', we may lose access to the tail of the alphabet.<\/p>\n

      To address this issue, I manually went through some blocks corresponding to various languages and indicated the offset of the auxiliary alphabet within the main one. The Latin alphabet was entirely reordered, akin to base64, as an exception.<\/p>\n

      \"Another<\/p>\n

      Final Touches<\/h2>\n

      \nLet's take one last look at where we can make further adjustments.<\/p>\n

      Note that the format 101xxxxx xxxxxxxx xxxxxxxx<\/b><\/code> allows for encoding numbers up to 0x1FFFFF<\/b><\/code>, while Unicode ends earlier, at 0x10FFFF<\/b><\/code>. In other words, the last code point will be represented as 10110000 11111111 11111111<\/b><\/code>. Thus, we can say that if the first byte appears as 1011xxxx<\/b><\/code> (where xxxx<\/b><\/code> is greater than 0), it signifies something else. For instance, we can add another 15 characters, constantly available for encoding in a single byte, but I chose a different approach.<\/p>\n

      Therefore, we select several chosen ranges corresponding to emojis, Hiragana, and Katakana, renumber them into a single continuous list, and encode them using two bytes instead of three:<\/p>\n

      1011xxxx xxxxxxxx<\/b><\/code> <\/p>\n

      Great: the aforementioned emoji \u200d\u200d\u200d\"Another<\/a><\/noindex>, consisting of 7 code points, takes up 25 bytes in UTF-8, and we managed to fit it into 14<\/strong> (exactly two bytes for each code point). By the way, Habr refused to process it (in both the old and the new editor), so I had to insert it as an image.<\/p>\n

      Let's try to fix another issue. As we remember, the main alphabet is essentially the upper 6 bits<\/strong>, which we keep in mind and attach to the code of each subsequent decoded symbol. In the case of Chinese characters, which are in the block 0x4E00<\/b><\/code> \u2013 0x9FFF<\/b><\/code>, it is either bit 0 or 1. This is not very convenient: we will need to constantly switch the alphabet between these two values (i.e., spending three bytes each time). However, we note that in long mode we can subtract the number of characters we are encoding with short mode (after all the tricks described above, this is 10240) \u2014 then the range of characters will shift to 0x2600<\/b><\/code> \u2013 0x77FF<\/b><\/code>, and in this case, the upper 6 bits (out of 21) over this entire range will be equal to 0. Thus, sequences of characters will use two bytes per character (which is optimal for such a large range), without requiring alphabet switching. <\/p>\n

      Alternative solutions: SCSU, BOCU-1<\/h2>\n

      \nUnicode experts, upon merely reading the title of the article, will likely rush to remind that directly among Unicode standards there is Standard Compression Scheme for Unicode<\/a><\/noindex> (SCSU), which describes an encoding method quite similar to what is described in the article.<\/p>\n

      I must confess: I only learned of its existence after deeply immersing myself in writing my own solution. Had I known about it from the very beginning, I would probably have tried to write its implementation instead of inventing my own approach.<\/p>\n

      Interestingly, SCSU uses ideas quite similar to those I arrived at independently (instead of the concept of 'alphabets', it uses 'windows', and there are more of them than I have). At the same time, this format also has its drawbacks: it is somewhat closer to compression algorithms rather than encoding. In particular, the standard provides many ways of representation but does not specify how to choose the optimal one \u2014 for this, the encoder must apply certain heuristics. Thus, an SCSU encoder that provides good packing will be more complex and cumbersome than my algorithm.<\/p>\n

      For comparison, I ported a relatively simple SCSU implementation to JavaScript \u2014 in terms of code volume, it turned out to be comparable to my UTF-C, but in some cases, it showed results that were tens of percent worse (though it might sometimes outperform it, but not by much). For example, texts in Hebrew and Greek were encoded by UTF-C a 60% better than SCSU<\/strong> (probably due to their compact alphabets).<\/p>\n

      I would like to add that besides SCSU, there is also another method for compact representation of Unicode \u2014 BOCU-1<\/a><\/noindex>, but it aims for compatibility with MIME (which wasn't required for me), and uses a slightly different approach to encoding. I haven't evaluated its efficiency, but I think it is unlikely to be higher than that of SCSU.<\/p>\n

      Possible enhancements<\/h2>\n

      \nThe algorithm I provided is not universal by design (this is where my goals diverge most from those of the Unicode consortium). I have already mentioned that it was primarily developed for one task (storing a multilingual dictionary in a prefix tree), and some of its features may not be well-suited for other tasks. However, the fact that it is not a standard can also be a plus \u2014 you can easily modify it to fit your needs<\/strong>.<\/p>\n

      For example, one can obviously eliminate state, making the encoding stateless \u2014 simply by not updating variables offs<\/b><\/code>, auxOffs<\/b><\/code> and is21Bit<\/b><\/code> in the encoder and decoder. In this case, it won't be possible to efficiently pack sequences of characters from the same alphabet, but there will be a guarantee that the same character is always encoded with the same bytes, regardless of context.<\/p>\n

      Furthermore, you can tailor the encoder to a specific language by changing the default state \u2014 for example, focusing on Russian texts, you can set at the beginning of the encoder and decoder offs = 0x0400<\/b><\/code> and auxOffs = 0<\/b><\/code>. This especially makes sense in a stateless mode. Overall, this will resemble the use of an old eight-bit encoding, but it does not exclude the possibility of inserting characters from the entire Unicode as needed.<\/p>\n

      Another drawback mentioned earlier \u2014 in large text encoded in UTF-C, there is no quick way to find the boundary of a character nearest to an arbitrary byte. Cutting off the last, say, 100 bytes from the encoded buffer risks getting garbage that cannot be processed. While the encoding is not designed for storing multi-gigabyte logs, this can generally be corrected. A byte 0xBF<\/b><\/code> should never occur as the first byte (but can be the second or third). Therefore, during encoding, a sequence can be inserted \u00bf\u00bf\u00bf<\/b><\/code> Every, say, 10 KB \u2014 then, if necessary, finding the boundary will be enough to scan the selected chunk until a similar marker is found. Following the last 0xBF<\/b><\/code> will guarantee the start of the character. (When decoding, this three-byte sequence will, of course, need to be ignored.)<\/p>\n

      In summary<\/h2>\n

      \nIf you've read this far \u2014 congratulations! I hope you, like me, learned something new (or refreshed your memory) about how Unicode works.<\/p>\n

      \"Another
      \nDemonstration page. The advantages over both UTF-8 and SCSU are visible through the example of Hebrew.<\/i><\/p>\n

      The above findings should not be regarded as encroachments on standards. However, I am generally satisfied with the results of my work, and therefore I am happy to share<\/a><\/noindex>: for example, the JS library in minified form weighs only 1710 bytes (and has no dependencies, of course). As I mentioned earlier, you can familiarize yourself with its operation on the demo page<\/a><\/noindex> (there's also a set of texts that you can compare it with against UTF-8 and SCSU).<\/p>\n

      Lastly, I would like to reiterate the cases in which using UTF-C is not advisable<\/b>:<\/p>\n

      Let's take a look at the Unicode blocks that currently require three bytes. Mostly, as mentioned, these are Chinese characters \u2014 but it's difficult to do anything with them, there are 21,000. Additionally, there's also Hiragana and Katakana \u2014 they are fewer, less than two hundred. And since we're mentioning the Japanese, emojis are also included (actually, they're scattered in many places within Unicode, but the main blocks fall within the range of 0x1F300<\/b><\/code> \u2013 0x1FBFF<\/b><\/code>). Considering that there are emojis that consist of several code points (for example, the emoji \u200d\u200d\u200d\"Another<\/a><\/noindex> is made up of 7 codes!), it's quite unfortunate to spend three bytes on each (7\u00d73 = 21 bytes for a single symbol, it's a nightmare).<\/p>\n