{"id":95911,"date":"2020-10-05T01:42:09","date_gmt":"2020-10-04T23:42:09","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/eshhyo-odin-velosiped-hranim-yunikodnye-stroki-na-30-60-kompaktnee-chem-utf-8"},"modified":"2020-10-05T01:42:09","modified_gmt":"2020-10-04T23:42:09","slug":"eshhyo-odin-velosiped-hranim-yunikodnye-stroki-na-30-60-kompaktnee-chem-utf-8","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/eshhyo-odin-velosiped-hranim-yunikodnye-stroki-na-30-60-kompaktnee-chem-utf-8","title":{"rendered":"Nj\u00eb tjet\u00ebr bi\u00e7iklet\u00eb: ruajm\u00eb stringjet Unicode deri n\u00eb 30-60% m\u00eb kompakt se UTF-8","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"Nj\u00eb
\n
\nN\u00ebse jeni zhvillues dhe p\u00ebrballeni me detyr\u00ebn e zgjedhjes s\u00eb kodimit, pothuajse gjithmon\u00eb zgjidhja e duhur do t\u00eb jet\u00eb Unicode. M\u00ebnyra specifike e paraqitjes varet nga konteksti, por shpesh ka nj\u00eb p\u00ebrgjigje universale \u2014 UTF-8. Ai \u00ebsht\u00eb i mir\u00eb sepse lejon p\u00ebrdorimin e t\u00eb gjitha simboleve Unicode, pa shpenzuar shum\u00eb<\/em> byte n\u00eb shumic\u00ebn e rasteve. Megjithat\u00eb, p\u00ebr gjuh\u00ebt q\u00eb p\u00ebrdorin m\u00eb shum\u00eb se vet\u00ebm alfabetin latin\u00eb, \u00abjo shum\u00eb\u00bb do t\u00eb thot\u00eb t\u00eb pakt\u00ebn dy byte p\u00ebr simbol<\/strong>. A \u00ebsht\u00eb e mundur t\u00eb kemi m\u00eb mir\u00eb, pa u rikthyer n\u00eb kodimet parahistorike q\u00eb na kufizojn\u00eb n\u00eb vet\u00ebm 256 simbole t\u00eb disponueshme?<\/p>\n

M\u00eb posht\u00eb propozoj t\u00eb njihemi me p\u00ebrpjekjen time p\u00ebr t\u00eb dh\u00ebn\u00eb nj\u00eb p\u00ebrgjigje p\u00ebr k\u00ebt\u00eb pyetje dhe realizimin e nj\u00eb algoritmi relativisht t\u00eb thjesht\u00eb, q\u00eb lejon ruajtjen e stringjeve n\u00eb shumic\u00ebn e gjuh\u00ebve t\u00eb bot\u00ebs, pa shtuar at\u00eb tep\u00ebrsin\u00eb q\u00eb ka UTF-8.<\/a><\/noindex><\/p>\n

Diskleimer.<\/i> Menj\u00ebher\u00eb do t\u00eb b\u00ebj disa sqarime t\u00eb r\u00ebnd\u00ebsishme: zgjidhja e p\u00ebrshkruar nuk ofrohet si nj\u00eb z\u00ebvend\u00ebsim universale p\u00ebr UTF-8<\/strong>, \u00ebsht\u00eb i p\u00ebrshtatsh\u00ebm vet\u00ebm n\u00eb nj\u00eb list\u00eb t\u00eb ngusht\u00eb rastesh (p\u00ebr t\u00eb cilat do t\u00eb flas m\u00eb posht\u00eb), dhe n\u00eb asnj\u00eb rast nuk duhet p\u00ebrdorur p\u00ebr nd\u00ebrveprimin me API t\u00eb jashtme (t\u00eb cilat as nuk e din\u00eb p\u00ebr t\u00eb). M\u00eb s\u00eb shpeshti, p\u00ebr ruajtjen kompakte t\u00eb sasi t\u00eb m\u00ebdha t\u00eb t\u00eb dh\u00ebnave tekstuale, algoritmet e kompresimit t\u00eb p\u00ebrdorura p\u00ebr q\u00ebllime t\u00eb p\u00ebrgjithshme do t\u00eb jen\u00eb t\u00eb p\u00ebrshtatshme (p.sh., deflate). P\u00ebr m\u00eb tep\u00ebr, gjat\u00eb procesit t\u00eb krijimit t\u00eb zgjidhjes sime, kam gjetur nj\u00eb standard ekzistues n\u00eb vet\u00eb Unicode q\u00eb zgjidh t\u00eb nj\u00ebjt\u00ebn \u00e7\u00ebshtje \u2014 \u00ebsht\u00eb pak m\u00eb i komplikuar (dhe shpeshher\u00eb m\u00eb i dob\u00ebt), por gjithsesi \u00ebsht\u00eb nj\u00eb standard i pranuar, dhe jo di\u00e7ka e mbledhur nxitimthi. P\u00ebr at\u00eb do t\u00eb flas gjithashtu.<\/p>\n

P\u00ebr Unicode dhe UTF-8<\/h2>\n

\nS\u00eb pari \u2014 disa fjal\u00eb rreth asaj se \u00e7far\u00eb \u00ebsht\u00eb n\u00eb t\u00eb v\u00ebrtet\u00eb Unicode<\/strong> dhe UTF-8<\/strong>.<\/p>\n

Si e dim\u00eb, m\u00eb par\u00eb ishin t\u00eb njohura kodifikimet 8-bit. Me to gjith\u00e7ka ishte e thjesht\u00eb: 256 simbole mund t\u00eb num\u00ebroheshin me numra nga 0 deri n\u00eb 255, dhe numrat nga 0 deri n\u00eb 255 paraqiten qart\u00eb si nj\u00eb bajt. N\u00ebse kthehemi n\u00eb origjinat m\u00eb t\u00eb hershme, kodifikimi ASCII \u00ebsht\u00eb i kufizuar n\u00eb 7 bit, prandaj biti m\u00eb i lart\u00eb n\u00eb p\u00ebrfaq\u00ebsimin e saj n\u00eb bajt \u00ebsht\u00eb zero, dhe shumica e kodifikimeve 8-bit jan\u00eb t\u00eb p\u00ebrputhshme me t\u00eb (ato ndryshojn\u00eb vet\u00ebm n\u00eb pjes\u00ebn e \u201csip\u00ebrme\u201d, ku biti m\u00eb i lart\u00eb \u00ebsht\u00eb nj\u00eb).<\/p>\n

\u00c7far\u00eb e ndan Unicode nga ato kodifikime dhe pse lidhen me t\u00eb shum\u00eb p\u00ebrfaq\u00ebsime specifike \u2014 UTF-8, UTF-16 (BE dhe LE), UTF-32? Le t\u00eb shqyrtojm\u00eb radhazi.<\/p>\n

Standardi kryesor i Unicode p\u00ebrshkruan vet\u00ebm korrespondenc\u00ebn midis simboleve (dhe n\u00eb disa raste \u2014 komponent\u00ebve t\u00eb ve\u00e7ant\u00eb t\u00eb simboleve) dhe numrat e tyre. Dhe numrat e mundsh\u00ebm n\u00eb k\u00ebt\u00eb standard jan\u00eb shum\u00eb t\u00eb shumt\u00eb \u2014 nga 0x00<\/b><\/code> deri n\u00eb 0x10FFFF<\/b><\/code> (1 114 112 copje). N\u00ebse do t\u00eb donim t\u00eb vendosnim nj\u00eb num\u00ebr n\u00eb nj\u00eb gam\u00eb t\u00eb till\u00eb n\u00eb nj\u00eb variab\u00ebl, as 1 as 2 byte nuk do t\u00eb ishin t\u00eb mjaftueshme. Dhe, duke qen\u00eb se procesor\u00ebt tan\u00eb nuk jan\u00eb shum\u00eb t\u00eb adaptuar p\u00ebr pun\u00eb me numra tre-byte, do t\u00eb ishim t\u00eb detyruar t\u00eb p\u00ebrdorim plot 4 byte p\u00ebr nj\u00eb simbol! Ky \u00ebsht\u00eb UTF-32, por pik\u00ebrisht p\u00ebr k\u00ebt\u00eb \u00abshpenzim\u00bb ky format nuk \u00ebsht\u00eb popullor.<\/p>\n

Fatmir\u00ebsisht, simbol\u00ebt brenda Unicode-it nuk jan\u00eb renditur rast\u00ebsisht. T\u00eb gjith\u00eb numri i tyre \u00ebsht\u00eb i ndar\u00eb n\u00eb 17 \u00abplane<\/em>\u00bb, secila prej t\u00eb cilave p\u00ebrmban 65536 (0x10000<\/b><\/code>) \u00abpik\u00eb kodimi<\/em>\u00bb. Koncepti \u00abpik\u00eb kodimi\u00bb k\u00ebtu \u00ebsht\u00eb thjesht numri i simbolit<\/em>, i caktuar atij nga Unicode. Por, si\u00e7 u tha m\u00eb lart, n\u00eb Unicode nuk jan\u00eb numeruar vet\u00ebm simbol\u00ebt e ve\u00e7ant\u00eb, por edhe komponent\u00ebt e tyre dhe sh\u00ebnimet sh\u00ebrbimore (ndonj\u00ebher\u00eb as q\u00eb i p\u00ebrputhen ndonj\u00eb numri \u2014 ndoshta p\u00ebr nj\u00eb periudh\u00eb t\u00eb caktuar, por p\u00ebr ne nuk \u00ebsht\u00eb kaq e r\u00ebnd\u00ebsishme), prandaj \u00ebsht\u00eb m\u00eb korrekte t\u00eb flitet gjithmon\u00eb p\u00ebr numrin e numrave, e jo t\u00eb simbol\u00ebve. Megjithat\u00eb, m\u00eb posht\u00eb p\u00ebr shkak t\u00eb shkurtimit shpesh do t\u00eb p\u00ebrdor fjal\u00ebn \u00absimbol\u00bb, duke n\u00ebnkuptuar termin \u00abpik\u00eb kodimi\u00bb.<\/p>\n

\"Nj\u00eb
\nPlane t\u00eb Unicode-it. Si\u00e7 duket, pjesa m\u00eb e madhe (planet nga 4 n\u00eb 13) ende nuk \u00ebsht\u00eb p\u00ebrdorur.<\/i><\/p>\n

E gjith\u00eb thelbi i r\u00ebnd\u00ebsish\u00ebm ndodhet n\u00eb nivelin zero, i njohur si "Plani Multilinguistik Bazik<\/em>". N\u00ebse rreshti ka tekst n\u00eb nj\u00eb nga gjuh\u00ebt moderne (p\u00ebrfshir\u00eb kinezishten), nuk do t\u00eb dal\u00ebsh p\u00ebrtej k\u00ebtij niveli. Por nuk \u00ebsht\u00eb e mundur t\u00eb largosh pjes\u00ebn tjet\u00ebr t\u00eb Unicode gjithashtu \u2014 p\u00ebr shembull, emotikoni zakonisht ndodhen n\u00eb fund t\u00eb nivelit tjet\u00ebr, "Plani Multilinguistik Shtes\u00eb<\/em>" (ai shtrihen nga 0x10000<\/b><\/code> deri n\u00eb 0x1FFFF<\/b><\/code>). Prandaj, UTF-16 vepron k\u00ebshtu: t\u00eb gjitha simbolet q\u00eb bien n\u00eb Plani Multilinguistik Bazik<\/em>, kodifikohen \"ashtu si jan\u00eb\", me numrin e tyre dy-bajt\u00ebsh p\u00ebrkat\u00ebs. Megjithat\u00eb, disa nga numrat n\u00eb k\u00ebt\u00eb gam\u00eb nuk tregojn\u00eb asnj\u00eb simbol t\u00eb ve\u00e7ant\u00eb, por tregojn\u00eb se pas k\u00ebsaj \u00e7ifti bajt duhet t\u00eb shqyrtohet nj\u00eb tjet\u00ebr \u2014 duke kombinuar vlerat e k\u00ebtyre kat\u00ebr bajt\u00ebve s\u00eb bashku, ne do t\u00eb kemi nj\u00eb num\u00ebr q\u00eb mbulon t\u00eb gjith\u00eb gam\u00ebn e lejueshme t\u00eb Unicode. Ky paraqitje quhet \"\u00e7iftet z\u00ebvend\u00ebsuese\" \u2014 ndoshta keni d\u00ebgjuar p\u00ebr to.<\/p>\n

K\u00ebshtu, UTF-16 k\u00ebrkon dy ose (n\u00eb raste shum\u00eb t\u00eb rralla) kat\u00ebr byte p\u00ebr nj\u00eb \"pik\u00eb kodimi\". Kjo \u00ebsht\u00eb m\u00eb mir\u00eb se sa t\u00eb p\u00ebrdor\u00ebsh vazhdimisht kat\u00ebr byte, por latinishtja (dhe simbolet e tjera ASCII) n\u00eb k\u00ebt\u00eb kodim shpenzojn\u00eb gjysm\u00ebn e hap\u00ebsir\u00ebs p\u00ebr zero. UTF-8 \u00ebsht\u00eb krijuar p\u00ebr ta rregulluar k\u00ebt\u00eb: ASCII n\u00eb t\u00eb z\u00eb, si m\u00eb par\u00eb, vet\u00ebm nj\u00eb byte; kodet nga 0x80<\/b><\/code> deri n\u00eb 0x7FF<\/b><\/code> \u2014 dy byte; nga 0x800<\/b><\/code> deri n\u00eb 0xFFFF<\/b><\/code> \u2014 tre, dhe nga 0x10000<\/b><\/code> deri n\u00eb 0x10FFFF<\/b><\/code> \u2014 kat\u00ebr. Nga nj\u00ebra an\u00eb, latinishtja ka p\u00ebrfituar: u rikthye kompatibiliteti me ASCII, dhe shp\u00ebrndarja \u00ebsht\u00eb m\u00eb e shp\u00ebrndar\u00eb \"e shp\u00ebrndar\u00eb\" nga 1 deri n\u00eb 4 byte. Por alfabetet e ndryshme nga latinishtja, fatkeq\u00ebsisht, nuk p\u00ebrfitojn\u00eb asgj\u00eb krahasuar me UTF-16, dhe shum\u00eb tani k\u00ebrkojn\u00eb n\u00eb fakt tre byte n\u00eb vend t\u00eb dy \u2014 diapazoni i mbuluar nga regjistrimi dy-byte \u00ebsht\u00eb ngushtuar 32 her\u00eb, nga 0xFFFF<\/b><\/code> deri n\u00eb 0x7FF<\/b><\/code>, dhe n\u00eb t\u00eb nuk p\u00ebrfshihet as kinezishtja, as, p\u00ebr shembull, gjeorgjishtja. Kirilika dhe pes\u00eb alfabete t\u00eb tjera \u2014 urime \u2014 kan\u00eb fat, 2 byte p\u00ebr simbol.<\/p>\n

Pse ndodh k\u00ebshtu? Le t\u00eb shohim se si UTF-8 paraqet kodet e simboleve:
\n\"Nj\u00eb
\nP\u00ebr t\u00eb paraqitur numrat k\u00ebtu jan\u00eb p\u00ebrdorur drejtp\u00ebrdrejt bit\u00ebt, t\u00eb sh\u00ebnuar me simbolin x<\/b><\/code>. \u00cbsht\u00eb e dukshme se n\u00eb regjistrimin me dy byte ka vet\u00ebm 11 bit\u00eb (nga 16). Bit\u00ebt udh\u00ebheq\u00ebs k\u00ebtu kan\u00eb vet\u00ebm funksion sh\u00ebrbimi. N\u00eb rastin e regjistrimit me kat\u00ebr byte, numri i pik\u00ebs koduese merr 21 bit\u00eb nga 32 \u2014 duket se k\u00ebtu do t\u00eb mjaftonin tre byte (t\u00eb cilat japin gjithsej 24 bit\u00eb), por markerat sh\u00ebrbimi konsumojn\u00eb shum\u00eb.<\/p>\n

A \u00ebsht\u00eb kjo nj\u00eb gj\u00eb e keqe? N\u00eb t\u00eb v\u00ebrtet\u00eb, jo shum\u00eb. Nga nj\u00ebra an\u00eb \u2014 n\u00ebse na intereson shum\u00eb hap\u00ebsira q\u00eb z\u00ebm\u00eb, kemi algoritme kompresimi q\u00eb leht\u00ebsisht do t\u00eb eliminojn\u00eb gjith\u00eb entropin\u00eb dhe tepric\u00ebn. Nga ana tjet\u00ebr \u2014 q\u00ebllimi i Unicode \u00ebsht\u00eb t\u00eb ofroj\u00eb kodim sa m\u00eb universale. P\u00ebr shembull, nj\u00eb varg i koduar n\u00eb UTF-8 mund t'i besohet nj\u00eb kodi q\u00eb m\u00eb par\u00eb punonte vet\u00ebm me ASCII, dhe nuk kemi frik\u00eb se ai do t\u00eb shoh\u00eb nj\u00eb karakter nga diapazoni ASCII q\u00eb n\u00eb t\u00eb v\u00ebrtet\u00eb nuk ekziston (sepse n\u00eb UTF-8, t\u00eb gjitha bajtat q\u00eb fillojn\u00eb me bitin zero jan\u00eb n\u00eb t\u00eb v\u00ebrtet\u00eb ASCII). Dhe n\u00ebse do t\u00eb donim t\u00eb prisnim nj\u00eb bisht t\u00eb vog\u00ebl nga nj\u00eb varg t\u00eb madh, pa e dekoduar at\u00eb nga fillimi (ose t\u00eb rikuperonim disa informacione pas nj\u00eb pjese t\u00eb d\u00ebmtuar) \u2014 nuk \u00ebsht\u00eb e v\u00ebshtir\u00eb t\u00eb gjendet duke filluar nga ndonj\u00eb simbol (mjafton t\u00eb kalosh mbi bajtat q\u00eb kan\u00eb prefiksin e bitit) 10<\/b><\/code>).<\/p>\n

Pse duhet t\u00eb shpikim di\u00e7ka t\u00eb re?<\/h2>\n

\nN\u00eb t\u00eb nj\u00ebjt\u00ebn koh\u00eb, ndonj\u00ebher\u00eb ndodhin situata kur algoritmet e kompresimit si deflate nuk jan\u00eb shum\u00eb t\u00eb aplikueshme, dhe d\u00ebshirojm\u00eb t\u00eb arrijm\u00eb ruajtjen kompakte t\u00eb vargjeve. Personal, kam p\u00ebrballur nj\u00eb detyr\u00eb t\u00eb till\u00eb, duke reflektuar mbi nd\u00ebrtimin t\u00eb nj\u00eb peme prefiks t\u00eb kompresuar<\/a><\/noindex> p\u00ebr nj\u00eb fjalor t\u00eb madh, q\u00eb p\u00ebrfshin fjal\u00eb n\u00eb gjuh\u00eb t\u00eb ndryshme. Nga nj\u00ebra an\u00eb \u2014 \u00e7do fjal\u00eb \u00ebsht\u00eb shum\u00eb e shkurt\u00ebr, k\u00ebshtu q\u00eb shkrirja e saj do t\u00eb ishte e pamjaftueshme. Nga ana tjet\u00ebr \u2014 implementimi i pem\u00ebs q\u00eb kam shqyrtuar, ishte parashikuar q\u00eb \u00e7do bajt i vargut t\u00eb ruajtur t\u00eb krijonte nj\u00eb maj\u00eb t\u00eb ve\u00e7ant\u00eb t\u00eb pem\u00ebs, k\u00ebshtu q\u00eb minimizimi i numrit t\u00eb tyre ishte shum\u00eb i dobish\u00ebm. N\u00eb bibliotek\u00ebn time Az.js<\/a><\/noindex> (ashtu si n\u00eb pymorphy2<\/a><\/noindex>, n\u00eb t\u00eb cil\u00ebn \u00ebsht\u00eb e bazuar) nj\u00eb problem i till\u00eb zgjidhet leht\u00ebsisht \u2014 vargjet e paketuar n\u00eb DAWG<\/a><\/noindex>-fjalor, ruhen atje n\u00eb CP1251 t\u00eb vjet\u00ebr t\u00eb mir\u00eb<\/a><\/noindex>. Por, si\u00e7 \u00ebsht\u00eb e leht\u00eb t\u00eb kuptohet, kjo funksionon mir\u00eb vet\u00ebm p\u00ebr nj\u00eb alfabet t\u00eb kufizuar \u2014 nj\u00eb varg n\u00eb kinezisht nuk mund t\u00eb ruhen n\u00eb nj\u00eb fjalor t\u00eb till\u00eb.<\/p>\n

Ve\u00e7mas do t\u00eb p\u00ebrmend nj\u00eb tjet\u00ebr nuanc\u00eb t\u00eb pak\u00ebndshme q\u00eb shfaqet kur p\u00ebrdoret UTF-8 n\u00eb nj\u00eb struktur\u00eb t\u00eb till\u00eb t\u00eb dh\u00ebnash. N\u00eb figur\u00ebn e m\u00ebsip\u00ebrme shihet se, kur regjistrohet simboli si dy bajta, bit\u00ebt q\u00eb i p\u00ebrkasin numrit t\u00eb tij, nuk ndodhin t\u00eb gjith\u00eb s\u00eb bashku, por jan\u00eb t\u00eb ndara nga nj\u00eb \u00e7ift bit\u00ebsh 10<\/b><\/code> n\u00eb mes: 110xxxxx 10xxxxxx<\/b><\/code>. P\u00ebr shkak t\u00eb k\u00ebsaj, kur n\u00eb kodin e simbolit mbushen 6 bit\u00ebt m\u00eb t\u00eb ul\u00ebt t\u00eb bajt\u00ebve t\u00eb dyt\u00eb (dmth. ndodh kalimi 10111111<\/b><\/code> \u2192 10000000<\/b><\/code>), at\u00ebher\u00eb nd\u00ebrlikohet edhe bajti i par\u00eb. Si rezultat, letra \u00ab\u043f\u00bb p\u00ebrfaq\u00ebsohet nga bajt\u00ebt 0xD0 0xBF<\/b><\/code>, dhe e ardhshme pas saj \"r\" \u2014 tashm\u00eb 0xD1 0x80<\/b><\/code>. N\u00eb pem\u00ebn prefiks, kjo \u00e7on n\u00eb shp\u00ebrb\u00ebrjen e maj\u00ebs prind n\u00eb dy \u2014 nj\u00eb p\u00ebr prefiksin 0xD0<\/b><\/code>, dhe nj\u00eb tjet\u00ebr p\u00ebr 0xD1<\/b><\/code> (edhe pse e gjith\u00eb cirilika mund t\u00eb kodifikohej vet\u00ebm me bajtin e dyt\u00eb).<\/p>\n

\u00c7far\u00eb kam arritur<\/h2>\n

\nDuke u p\u00ebrballur me k\u00ebt\u00eb sfid\u00eb, vendosa t\u00eb ushtrohem me lojra me bits, dhe gjithashtu t\u00eb njoh pak m\u00eb mir\u00eb struktur\u00ebn e Unicode-it n\u00eb p\u00ebrgjith\u00ebsi. Rezultati ishte formati i kodimit UTF-C (\"C\" nga kompakt<\/em>), q\u00eb shpenzon jo m\u00eb shum\u00eb se 3 bajta p\u00ebr nj\u00eb pik\u00eb kodimi, dhe shum\u00eb shpesh lejon q\u00eb t\u00eb shpenzoj\u00eb vet\u00ebm nj\u00eb bajt shtes\u00eb p\u00ebr t\u00ebr\u00eb rreshtin e koduar<\/strong>. Kjo b\u00ebn q\u00eb n\u00eb shum\u00eb alfabetet jo-ASCII, ky kodim t\u00eb jet\u00eb 30-60% m\u00eb kompakt se UTF-8<\/strong>.<\/p>\n

Kam organizuar shembuj t\u00eb zbatimeve t\u00eb algoritmeve t\u00eb kodimit dhe dekodimit n\u00eb form\u00ebn e bibliotekave n\u00eb JavaScript dhe Go<\/a><\/noindex>, ju mund t'i p\u00ebrdorni ato lirsh\u00ebm n\u00eb kodin tuaj. Por un\u00eb ende do t\u00eb theksoj q\u00eb n\u00eb nj\u00eb far\u00eb m\u00ebnyre ky format mbetet \"biciklet\u00eb\", dhe nuk e rekomandoj ta p\u00ebrdorni pa kuptuar se p\u00ebrse ju nevojitet<\/strong>. Kjo \u00ebsht\u00eb m\u00eb shum\u00eb nj\u00eb eksperiment sesa nj\u00eb \"p\u00ebrmir\u00ebsim serioz i UTF-8\". Megjithat\u00eb, kodi \u00ebsht\u00eb shkruar me kujdes, me p\u00ebrmbledhje t\u00eb qarta, komente t\u00eb shumta dhe mbulimin me teste.<\/p>\n

\"Nj\u00eb
\nRezultati i ekzekutimit t\u00eb testeve dhe krahasimi me UTF-8<\/i><\/p>\n

Gjithashtu kam b\u00ebr\u00eb nj\u00eb faqe demo<\/a><\/noindex>, ku mund t\u00eb vler\u00ebsoni pun\u00ebn e algoritmit, dhe m\u00eb pas do t\u00eb flas p\u00ebr parimet dhe procesin e zhvillimit m\u00eb n\u00eb detaje.<\/p>\n

Eliminimi i bit\u00ebve t\u00eb tep\u00ebrt<\/h2>\n

\nP\u00ebr baz\u00eb kam marr\u00eb, sigurisht, UTF-8. E para dhe m\u00eb e dukshme q\u00eb mund t\u00eb ndryshohet \u00ebsht\u00eb - t\u00eb reduktohet numri i bit\u00ebve ndihm\u00ebs n\u00eb \u00e7do byte. P\u00ebr shembull, byte i par\u00eb n\u00eb UTF-8 gjithmon\u00eb fillon ose me 0<\/b><\/code>, ose me 11<\/b><\/code> \u2014 dhe prefiksi 10<\/b><\/code> \u00ebsht\u00eb vet\u00ebm n\u00eb byte t\u00eb ardhsh\u00ebm. Do ta z\u00ebvend\u00ebsojm\u00eb prefiksin 11<\/b><\/code> n\u00eb 1<\/b><\/code>, dhe do t\u00eb heqim plot\u00ebsisht prefikset n\u00eb byte t\u00eb ardhsh\u00ebm. \u00c7far\u00eb do t\u00eb nxjerrim?<\/p>\n

0xxxxxxx<\/b><\/code> \u2014 1 byte
\n10xxxxxx xxxxxxxx<\/b><\/code> \u2014 2 byte
\n110xxxxx xxxxxxxx xxxxxxxx<\/b><\/code> \u2014 3 byte<\/p>\n

Stop, ku \u00ebsht\u00eb regjistrimi me kat\u00ebr byte? Nuk \u00ebsht\u00eb m\u00eb e nevojshme - me regjistrimin me tre byte tani kemi n\u00eb dispozicion 21 bit dhe k\u00ebt\u00eb e kemi m\u00ebse mjaftuesh\u00ebm p\u00ebr t\u00eb gjith\u00eb numrat deri n\u00eb 0x10FFFF<\/b><\/code>.<\/p>\n

\u00c7far\u00eb ndihmuam k\u00ebtu? E r\u00ebnd\u00ebsishmja \u00ebsht\u00eb zb discoverimi i kufijve t\u00eb simboleve nga nj\u00eb vend t\u00eb rast\u00ebsish\u00ebm n\u00eb tampon. Nuk mund t\u00eb gjuajm\u00eb n\u00eb nj\u00eb bajt t\u00eb rast\u00ebsish\u00ebm dhe t\u00eb gjejm\u00eb fillimin e simbolit tjet\u00ebr. Ky \u00ebsht\u00eb nj\u00eb kufizim i formatit ton\u00eb, por n\u00eb praktik\u00eb nevoja p\u00ebr nj\u00eb gj\u00eb t\u00eb till\u00eb ndodh rrall\u00eb. Zakonisht, jemi n\u00eb gjendje t\u00eb kalojm\u00eb tamponin nga fillimi (sidomos kur b\u00ebhet fjal\u00eb p\u00ebr rreshta t\u00eb shkurt\u00ebr).<\/p>\n

Situata me mbulimin e gjuh\u00ebve me 2 bajta gjithashtu \u00ebsht\u00eb p\u00ebrmir\u00ebsuar: tani formati dy-bajt\u00ebsh ofron nj\u00eb gam\u00eb prej 14 bit\u00ebsh, q\u00eb do t\u00eb thot\u00eb kode deri n\u00eb 0x3FFF<\/b><\/code>. Kinez\u00ebt nuk kan\u00eb fat (hieroglifet e tyre kryesisht bien n\u00eb gam\u00ebn nga 0x4E00<\/b><\/code> deri n\u00eb 0x9FFF<\/b><\/code>), por georgian\u00ebt dhe shum\u00eb popuj t\u00eb tjer\u00eb kan\u00eb pasur m\u00eb shum\u00eb g\u00ebzim \u2014 gjuh\u00ebt e tyre gjithashtu ndodhen n\u00eb 2 bajta p\u00ebr simbol.<\/p>\n

Jemi duke futur gjendjen e koduesit<\/h2>\n

\nLe t\u00eb mendojm\u00eb tani p\u00ebr vetit\u00eb e vet\u00eb rreshtave. N\u00eb fjalor zakonisht gjenden fjal\u00eb t\u00eb shkruara me simbole t\u00eb nj\u00eb alfabete, dhe kjo \u00ebsht\u00eb e v\u00ebrtet\u00eb p\u00ebr shum\u00eb tekste t\u00eb tjera gjithashtu. Do t\u00eb ishte mir\u00eb t\u00eb p\u00ebrcaktonim nj\u00eb her\u00eb k\u00ebt\u00eb alfabet, dhe pastaj t\u00eb tregojm\u00eb vet\u00ebm numrin e shkronj\u00ebs brenda tij. Le t\u00eb shohim n\u00ebse na ndihmon vendosja e simboleve n\u00eb tabel\u00ebn Unicode.<\/p>\n

Si\u00e7 u tha m\u00eb sip\u00ebr, Unicode \u00ebsht\u00eb ndar\u00eb n\u00eb planesh<\/em> me 65536 kodet p\u00ebr secil\u00ebn. Por kjo ndarje nuk \u00ebsht\u00eb shum\u00eb e dobishme (si\u00e7 \u00ebsht\u00eb th\u00ebn\u00eb, m\u00eb shpesh ne jemi n\u00eb planin zero). Nj\u00eb ndarje m\u00eb interesante \u00ebsht\u00eb ajo n\u00eb bloke.<\/em> K\u00ebto intervale tani nuk kan\u00eb nj\u00eb gjat\u00ebsi fikse dhe kan\u00eb m\u00eb shum\u00eb kuptim \u2014 si rregull, secili bashkon simbole t\u00eb nj\u00eb alfabeti.<\/p>\n

\"Nj\u00eb
\nBlloku q\u00eb p\u00ebrmban simbole t\u00eb alfabetit bengali. Fatkeq\u00ebsisht, p\u00ebr shkaqe historike, ky \u00ebsht\u00eb nj\u00eb shembull i paketimit t\u00eb ngjeshur jo shum\u00eb mir\u00eb \u2014 96 simbole jan\u00eb shp\u00ebrndar\u00eb n\u00eb m\u00ebnyr\u00eb kaotike n\u00eb 128 kodet e bllokut.<\/i><\/p>\n

Fillimet e bllokut dhe p\u00ebrmasat e tyre gjithmon\u00eb jan\u00eb shum\u00ebfish t\u00eb 16 \u2014 kjo u b\u00eb vet\u00ebm p\u00ebr leht\u00ebsi. P\u00ebr m\u00eb tep\u00ebr, shum\u00eb blloqe fillojn\u00eb dhe p\u00ebrfundohen n\u00eb vlera shum\u00ebfish t\u00eb 128 ose madje 256 \u2014 p\u00ebr shembull, kyrilika kryesore z\u00eb 256 byte nga 0x0400<\/b><\/code> deri n\u00eb 0x04FF<\/b><\/code>. Kjo \u00ebsht\u00eb shum\u00eb e p\u00ebrshtatshme: n\u00ebse nj\u00eb her\u00eb ruajm\u00eb prefiksin 0x04<\/b><\/code>, at\u00ebher\u00eb \u00e7do simbol kyrilik mund t\u00eb shkruhet me nj\u00eb byte. Megjithat\u00eb, k\u00ebshtu ne do t\u00eb humbim mund\u00ebsin\u00eb p\u00ebr t\u00eb kthyer n\u00eb ASCII (dhe \u00e7do simbol tjet\u00ebr n\u00eb p\u00ebrgjith\u00ebsi). Prandaj veprojm\u00eb k\u00ebshtu:<\/p>\n

    \n
  1. Dy byte 10yyyyyy yxxxxxxx<\/b><\/code> jo vet\u00ebm q\u00eb tregojn\u00eb simbolin me num\u00ebr yyyyyy yxxxxxxx<\/b><\/code>, por gjithashtu ndryshojn\u00eb alfabetin aktual<\/em> n\u00eb yyyyyy y0000000<\/b><\/code> (dmth. mbajm\u00eb t\u00eb gjith\u00eb bit\u00ebt p\u00ebrve\u00e7 atyre m\u00eb t\u00eb rinjve 7 bit<\/strong>);<\/li>\n
  2. Nj\u00eb byte 0xxxxxxx<\/b><\/code> ky \u00ebsht\u00eb simboli i alfabetit aktual. Thjesht duhet ta shtojm\u00eb me at\u00eb zhvendosje q\u00eb e mbajm\u00eb mend n\u00eb hapin 1. Deri tani nuk e kemi ndryshuar alfabetin, prandaj zhvendosja \u00ebsht\u00eb zero, k\u00ebshtu q\u00eb kemi ruajtur p\u00ebrputhshm\u00ebri me ASCII.<\/li>\n<\/ol>\n

    \nNj\u00ebsoj p\u00ebr kodet q\u00eb k\u00ebrkojn\u00eb 3 byte:<\/p>\n

      \n
    1. T\u00eb tre byte 110yyyyy yxxxxxxx xxxxxxxx<\/b><\/code> tregon simbolin me num\u00ebr yyyyyy yxxxxxxx xxxxxxxx<\/b><\/code>, ndryshon alfabetin aktual<\/em> n\u00eb yyyyyy y0000000 00000000<\/b><\/code> (mbajm\u00eb mend gjith\u00e7ka, p\u00ebrve\u00e7 m\u00eb t\u00eb riut 15 bit<\/strong>), dhe vendosim nj\u00eb flamur q\u00eb tani jemi n\u00eb modalitet t\u00eb gjat\u00eb<\/em> kjo do t\u00eb thot\u00eb gjat\u00eb p\u00ebrdorimit t\u00eb alfabetit t\u00eb dyta byte do ta reshtojm\u00eb k\u00ebt\u00eb flamur);<\/li>\n
    2. Dy byte 0xxxxxxx xxxxxxxx<\/b><\/code> n\u00eb modalitetin e gjat\u00eb, ky \u00ebsht\u00eb simboli i alfabetit aktual. Nj\u00ebsoj, ne e shtojm\u00eb at\u00eb me zhvendosjen nga hapi 1. E gjith\u00eb ndryshimi \u00ebsht\u00eb se tani lexojm\u00eb dy byte (sepse kemi kaluar n\u00eb k\u00ebt\u00eb modalitet).<\/li>\n<\/ol>\n

      \nD\u00ebgjon mir\u00eb: tani kur na nevojitet t\u00eb kodojm\u00eb simbolet nga nj\u00eb gam\u00eb t\u00eb nj\u00ebjt\u00eb 7-bit t\u00eb Unicode, ne harxhojm\u00eb 1 byte shtes\u00eb n\u00eb fillim dhe vet\u00ebm nj\u00eb byte p\u00ebr \u00e7do simbol.<\/p>\n

      \"Nj\u00eb
      \nFunksionimi i nj\u00eb versioni t\u00eb hersh\u00ebm. Tani shpesh kalon UTF-8, por ende ka hap\u00ebsira p\u00ebr p\u00ebrmir\u00ebsim.<\/i><\/p>\n

      \u00c7far\u00eb ka shkuar m\u00eb keq? S\u00eb pari, kemi nj\u00eb gjendje, dometh\u00ebn\u00eb zhvendosjen e alfabetit aktual<\/em> dhe flamurin modalitet t\u00eb gjat\u00eb<\/em>. Kjo na kufizon m\u00eb tej: tani t\u00eb nj\u00ebjtat simbole mund t\u00eb kodifikohen n\u00eb m\u00ebnyra t\u00eb ndryshme n\u00eb kontekste t\u00eb ndryshme. K\u00ebrkimi i n\u00ebnshkronjave, p\u00ebr shembull, do t\u00eb duhet t\u00eb b\u00ebhet duke marr\u00eb parasysh k\u00ebt\u00eb, e jo thjesht duke krahasuar bajt\u00ebt. N\u00eb t\u00eb dyt\u00ebn, sapo nd\u00ebrruam alfabetin, pati probleme me kodimin e simboleve ASCII (dhe kjo \u00ebsht\u00eb jo vet\u00ebm latinisht, por edhe pikaturat baz\u00eb, duke p\u00ebrfshir\u00eb hap\u00ebsirat) - ato k\u00ebrkojn\u00eb nj\u00eb nd\u00ebrrim t\u00eb p\u00ebrs\u00ebritur t\u00eb alfabetit n\u00eb 0, pra p\u00ebrs\u00ebri nj\u00eb bajt t\u00eb tep\u00ebrt (dhe pastaj edhe nj\u00eb tjet\u00ebr, p\u00ebr t'u kthyer te baza jon\u00eb).<\/p>\n

      Nj\u00eb alfabet \u00ebsht\u00eb mir\u00eb, dy - m\u00eb mir\u00eb<\/h2>\n

      \nLe t\u00eb provom\u00eb ta ndryshojm\u00eb pak prefiksin ton\u00eb t\u00eb bajt\u00ebve, duke shtuar nj\u00eb tjet\u00ebr p\u00ebrkat\u00ebsisht tri q\u00eb u p\u00ebrmend\u00ebn m\u00eb lart:<\/p>\n

      0xxxxxxx<\/b><\/code> \u2014 1 bajt n\u00eb modin normal, 2 n\u00eb at\u00eb t\u00eb gjat\u00eb
      \n11xxxxxx<\/b><\/code> \u2014 1 byte
      \n100xxxxx xxxxxxxx<\/b><\/code> \u2014 2 byte
      \n101xxxxx xxxxxxxx xxxxxxxx<\/b><\/code> \u2014 3 byte<\/p>\n

      \"Nj\u00eb
      \n
      \nTani, n\u00eb regjistrimin me dy bajt\u00eb, kemi nj\u00eb bajt m\u00eb t\u00eb pak\u00ebt t\u00eb disponuesh\u00ebm \u2014 kodohet deri te 0x1FFF<\/b><\/code>, jo 0x3FFF<\/b><\/code>. Megjithat\u00eb, \u00ebsht\u00eb ende ndjesh\u00ebm m\u00eb shum\u00eb se n\u00eb kodet me dy bajt\u00eb t\u00eb UTF-8, pjesa m\u00eb e madhe e gjuh\u00ebve t\u00eb zakonshme ende p\u00ebrfshihet, humbja m\u00eb e dukshme \u2014 ra       hiragana<\/a><\/noindex> dhe katakana<\/a><\/noindex>, japonez\u00ebt jan\u00eb n\u00eb trishtim.<\/p>\n

      \u00c7far\u00eb kod i ri 11xxxxxx<\/b><\/code>? \u042d\u0442\u043e \u043d\u0435\u0431\u043e\u043b\u044c\u0448\u043e\u0439 \u00ab\u0437\u0430\u0433\u0430\u0448\u043d\u0438\u043a\u00bb \u0440\u0430\u0437\u043c\u0435\u0440\u043e\u043c \u0432 64 \u0441\u0438\u043c\u0432\u043e\u043b\u0430, \u043e\u043d \u0434\u043e\u043f\u043e\u043b\u043d\u044f\u0435\u0442 \u043d\u0430\u0448 \u043e\u0441\u043d\u043e\u0432\u043d\u043e\u0439 \u0430\u043b\u0444\u0430\u0432\u0438\u0442, \u043f\u043e\u044d\u0442\u043e\u043c\u0443 \u044f \u043d\u0430\u0437\u0432\u0430\u043b \u0435\u0433\u043e \u0432\u0441\u043f\u043e\u043c\u043e\u0433\u0430\u0442\u0435\u043b\u044c\u043d\u044b\u043c (auxiliary<\/em>) alfabeti. Kur ne kalojm\u00eb alfabetin aktual, nj\u00eb pjes\u00eb e vjet\u00ebr e alfabetit b\u00ebhet ndihm\u00ebs. P\u00ebr shembull, kur kalojm\u00eb nga ASCII n\u00eb cirilik \u2014 tani n\u00eb \"dajnik\" kemi 64 simbole q\u00eb p\u00ebrmbajn\u00eb latinisht, numra, hap\u00ebsira dhe presje<\/strong> (inserton m\u00eb t\u00eb zakonshme n\u00eb tekstet jo-ASCII). Kur kthehemi p\u00ebrs\u00ebri n\u00eb ASCII \u2014 ndihm\u00ebs do t\u00eb b\u00ebhet pjesa kryesore e cirilikut.<\/p>\n

      Fal\u00eb aksesit n\u00eb dy alfabete, ne mund t\u00eb p\u00ebrballojm\u00eb shum\u00eb tekste, duke pasur shpenzime minimale p\u00ebr kalimin e alfabeteve (pik\u00ebrisht forma e shpesh do t\u00eb \u00e7oj\u00eb n\u00eb kthimin n\u00eb ASCII, por pas k\u00ebsaj shum\u00eb simbole jo-ASCII do t\u00eb nxirren nga alfabeti shtes\u00eb, pa nevoj\u00ebn p\u00ebr kalim t\u00eb p\u00ebrs\u00ebritur).<\/p>\n

      Bonus: duke e sh\u00ebnuar alfabetin ndihm\u00ebs me nj\u00eb prefiks 11xxxxxx<\/b><\/code> dhe duke zgjedhur offset-in e tij fillestar t\u00eb barabart\u00eb me 0xC0<\/b><\/code>, ne marrim pajtim t\u00eb pjessh\u00ebm me CP1252. Me fjal\u00eb t\u00eb tjera, shum\u00eb (por jo t\u00eb gjitha) tekste evropian\u00eb per\u00ebndimor\u00eb, t\u00eb koduara n\u00eb CP1252, do t\u00eb duken ashtu si edhe n\u00eb UTF-C.<\/p>\n

      K\u00ebtu, megjithat\u00eb, shfaqet nj\u00eb v\u00ebshtir\u00ebsi: si t\u00eb merrni alfabetin ndihm\u00ebs nga ai kryesor? Mund t\u00eb l\u00ebm\u00eb t\u00eb nj\u00ebjtin zhvendosje, por \u2014 fatkeq\u00ebsisht \u2014 struktura e Unicode tani punon kund\u00ebr nesh. Shum\u00eb shpesh, pjesa kryesore e alfabetit nuk ndodhet n\u00eb fillim t\u00eb bllokut (p\u00ebr shembull, shkronja e madhe ruse \u00ab\u0410\u00bb ka kod 0x0410<\/b><\/code>, megjithat\u00eb blloku kyriliks fillon me 0x0400<\/b><\/code>). K\u00ebshtu, duke marr\u00eb n\u00eb \u00abdokument\u00bb 64 simbolat e para, ndoshta do t\u00eb humbasim aksesin n\u00eb pjes\u00ebn e bishtit t\u00eb alfabetit.<\/p>\n

      P\u00ebr t\u00eb zgjidhur k\u00ebt\u00eb problem, kalova dorazi disa blloqe q\u00eb p\u00ebrputhen me gjuh\u00eb t\u00eb ndryshme, dhe tregova p\u00ebr to zhvendosjen e alfabetit ndihm\u00ebs brenda atij kryesor. Latinishten, p\u00ebrjashtim, e riordnon si nj\u00eb base64.<\/p>\n

      \"Nj\u00eb<\/p>\n

      Detajet p\u00ebrfundimtare<\/h2>\n

      \nLe t\u00eb mendojm\u00eb p\u00ebr fund se ku mund t\u00eb p\u00ebrmir\u00ebsojm\u00eb di\u00e7ka tjet\u00ebr.<\/p>\n

      V\u00ebrejm\u00eb se formati 101xxxxx xxxxxxxx xxxxxxxx<\/b><\/code> lejon kodimin e numrave deri n\u00eb 0x1FFFFF<\/b><\/code>, nd\u00ebrsa Unicode p\u00ebrfundon m\u00eb her\u00ebt, n\u00eb 0x10FFFF<\/b><\/code>. Me fjal\u00eb t\u00eb tjera, pika e fundit e kodit do t\u00eb p\u00ebrfaq\u00ebsohet si 10110000 11111111 11111111<\/b><\/code>. Pra, mund t\u00eb themi se n\u00ebse byte-i i par\u00eb ka pamjen 1011xxxx<\/b><\/code> (ku xxxx<\/b><\/code> n\u00ebse \u00ebsht\u00eb m\u00eb shum\u00eb se 0), at\u00ebher\u00eb ai p\u00ebrfaq\u00ebson di\u00e7ka tjet\u00ebr. P\u00ebr shembull, mund t\u00eb shtojm\u00eb edhe 15 simbole, q\u00eb jan\u00eb vazhdimisht t\u00eb disponueshme p\u00ebr kodim me nj\u00eb byte, por un\u00eb vendosa t\u00eb veproj ndryshe.<\/p>\n

      Prandaj zgjedhim disa gam\u00eb t\u00eb p\u00ebrzgjedhura, p\u00ebrkat\u00ebse p\u00ebr emoji, hiragana dhe katakana, i radhisim ato n\u00eb nj\u00eb list\u00eb t\u00eb vazhdueshme dhe i kodifikojm\u00eb n\u00eb form\u00eb t\u00eb dy byte n\u00eb vend t\u00eb tre:<\/p>\n

      1011xxxx xxxxxxxx<\/b><\/code> <\/p>\n

      Shk\u00eblqyesh\u00ebm: emoji i p\u00ebrmendur m\u00eb par\u00eb \u200d\u200d\u200d\"Nj\u00eb<\/a><\/noindex>, i cili p\u00ebrb\u00ebhet nga 7 pika kodimi, n\u00eb UTF-8 z\u00eb 25 byte, nd\u00ebrsa ne e p\u00ebrfshim\u00eb at\u00eb n\u00eb 14<\/strong> (pik\u00ebrisht dy byte p\u00ebr \u00e7do pik\u00eb kodi). P\u00ebrve\u00e7 k\u00ebsaj, Habr refuzoi ta p\u00ebrballonte (si n\u00eb redaktorin e vjet\u00ebr ashtu edhe n\u00eb at\u00eb t\u00eb ri), k\u00ebshtu q\u00eb u detyruam ta vendosim si imazh.<\/p>\n

      Do t\u00eb p\u00ebrpiqemi t\u00eb rregullojm\u00eb nj\u00eb problem tjet\u00ebr. Si\u00e7 e dim\u00eb, alfabeti kryesor \u00ebsht\u00eb n\u00eb thelb 6 bit\u00ebt e lart\u00eb<\/strong>, t\u00eb cilat i mbajm\u00eb mend, dhe i ngjisim me kodin e \u00e7do simbole t\u00eb dekoduara radhazi. N\u00eb rastin e karaktereve kineze, t\u00eb cilat gjenden n\u00eb bllokun 0x4E00<\/b><\/code> \u2013 0x9FFF<\/b><\/code>, jan\u00eb ose bit 0, ose 1. Kjo nuk \u00ebsht\u00eb shum\u00eb e rehatshme: do t\u00eb na duhet vazhdimisht t\u00eb kalojm\u00eb alfabetin midis k\u00ebtyre dy vlerave (dmth. t\u00eb shpenzojm\u00eb tre byte). Por le t\u00eb v\u00ebrejm\u00eb se n\u00eb modin e gjat\u00eb nga vet\u00eb kodi mund t\u00eb zbresim numrin e simboleve q\u00eb kodojm\u00eb me modin e shkurt\u00ebr (pas t\u00eb gjitha trukeve t\u00eb p\u00ebrshkruara m\u00eb sip\u00ebr, kjo \u00ebsht\u00eb 10240) \u2014 at\u00ebher\u00eb diapazoni i karaktereve do t\u00eb zhvendoset n\u00eb 0x2600<\/b><\/code> \u2013 0x77FF<\/b><\/code>, dhe n\u00eb k\u00ebt\u00eb rast n\u00eb t\u00eb gjith\u00eb k\u00ebt\u00eb diapazon 6 bit\u00ebt e lart\u00eb (nga 21) do t\u00eb jen\u00eb t\u00eb barabarta me 0. K\u00ebshtu, sekuencat e karaktereve do t\u00eb p\u00ebrdorin dy byte p\u00ebr karakter (\u00e7far\u00eb \u00ebsht\u00eb optimal p\u00ebr nj\u00eb diapazon kaq t\u00eb madh), pa shkaktuar kalime t\u00eb alfabetit. <\/p>\n

      Zgjidhje alternative: SCSU, BOCU-1<\/h2>\n

      \nEkspert\u00ebt e Unicode, vet\u00ebm duke lexuar titullin e artikullit, p\u00ebr siguri do t\u00eb nxitojn\u00eb t\u00eb kujtojn\u00eb se direkt n\u00eb mesin e standarteve t\u00eb Unicode ka Standard Compression Scheme for Unicode<\/a><\/noindex> (SCSU), i cili p\u00ebrshkruan nj\u00eb m\u00ebnyr\u00eb kodimi, p\u00ebr t\u00eb cilin \u00ebsht\u00eb shum\u00eb e ngjashme me at\u00eb t\u00eb p\u00ebrshkruar n\u00eb artikull.<\/p>\n

      E pranoj sinqerisht: p\u00ebr ekzistenc\u00ebn e tij kam m\u00ebsuar vet\u00ebm pasi u angazhova thell\u00ebsisht n\u00eb shkrimin e zgjidhjes s\u00eb vet. Sikur ta kisha ditur q\u00eb n\u00eb fillim, ndoshta do t\u00eb kisha provuar t\u00eb shkruaja implementimin e tij n\u00eb vend t\u00eb shpikjes s\u00eb nj\u00eb qasjeje t\u00eb re.<\/p>\n

      E \u00e7uditshme, SCSU p\u00ebrdor ide q\u00eb jan\u00eb shum\u00eb t\u00eb ngjashme me ato n\u00eb t\u00eb cilat kam arritur vet\u00eb (n\u00eb vend t\u00eb konceptit t\u00eb \"alfave\" aty p\u00ebrdoren \"dritaret\", dhe ka m\u00eb shum\u00eb se sa kam un\u00eb). Nd\u00ebrkoh\u00eb, ky format ka gjithashtu disavantazhe: \u00ebsht\u00eb pak m\u00eb af\u00ebr algoritmeve t\u00eb kompresimit, sesa kodimit. Sidomos, standardi ofron shum\u00eb m\u00ebnyra p\u00ebr p\u00ebrfaq\u00ebsim, por nuk thot\u00eb se si t\u00eb zgjedh\u00ebsh nga to optimalin \u2014 p\u00ebr k\u00ebt\u00eb, encoder-i duhet t\u00eb aplikoj\u00eb disa heuristika. Prandaj, encoder-i SCSU q\u00eb jep nj\u00eb paketim t\u00eb mir\u00eb, do t\u00eb jet\u00eb m\u00eb i komplikuar dhe m\u00eb i ngarkuar se algoritmi im.<\/p>\n

      P\u00ebr krahasim, kam transferuar nj\u00eb implementim relativisht t\u00eb thjesht\u00eb t\u00eb SCSU n\u00eb JavaScript \u2014 nga volumi i kodit, ai doli t\u00eb ishte i ngjash\u00ebm me UTF-C tim, por n\u00eb disa raste tregoi rezultate deri n\u00eb disa p\u00ebr qind m\u00eb keq (ndonj\u00ebher\u00eb mund t\u00eb kaloj\u00eb at\u00eb, por jo shum\u00eb). P\u00ebr shembull, tekstet n\u00eb hebraisht dhe greqisht UTF-C i kodova madje 60% m\u00eb mir\u00eb se SCSU<\/strong> (ndoshta p\u00ebr shkak t\u00eb alfabetit t\u00eb tyre t\u00eb kompakt).<\/p>\n

      Ve\u00e7mas do t\u00eb shtoj se p\u00ebrve\u00e7 SCSU ekziston gjithashtu nj\u00eb m\u00ebnyr\u00eb tjet\u00ebr p\u00ebr p\u00ebrfaq\u00ebsimin kompakt t\u00eb Unicode \u2014 BOCU-1<\/a><\/noindex>, por ai synon kompatibilitetin me MIME (\u00e7far\u00eb nuk m\u00eb nevojitej), dhe p\u00ebrdor nj\u00eb qasje pak m\u00eb t\u00eb ndryshme p\u00ebr kodimin. Efektivitetin e tij nuk e kam vler\u00ebsuar, por mendoj se nuk do t\u00eb jet\u00eb m\u00eb i lart\u00eb se SCSU.<\/p>\n

      P\u00ebrmir\u00ebsimet e mundshme<\/h2>\n

      \nAlgoritmi q\u00eb kam paraqitur nuk \u00ebsht\u00eb universale nga dizajni (n\u00eb k\u00ebt\u00eb, ndoshta, q\u00ebllimet e mia ndahen m\u00eb fort nga ato t\u00eb konsorciumit Unicode). E p\u00ebrmenda tashm\u00eb se ai \u00ebsht\u00eb zhvilluar kryesisht p\u00ebr nj\u00eb detyr\u00eb (ruajtjen e nj\u00eb fjalori shum\u00ebgjuh\u00ebsh n\u00eb nj\u00eb pem\u00eb prefiksesh), dhe disa karakteristika t\u00eb tij mund t\u00eb mos p\u00ebrshtaten mir\u00eb p\u00ebr detyra t\u00eb tjera. Por fakti q\u00eb ai nuk \u00ebsht\u00eb standard mund t\u00eb jet\u00eb gjithashtu nj\u00eb avantazh \u2014 mund t'i adaptoni leht\u00ebsisht sipas nevojave tuaja<\/strong>.<\/p>\n

      P\u00ebr shembull, \u00ebsht\u00eb qart\u00eb se mund t\u00eb hiqni gjendjen, duke e b\u00ebr\u00eb kodimin stateless \u2014 thjesht mos e azhurnoni variablin off<\/b><\/code>, auxOffs<\/b><\/code> dhe is21Bit<\/b><\/code> n\u00eb encoder dhe decoder. n\u00eb k\u00ebt\u00eb rast, nuk do t\u00eb jepet mund\u00ebsia p\u00ebr t\u00eb paketa efektivisht sekuenca simbolesh t\u00eb nj\u00eb alfabeti, por do t\u00eb ket\u00eb garanci q\u00eb nj\u00eb simbol i nj\u00ebjt\u00eb do t\u00eb kodifikohet gjithmon\u00eb me t\u00eb nj\u00ebjtat byte, pavar\u00ebsisht nga konteksti.<\/p>\n

      P\u00ebr m\u00eb tep\u00ebr, mund ta rregulloni koduesin p\u00ebr nj\u00eb gjuh\u00eb t\u00eb ve\u00e7ant\u00eb, duke ndryshuar gjendjen e paracaktuar \u2014 p\u00ebr shembull, duke u bazuar n\u00eb tekstet ruse, t\u00eb vendosni n\u00eb fillim t\u00eb encoder dhe decoder offs = 0x0400<\/b><\/code> dhe auxOffs = 0<\/b><\/code>. Kjo ka kuptim ve\u00e7an\u00ebrisht n\u00eb rastin e modit stateless. N\u00eb p\u00ebrgjith\u00ebsi, kjo do t\u00eb ishte e ngjashme me p\u00ebrdorimin e kodimit t\u00eb vjet\u00ebr t\u00eb tet\u00eb bit\u00ebve, vet\u00ebm se nuk e humbet mund\u00ebsin\u00eb p\u00ebr t\u00eb futur simbole nga e gjith\u00eb Unicode sipas nevoj\u00ebs.<\/p>\n

      Nj\u00eb tjet\u00ebr mang\u00ebsi, e p\u00ebrmendur m\u00eb par\u00eb - n\u00eb tekstin voluminoz t\u00eb koduar n\u00eb UTF-C, nuk ka nj\u00eb m\u00ebnyr\u00eb t\u00eb shpejt\u00eb p\u00ebr t\u00eb gjetur kufirin e simbolit m\u00eb t\u00eb af\u00ebrt me nj\u00eb byte t\u00eb rast\u00ebsish\u00ebm. Duke prer\u00eb nga bufferi i koduar, le t\u00eb themi, 100 byte t\u00eb fundit, rrezikoni t\u00eb merrni plehra, t\u00eb cilat nuk mund t\u00eb b\u00ebni asgj\u00eb me to. Kodimi nuk \u00ebsht\u00eb i p\u00ebrshtatur p\u00ebr ruajtjen e log-eve disa gigabajt, por p\u00ebrgjith\u00ebsisht, kjo mund t\u00eb korrigjohet. Byte 0xBF<\/b><\/code> kurr\u00eb nuk duhet t\u00eb ndodh\u00eb si byte i par\u00eb (por mund t\u00eb jet\u00eb i dyti ose i tret\u00eb). Prandaj, gjat\u00eb kodimit, mund t\u00eb futni nj\u00eb sekuenc\u00eb 0xBF 0xBF 0xBF<\/b><\/code> \u00e7do 10 KB, le t\u00eb themi - at\u00ebher\u00eb, n\u00ebse \u00ebsht\u00eb e nevojshme, p\u00ebr t\u00eb gjetur kufirin do t\u00eb mjaftoj\u00eb t\u00eb skanoni cop\u00ebn e zgjedhur derisa t\u00eb gjeni nj\u00eb marker t\u00eb till\u00eb. Pas t\u00eb fundit 0xBF<\/b><\/code> garantohet se do t\u00eb jet\u00eb fillimi i simbolit. (Gjat\u00eb dekodimit, sigurisht q\u00eb kjo sekuenc\u00eb prej tre byteve do t\u00eb injorohet.)<\/p>\n

      N\u00eb p\u00ebrfundim<\/h2>\n

      \nN\u00ebse keni lexuar deri k\u00ebtu - urime! Shpresoj se, ashtu si un\u00eb, keni m\u00ebsuar di\u00e7ka t\u00eb re (ose keni rifreskuar di\u00e7ka t\u00eb vjet\u00ebr) rreth funksionit t\u00eb Unicode.<\/p>\n

      \"Nj\u00eb
      \nFaqja demonstruese. N\u00eb shembullin e hebraisht, shihen avantazhet si ndaj UTF-8, ashtu edhe ndaj SCSU.<\/i><\/p>\n

      Mos t\u00eb shikoni hulumtimet e p\u00ebrmendura m\u00eb sip\u00ebr si nj\u00eb shqet\u00ebsim p\u00ebr standardet. Megjithat\u00eb, n\u00eb p\u00ebrgjith\u00ebsi jam i k\u00ebnaqur me rezultatet e pun\u00ebs sime, prandaj kam k\u00ebnaq\u00ebsi q\u00eb i ndaj ato. ndaj<\/a><\/noindex>: p\u00ebr shembull, biblioteka JS n\u00eb form\u00eb t\u00eb minimizuar peshon vet\u00ebm 1710 byte (pa var\u00ebsi, sigurisht). Si\u00e7 e p\u00ebrmenda m\u00eb sip\u00ebr, mund t\u00eb njiheni me funksionimin e saj n\u00eb faqen_demo<\/a><\/noindex> (aty ka edhe nj\u00eb set tekstit me t\u00eb cilin mund ta krahasoni me UTF-8 dhe SCSU).<\/p>\n

      N\u00eb fund, do t\u00eb theksoj p\u00ebrs\u00ebri rastet e p\u00ebrdorimit t\u00eb UTF-C. nuk duhet<\/b>:<\/p>\n

      Le t\u00eb shohim ato blloqe t\u00eb Unicode q\u00eb k\u00ebrkojn\u00eb tani tre byte. Kryesisht, si\u00e7 u tha m\u00eb par\u00eb, k\u00ebto jan\u00eb hieroglifet kineze \u2014 por me to \u00ebsht\u00eb e v\u00ebshtir\u00eb t\u00eb b\u00ebsh di\u00e7ka, ka 21 mij\u00eb. Por gjithashtu, aty fluturoi hiragana dhe katakana \u2014 dhe ato nuk jan\u00eb aq shum\u00eb, m\u00eb pak se dyqind. Dhe, duke qen\u00eb se p\u00ebrmend\u00ebm japonez\u00ebt \u2014 aty ndodhen edhe emoji (n\u00eb t\u00eb v\u00ebrtet\u00eb ata jan\u00eb shp\u00ebrndar\u00eb shum\u00eb kudo n\u00eb Unicode, por blloqet kryesore jan\u00eb n\u00eb gam\u00ebn 0x1F300<\/b><\/code> \u2013 0x1FBFF<\/b><\/code>). N\u00ebse mendojm\u00eb p\u00ebr at\u00eb q\u00eb tani ekzistojn\u00eb emoji q\u00eb nd\u00ebrtohen nga disa pika kodimi (p\u00ebr shembull, emoji \u200d\u200d\u200d\"Nj\u00eb<\/a><\/noindex> p\u00ebrdor 7 kode!), \u00ebsht\u00eb v\u00ebrtet e trishtueshme t\u00eb shpenzojm\u00eb p\u00ebr \u00e7do nj\u00eb nga tre byte (7\u00d73 = 21 byte p\u00ebr nj\u00eb simbol, \u00ebsht\u00eb nj\u00eb makth).<\/p>\n