{"id":95911,"date":"2020-10-05T01:42:09","date_gmt":"2020-10-04T23:42:09","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/eshhyo-odin-velosiped-hranim-yunikodnye-stroki-na-30-60-kompaktnee-chem-utf-8"},"modified":"2020-10-05T01:42:09","modified_gmt":"2020-10-04T23:42:09","slug":"eshhyo-odin-velosiped-hranim-yunikodnye-stroki-na-30-60-kompaktnee-chem-utf-8","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/eshhyo-odin-velosiped-hranim-yunikodnye-stroki-na-30-60-kompaktnee-chem-utf-8","title":{"rendered":"Edhe nj\u00eb \"shpikje rrote\": ruajm\u00eb vargjet Unicode 30\u201360% m\u00eb kompakt se UTF-8","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

\"Edhe
\n
\nN\u00ebse jeni zhvillues dhe duhet t\u00eb zgjidhni nj\u00eb kodim, pothuajse gjithmon\u00eb zgjedhja e duhur \u00ebsht\u00eb Unicode. M\u00ebnyra konkrete e p\u00ebrfaq\u00ebsimit varet nga konteksti, por n\u00eb shumic\u00ebn e rasteve edhe k\u00ebtu ka nj\u00eb p\u00ebrgjigje universale \u2014 UTF-8. Ai \u00ebsht\u00eb i mir\u00eb sepse ju lejon t\u00eb p\u00ebrdorni t\u00eb gjitha simbolet e Unicode pa shpenzuar tep\u00ebr<\/em> shum\u00eb byte n\u00eb shumic\u00ebn e rasteve. Megjithat\u00eb, p\u00ebr gjuh\u00ebt q\u00eb nuk p\u00ebrdorin vet\u00ebm alfabetin latin, \u201cjo shum\u00eb\u201d do t\u00eb thot\u00eb t\u00eb pakt\u00ebn dy byte p\u00ebr simbol<\/strong>. A mund t\u00eb b\u00ebhet m\u00eb mir\u00eb, pa u kthyer te kodimet parahistorike q\u00eb na kufizojn\u00eb n\u00eb vet\u00ebm 256 simbole t\u00eb disponueshme?<\/p>\n

M\u00eb posht\u00eb po ju prezantoj p\u00ebrpjekjen time p\u00ebr t\u2019iu p\u00ebrgjigjur k\u00ebsaj pyetjeje dhe nj\u00eb zbatim t\u00eb nj\u00eb algoritmi relativisht t\u00eb thjesht\u00eb, q\u00eb lejon ruajtjen e vargjeve n\u00eb shumic\u00ebn e gjuh\u00ebve t\u00eb bot\u00ebs pa shtuar at\u00eb tepric\u00eb q\u00eb ekziston n\u00eb UTF-8.<\/a><\/noindex><\/p>\n

Diskleimer.<\/i> Q\u00eb n\u00eb fillim do t\u00eb b\u00ebj disa sqarime t\u00eb r\u00ebnd\u00ebsishme: zgjidhja e p\u00ebrshkruar nuk propozohet si z\u00ebvend\u00ebsim universal p\u00ebr UTF-8<\/strong>, ajo \u00ebsht\u00eb e p\u00ebrshtatshme vet\u00ebm p\u00ebr nj\u00eb num\u00ebr t\u00eb kufizuar rastesh (p\u00ebr to m\u00eb posht\u00eb) dhe n\u00eb asnj\u00eb m\u00ebnyr\u00eb nuk duhet p\u00ebrdorur p\u00ebr nd\u00ebrveprim me API t\u00eb pal\u00ebve t\u00eb treta (t\u00eb cilat as q\u00eb din\u00eb p\u00ebr t\u00eb). N\u00eb shumic\u00ebn e rasteve, p\u00ebr ruajtje kompakte t\u00eb v\u00ebllimeve t\u00eb m\u00ebdha t\u00eb t\u00eb dh\u00ebnave tekstuale, jan\u00eb m\u00eb t\u00eb p\u00ebrshtatsh\u00ebm algoritmet e kompresimit me p\u00ebrdorim t\u00eb p\u00ebrgjithsh\u00ebm (p\u00ebr shembull, deflate). P\u00ebr m\u00eb tep\u00ebr, tashm\u00eb gjat\u00eb krijimit t\u00eb zgjidhjes sime gjeta nj\u00eb standard ekzistues brenda vet\u00eb Unicode q\u00eb zgjidh t\u00eb nj\u00ebjt\u00ebn detyr\u00eb \u2014 ai \u00ebsht\u00eb disi m\u00eb kompleks (dhe jo rrall\u00eb m\u00eb i dob\u00ebt), por gjithsesi \u00ebsht\u00eb nj\u00eb standard i pranuar, jo nj\u00eb zgjidhje e improvizuar. Edhe p\u00ebr t\u00eb do t\u00eb flas.<\/p>\n

P\u00ebr Unicode dhe UTF-8<\/h2>\n

\nP\u00ebr t\u00eb filluar \u2014 disa fjal\u00eb se \u00e7far\u00eb jan\u00eb n\u00eb t\u00eb v\u00ebrtet\u00eb Unicode<\/strong> dhe UTF-8<\/strong>.<\/p>\n

Si\u00e7 dihet, dikur kodimet 8-bit\u00ebshe ishin shum\u00eb t\u00eb p\u00ebrhapura. Me to gjith\u00e7ka ishte e thjesht\u00eb: 256 simbole mund t\u00eb num\u00ebroheshin me vlera nga 0 deri n\u00eb 255, dhe numrat nga 0 deri n\u00eb 255 p\u00ebrfaq\u00ebsohen natyrsh\u00ebm me nj\u00eb byte. N\u00ebse kthehemi te fillesat, at\u00ebher\u00eb kodimi ASCII madje kufizohet n\u00eb 7 bite, prandaj biti m\u00eb i lart\u00eb n\u00eb paraqitjen e tij me byte \u00ebsht\u00eb zero, dhe shumica e kodimeve 8-bit\u00ebshe jan\u00eb n\u00eb p\u00ebrputhje me t\u00eb (ato ndryshojn\u00eb vet\u00ebm n\u00eb pjes\u00ebn \u201ce sip\u00ebrme\u201d, ku biti m\u00eb i lart\u00eb \u00ebsht\u00eb nj\u00eb).<\/p>\n

\u00c7far\u00eb e dallon Unicode nga ato kodime dhe pse me t\u00eb lidhen disa formate konkrete nj\u00ebher\u00ebsh \u2014 UTF-8, UTF-16 (BE dhe LE), UTF-32? Le ta shqyrtojm\u00eb me radh\u00eb.<\/p>\n

Standardi baz\u00eb i Unicode p\u00ebrshkruan vet\u00ebm p\u00ebrputhjen midis simboleve (dhe n\u00eb disa raste edhe midis p\u00ebrb\u00ebr\u00ebsve t\u00eb ve\u00e7ant\u00eb t\u00eb simboleve) dhe numrave t\u00eb tyre. Dhe numrat e mundsh\u00ebm n\u00eb k\u00ebt\u00eb standard jan\u00eb shum\u00eb t\u00eb shumt\u00eb \u2014 nga 0x00<\/b><\/code> n\u00eb 0x10FFFF<\/b><\/code> (1 114 112 gjithsej). N\u00ebse do t\u00eb donim t\u00eb ruanim nj\u00eb num\u00ebr nga ky interval n\u00eb nj\u00eb variab\u00ebl, as 1 dhe as 2 bajt nuk do t\u00eb mjaftonin. Dhe meq\u00eb procesor\u00ebt tan\u00eb nuk jan\u00eb fort t\u00eb optimizuar p\u00ebr pun\u00eb me numra trebajt\u00ebsh, do t\u00eb detyroheshim t\u00eb p\u00ebrdornim plot 4 bajt p\u00ebr nj\u00eb simbol t\u00eb vet\u00ebm! Kjo \u00ebsht\u00eb pik\u00ebrisht UTF-32, por p\u00ebr shkak t\u00eb k\u00ebtij \u201cshpenzimi\u201d t\u00eb tepruar ky format nuk \u00ebsht\u00eb i p\u00ebrhapur.<\/p>\n

P\u00ebr fat t\u00eb mir\u00eb, simbolet brenda Unicode nuk jan\u00eb renditur rast\u00ebsisht. I gjith\u00eb grupi i tyre ndahet n\u00eb 17 \u201cplane<\/em>\u201d, secila prej t\u00eb cilave p\u00ebrmban 65536 (0x10000<\/b><\/code>) \u00abpika kodi<\/em>\u201d. Koncepti i \u201cpik\u00ebs s\u00eb kodit\u201d k\u00ebtu \u00ebsht\u00eb thjesht numri i simbolit<\/em>, q\u00eb i \u00ebsht\u00eb caktuar nga Unicode. Por, si\u00e7 u p\u00ebrmend m\u00eb sip\u00ebr, n\u00eb Unicode jan\u00eb t\u00eb num\u00ebruar jo vet\u00ebm simbolet e ve\u00e7anta, por edhe p\u00ebrb\u00ebr\u00ebsit e tyre dhe shenjat sh\u00ebrbyese (madje ndonj\u00ebher\u00eb nj\u00eb numri nuk i korrespondon asgj\u00eb \u2014 ndoshta vet\u00ebm p\u00ebrkoh\u00ebsisht, por p\u00ebr ne kjo nuk ka shum\u00eb r\u00ebnd\u00ebsi), prandaj \u00ebsht\u00eb m\u00eb e sakt\u00eb t\u00eb flitet gjithmon\u00eb p\u00ebr vet\u00eb numrat, jo p\u00ebr simbolet. Megjithat\u00eb, m\u00eb posht\u00eb p\u00ebr shkurt\u00ebsi do ta p\u00ebrdor shpesh fjal\u00ebn \u201csimbol\u201d, duke pasur parasysh termin \u201cpik\u00eb kodi\u201d.<\/p>\n

\"Edhe
\nPlanet e Unicode. Si\u00e7 shihet, pjesa m\u00eb e madhe (planet nga 4 deri n\u00eb 13) ende nuk \u00ebsht\u00eb p\u00ebrdorur.<\/i><\/p>\n

Ekzistencialisht, e gjith\u00eb \"mishi\" q\u00ebndron n\u00eb nivelin zero, i quajtur \"Basic Multilingual Plane<\/em>\". N\u00ebse rreshti p\u00ebrmban tekst n\u00eb nj\u00eb nga gjuh\u00ebt moderne (duke p\u00ebrfshir\u00eb mandarin), nuk do t\u00eb dilni p\u00ebrtej k\u00ebtij niveli. Por nuk mund ta prisni pjes\u00ebn tjet\u00ebr t\u00eb Unicode t\u00eb gjithashtu \u2014 p\u00ebr shembull, emotikon\u00ebt zakonisht ndodhen n\u00eb fund t\u00eb nivelit q\u00eb vjen pas, \"Supplementary Multilingual Plane<\/em>\" (ai shtrihet nga 0x10000<\/b><\/code> n\u00eb 0x1FFFF<\/b><\/code>). Prandaj UTF-16 vepron k\u00ebshtu: t\u00eb gjitha simbolet q\u00eb bien n\u00eb Basic Multilingual Plane<\/em>, kodohen \u00absi\u00e7 jan\u00eb\u00bb, me numrin dybajt\u00ebsh q\u00eb u korrespondon. Megjithat\u00eb, nj\u00eb pjes\u00eb e numrave n\u00eb k\u00ebt\u00eb interval nuk p\u00ebrfaq\u00ebsojn\u00eb fare karaktere t\u00eb caktuara, por tregojn\u00eb se pas k\u00ebsaj dysheje bajt\u00ebsh duhet t\u00eb merret n\u00eb shqyrtim edhe nj\u00eb tjet\u00ebr \u2014 duke kombinuar vlerat e k\u00ebtyre kat\u00ebr bajt\u00ebve, marrim nj\u00eb num\u00ebr q\u00eb mbulon t\u00eb gjith\u00eb intervalin e lejuar t\u00eb Unicode. Ky paraqitje quhet \u00ab\u00e7ifte surrogate\u00bb \u2014 ndoshta keni d\u00ebgjuar p\u00ebr to.<\/p>\n

K\u00ebshtu, UTF-16 k\u00ebrkon dy ose (n\u00eb raste shum\u00eb t\u00eb rralla) kat\u00ebr bajt\u00eb p\u00ebr nj\u00eb \u00abpik\u00eb kodi\u00bb. Kjo \u00ebsht\u00eb m\u00eb mir\u00eb sesa t\u00eb p\u00ebrdoren vazhdimisht kat\u00ebr bajt\u00eb, por alfabeti latin (dhe simbolet e tjera ASCII) me k\u00ebt\u00eb kodim shpenzon gjysm\u00ebn e hap\u00ebsir\u00ebs p\u00ebr zero. UTF-8 synon ta korrigjoj\u00eb k\u00ebt\u00eb: ASCII n\u00eb t\u00eb z\u00eb, si m\u00eb par\u00eb, vet\u00ebm nj\u00eb bajt; kodet nga 0x80<\/b><\/code> n\u00eb 0x7FF<\/b><\/code> \u2014 dy bajt\u00eb; nga 0x800<\/b><\/code> n\u00eb 0xFFFF<\/b><\/code> \u2014 tre, nd\u00ebrsa nga 0x10000<\/b><\/code> n\u00eb 0x10FFFF<\/b><\/code> \u2014 kat\u00ebr. Nga nj\u00ebra an\u00eb, alfabeti latin p\u00ebrfitoi: u rikthye p\u00ebrputhshm\u00ebria me ASCII dhe shp\u00ebrndarja u b\u00eb m\u00eb e nj\u00ebtrajtshme, e \u00abshtrir\u00eb\u00bb nga 1 deri n\u00eb 4 bajt\u00eb. Por alfabetet jo latine, p\u00ebr fat t\u00eb keq, nuk fitojn\u00eb asgj\u00eb krahasuar me UTF-16, madje shum\u00eb prej tyre tani k\u00ebrkojn\u00eb tre bajt\u00eb n\u00eb vend t\u00eb dyve \u2014 intervali q\u00eb mbulohet nga shkrimi dybajt\u00ebsh \u00ebsht\u00eb ngushtuar 32 her\u00eb, nga 0xFFFF<\/b><\/code> n\u00eb 0x7FF<\/b><\/code>, dhe n\u00eb t\u00eb nuk p\u00ebrfshihen as mandarin, as, p\u00ebr shembull, gjeorgjian. Kirilika dhe pes\u00eb alfabete t\u00eb tjera \u2014 hurra \u2014 kan\u00eb fat, 2 byte p\u00ebr simbol.<\/p>\n

Pse ndodh k\u00ebshtu? Le t\u00eb shohim se si UTF-8 i paraqet kodet e karaktereve:
\n\"Edhe
\nP\u00ebr paraqitjen e drejtp\u00ebrdrejt\u00eb t\u00eb numrave k\u00ebtu jan\u00eb p\u00ebrdorur bit\u00ebt e sh\u00ebnuar me simbolin x<\/b><\/code>. Shihet se n\u00eb shkrimin dybajt\u00ebsh ka vet\u00ebm 11 bit\u00eb t\u00eb till\u00eb (nga 16). Bit\u00ebt udh\u00ebheq\u00ebs k\u00ebtu kan\u00eb vet\u00ebm funksion sh\u00ebrbyes. N\u00eb rastin e shkrimit kat\u00ebrbajt\u00ebsh, p\u00ebr numrin e pik\u00ebs s\u00eb kodit jan\u00eb caktuar vet\u00ebm 21 bit\u00eb nga 32 \u2014 n\u00eb dukje, k\u00ebtu do t\u00eb mjaftonin edhe tre bajt\u00eb (q\u00eb japin gjithsej 24 bit\u00eb), por sh\u00ebnuesit sh\u00ebrbyes z\u00ebn\u00eb tep\u00ebr shum\u00eb.<\/p>\n

A \u00ebsht\u00eb kjo keq? N\u00eb fakt, jo edhe aq. Nga nj\u00ebra an\u00eb, n\u00ebse na shqet\u00ebson shum\u00eb hap\u00ebsira e z\u00ebn\u00eb, kemi algoritme kompresimi q\u00eb e eliminojn\u00eb leht\u00ebsisht gjith\u00eb entropin\u00eb dhe tepric\u00ebn e panevojshme. Nga ana tjet\u00ebr, q\u00ebllimi i Unicode ishte t\u00eb ofronte nj\u00eb kodim sa m\u00eb universal. P\u00ebr shembull, nj\u00eb varg i koduar n\u00eb UTF-8 mund t\u2019ia besojm\u00eb kodit q\u00eb m\u00eb par\u00eb punonte vet\u00ebm me ASCII, pa u shqet\u00ebsuar se ai do t\u00eb ndesh\u00eb aty nj\u00eb simbol nga diapazoni ASCII q\u00eb n\u00eb t\u00eb v\u00ebrtet\u00eb nuk ekziston aty (sepse n\u00eb UTF-8 t\u00eb gjith\u00eb bajtat q\u00eb fillojn\u00eb me bit zero jan\u00eb pik\u00ebrisht ASCII). Dhe n\u00ebse papritur duam t\u00eb presim nj\u00eb bisht t\u00eb vog\u00ebl nga nj\u00eb varg i madh, pa e dekoduar nga fillimi (ose t\u00eb rikuperojm\u00eb nj\u00eb pjes\u00eb t\u00eb informacionit pas nj\u00eb segmenti t\u00eb d\u00ebmtuar) \u2014 nuk \u00ebsht\u00eb e v\u00ebshtir\u00eb t\u00eb gjejm\u00eb zhvendosjen ku fillon nj\u00eb simbol (mjafton t\u00eb anashkalohen bajtat q\u00eb kan\u00eb prefiksin bitor 10<\/b><\/code>).<\/p>\n

At\u00ebher\u00eb pse t\u00eb shpikim di\u00e7ka t\u00eb re?<\/h2>\n

\nMegjithat\u00eb, her\u00eb pas here ka situata kur algoritmet e kompresimit si deflate nuk zbatohen mir\u00eb, nd\u00ebrsa ruajtja kompakte e vargjeve mbetet e d\u00ebshirueshme. Personalisht u ndesha me nj\u00eb detyr\u00eb t\u00eb till\u00eb, duke menduar p\u00ebr nd\u00ebrtimin e nj\u00eb trie t\u00eb kompresuar me prefikse<\/a><\/noindex> p\u00ebr nj\u00eb fjalor t\u00eb madh q\u00eb p\u00ebrfshin fjal\u00eb n\u00eb gjuh\u00eb t\u00eb ndryshme. Nga nj\u00ebra an\u00eb, \u00e7do fjal\u00eb \u00ebsht\u00eb shum\u00eb e shkurt\u00ebr, prandaj kompresimi i saj nuk do t\u00eb ishte efikas. Nga ana tjet\u00ebr, implementimi i pem\u00ebs q\u00eb po shqyrtoja ishte nd\u00ebrtuar mbi parimin q\u00eb \u00e7do bajt i vargut t\u00eb ruajtur krijonte nj\u00eb nyje t\u00eb ve\u00e7ant\u00eb t\u00eb pem\u00ebs, ndaj minimizimi i numrit t\u00eb tyre ishte shum\u00eb i dobish\u00ebm. N\u00eb bibliotek\u00ebn time Az.js<\/a><\/noindex> (si edhe n\u00eb pymorphy2<\/a><\/noindex>, mbi t\u00eb cil\u00ebn ajo bazohet) ky problem zgjidhet thjesht \u2014 vargjet e paketuara n\u00eb DAWG<\/a><\/noindex>-fjalor ruhen aty n\u00eb CP1251 t\u00eb vjet\u00ebr dhe t\u00eb mir\u00eb<\/a><\/noindex>. Por, si\u00e7 kuptohet leht\u00eb, kjo funksionon mir\u00eb vet\u00ebm p\u00ebr nj\u00eb alfabet t\u00eb kufizuar \u2014 nj\u00eb varg n\u00eb kinezisht nuk mund t\u00eb futet m\u00eb n\u00eb nj\u00eb fjalor t\u00eb till\u00eb.<\/p>\n

Ve\u00e7mas dua t\u00eb theksoj edhe nj\u00eb nuanc\u00eb tjet\u00ebr t\u00eb pak\u00ebndshme q\u00eb shfaqet kur p\u00ebrdoret UTF-8 n\u00eb nj\u00eb struktur\u00eb t\u00eb till\u00eb t\u00eb dh\u00ebnash. N\u00eb figur\u00ebn m\u00eb sip\u00ebr shihet se kur nj\u00eb simbol shkruhet me dy bajta, bitet q\u00eb i p\u00ebrkasin numrit t\u00eb tij nuk vendosen nj\u00ebri pas tjetrit, por nd\u00ebrpriten nga nj\u00eb pal\u00eb bitesh 10<\/b><\/code> n\u00eb mes: 110xxxxx 10xxxxxx<\/b><\/code>. P\u00ebr shkak t\u00eb k\u00ebsaj, kur n\u00eb kodin e simbolit mbushen 6 bitet m\u00eb t\u00eb ul\u00ebta t\u00eb bajtit t\u00eb dyt\u00eb (dometh\u00ebn\u00eb ndodh kalimi 10111111<\/b><\/code> \u2192 10000000<\/b><\/code>), at\u00ebher\u00eb ndryshon edhe bajti i par\u00eb. Si rezultat, shkronja \u00ab\u043f\u00bb p\u00ebrfaq\u00ebsohet nga bajtat 0xD0 0xBF<\/b><\/code>, nd\u00ebrsa \u00ab\u0440\u00bb q\u00eb vjen pas saj \u00ebsht\u00eb tashm\u00eb 0xD1 0x80<\/b><\/code>. N\u00eb pem\u00ebn e prefikseve kjo \u00e7on n\u00eb ndarjen e nyj\u00ebs prind n\u00eb dy pjes\u00eb \u2014 nj\u00ebra p\u00ebr prefiksin 0xD0<\/b><\/code>, dhe tjetra p\u00ebr 0xD1<\/b><\/code> (edhe pse i gjith\u00eb alfabeti cirilik mund t\u00eb kodohej vet\u00ebm me bajtin e dyt\u00eb).<\/p>\n

\u00c7far\u00eb arrita t\u00eb krijoj<\/h2>\n

\nKur u p\u00ebrballa me k\u00ebt\u00eb detyr\u00eb, vendosa t\u00eb ushtrohesha pak me manipulimin e biteve dhe nj\u00ebkoh\u00ebsisht t\u00eb njihesha m\u00eb mir\u00eb me struktur\u00ebn e Unicode n\u00eb p\u00ebrgjith\u00ebsi. Si rezultat doli formati i kodimit UTF-C (\u00abC\u00bb nga compact<\/em>), i cili p\u00ebrdor jo m\u00eb shum\u00eb se 3 bajt p\u00ebr nj\u00eb pik\u00eb kodi dhe shum\u00eb shpesh lejon t\u00eb p\u00ebrdoret vet\u00ebm nj\u00eb bajt shtes\u00eb p\u00ebr gjith\u00eb vargun q\u00eb po kodohet<\/strong>. Kjo b\u00ebn q\u00eb, p\u00ebr shum\u00eb alfabete jo-ASCII, ky kodim t\u00eb jet\u00eb 30-60% m\u00eb kompakt se UTF-8<\/strong>.<\/p>\n

I kam p\u00ebrgatitur shembujt e implementimit t\u00eb algoritmeve t\u00eb kodimit dhe dekodimit n\u00eb form\u00ebn e bibliotekave p\u00ebr JavaScript dhe Go<\/a><\/noindex>, t\u00eb cilat mund t\u2019i p\u00ebrdorni lirisht n\u00eb kodin tuaj. Megjithat\u00eb, dua t\u00eb theksoj se n\u00eb nj\u00ebfar\u00eb kuptimi ky format mbetet nj\u00eb \u00abzgjidhje artizanale\u00bb, dhe nuk rekomandoj ta p\u00ebrdorni pa e kuptuar qart\u00eb pse ju nevojitet<\/strong>. N\u00eb fund t\u00eb fundit, ky \u00ebsht\u00eb m\u00eb shum\u00eb nj\u00eb eksperiment sesa nj\u00eb \u00abp\u00ebrmir\u00ebsim serioz i UTF-8\u00bb. Megjithat\u00eb, kodi aty \u00ebsht\u00eb shkruar me kujdes, n\u00eb m\u00ebnyr\u00eb koncize, me shum\u00eb komente dhe me mbulim testesh.<\/p>\n

\"Edhe
\nRezultati i ekzekutimit t\u00eb testeve dhe krahasimi me UTF-8<\/i><\/p>\n

Gjithashtu kam krijuar nj\u00eb faqe demo<\/a><\/noindex>, ku mund t\u00eb vler\u00ebsoni funksionimin e algoritmit, nd\u00ebrsa m\u00eb posht\u00eb do t\u00eb tregoj m\u00eb holl\u00ebsisht p\u00ebr parimet e tij dhe procesin e zhvillimit.<\/p>\n

Heqim bit\u00ebt e tep\u00ebrt<\/h2>\n

\nSi baz\u00eb mora, sigurisht, UTF-8. Gj\u00ebja e par\u00eb dhe m\u00eb e dukshme q\u00eb mund t\u00eb ndryshohet aty \u00ebsht\u00eb ulja e numrit t\u00eb biteve sh\u00ebrbyes n\u00eb \u00e7do bajt. P\u00ebr shembull, bajti i par\u00eb n\u00eb UTF-8 gjithmon\u00eb fillon ose me 0<\/b><\/code>, ose me 11<\/b><\/code> \u2014 nd\u00ebrsa prefiksi 10<\/b><\/code> gjendet vet\u00ebm te bajtet pasuese. Le ta z\u00ebvend\u00ebsojm\u00eb prefiksin 11<\/b><\/code> n\u00eb 1<\/b><\/code>, nd\u00ebrsa te bajtet pasuese t\u2019i heqim fare prefikset. \u00c7far\u00eb do t\u00eb dal\u00eb?<\/p>\n

0xxxxxxx<\/b><\/code> \u2014 1 bajt
\n10xxxxxx xxxxxxxx<\/b><\/code> \u2014 2 bajte
\n110xxxxx xxxxxxxx xxxxxxxx<\/b><\/code> \u2014 3 bajte<\/p>\n

Ndal, po ku shkoi shkrimi me kat\u00ebr bajte? Ai nuk nevojitet m\u00eb \u2014 me shkrimin n\u00eb tre bajte tani kemi n\u00eb dispozicion 21 bit dhe kjo mjafton me tepric\u00eb p\u00ebr t\u00eb gjith\u00eb numrat deri n\u00eb 0x10FFFF<\/b><\/code>.<\/p>\n

\u00c7far\u00eb sakrifikuam k\u00ebtu? M\u00eb kryesorja \u2014 zbulimin e kufijve t\u00eb simboleve nga nj\u00eb pozicion arbitrar n\u00eb buffer. Nuk mund t\u00eb tregojm\u00eb nj\u00eb bajt \u00e7far\u00ebdo dhe prej tij t\u00eb gjejm\u00eb fillimin e simbolit pasues. Ky \u00ebsht\u00eb nj\u00eb kufizim i formatit ton\u00eb, por n\u00eb praktik\u00eb nevoja p\u00ebr k\u00ebt\u00eb lind rrall\u00eb. Zakonisht mund ta p\u00ebrshkojm\u00eb buffer-in q\u00eb nga fillimi (sidomos kur b\u00ebhet fjal\u00eb p\u00ebr vargje t\u00eb shkurtra).<\/p>\n

Situata me mbulimin e gjuh\u00ebve me 2 bajte \u00ebsht\u00eb p\u00ebrmir\u00ebsuar gjithashtu: tani formati dybajt\u00ebsh jep nj\u00eb diapazon prej 14 bit\u00ebsh, pra kode deri n\u00eb 0x3FFF<\/b><\/code>. Kinez\u00ebt nuk kan\u00eb shum\u00eb fat (hieroglifet e tyre ndodhen kryesisht n\u00eb intervalin nga 0x4E00<\/b><\/code> n\u00eb 0x9FFF<\/b><\/code>), por p\u00ebr gjeorgjian\u00ebt dhe shum\u00eb popuj t\u00eb tjer\u00eb gj\u00ebrat jan\u00eb b\u00ebr\u00eb m\u00eb t\u00eb mira \u2014 gjuh\u00ebt e tyre gjithashtu futen n\u00eb 2 bajte p\u00ebr simbol.<\/p>\n

Vendosim gjendjen e enkoderit<\/h2>\n

\nTani le t\u00eb mendojm\u00eb p\u00ebr vetit\u00eb e vet\u00eb vargjeve. N\u00eb fjalor zakonisht gjenden fjal\u00eb t\u00eb shkruara me simbole t\u00eb t\u00eb nj\u00ebjtit alfabet, dhe kjo vlen edhe p\u00ebr shum\u00eb tekste t\u00eb tjera. Do t\u00eb ishte mir\u00eb ta p\u00ebrcaktonim k\u00ebt\u00eb alfabet nj\u00eb her\u00eb, e m\u00eb pas t\u00eb tregonim vet\u00ebm numrin e shkronj\u00ebs brenda tij. Le t\u00eb shohim n\u00ebse mund t\u00eb na ndihmoj\u00eb vendosja e simboleve n\u00eb tabel\u00ebn Unicode.<\/p>\n

Si\u00e7 u tha m\u00eb sip\u00ebr, Unicode ndahet n\u00eb plane<\/em> me nga 65536 kode secili. Por kjo nuk \u00ebsht\u00eb nj\u00eb ndarje shum\u00eb e dobishme (si\u00e7 u p\u00ebrmend, m\u00eb shpesh ndodhemi n\u00eb planin zero). M\u00eb interesant \u00ebsht\u00eb ndarja n\u00eb blloqe.<\/em> K\u00ebto intervale nuk kan\u00eb m\u00eb gjat\u00ebsi fikse dhe mbartin m\u00eb shum\u00eb kuptim \u2014 si rregull, secili bashkon simbolet e nj\u00eb alfabeti.<\/p>\n

\"Edhe
\nBlloku q\u00eb p\u00ebrmban simbolet e alfabetit bengalez. Fatkeq\u00ebsisht, p\u00ebr arsye historike, ky \u00ebsht\u00eb nj\u00eb shembull i paketimit jo shum\u00eb t\u00eb dendur \u2014 96 simbole jan\u00eb shp\u00ebrndar\u00eb n\u00eb m\u00ebnyr\u00eb kaotike n\u00eb 128 pika kodi t\u00eb bllokut.<\/i><\/p>\n

Fillimet e blloqeve dhe madh\u00ebsit\u00eb e tyre jan\u00eb gjithmon\u00eb shum\u00ebfish i 16 \u2014 kjo \u00ebsht\u00eb b\u00ebr\u00eb thjesht p\u00ebr leht\u00ebsi. P\u00ebrve\u00e7 k\u00ebsaj, shum\u00eb blloqe fillojn\u00eb dhe mbarojn\u00eb me vlera shum\u00ebfish i 128 apo edhe 256 \u2014 p\u00ebr shembull, \u043a\u0438\u0440\u0438\u043b\u0438\u0446\u0430 baz\u00eb z\u00eb 256 bajte nga 0x0400<\/b><\/code> n\u00eb 0x04FF<\/b><\/code>. Kjo \u00ebsht\u00eb mjaft e p\u00ebrshtatshme: n\u00ebse ruajm\u00eb nj\u00eb her\u00eb prefiksin 0x04<\/b><\/code>, at\u00ebher\u00eb m\u00eb pas \u00e7do simbol \u043a\u0438\u0440\u0438\u043b\u0438\u043a mund t\u00eb shkruhet me nj\u00eb bajt. V\u00ebrtet, k\u00ebshtu humbasim mund\u00ebsin\u00eb p\u00ebr t'u kthyer te ASCII (dhe te \u00e7do simbol tjet\u00ebr n\u00eb p\u00ebrgjith\u00ebsi). Prandaj veprojm\u00eb k\u00ebshtu:<\/p>\n

    \n
  1. Dy bajte 10yyyyyy yxxxxxxx<\/b><\/code> jo vet\u00ebm q\u00eb tregojn\u00eb simbolin me numrin Yyyyyy yxxxxxxx<\/b><\/code>, por edhe ndryshojn\u00eb alfabetin aktual<\/em> n\u00eb yyyyyy y0000000<\/b><\/code> (d.m.th. ruajm\u00eb t\u00eb gjith\u00eb bit\u00ebt, p\u00ebrve\u00e7 7 bit\u00ebve<\/strong>);<\/li>\n
  2. Nj\u00eb bajt 0xxxxxxx<\/b><\/code> \u00ebsht\u00eb simbol i alfabetit aktual. Mjafton ta mbledhim me zhvendosjen q\u00eb ruajt\u00ebm n\u00eb hapin 1. P\u00ebr sa koh\u00eb q\u00eb alfabetin nuk e kemi ndryshuar, zhvendosja \u00ebsht\u00eb zero, k\u00ebshtu q\u00eb p\u00ebrputhshm\u00ebrin\u00eb me ASCII e kemi ruajtur.<\/li>\n<\/ol>\n

    \nN\u00eb m\u00ebnyr\u00eb t\u00eb ngjashme p\u00ebr kodet q\u00eb k\u00ebrkojn\u00eb 3 bajte:<\/p>\n

      \n
    1. Tre bajte 110yyyyy yxxxxxxx xxxxxxxx<\/b><\/code> tregojn\u00eb simbolin me numrin yyyyyy yxxxxxxx xxxxxxxx<\/b><\/code>, ndryshojn\u00eb alfabetin aktual<\/em> n\u00eb yyyyyy y0000000 00000000<\/b><\/code> (ruajt\u00ebm gjith\u00e7ka, p\u00ebrve\u00e7 15 bit\u00ebve<\/strong>), dhe vendosin nj\u00eb flamur q\u00eb tani jemi n\u00eb regjimin t\u00eb gjat\u00eb<\/em> n\u00eb regjim (kur alfabeti kthehet s\u00ebrish n\u00eb dybajt\u00ebsh, k\u00ebt\u00eb flamur do ta \u00e7aktivizojm\u00eb);<\/li>\n
    2. Dy bajte 0xxxxxxx xxxxxxxx<\/b><\/code> n\u00eb regjimin e gjat\u00eb ky \u00ebsht\u00eb simboli i alfabetit aktual. Po ashtu, e mbledhim me zhvendosjen nga hapi 1. I vetmi ndryshim \u00ebsht\u00eb se tani lexojm\u00eb nga dy bajte (sepse kemi kaluar n\u00eb k\u00ebt\u00eb regjim).<\/li>\n<\/ol>\n

      \nTing\u00ebllon mir\u00eb: tani, p\u00ebr sa koh\u00eb q\u00eb duhet t\u00eb kodojm\u00eb simbole nga i nj\u00ebjti diapazon 7-bit\u00ebsh i Unicode, harxhojm\u00eb 1 bajt shtes\u00eb n\u00eb fillim dhe vet\u00ebm nga nj\u00eb bajt p\u00ebr \u00e7do simbol.<\/p>\n

      \"Edhe
      \nPuna e nj\u00ebrit prej versioneve t\u00eb hershme. Tashm\u00eb jo rrall\u00eb e kalon UTF-8, por ka ende vend p\u00ebr p\u00ebrmir\u00ebsim.<\/i><\/p>\n

      \u00c7far\u00eb u p\u00ebrkeq\u00ebsua? S\u00eb pari, u shfaq gjendja, p\u00ebrkat\u00ebsisht zhvendosja e alfabetit aktual<\/em> dhe flamuri i regjimit t\u00eb gjat\u00eb<\/em>. Kjo na kufizon m\u00eb shum\u00eb: tani t\u00eb nj\u00ebjtat simbole mund t\u00eb kodohen ndryshe n\u00eb kontekste t\u00eb ndryshme. K\u00ebrkimi i n\u00ebnvargjeve, p\u00ebr shembull, tashm\u00eb duhet t\u00eb b\u00ebhet duke e marr\u00eb parasysh k\u00ebt\u00eb, e jo thjesht duke krahasuar bajtet. S\u00eb dyti, sapo e ndryshuam alfabetin, kodimi i simboleve ASCII u b\u00eb m\u00eb i pap\u00ebrshtatsh\u00ebm (dhe kjo nuk \u00ebsht\u00eb vet\u00ebm latinishtja, por edhe pik\u00ebsimi baz\u00eb, p\u00ebrfshir\u00eb hap\u00ebsirat) \u2014 ato k\u00ebrkojn\u00eb rikthim t\u00eb alfabetit n\u00eb 0, pra s\u00ebrish nj\u00eb bajt shtes\u00eb (dhe m\u00eb pas edhe nj\u00eb tjet\u00ebr p\u00ebr t'u kthyer te alfabeti yn\u00eb kryesor).<\/p>\n

      Nj\u00eb alfabet \u00ebsht\u00eb mir\u00eb, dy jan\u00eb m\u00eb mir\u00eb<\/h2>\n

      \nLe t\u00eb provojm\u00eb t\u2019i ndryshojm\u00eb pak prefikset tona bit, duke i shtuar edhe nj\u00eb tjet\u00ebr tre t\u00eb p\u00ebrshkruara m\u00eb sip\u00ebr:<\/p>\n

      0xxxxxxx<\/b><\/code> \u2014 1 bajt n\u00eb regjim normal, 2 n\u00eb t\u00eb gjat\u00eb
      \n11xxxxxx<\/b><\/code> \u2014 1 bajt
      \n100xxxxx xxxxxxxx<\/b><\/code> \u2014 2 bajte
      \n101xxxxx xxxxxxxx xxxxxxxx<\/b><\/code> \u2014 3 bajte<\/p>\n

      \"Edhe
      \n
      \nTani, n\u00eb shkrimin dybajt\u00ebsh ka nj\u00eb bit m\u00eb pak t\u00eb disponuesh\u00ebm \u2014 futen code point-et deri te 0x1FFF<\/b><\/code>, e jo 0x3FFF<\/b><\/code>. Megjithat\u00eb, kjo \u00ebsht\u00eb ende duksh\u00ebm m\u00eb shum\u00eb se n\u00eb kodet dybajt\u00ebshe t\u00eb UTF-8, shumica e gjuh\u00ebve t\u00eb zakonshme ende p\u00ebrfshihen, humbja m\u00eb e dukshme \u00ebsht\u00eb se mbet\u00ebn jasht\u00eb       hiragana<\/a><\/noindex> dhe katakana<\/a><\/noindex>, japonez\u00ebt jan\u00eb t\u00eb m\u00ebrzitur.<\/p>\n

      Po \u00e7far\u00eb \u00ebsht\u00eb kodi i ri 11xxxxxx<\/b><\/code>? \u042d\u0442\u043e \u043d\u0435\u0431\u043e\u043b\u044c\u0448\u043e\u0439 \u00ab\u0437\u0430\u0433\u0430\u0448\u043d\u0438\u043a\u00bb \u0440\u0430\u0437\u043c\u0435\u0440\u043e\u043c \u0432 64 \u0441\u0438\u043c\u0432\u043e\u043b\u0430, \u043e\u043d \u0434\u043e\u043f\u043e\u043b\u043d\u044f\u0435\u0442 \u043d\u0430\u0448 \u043e\u0441\u043d\u043e\u0432\u043d\u043e\u0439 \u0430\u043b\u0444\u0430\u0432\u0438\u0442, \u043f\u043e\u044d\u0442\u043e\u043c\u0443 \u044f \u043d\u0430\u0437\u0432\u0430\u043b \u0435\u0433\u043e \u0432\u0441\u043f\u043e\u043c\u043e\u0433\u0430\u0442\u0435\u043b\u044c\u043d\u044b\u043c (auxiliary<\/em>) alfabet. Kur nd\u00ebrrojm\u00eb alfabetin aktual, nj\u00eb pjes\u00eb e alfabetit t\u00eb vjet\u00ebr b\u00ebhet ndihm\u00ebs. P\u00ebr shembull, kaluam nga ASCII n\u00eb cirilik \u2014 tani \u201cn\u00eb rezerv\u00eb\u201d kemi 64 simbole, q\u00eb p\u00ebrmbajn\u00eb latinishten, shifrat, hap\u00ebsir\u00ebn dhe presjen<\/strong> (futjet m\u00eb t\u00eb shpeshta n\u00eb tekstet jo-ASCII). U kthyem s\u00ebrish te ASCII \u2014 dhe alfabet ndihm\u00ebs do t\u00eb b\u00ebhet pjesa kryesore e cirilikut.<\/p>\n

      Fal\u00eb qasjes n\u00eb dy alfabete, mund t\u00eb p\u00ebrpunojm\u00eb m\u00eb shum\u00eb tekste me kosto minimale p\u00ebr kalimin mes alfabeteve (shenjat e pik\u00ebsimit shpesh do t\u00eb sjellin kthim n\u00eb ASCII, por pas k\u00ebsaj shum\u00eb simbole jo-ASCII do t\u2019i marrim tashm\u00eb nga alfabeti shtes\u00eb, pa kaluar s\u00ebrish).<\/p>\n

      Bonus: duke e sh\u00ebnuar alfabetin shtes\u00eb me prefiksin 11xxxxxx<\/b><\/code> dhe duke zgjedhur zhvendosjen e tij fillestare t\u00eb barabart\u00eb me 0xC0<\/b><\/code>, marrim p\u00ebrputhshm\u00ebri t\u00eb pjesshme me CP1252. Me fjal\u00eb t\u00eb tjera, shum\u00eb (por jo t\u00eb gjitha) tekste t\u00eb Evrop\u00ebs Per\u00ebndimore t\u00eb koduara n\u00eb CP1252 do t\u00eb duken nj\u00ebsoj edhe n\u00eb UTF-C.<\/p>\n

      K\u00ebtu, megjithat\u00eb, lind nj\u00eb v\u00ebshtir\u00ebsi: si ta marrim alfabetin ndihm\u00ebs nga ai kryesor? Mund t\u00eb l\u00ebm\u00eb t\u00eb nj\u00ebjt\u00ebn zhvendosje, por, p\u00ebr fat t\u00eb keq, k\u00ebtu struktura e Unicode tashm\u00eb punon kund\u00ebr nesh. Shum\u00eb shpesh pjesa kryesore e alfabetit nuk ndodhet n\u00eb fillim t\u00eb bllokut (p\u00ebr shembull, shkronja e madhe ruse \u00ab\u0410\u00bb ka kodin 0x0410<\/b><\/code>, megjith\u00ebse blloku cirilik fillon me 0x0400<\/b><\/code>). K\u00ebshtu, duke rezervuar 64 simbolet e para, mund t\u00eb humbasim qasjen te pjesa fundore e alfabetit.<\/p>\n

      P\u00ebr ta zgjidhur k\u00ebt\u00eb problem, kalova manualisht n\u00ebp\u00ebr disa blloqe q\u00eb u korrespondojn\u00eb gjuh\u00ebve t\u00eb ndryshme dhe p\u00ebrcaktova p\u00ebr to zhvendosjen e alfabetit ndihm\u00ebs brenda atij kryesor. Latinishten, si p\u00ebrjashtim, madje e riorganizova sipas modelit t\u00eb base64.<\/p>\n

      \"Edhe<\/p>\n

      Prekjet p\u00ebrfundimtare<\/h2>\n

      \nLe t\u00eb mendojm\u00eb n\u00eb fund se ku tjet\u00ebr mund t\u00eb p\u00ebrmir\u00ebsojm\u00eb di\u00e7ka.<\/p>\n

      V\u00ebrejm\u00eb se formati 101xxxxx xxxxxxxx xxxxxxxx<\/b><\/code> lejon kodimin e numrave deri n\u00eb 0x1FFFFF<\/b><\/code>, nd\u00ebrsa Unicode p\u00ebrfundon m\u00eb her\u00ebt, te 0x10FFFF<\/b><\/code>. Me fjal\u00eb t\u00eb tjera, pika e fundit e kodit do t\u00eb paraqitet si 10110000 11111111 11111111<\/b><\/code>. Prandaj, mund t\u00eb themi se n\u00ebse bajti i par\u00eb ka form\u00ebn 1011xxxx<\/b><\/code> (ku xxxx<\/b><\/code> \u00ebsht\u00eb m\u00eb i madh se 0), at\u00ebher\u00eb ai sh\u00ebnon di\u00e7ka tjet\u00ebr. P\u00ebr shembull, aty mund t\u00eb shtohen edhe 15 simbole t\u00eb tjera, gjithmon\u00eb t\u00eb disponueshme p\u00ebr kodim me nj\u00eb bajt, por un\u00eb vendosa t\u00eb veproj ndryshe.<\/p>\n

      Prandaj zgjedhim disa intervale t\u00eb ve\u00e7anta q\u00eb u korrespondojn\u00eb emoji-ve, hiragan\u00ebs dhe katakan\u00ebs, i rinum\u00ebrojm\u00eb n\u00eb nj\u00eb list\u00eb t\u00eb vetme t\u00eb pand\u00ebrprer\u00eb dhe i kodojm\u00eb me dy bajte n\u00eb vend t\u00eb treve:<\/p>\n

      1011xxxx xxxxxxxx<\/b><\/code> <\/p>\n

      Shk\u00eblqyesh\u00ebm: emoji i p\u00ebrmendur m\u00eb sip\u00ebr \u200d\u200d\u200d\"Edhe<\/a><\/noindex>, i cili p\u00ebrb\u00ebhet nga 7 pika kodi, n\u00eb UTF-8 z\u00eb 25 bajte, nd\u00ebrsa ne e fut\u00ebm n\u00eb 14<\/strong> (sakt\u00ebsisht nga dy bajte p\u00ebr \u00e7do pik\u00eb kodi). Meq\u00eb ra fjala, Habr refuzoi ta p\u00ebrpunonte (si n\u00eb editorin e vjet\u00ebr, ashtu edhe n\u00eb t\u00eb riun), ndaj m'u desh ta fus si figur\u00eb.<\/p>\n

      Le t\u00eb p\u00ebrpiqemi t\u00eb zgjidhim edhe nj\u00eb problem tjet\u00ebr. Si\u00e7 kujtojm\u00eb, alfabeti baz\u00eb \u00ebsht\u00eb n\u00eb thelb 6 bit\u00ebt e sip\u00ebrm<\/strong>, t\u00eb cil\u00ebt i mbajm\u00eb mend dhe ia ngjisim kodit t\u00eb \u00e7do simboli t\u00eb radh\u00ebs q\u00eb dekodojm\u00eb. N\u00eb rastin e hieroglifeve kineze, t\u00eb cilat ndodhen n\u00eb bllokun 0x4E00<\/b><\/code> \u2013 0x9FFF<\/b><\/code>, kjo \u00ebsht\u00eb ose biti 0, ose 1. Kjo nuk \u00ebsht\u00eb shum\u00eb e p\u00ebrshtatshme: do t\u00eb na duhet ta nd\u00ebrrojm\u00eb vazhdimisht alfabetin midis k\u00ebtyre dy vlerave (dometh\u00ebn\u00eb t\u00eb shpenzojm\u00eb nga tre bajte). Por v\u00ebm\u00eb re se n\u00eb modalitetin e gjat\u00eb nga vet\u00eb kodi mund t\u00eb zbresim numrin e simboleve q\u00eb i kodojm\u00eb me modalitetin e shkurt\u00ebr (pas t\u00eb gjitha marifeteve t\u00eb p\u00ebrshkruara m\u00eb sip\u00ebr, kjo \u00ebsht\u00eb 10240) \u2014 at\u00ebher\u00eb intervali i hieroglifeve do t\u00eb zhvendoset te 0x2600<\/b><\/code> \u2013 0x77FF<\/b><\/code>, dhe n\u00eb k\u00ebt\u00eb rast n\u00eb t\u00eb gjith\u00eb k\u00ebt\u00eb interval 6 bit\u00ebt e sip\u00ebrm (nga 21) do t\u00eb jen\u00eb t\u00eb barabart\u00eb me 0. K\u00ebshtu, sekuencat e hieroglifeve do t\u00eb p\u00ebrdorin nga dy bajte p\u00ebr hieroglif (\u00e7ka \u00ebsht\u00eb optimale p\u00ebr nj\u00eb interval kaq t\u00eb madh), pa shkaktuar nd\u00ebrrime t\u00eb alfabetit. <\/p>\n

      Zgjidhje alternative: SCSU, BOCU-1<\/h2>\n

      \nNjoh\u00ebsit e Unicode, vet\u00ebm duke lexuar titullin e artikullit, me shum\u00eb gjas\u00eb do t\u00eb nxitojn\u00eb t\u00eb kujtojn\u00eb se drejtp\u00ebrdrejt mes standardeve t\u00eb Unicode ekziston Standard Compression Scheme for Unicode<\/a><\/noindex> (SCSU), i cili p\u00ebrshkruan nj\u00eb m\u00ebnyr\u00eb kodimi mjaft t\u00eb ngjashme me at\u00eb t\u00eb p\u00ebrshkruar n\u00eb artikull.<\/p>\n

      E pranoj sinqerisht: p\u00ebr ekzistenc\u00ebn e tij m\u00ebsova vet\u00ebm pasi isha zhytur thell\u00eb n\u00eb shkrimin e zgjidhjes sime. Po ta kisha ditur q\u00eb n\u00eb fillim, me gjas\u00eb do t\u00eb isha p\u00ebrpjekur t\u00eb shkruaja implementimin e tij n\u00eb vend q\u00eb t\u00eb shpikja qasjen time.<\/p>\n

      \u00cbsht\u00eb interesante q\u00eb SCSU p\u00ebrdor ide shum\u00eb t\u00eb ngjashme me ato te t\u00eb cilat arrita vet\u00eb (n\u00eb vend t\u00eb konceptit t\u00eb \u00abalfabeteve\u00bb aty p\u00ebrdoren \u00abdritare\u00bb, dhe ato jan\u00eb m\u00eb t\u00eb shumta se n\u00eb zgjidhjen time). N\u00eb t\u00eb nj\u00ebjt\u00ebn koh\u00eb, ky format ka edhe mang\u00ebsi: ai \u00ebsht\u00eb pak m\u00eb af\u00ebr algoritmeve t\u00eb kompresimit sesa kodimit. N\u00eb ve\u00e7anti, standardi ofron shum\u00eb m\u00ebnyra paraqitjeje, por nuk thot\u00eb si t\u00eb zgjidhet varianti optimal \u2014 p\u00ebr k\u00ebt\u00eb, enkoderi duhet t\u00eb p\u00ebrdor\u00eb disa heuristika. Si rrjedhoj\u00eb, nj\u00eb enkoder SCSU q\u00eb jep kompresim t\u00eb mir\u00eb do t\u00eb jet\u00eb m\u00eb kompleks dhe m\u00eb i r\u00ebnd\u00eb se algoritmi im.<\/p>\n

      P\u00ebr krahasim, un\u00eb portova n\u00eb JavaScript nj\u00eb implementim relativisht t\u00eb thjesht\u00eb t\u00eb SCSU-s\u00eb \u2014 p\u00ebr nga v\u00ebllimi i kodit doli i krahasuesh\u00ebm me UTF-C tim, por n\u00eb nj\u00eb s\u00ebr\u00eb rastesh tregoi rezultat dhjet\u00ebra p\u00ebr qind m\u00eb t\u00eb dob\u00ebt (ndonj\u00ebher\u00eb mund ta tejkaloj\u00eb edhe ai, por jo me shum\u00eb). P\u00ebr shembull, tekstet n\u00eb hebraisht dhe greqisht UTF-C i kodoi madje 60% m\u00eb mir\u00eb se SCSU<\/strong> (me shum\u00eb gjasa p\u00ebr shkak t\u00eb alfabeteve t\u00eb tyre kompakte).<\/p>\n

      Ve\u00e7mas do t\u00eb shtoj se, p\u00ebrve\u00e7 SCSU, ekziston edhe nj\u00eb m\u00ebnyr\u00eb tjet\u00ebr p\u00ebr paraqitje kompakte t\u00eb Unicode \u2014 BOCU-1<\/a><\/noindex>, por ai synon p\u00ebrputhshm\u00ebrin\u00eb me MIME (gj\u00eb q\u00eb mua nuk m\u00eb nevojitej) dhe p\u00ebrdor nj\u00eb qasje disi tjet\u00ebr ndaj kodimit. Efikasitetin e tij nuk e kam vler\u00ebsuar, por m\u00eb duket se v\u00ebshtir\u00eb se do t\u00eb jet\u00eb m\u00eb i lart\u00eb se ai i SCSU-s\u00eb.<\/p>\n

      P\u00ebrmir\u00ebsime t\u00eb mundshme<\/h2>\n

      \nAlgoritmi q\u00eb kam paraqitur nuk \u00ebsht\u00eb universal by design (k\u00ebtu, ndoshta, q\u00ebllimet e mia dallohen m\u00eb shum\u00eb se kudo nga ato t\u00eb konsorciumit Unicode). E kam p\u00ebrmendur tashm\u00eb se ai u zhvillua kryesisht p\u00ebr nj\u00eb detyr\u00eb t\u00eb vetme (ruajtjen e nj\u00eb fjalori shum\u00ebgjuh\u00ebsh n\u00eb nj\u00eb pem\u00eb prefikse), dhe disa nga ve\u00e7orit\u00eb e tij mund t\u00eb mos p\u00ebrshtaten mir\u00eb p\u00ebr detyra t\u00eb tjera. Por fakti q\u00eb nuk \u00ebsht\u00eb standard mund t\u00eb jet\u00eb edhe plus \u2014 mund ta p\u00ebrshtatni leht\u00ebsisht sipas nevojave tuaja<\/strong>.<\/p>\n

      P\u00ebr shembull, n\u00eb m\u00ebnyr\u00eb krejt t\u00eb qart\u00eb mund t\u00eb hiqet prania e gjendjes dhe kodimi t\u00eb b\u00ebhet stateless \u2014 thjesht duke mos p\u00ebrdit\u00ebsuar variablat zbritje<\/b><\/code>, auxOffs<\/b><\/code> dhe is21Bit<\/b><\/code> n\u00eb enkoder dhe dekoder. N\u00eb k\u00ebt\u00eb rast nuk do t\u00eb jet\u00eb e mundur t\u00eb paketohen me efikasitet sekuencat e simboleve t\u00eb t\u00eb nj\u00ebjtit alfabet, por do t\u00eb ket\u00eb garanci q\u00eb i nj\u00ebjti simbol kodohet gjithmon\u00eb me t\u00eb nj\u00ebjtat bajte, pavar\u00ebsisht nga konteksti.<\/p>\n

      P\u00ebrve\u00e7 k\u00ebsaj, koduesi mund t\u00eb p\u00ebrshtatet p\u00ebr nj\u00eb gjuh\u00eb t\u00eb caktuar duke ndryshuar gjendjen e parazgjedhur \u2014 p\u00ebr shembull, duke u orientuar te tekstet ruse, t\u00eb vendosen n\u00eb fillim t\u00eb enkoderit dhe dekoderit offs = 0x0400<\/b><\/code> dhe auxOffs = 0<\/b><\/code>. Ve\u00e7an\u00ebrisht kjo ka kuptim pik\u00ebrisht n\u00eb rastin e modalitetit stateless. N\u00eb p\u00ebrgjith\u00ebsi, kjo do t'i ngjaj\u00eb p\u00ebrdorimit t\u00eb kodimit t\u00eb vjet\u00ebr 8-bit\u00ebsh, vet\u00ebm se nuk ju privon nga mund\u00ebsia p\u00ebr t\u00eb futur simbole nga i gjith\u00eb Unicode sipas nevoj\u00ebs.<\/p>\n

      Nj\u00eb mang\u00ebsi tjet\u00ebr, e p\u00ebrmendur m\u00eb her\u00ebt, \u00ebsht\u00eb se n\u00eb nj\u00eb tekst voluminoz t\u00eb koduar n\u00eb UTF-C nuk ka nj\u00eb m\u00ebnyr\u00eb t\u00eb shpejt\u00eb p\u00ebr t\u00eb gjetur kufirin e simbolit m\u00eb t\u00eb af\u00ebrt me nj\u00eb bajt arbitrar. Duke prer\u00eb nga buffer-i i koduar 100 bajtet e fundit, fjala vjen, rrezikoni t\u00eb merrni mbeturina me t\u00eb cilat nuk mund t\u00eb b\u00ebni asgj\u00eb. Kodimi nuk \u00ebsht\u00eb menduar p\u00ebr ruajtjen e logeve shum\u00eb-gigabajt\u00ebshe, por n\u00eb p\u00ebrgjith\u00ebsi kjo mund t\u00eb rregullohet. Bajti 0xBF<\/b><\/code> nuk duhet t\u00eb shfaqet kurr\u00eb si bajti i par\u00eb (por mund t\u00eb jet\u00eb i dyti ose i treti). Prandaj, gjat\u00eb kodimit mund t\u00eb futet sekuenca 0xBF 0xBF 0xBF<\/b><\/code> \u00e7do, t\u00eb themi, 10 KB \u2014 at\u00ebher\u00eb, kur t\u00eb jet\u00eb e nevojshme t\u00eb gjendet kufiri, do t\u00eb mjaftoj\u00eb t\u00eb skanohet pjesa e zgjedhur derisa t\u00eb gjendet nj\u00eb marker i till\u00eb. Menj\u00ebher\u00eb pas t\u00eb fundit 0xBF<\/b><\/code> do t\u00eb ket\u00eb me siguri fillimin e simbolit. (Gjat\u00eb dekodimit, kjo sekuenc\u00eb prej tre bajtesh, natyrisht, duhet t\u00eb shp\u00ebrfillet.)<\/p>\n

      N\u00eb p\u00ebrfundim<\/h2>\n

      \nN\u00ebse keni lexuar deri k\u00ebtu \u2014 urime! Shpresoj q\u00eb, ashtu si un\u00eb, t\u00eb keni m\u00ebsuar di\u00e7ka t\u00eb re (ose t'ju jen\u00eb rifreskuar disa njohuri t\u00eb vjetra) rreth m\u00ebnyr\u00ebs si funksionon Unicode.<\/p>\n

      \"Edhe
      \nFaqe demonstrimi. Me shembullin e hebraishtes duken qart\u00eb p\u00ebrpar\u00ebsit\u00eb si ndaj UTF-8, ashtu edhe ndaj SCSU.<\/i><\/p>\n

      Nuk duhet t'i shihni k\u00ebrkimet e p\u00ebrshkruara m\u00eb sip\u00ebr si cenim t\u00eb standardeve. Megjithat\u00eb, n\u00eb p\u00ebrgjith\u00ebsi jam i k\u00ebnaqur me rezultatet e pun\u00ebs sime, ndaj kam k\u00ebnaq\u00ebsi t'i ndaj<\/a><\/noindex>: p\u00ebr shembull, biblioteka JS n\u00eb form\u00eb t\u00eb minifikuar z\u00eb vet\u00ebm 1710 bajte (dhe, sigurisht, nuk ka var\u00ebsi). Si\u00e7 e p\u00ebrmenda m\u00eb sip\u00ebr, me funksionimin e saj mund t\u00eb njiheni n\u00eb faqen demo<\/a><\/noindex> (aty gjendet edhe nj\u00eb grup tekstesh me t\u00eb cilat mund ta krahasoni me UTF-8 dhe SCSU).<\/p>\n

      N\u00eb fund, dua t\u00eb theksoj edhe nj\u00eb her\u00eb rastet kur p\u00ebrdorimi i UTF-C nuk rekomandohet<\/b>:<\/p>\n

      Le t\u00eb shohim ato blloqe t\u00eb Unicode q\u00eb tani k\u00ebrkojn\u00eb tre bajte. Kryesisht, si\u00e7 u p\u00ebrmend tashm\u00eb, k\u00ebto jan\u00eb ideogramet kineze, por me to \u00ebsht\u00eb e v\u00ebshtir\u00eb t\u00eb b\u00ebhet di\u00e7ka, sepse jan\u00eb 21 mij\u00eb. Por aty kan\u00eb p\u00ebrfunduar edhe hiragana me katakan\u00ebn, dhe ato nuk jan\u00eb aq shum\u00eb, m\u00eb pak se dyqind. Dhe, meq\u00eb p\u00ebrmend\u00ebm japonez\u00ebt, aty gjenden edhe emoji (n\u00eb fakt ato jan\u00eb t\u00eb shp\u00ebrndara n\u00eb shum\u00eb vende t\u00eb Unicode, por blloqet kryesore n\u00eb intervalin 0x1F300<\/b><\/code> \u2013 0x1FBFF<\/b><\/code>). N\u00ebse mendon se sot ekzistojn\u00eb emoji q\u00eb p\u00ebrb\u00ebhen nga disa pika kodi nj\u00ebher\u00ebsh (p\u00ebr shembull, emoji \u200d\u200d\u200d\"Edhe<\/a><\/noindex> p\u00ebrb\u00ebhet plot nga 7 kode!), at\u00ebher\u00eb b\u00ebhet edhe m\u00eb e dhimbshme t\u00eb shpenzosh nga tre bajte p\u00ebr secil\u00ebn (7\u00d73 = 21 bajte p\u00ebr nj\u00eb simbol t\u00eb vet\u00ebm, tmerr fare).<\/p>\n