{"id":74737,"date":"2020-03-20T08:43:14","date_gmt":"2020-03-20T05:43:14","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/optimizacziya-strok-v-clickhouse-doklad-yandeksa"},"modified":"2020-03-20T08:43:14","modified_gmt":"2020-03-20T05:43:14","slug":"optimizacziya-strok-v-clickhouse-doklad-yandeksa","status":"publish","type":"post","link":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/optimizacziya-strok-v-clickhouse-doklad-yandeksa","title":{"rendered":"Optimizarea \u0219irurilor \u00een ClickHouse. Prezentarea Yandex","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p>Sistemul analitic ClickHouse proceseaz\u0103 o varietate de \u0219iruri, consum\u00e2nd resurse. Pentru a accelera func\u021bionarea sistemului, sunt ad\u0103ugate constant noi optimiz\u0103ri. Dezvoltatorul ClickHouse, Nikolai Kochetov, vorbe\u0219te despre tipul de date \u0219ir, inclusiv despre noul tip, LowCardinality, \u0219i explic\u0103 cum se poate \u00eembun\u0103t\u0103\u021bi performan\u021ba lucrului cu \u0219irurile. <\/p>\n<p><center><div class=\"youtube-placeholder\" data-id=\"rqf-ILRgBdY\" onclick=\"loadVideo(this)\">\r\n        <img decoding=\"async\" src=\"https:\/\/img.youtube.com\/vi\/rqf-ILRgBdY\/hqdefault.jpg\" alt=\"Reda\u021bi video\" loading=\"lazy\" width=\"480\" height=\"360\" style=\"width:100%;height:auto;\">\r\n        <div class=\"play-button\"><\/div>\r\n    <\/div><\/center><br \/>\n\u2014 Mai \u00eent\u00e2i, s\u0103 \u00een\u021belegem cum putem stoca \u0219iruri. <br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><br \/>\n<img decoding=\"async\" alt=\"Optimizarea \u0219irurilor \u00een ClickHouse. Prezentarea Yandex\" src=\"\/wp-content\/uploads\/2020\/03\/5e38e903aaaf54a533bd3026e76a14be.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nAvem tipuri de date pentru \u0219iruri. String este potrivit ca op\u021biune implicit\u0103 \u0219i ar trebui folosit aproape \u00eentotdeauna. Are un overhead mic \u2014 9 octe\u021bi pentru un \u0219ir. Dac\u0103 dorim ca dimensiunea \u0219irurilor s\u0103 fie fix\u0103 \u0219i cunoscut\u0103 dinainte, este mai bine s\u0103 folosim FixedString. Aici putem specifica num\u0103rul dorit de octe\u021bi, ceea ce este convenabil pentru date precum adrese IP sau func\u021bii hash. <\/p>\n<p><img decoding=\"async\" alt=\"Optimizarea \u0219irurilor \u00een ClickHouse. Prezentarea Yandex\" src=\"\/wp-content\/uploads\/2020\/03\/eaae898fb5163cb1c070c34a13f6f160.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nDesigur, uneori ceva poate s\u0103 fie mai lent. S\u0103 spunem c\u0103 efectua\u021bi o interogare asupra unei tabele. ClickHouse cite\u0219te o cantitate destul de mare de date, s\u0103 zicem, cu o vitez\u0103 de 100 GB\/s, \u00een timp ce \u0219irurile sunt procesate foarte pu\u021bin. Avem dou\u0103 tabele care stocheaz\u0103 aproape acelea\u0219i date. Din a doua tabel\u0103, ClickHouse cite\u0219te datele cu o vitez\u0103 mai mare, dar num\u0103rul de \u0219iruri citite pe secund\u0103 este de trei ori mai mic. <\/p>\n<p><img decoding=\"async\" alt=\"Optimizarea \u0219irurilor \u00een ClickHouse. Prezentarea Yandex\" src=\"\/wp-content\/uploads\/2020\/03\/be7cc47e628112bb212ed0e8ff9091ed.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nDac\u0103 ne uit\u0103m la dimensiunea datelor comprimate, aceasta va fi aproape egal\u0103. De fapt, \u00een tabele sunt stocate acelea\u0219i date \u2014 primul miliard de numere \u2014 doar c\u0103 \u00een prima coloan\u0103 sunt \u00eenregistrate ca UInt64, iar \u00een a doua coloan\u0103 ca String. Din acest motiv, a doua interogare dureaz\u0103 mai mult pentru a citi datele de pe disc \u0219i a le decomprima. <\/p>\n<p><img decoding=\"async\" alt=\"Optimizarea \u0219irurilor \u00een ClickHouse. Prezentarea Yandex\" src=\"\/wp-content\/uploads\/2020\/03\/c699103cecd972fcee73841cc35716d6.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nIat\u0103 un alt exemplu. S\u0103 presupunem c\u0103 avem un set de \u0219iruri cunoscut dinainte, limitat la o constant\u0103 de 1000 sau 10 000 \u0219i care nu se schimb\u0103 aproape niciodat\u0103. Pentru acest caz, tipul de date Enum este potrivit, iar \u00een ClickHouse avem dou\u0103 tipuri \u2014 Enum8 \u0219i Enum16. Prin stocarea \u00een Enum, proces\u0103m rapid interog\u0103rile. <\/p>\n<p>\u00cen ClickHouse exist\u0103 optimiz\u0103ri pentru GROUP BY, IN, DISTINCT \u0219i optimiz\u0103ri pentru anumite func\u021bii, de exemplu, pentru compara\u021bia cu un \u0219ir constant. Desigur, numerele din \u0219ir nu sunt convertite, ci, dimpotriv\u0103, \u0219irul constant este convertit \u00een valoarea Enum. Dup\u0103 aceea, toate sunt comparate rapid. <\/p>\n<p>Dar exist\u0103 \u0219i dezavantaje. Chiar dac\u0103 \u0219tim exact setul de \u0219iruri, uneori acesta trebuie s\u0103 fie extins. A ap\u0103rut un nou \u0219ir \u2014 trebuie s\u0103 facem un ALTER. <\/p>\n<p><img decoding=\"async\" alt=\"Optimizarea \u0219irurilor \u00een ClickHouse. Prezentarea Yandex\" src=\"\/wp-content\/uploads\/2020\/03\/064c35faaf3d9de2f12d29f83197e983.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nALTER pentru Enum \u00een ClickHouse este implementat optim. Nu rescriem datele pe disc, dar ALTER poate s\u0103 r\u00e2g\u00e2ie din cauza faptului c\u0103 structurile Enum sunt stocate \u00een schema tabelului \u00een sine. Prin urmare, trebuie s\u0103 a\u0219tept\u0103m cererile de citire din tabel, de exemplu. <\/p>\n<p>Se pune \u00eentrebarea, se poate face mai bine? Probabil, da. Putem salva structura Enum nu \u00een schema tabelului, ci \u00een ZooKeeper. Totu\u0219i, pot ap\u0103rea probleme legate de sincronizare. De exemplu, o replic\u0103 a primit datele, iar alta nu, iar dac\u0103 aceasta din urm\u0103 are un Enum vechi, atunci ceva se va strica. (\u00cen ClickHouse aproape am terminat de implementat cererile ALTER non-blocante. C\u00e2nd le finaliz\u0103m complet, nu va mai fi necesar s\u0103 a\u0219tept\u0103m cererile de citire.)<\/p>\n<p><img decoding=\"async\" alt=\"Optimizarea \u0219irurilor \u00een ClickHouse. Prezentarea Yandex\" src=\"\/wp-content\/uploads\/2020\/03\/f5d2fdf14778f6afdd4acc8c204e163c.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nPentru a nu ne ocupa cu ALTER Enum, putem folosi dic\u021bionarele externe ClickHouse. Reamintesc c\u0103 aceasta este o structur\u0103 de date key-value \u00een ClickHouse, prin care putem ob\u021bine date din surse externe, de exemplu din tabele MySQL. <\/p>\n<p>\u00cen dic\u021bionarul ClickHouse stoc\u0103m multe \u0219iruri diferite, iar \u00een tabel avem identificatorii lor sub form\u0103 de numere. Dac\u0103 trebuie s\u0103 ob\u021binem un \u0219ir, apel\u0103m func\u021bia dictGet \u0219i lucr\u0103m cu el. Dup\u0103 aceea, nu mai trebuie s\u0103 facem ALTER. Pentru a ad\u0103uga ceva \u00een Enum, \u00eel inser\u0103m \u00een aceea\u0219i tabel\u0103 MySQL. <\/p>\n<p>Dar aici apar alte probleme. \u00cen primul r\u00e2nd, sintaxa incomod\u0103. Dac\u0103 dorim s\u0103 ob\u021binem un \u0219ir, trebuie s\u0103 apel\u0103m dictGet. \u00cen al doilea r\u00e2nd, lipsa unor optimiz\u0103ri. Compara\u021bia cu un \u0219ir constant pentru dic\u021bionare nu se poate face la fel de repede. <\/p>\n<p>De asemenea, pot exista probleme cu actualizarea. S\u0103 presupunem c\u0103 am solicitat un \u0219ir \u00een dic\u021bionarul cache, dar acesta nu a fost inclus \u00een cache. Atunci va trebui s\u0103 a\u0219tept\u0103m p\u00e2n\u0103 c\u00e2nd datele sunt \u00eenc\u0103rcate din sursa extern\u0103. <\/p>\n<p><img decoding=\"async\" alt=\"Optimizarea \u0219irurilor \u00een ClickHouse. Prezentarea Yandex\" src=\"\/wp-content\/uploads\/2020\/03\/534818ee523a243623a5cb2babe08040.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nDezavantajul general al ambelor metode este c\u0103 stoc\u0103m toate cheile \u00eentr-un singur loc \u0219i le sincroniz\u0103m. De ce s\u0103 nu stoc\u0103m dic\u021bionarele local? F\u0103r\u0103 sincronizare, f\u0103r\u0103 probleme. Pute\u021bi stoca dic\u021bionarul local pe un segment de disc. Asta \u00eenseamn\u0103 c\u0103 am f\u0103cut Insert, am salvat dic\u021bionarul. Dac\u0103 lucr\u0103m cu date \u00een memorie, putem salva dic\u021bionarul fie \u00een blocul de date, fie \u00een segmentul de coloan\u0103, fie \u00eentr-un fel de cache pentru a accelera calcul\u0103rile.<\/p>\n<h3>Codificarea dic\u021bionarului pentru \u0219iruri <\/h3>\n<p>\nAstfel, am ajuns la crearea unui nou tip de date \u00een ClickHouse - LowCardinality. Acesta este un format de stocare a datelor: cum sunt scrise pe disc \u0219i cum sunt citite, cum sunt prezentate \u00een memorie \u0219i schema lor de procesare. <\/p>\n<p><img decoding=\"async\" alt=\"Optimizarea \u0219irurilor \u00een ClickHouse. Prezentarea Yandex\" src=\"\/wp-content\/uploads\/2020\/03\/e25b2eb12ff158b89b7d2dbdfb8e2e5a.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nPe diapozitiv exist\u0103 dou\u0103 coloane. \u00cen dreapta, r\u00e2ndurile sunt stocate standard, \u00een tipul String. Se observ\u0103 c\u0103 sunt modele de telefoane mobile. \u00cen st\u00e2nga este aceea\u0219i coloan\u0103, doar c\u0103 \u00een tipul LowCardinality. Aceasta const\u0103 dintr-un dic\u021bionar cu multe r\u00e2nduri diferite (r\u00e2nduri din coloana din dreapta) \u0219i o list\u0103 de pozi\u021bii (numerele r\u00e2ndurilor). <\/p>\n<p>Cu ajutorul acestor dou\u0103 structuri, putem reconstrui coloana ini\u021bial\u0103. De asemenea, exist\u0103 un index invers \u2014 o tabel\u0103 hash care ajut\u0103 s\u0103 g\u0103sim pozi\u021bia din dic\u021bionar pe baza unui r\u00e2nd. Aceasta este necesar\u0103 pentru a accelera anumite interog\u0103ri. De exemplu, dac\u0103 dorim s\u0103 compar\u0103m, s\u0103 c\u0103ut\u0103m un r\u00e2nd \u00een coloana noastr\u0103 sau s\u0103 le unim. <\/p>\n<p>LowCardinality este un tip de date parametric. Poate fi fie un num\u0103r, fie ceva care este stocat ca num\u0103r, fie un \u0219ir, fie Nullable de la acestea. <\/p>\n<p><img decoding=\"async\" alt=\"Optimizarea \u0219irurilor \u00een ClickHouse. Prezentarea Yandex\" src=\"\/wp-content\/uploads\/2020\/03\/d52ef0c99617238442d4f71e43af48af.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nParticularitatea LowCardinality este c\u0103 acesta poate fi p\u0103strat pentru anumite func\u021bii. Pe diapozitiv se poate vedea un exemplu de interogare. \u00cen prima linie, am creat o coloan\u0103 de tip LowCardinality de la String, numind-o S. Apoi am \u00eentrebat numele acesteia \u2014 ClickHouse a spus c\u0103 este LowCardinality de la String. Totul este corect.<\/p>\n<p>A treia linie este aproape aceea\u0219i, doar c\u0103 am apelat func\u021bia length. \u00cen ClickHouse, func\u021bia length returneaz\u0103 tipul de date UInt64. Dar am ob\u021binut LowCardinality de la UInt64. Care este scopul?<\/p>\n<p><img decoding=\"async\" alt=\"Optimizarea \u0219irurilor \u00een ClickHouse. Prezentarea Yandex\" src=\"\/wp-content\/uploads\/2020\/03\/a4a51a0f1ecd68f7fffd521ab786c8a5.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n\u00cen dic\u021bionar erau stocate denumirile telefoanelor mobile, am aplicat func\u021bia length. Acum avem un dic\u021bionar similar, const\u00e2nd doar din numere \u2014 acestea sunt lungimile \u0219irurilor. Coloana cu pozi\u021biile nu s-a schimbat. \u00cen final, am procesat mai pu\u021bine date, economisind timp pentru interogare. <\/p>\n<p>Pot exista \u0219i alte optimiz\u0103ri, de exemplu, ad\u0103ugarea unui cache simplu. Atunci c\u00e2nd se calculeaz\u0103 valoarea unei func\u021bii, o putem re\u021bine \u0219i construi aceea\u0219i, f\u0103r\u0103 a o recalcula. <\/p>\n<p>De asemenea, poate fi realizat\u0103 o optimizare a GROUP BY, deoarece coloana noastr\u0103 cu dic\u021bionarul este deja par\u021bial agregat\u0103 \u2014 se pot calcula mai repede valorile func\u021biilor hash \u0219i se poate g\u0103si aproximativ bucket-ul \u00een care s\u0103 fie plasat urm\u0103torul r\u00e2nd. De asemenea, se pot specializa unele func\u021bii agregate, de exemplu uniq, deoarece \u00een aceasta pot fi trimise doar dic\u021bionarele, iar pozi\u021biile pot r\u0103m\u00e2ne netocmite \u2014 astfel, totul va func\u021biona mai repede. Primele dou\u0103 optimiz\u0103ri au fost deja ad\u0103ugate \u00een ClickHouse.<\/p>\n<p><img decoding=\"async\" alt=\"Optimizarea \u0219irurilor \u00een ClickHouse. Prezentarea Yandex\" src=\"\/wp-content\/uploads\/2020\/03\/943e0a8ae5a37cd47164246db3a67893.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nCe ar fi dac\u0103 am crea o coloan\u0103 cu tipul nostru de date \u0219i am insera \u00een ea multe \u0219iruri diferite \u0219i proaste? Oare nu ne va umple memoria? Nu, pentru asta ClickHouse are dou\u0103 set\u0103ri speciale. Prima este low_cardinality_max_dictionary_size. Acesta este dimensiunea maxim\u0103 a dic\u021bionarului care poate fi scris pe disc. Inserarea se desf\u0103\u0219oar\u0103 astfel: atunci c\u00e2nd inser\u0103m date, primim un flux de \u0219iruri, din acestea form\u0103m un mare dic\u021bionar comun. Dac\u0103 dic\u021bionarul devine mai mare dec\u00e2t valoarea set\u0103rii, scriem dic\u021bionarul curent pe disc, iar celelalte \u0219iruri le stoc\u0103m undeva \u201epe l\u00e2ng\u0103\u201d, l\u00e2ng\u0103 indici. \u00cen cele din urm\u0103, niciodat\u0103 nu vom recalcula un dic\u021bionar mare \u0219i nu vom avea probleme de memorie. <\/p>\n<p>A doua setare se nume\u0219te low_cardinality_use_single_dictionary_for_part. Imagineaz\u0103-\u021bi c\u0103, \u00een schema anterioar\u0103, c\u00e2nd am inserat datele, dic\u021bionarul nostru s-a umplut \u0219i l-am scris pe disc. Se pune \u00eentrebarea, de ce s\u0103 nu form\u0103m acum un alt dic\u021bionar identic? <\/p>\n<p>C\u00e2nd se umple, \u00eel vom scrie din nou pe disc \u0219i vom \u00eencepe s\u0103 form\u0103m al treilea. Aceast\u0103 setare dezactiveaz\u0103 exact aceast\u0103 posibilitate din oficiu. <\/p>\n<p>De fapt, multe dic\u021bionare pot fi utile dac\u0103 dorim s\u0103 inser\u0103m un anumit set de \u0219iruri, dar am inserat din gre\u0219eal\u0103 \u201egunoi\u201d. S\u0103 spunem c\u0103 mai \u00eent\u00e2i am inserat \u0219iruri proaste \u0219i apoi am inserat unele bune. Atunci dic\u021bionarul se va \u00eemp\u0103r\u021bi \u00een multe dic\u021bionare mici. O parte dintre ele vor con\u021bine \u201egunoi\u201d, dar ultimele vor avea \u0219iruri bune. \u0218i dac\u0103 citim, s\u0103 spunem, doar ultima granula, atunci totul va func\u021biona rapid. <\/p>\n<p><img decoding=\"async\" alt=\"Optimizarea \u0219irurilor \u00een ClickHouse. Prezentarea Yandex\" src=\"\/wp-content\/uploads\/2020\/03\/834ba3295ace98870334aa10a78e4781.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n\u00cenainte de a vorbi despre avantajele LowCardinality, trebuie s\u0103 spun c\u0103 este pu\u021bin probabil s\u0103 ob\u021binem o reducere a datelor pe disc (de\u0219i acest lucru poate s\u0103 se \u00eent\u00e2mple), deoarece ClickHouse comprim\u0103 datele. Exist\u0103 o op\u021biune implicit\u0103 - LZ4. De asemenea, se poate face comprimare folosind ZSTD. Dar ambele algoritmi deja implementeaz\u0103 comprimarea dic\u021bionarului, a\u0219a c\u0103 dic\u021bionarul nostru extern ClickHouse nu va ajuta foarte mult. <\/p>\n<p>Pentru a nu fi nedrept, am luat unele date din metric\u0103 \u2014 String, LowCardinality(String) \u0219i Enum \u2014 \u0219i le-am salvat \u00een tipuri diferite de date. Am ob\u021binut trei coloane, fiecare av\u00e2nd un miliard de r\u00e2nduri. \u00cen prima coloan\u0103, CodePage, sunt doar 62 de valori. Se observ\u0103 c\u0103 LowCardinality(String) ne-a comprimat rezultatele mai bine. String este pu\u021bin mai pu\u021bin eficient, dar asta se datoreaz\u0103, probabil, faptului c\u0103 \u0219irurile sunt scurte, p\u0103str\u0103m lungimile acestora, iar acestea ocup\u0103 mult loc, fiind mai pu\u021bin comprimabile. <\/p>\n<p>Dac\u0103 lu\u0103m PhoneModel, avem 48 de mii \u2014 deja mai mult, iar diferen\u021bele \u00eentre String \u0219i LowCardinality(String) sunt aproape inexistente. Pentru URL, am economisit doar 2 GB \u2014 cred c\u0103 nu merit\u0103 s\u0103 ne baz\u0103m pe asta. <\/p>\n<h3>Evaluarea vitezei de execu\u021bie<\/h3>\n<p>\n<img decoding=\"async\" alt=\"Optimizarea \u0219irurilor \u00een ClickHouse. Prezentarea Yandex\" src=\"\/wp-content\/uploads\/2020\/03\/0b646570639f8f2e29b599473b84e25c.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<b><noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/toddwschneider\/nyc-taxi-data\">Link de la diapozitiv<\/a><\/noindex><\/b><\/p>\n<p>Acum s\u0103 evalu\u0103m viteza de execu\u021bie. Pentru a o evalua, am folosit un dataset cu descrierea c\u0103l\u0103toriilor cu taxiul din New York. Acesta <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/toddwschneider\/nyc-taxi-data\">disponibil<\/a><\/noindex> se g\u0103se\u0219te pe GitHub. Con\u021bine pu\u021bin peste un miliard de c\u0103l\u0103torii. Reflect\u0103 loca\u021bia, ora de \u00eenceput \u0219i de sf\u00e2r\u0219it a c\u0103l\u0103toriei, metoda de plat\u0103, num\u0103rul de pasageri \u0219i chiar tipul de taxi \u2014 verde, galben \u0219i Uber. <\/p>\n<p><img decoding=\"async\" alt=\"Optimizarea \u0219irurilor \u00een ClickHouse. Prezentarea Yandex\" src=\"\/wp-content\/uploads\/2020\/03\/0901669dec01b0f956d6976e3aa8b741.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nPrima \u00eentrebare am formulat-o destul de simplu \u2014 am \u00eentrebat unde se comand\u0103 cel mai des taxiuri. Pentru aceasta, trebuie s\u0103 lu\u0103m loca\u021bia de unde s-au f\u0103cut comenzile, s\u0103 facem un GROUP BY pe ea \u0219i s\u0103 num\u0103r\u0103m func\u021bia count. Iat\u0103 ce produce ClickHouse. <\/p>\n<p><img decoding=\"async\" alt=\"Optimizarea \u0219irurilor \u00een ClickHouse. Prezentarea Yandex\" src=\"\/wp-content\/uploads\/2020\/03\/50983341e20bcbc1b5adbaed35f8624d.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nPentru a m\u0103sura viteza de procesare a interog\u0103rii, am creat trei tabele cu date identice, dar am folosit pentru loca\u021bia noastr\u0103 de start trei tipuri diferite de date \u2014 String, LowCardinality \u0219i Enum. LowCardinality \u0219i Enum s-au dovedit a fi de cinci ori mai rapide dec\u00e2t String. Enum este mai rapid pentru c\u0103 lucreaz\u0103 cu numere. LowCardinality este mai rapid datorit\u0103 optimiz\u0103rii GROUP BY. <\/p>\n<p><img decoding=\"async\" alt=\"Optimizarea \u0219irurilor \u00een ClickHouse. Prezentarea Yandex\" src=\"\/wp-content\/uploads\/2020\/03\/e2ce4f2127d8728287488c85c92085fc.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nHai s\u0103 complic\u0103m un pic interogarea \u2014 s\u0103 \u00eentreb\u0103m unde se afl\u0103 cel mai popular parc din New York. Din nou vom m\u0103sura aceasta pe baza locurilor unde se comand\u0103 cel mai des taxiuri, dar vom filtra doar acele loca\u021bii care con\u021bin cuv\u00e2ntul \u201eparc\u201d. De asemenea, vom ad\u0103uga func\u021bia like. <\/p>\n<p><img decoding=\"async\" alt=\"Optimizarea \u0219irurilor \u00een ClickHouse. Prezentarea Yandex\" src=\"\/wp-content\/uploads\/2020\/03\/7cf6fd64751f3ae578581caddde17aaf.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nNe uit\u0103m la timp \u2014 observ\u0103m c\u0103 Enum a \u00eenceput brusc s\u0103 \u00eencetineasc\u0103. \u0218i func\u021bioneaz\u0103 chiar mai \u00eencet dec\u00e2t tipul de date standard, String. Acest lucru se \u00eent\u00e2mpl\u0103 pentru c\u0103 func\u021bia like nu este deloc optimizat\u0103 pentru Enum. Trebuie s\u0103 convertim \u0219irurile noastre din Enum \u00een \u0219iruri obi\u0219nuite \u2014 facem mai mult\u0103 munc\u0103. LowCardinality(String) nu este optimizat \u00een mod implicit, dar acolo like func\u021bioneaz\u0103 pe baza unui dic\u021bionar, astfel c\u0103 interogarea se accelereaz\u0103 fa\u021b\u0103 de String. <\/p>\n<p>C\u00e2nd lucr\u0103m cu Enum, exist\u0103 o problem\u0103 mai global\u0103. Dac\u0103 dorim s\u0103-l optimiz\u0103m, trebuie s\u0103 facem acest lucru \u00een fiecare loc din cod. S\u0103 presupunem c\u0103 am scris o nou\u0103 func\u021bie \u2014 trebuie neap\u0103rat s\u0103 venim cu o optimizare pentru Enum. \u00cen schimb, LowCardinality este optimizat implicit.<\/p>\n<p><img decoding=\"async\" alt=\"Optimizarea \u0219irurilor \u00een ClickHouse. Prezentarea Yandex\" src=\"\/wp-content\/uploads\/2020\/03\/459243eb5b3dd993393d0184b224ea09.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nS\u0103 ne uit\u0103m la ultima cerere, care este mai artificial\u0103. Vom calcula pur \u0219i simplu func\u021bia hash a loca\u021biei noastre. Func\u021bia hash este o cerere destul de lent\u0103, dureaz\u0103 mult, a\u0219a c\u0103 totul va fi \u00eencetinit de aproximativ trei ori.<\/p>\n<p><img decoding=\"async\" alt=\"Optimizarea \u0219irurilor \u00een ClickHouse. Prezentarea Yandex\" src=\"\/wp-content\/uploads\/2020\/03\/a5f49c22748971646ec358eb52d3feda.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nLowCardinality func\u021bioneaz\u0103 \u00een continuare mai rapid, de\u0219i nu exist\u0103 filtrare aici. Aceasta se \u00eent\u00e2mpl\u0103 deoarece func\u021biile noastre lucreaz\u0103 doar cu dic\u021bionarul. Func\u021bia de calcul a hash-ului are un singur argument \u2014 poate procesa mai pu\u021bine date \u0219i poate returna, de asemenea, LowCardinality. <\/p>\n<p><img decoding=\"async\" alt=\"Optimizarea \u0219irurilor \u00een ClickHouse. Prezentarea Yandex\" src=\"\/wp-content\/uploads\/2020\/03\/7842b7eb2debb4a4b1d4cbc3488deb32.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nPlanul nostru global este de a ob\u021bine o vitez\u0103 de lucru nu mai mic\u0103 dec\u00e2t String \u00een orice caz \u0219i s\u0103 men\u021binem accelera\u021bia. \u0218i, poate, c\u00e2ndva vom \u00eenlocui String cu LowCardinality, ve\u021bi actualiza ClickHouse, iar totul va func\u021biona pu\u021bin mai repede.<br \/>\n<br \/>Sursa: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/yandex\/blog\/492868\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0410\u043d\u0430\u043b\u0438\u0442\u0438\u0447\u0435\u0441\u043a\u0430\u044f \u0421\u0423\u0411\u0414 ClickHouse \u043e\u0431\u0440\u0430\u0431\u0430\u0442\u044b\u0432\u0430\u0435\u0442 \u043c\u043d\u043e\u0436\u0435\u0441\u0442\u0432\u043e \u0440\u0430\u0437\u043d\u044b\u0445 \u0441\u0442\u0440\u043e\u043a, \u043f\u043e\u0442\u0440\u0435\u0431\u043b\u044f\u044f \u0440\u0435\u0441\u0443\u0440\u0441\u044b. \u0414\u043b\u044f \u0443\u0441\u043a\u043e\u0440\u0435\u043d\u0438\u044f \u0440\u0430\u0431\u043e\u0442\u044b \u0441\u0438\u0441\u0442\u0435\u043c\u044b \u043f\u043e\u0441\u0442\u043e\u044f\u043d\u043d\u043e \u0434\u043e\u0431\u0430\u0432\u043b\u044f\u044e\u0442\u0441\u044f \u043d\u043e\u0432\u044b\u0435 \u043e\u043f\u0442\u0438\u043c\u0438\u0437\u0430\u0446\u0438\u0438. \u0420\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u0447\u0438\u043a ClickHouse \u041d\u0438\u043a\u043e\u043b\u0430\u0439 \u041a\u043e\u0447\u0435\u0442\u043e\u0432 \u0440\u0430\u0441\u0441\u043a\u0430\u0437\u044b\u0432\u0430\u0435\u0442 \u043e \u0441\u0442\u0440\u043e\u043a\u043e\u0432\u043e\u043c \u0442\u0438\u043f\u0435 \u0434\u0430\u043d\u043d\u044b\u0445, \u0432 \u0442\u043e\u043c \u0447\u0438\u0441\u043b\u0435 \u043e \u043d\u043e\u0432\u043e\u043c \u0442\u0438\u043f\u0435, LowCardinality, \u0438 \u043e\u0431\u044a\u044f\u0441\u043d\u044f\u0435\u0442, \u043a\u0430\u043a \u043c\u043e\u0436\u043d\u043e \u0443\u0441\u043a\u043e\u0440\u0438\u0442\u044c \u0440\u0430\u0431\u043e\u0442\u0443 \u0441\u043e \u0441\u0442\u0440\u043e\u043a\u0430\u043c\u0438. \u2014 \u0421\u043d\u0430\u0447\u0430\u043b\u0430 \u0434\u0430\u0432\u0430\u0439\u0442\u0435 \u0440\u0430\u0437\u0431\u0435\u0440\u0435\u043c\u0441\u044f, \u043a\u0430\u043a \u043c\u043e\u0436\u043d\u043e \u0445\u0440\u0430\u043d\u0438\u0442\u044c \u0441\u0442\u0440\u043e\u043a\u0438. \u0423 \u043d\u0430\u0441 \u0435\u0441\u0442\u044c \u0441\u0442\u0440\u043e\u043a\u043e\u0432\u044b\u0435 \u0442\u0438\u043f\u044b \u0434\u0430\u043d\u043d\u044b\u0445. [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":74738,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-74737","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.2.1 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u0410\u043d\u0430\u043b\u0438\u0442\u0438\u0447\u0435\u0441\u043a\u0430\u044f \u0421\u0423\u0411\u0414 ClickHouse \u043e\u0431\u0440\u0430\u0431\u0430\u0442\u044b\u0432\u0430\u0435\u0442 \u043c\u043d\u043e\u0436\u0435\u0441\u0442\u0432\u043e \u0440\u0430\u0437\u043d\u044b\u0445 \u0441\u0442\u0440\u043e\u043a, \u043f\u043e\u0442\u0440\u0435\u0431\u043b\u044f\u044f \u0440\u0435\u0441\u0443\u0440\u0441\u044b. \u0414\u043b\u044f \u0443\u0441\u043a\u043e\u0440\u0435\u043d\u0438\u044f \u0440\u0430\u0431\u043e\u0442\u044b \u0441\u0438\u0441\u0442\u0435\u043c\u044b \u043f\u043e\u0441\u0442\u043e\u044f\u043d\u043d\u043e \u0434\u043e\u0431\u0430\u0432\u043b\u044f\u044e\u0442\u0441\u044f \u043d\u043e\u0432\u044b\u0435 \u043e\u043f\u0442\u0438\u043c\u0438\u0437\u0430\u0446\u0438\u0438.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/optimizacziya-strok-v-clickhouse-doklad-yandeksa\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.2.1\" \/>\n\t\t<meta property=\"og:locale\" content=\"ro_RO\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041e\u043f\u0442\u0438\u043c\u0438\u0437\u0430\u0446\u0438\u044f \u0441\u0442\u0440\u043e\u043a \u0432 ClickHouse. \u0414\u043e\u043a\u043b\u0430\u0434 \u042f\u043d\u0434\u0435\u043a\u0441\u0430 | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u0410\u043d\u0430\u043b\u0438\u0442\u0438\u0447\u0435\u0441\u043a\u0430\u044f \u0421\u0423\u0411\u0414 ClickHouse \u043e\u0431\u0440\u0430\u0431\u0430\u0442\u044b\u0432\u0430\u0435\u0442 \u043c\u043d\u043e\u0436\u0435\u0441\u0442\u0432\u043e \u0440\u0430\u0437\u043d\u044b\u0445 \u0441\u0442\u0440\u043e\u043a, \u043f\u043e\u0442\u0440\u0435\u0431\u043b\u044f\u044f \u0440\u0435\u0441\u0443\u0440\u0441\u044b. \u0414\u043b\u044f \u0443\u0441\u043a\u043e\u0440\u0435\u043d\u0438\u044f \u0440\u0430\u0431\u043e\u0442\u044b \u0441\u0438\u0441\u0442\u0435\u043c\u044b \u043f\u043e\u0441\u0442\u043e\u044f\u043d\u043d\u043e \u0434\u043e\u0431\u0430\u0432\u043b\u044f\u044e\u0442\u0441\u044f \u043d\u043e\u0432\u044b\u0435 \u043e\u043f\u0442\u0438\u043c\u0438\u0437\u0430\u0446\u0438\u0438.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/optimizacziya-strok-v-clickhouse-doklad-yandeksa\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-03-20T05:43:14+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-03-20T05:43:14+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Optimizarea \u0219irurilor \u00een ClickHouse. Prezentare Yandex | ProHoster","description":"Sistemul de gestionare a bazelor de date analitice ClickHouse proceseaz\u0103 multe \u0219iruri diferite, consum\u00e2nd resurse. Pentru a accelera func\u021bionarea sistemului, sunt ad\u0103ugate constant noi optimiz\u0103ri.","canonical_url":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/optimizacziya-strok-v-clickhouse-doklad-yandeksa","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"ro_RO","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041e\u043f\u0442\u0438\u043c\u0438\u0437\u0430\u0446\u0438\u044f \u0441\u0442\u0440\u043e\u043a \u0432 ClickHouse. \u0414\u043e\u043a\u043b\u0430\u0434 \u042f\u043d\u0434\u0435\u043a\u0441\u0430 | ProHoster","og:description":"\u0410\u043d\u0430\u043b\u0438\u0442\u0438\u0447\u0435\u0441\u043a\u0430\u044f \u0421\u0423\u0411\u0414 ClickHouse \u043e\u0431\u0440\u0430\u0431\u0430\u0442\u044b\u0432\u0430\u0435\u0442 \u043c\u043d\u043e\u0436\u0435\u0441\u0442\u0432\u043e \u0440\u0430\u0437\u043d\u044b\u0445 \u0441\u0442\u0440\u043e\u043a, \u043f\u043e\u0442\u0440\u0435\u0431\u043b\u044f\u044f \u0440\u0435\u0441\u0443\u0440\u0441\u044b. \u0414\u043b\u044f \u0443\u0441\u043a\u043e\u0440\u0435\u043d\u0438\u044f \u0440\u0430\u0431\u043e\u0442\u044b \u0441\u0438\u0441\u0442\u0435\u043c\u044b \u043f\u043e\u0441\u0442\u043e\u044f\u043d\u043d\u043e \u0434\u043e\u0431\u0430\u0432\u043b\u044f\u044e\u0442\u0441\u044f \u043d\u043e\u0432\u044b\u0435 \u043e\u043f\u0442\u0438\u043c\u0438\u0437\u0430\u0446\u0438\u0438.","og:url":"https:\/\/prohoster.info\/ro\/blog\/administrirovanie\/optimizacziya-strok-v-clickhouse-doklad-yandeksa","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-03-20T05:43:14+00:00","article:modified_time":"2020-03-20T05:43:14+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"74737","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 17:23:10","updated":"2022-09-27 14:48:44","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts\/74737","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/comments?post=74737"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/posts\/74737\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/media\/74738"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/media?parent=74737"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/categories?post=74737"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/ro\/wp-json\/wp\/v2\/tags?post=74737"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}