{"id":74737,"date":"2020-03-20T08:43:14","date_gmt":"2020-03-20T05:43:14","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/optimizacziya-strok-v-clickhouse-doklad-yandeksa"},"modified":"2020-03-20T08:43:14","modified_gmt":"2020-03-20T05:43:14","slug":"optimizacziya-strok-v-clickhouse-doklad-yandeksa","status":"publish","type":"post","link":"https:\/\/prohoster.info\/lv\/blog\/administrirovanie\/optimizacziya-strok-v-clickhouse-doklad-yandeksa","title":{"rendered":"Rindkopu optimiz\u0101cija ClickHouse. Yandex prezent\u0101cija.","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p>Anal\u012btisk\u0101 DB ClickHouse apstr\u0101d\u0101 daudzveid\u012bgas rindas, pat\u0113r\u0113jot resursus. Lai pa\u0101trin\u0101tu sist\u0113mas darb\u012bbu, nep\u0101rtraukti tiek pievienotas jaunas optimiz\u0101cijas. ClickHouse izstr\u0101d\u0101t\u0101js Nikolajs Ko\u010detovs st\u0101sta par virkn\u0113\u0161anas datu tipu, tostarp par jauno tipu LowCardinality, un skaidro, k\u0101 pa\u0101trin\u0101t darba procesu ar rind\u0101m. <\/p>\n<p><center><div class=\"youtube-placeholder\" data-id=\"rqf-ILRgBdY\" onclick=\"loadVideo(this)\">\r\n        <img decoding=\"async\" src=\"https:\/\/img.youtube.com\/vi\/rqf-ILRgBdY\/hqdefault.jpg\" alt=\"Atska\u0146ot video\" loading=\"lazy\" width=\"480\" height=\"360\" style=\"width:100%;height:auto;\">\r\n        <div class=\"play-button\"><\/div>\r\n    <\/div><\/center><br \/>\n\u2014 S\u0101kum\u0101 izskat\u012bsim, k\u0101 var uzglab\u0101t rindas. <br \/>\n<noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><br \/>\n<img decoding=\"async\" alt=\"Rindkopu optimiz\u0101cija ClickHouse. Yandex prezent\u0101cija.\" src=\"\/wp-content\/uploads\/2020\/03\/5e38e903aaaf54a533bd3026e76a14be.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nMums ir virkn\u0113\u0161anas datu tipi. String labi der k\u0101 noklus\u0113juma variants, to vajadz\u0113tu izmantot gandr\u012bz vienm\u0113r. Tam ir neliels Overhead \u2014 9 baiti vienai rindai. Ja v\u0113lamies, lai rindu izm\u0113rs b\u016btu fiks\u0113ts un zin\u0101ms iepriek\u0161, vislab\u0101k izmantot FixedString. Taj\u0101 var noteikt nepiecie\u0161amo bajtu skaitu, tas ir \u0113rti t\u0101diem datiem k\u0101 IP adreses vai hashu funkcijas. <\/p>\n<p><img decoding=\"async\" alt=\"Rindkopu optimiz\u0101cija ClickHouse. Yandex prezent\u0101cija.\" src=\"\/wp-content\/uploads\/2020\/03\/eaae898fb5163cb1c070c34a13f6f160.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nProtams, reiz\u0113m kaut kas pal\u0113nin\u0101s. Pie\u0146emsim, ka j\u016bs veicat vaic\u0101jumu uz tabulu. ClickHouse lasa sal\u012bdzino\u0161i lielu datu apjomu, sakot, ar \u0101trumu 100 GB\/s, bet apstr\u0101d\u0101 maz rindu. Mums ir divas tabulas, kuras glab\u0101 gandr\u012bz vien\u0101dus datus. Otraj\u0101 tabul\u0101 ClickHouse lasa datus liel\u0101k\u0101 \u0101trum\u0101, ta\u010du rindu skaits sekund\u0113 tiek las\u012bts tr\u012bs reizes maz\u0101k. <\/p>\n<p><img decoding=\"async\" alt=\"Rindkopu optimiz\u0101cija ClickHouse. Yandex prezent\u0101cija.\" src=\"\/wp-content\/uploads\/2020\/03\/be7cc47e628112bb212ed0e8ff9091ed.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nJa m\u0113s paskat\u0101mies uz saspiesto datu izm\u0113ru, tas izr\u0101d\u012bsies gandr\u012bz vien\u0101ds. Paties\u012bb\u0101 tabul\u0101s ir ierakst\u012bti tie pa\u0161i dati \u2014 pirmais miljards skait\u013cu \u2014 tikai pirmaj\u0101 kolonn\u0101 tie ir ierakst\u012bti k\u0101 UInt64, bet otraj\u0101 \u2014 k\u0101 String. T\u0101p\u0113c otrais vaic\u0101jums ilg\u0101k lasa datus no diska un tos atspie\u017e. <\/p>\n<p><img decoding=\"async\" alt=\"Rindkopu optimiz\u0101cija ClickHouse. Yandex prezent\u0101cija.\" src=\"\/wp-content\/uploads\/2020\/03\/c699103cecd972fcee73841cc35716d6.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\n\u0160eit ir cits piem\u0113rs. Pie\u0146emsim, ka ir iepriek\u0161 zin\u0101ms rindas kopums, kas ir ierobe\u017eots ar konstantu 1000 vai 10 000 un praktiski nekad nemain\u0101s. \u0160aj\u0101 gad\u012bjum\u0101 mums der datu tips Enum, ClickHouse ir divi \u2014 Enum8 un Enum16. Izmantojot Enum, m\u0113s \u0101tri apstr\u0101d\u0101jam vaic\u0101jumus. <\/p>\n<p>ClickHouse ir pa\u0101trin\u0101jumi GROUP BY, IN, DISTINCT un optimiz\u0101cijas da\u017e\u0101m funkcij\u0101m, piem\u0113ram, sal\u012bdzin\u0101\u0161anai ar konstantu virkni. Protams, skait\u013ci virkn\u0113 netiek p\u0101rveidoti, bet, glu\u017ei pret\u0113ji, konstanta virkne tiek p\u0101rveidota uz Enum v\u0113rt\u012bbu. P\u0113c tam viss \u0101tri sal\u012bdzina. <\/p>\n<p>Bet ir ar\u012b m\u012bnusi. Pat ja m\u0113s zin\u0101m prec\u012bzu rindu kopumu, da\u017ek\u0101rt tas j\u0101papildina. Ja iest\u0101jas jauna rinda \u2014 mums j\u0101veic ALTER. <\/p>\n<p><img decoding=\"async\" alt=\"Rindkopu optimiz\u0101cija ClickHouse. Yandex prezent\u0101cija.\" src=\"\/wp-content\/uploads\/2020\/03\/064c35faaf3d9de2f12d29f83197e983.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nALTER komanda Enum ClickHouse ir optimiz\u0113ta. M\u0113s nep\u0101rrakst\u0101m datus disk\u0101, bet ALTER var mazliet pal\u0113nin\u0101t izpildi, jo Enum strukt\u016bras tiek glab\u0101tas pa\u0161as tabulas sh\u0113m\u0101. T\u0101p\u0113c mums j\u0101gaida las\u012b\u0161anas piepras\u012bjumi no tabulas, piem\u0113ram. <\/p>\n<p>Rod\u0101s jaut\u0101jums, vai ir iesp\u0113jams lab\u0101k? Iesp\u0113jams, j\u0101. Enum strukt\u016bru var saglab\u0101t nevis tabulas sh\u0113m\u0101, bet gan ZooKeeper. Tom\u0113r var rasties probl\u0113mas, kas saist\u012btas ar sinhroniz\u0101ciju. Piem\u0113ram, viena replika sa\u0146em datus, bet otra \u2014 n\u0113, un, ja tai ir vecais Enum, tad kaut kas var saboj\u0101ties. (ClickHouse m\u0113s gandr\u012bz esam pabeigu\u0161i neblo\u0137\u0113jo\u0161os ALTER piepras\u012bjumus. Kad m\u0113s tos pabeigsim piln\u012bgi, neb\u016bs nepiecie\u0161ams gaid\u012bt las\u012b\u0161anas piepras\u012bjumus.)<\/p>\n<p><img decoding=\"async\" alt=\"Rindkopu optimiz\u0101cija ClickHouse. Yandex prezent\u0101cija.\" src=\"\/wp-content\/uploads\/2020\/03\/f5d2fdf14778f6afdd4acc8c204e163c.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nLai nesaskartos ar ALTER Enum, var izmantot ClickHouse \u0101r\u0113jos v\u0101rdus. Atg\u0101dinu, ka tas ir key-value datu strukt\u016bra ClickHouse, ar kuru var ieg\u016bt datus no \u0101r\u0113jiem avotiem, piem\u0113ram, no MySQL tabul\u0101m. <\/p>\n<p>ClickHouse v\u0101rdn\u012bc\u0101 m\u0113s uzglab\u0101jam daudz da\u017e\u0101du rindu, bet tabul\u0101 \u2014 vi\u0146u identifikatori ciparu form\u0101. Ja mums ir nepiecie\u0161ams ieg\u016bt rindu, m\u0113s izsaucam funkciju dictGet un str\u0101d\u0101jam ar to. P\u0113c tam mums vairs nav j\u0101veic ALTER. Lai ko pievienotu Enum, m\u0113s to ievietojam taj\u0101 pa\u0161\u0101 MySQL tabul\u0101. <\/p>\n<p>Tom\u0113r \u0161eit rodas citas probl\u0113mas. Pirmk\u0101rt, ne\u0113rta sintakse. Ja m\u0113s v\u0113lamies ieg\u016bt rindu, mums ir j\u0101sauc dictGet. Otrk\u0101rt, tr\u016bkst da\u017eu optimiz\u0101ciju. Sal\u012bdzin\u0101jums ar konstanto rindu v\u0101rdn\u012bc\u0101m nav tik \u0101trs. <\/p>\n<p>Var b\u016bt ar\u012b probl\u0113mas ar atjaunin\u0101\u0161anu. Pie\u0146emsim, ka m\u0113s piepras\u012bj\u0101m rindu ke\u0161 v\u0101rdn\u012bc\u0101, bet t\u0101 neiek\u013cuva ke\u0161\u0101. Tad mums j\u0101gaida, kam\u0113r dati tiks iel\u0101d\u0113ti no \u0101r\u0113j\u0101 avota. <\/p>\n<p><img decoding=\"async\" alt=\"Rindkopu optimiz\u0101cija ClickHouse. Yandex prezent\u0101cija.\" src=\"\/wp-content\/uploads\/2020\/03\/534818ee523a243623a5cb2babe08040.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nAbu meto\u017eu kop\u0113jais tr\u016bkums ir tas, ka m\u0113s glab\u0101jam visus atsl\u0113gas vien\u0101 viet\u0101 un tos sinhroniz\u0113jam. T\u0101tad, k\u0101p\u0113c neglab\u0101t v\u0101rdn\u012bcas lok\u0101li? Nav sinhroniz\u0101cijas \u2014 nav probl\u0113mu. V\u0101rdn\u012bcu var glab\u0101t lok\u0101li disk\u0101. Tas ir, m\u0113s veicam Insert, ierakst\u0101m v\u0101rdn\u012bcu. Ja m\u0113s str\u0101d\u0101jam ar datiem atmi\u0146\u0101, tad varam ierakst\u012bt v\u0101rdn\u012bcu vai nu datu blok\u0101, vai kolonnas gabal\u0101, vai k\u0101d\u0101 ke\u0161\u0101, lai pa\u0101trin\u0101tu apr\u0113\u0137inus.<\/p>\n<h3>V\u0101rdu kodo\u0161ana rind\u0101m <\/h3>\n<p>\nT\u0101d\u0113j\u0101di m\u0113s non\u0101c\u0101m pie jauna datu tipa izveides ClickHouse \u2014 LowCardinality. Tas ir datu uzglab\u0101\u0161anas form\u0101ts: k\u0101 tie tiek ierakst\u012bti disk\u0101 un k\u0101 tie tiek nolas\u012bti, k\u0101 tie tiek att\u0113loti atmi\u0146\u0101 un sh\u0113ma vi\u0146u apstr\u0101dei. <\/p>\n<p><img decoding=\"async\" alt=\"Rindkopu optimiz\u0101cija ClickHouse. Yandex prezent\u0101cija.\" src=\"\/wp-content\/uploads\/2020\/03\/e25b2eb12ff158b89b7d2dbdfb8e2e5a.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nSlide satur contains two columns. On the right, the rows are stored in the standard type String. It is clear that these are some model of mobile phones. On the left, there is an identical column, but in the type LowCardinality. It consists of a dictionary with numerous different strings (strings from the column on the right) and a list of positions (row numbers). <\/p>\n<p>With the help of these two structures, the original column can be restored. There is also a reverse mapping index \u2014 a hash table that helps find the position in the dictionary by the string. It is necessary to speed up certain queries. For example, if we want to compare, search for a string in our column, or merge them together. <\/p>\n<p>LowCardinality is a parameterized data type. It can be either a number, or something stored as a number, or a string, or Nullable types of them. <\/p>\n<p><img decoding=\"async\" alt=\"Rindkopu optimiz\u0101cija ClickHouse. Yandex prezent\u0101cija.\" src=\"\/wp-content\/uploads\/2020\/03\/d52ef0c99617238442d4f71e43af48af.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nThe feature of LowCardinality is that it can be saved for some functions. The slide shows an example of a query. In the first line, I created a column of type LowCardinality from String, naming it S. Then I asked for its name \u2014 ClickHouse said it is LowCardinality from String. That's correct.<\/p>\n<p>The third line is almost the same, except we called the length function. In ClickHouse, the length function returns the data type UInt64. But now we have it as LowCardinality from UInt64. What is the point?<\/p>\n<p><img decoding=\"async\" alt=\"Rindkopu optimiz\u0101cija ClickHouse. Yandex prezent\u0101cija.\" src=\"\/wp-content\/uploads\/2020\/03\/a4a51a0f1ecd68f7fffd521ab786c8a5.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nThe dictionary stored the names of mobile phones; we applied the length function. Now we have an equivalent dictionary consisting only of numbers \u2014 these are the string lengths. The column with positions remained unchanged. As a result, we processed less data and saved on query time. <\/p>\n<p>There can be other optimizations, such as adding a simple cache. When calculating the value of a function, it can be remembered and constructed without recalculating. <\/p>\n<p>There may also be optimization for GROUP BY, because our column with a dictionary is already partially aggregated \u2014 it is possible to compute hash function values faster and roughly find the bucket to place the next string. Additionally, some aggregate functions can be specialized, for example uniq, since it can only receive the dictionary while leaving the positions untouched \u2014 this way everything will work faster. The first two optimizations have already been added to ClickHouse.<\/p>\n<p><img decoding=\"async\" alt=\"Rindkopu optimiz\u0101cija ClickHouse. Yandex prezent\u0101cija.\" src=\"\/wp-content\/uploads\/2020\/03\/943e0a8ae5a37cd47164246db3a67893.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nUn, ja m\u0113s izveidosim kolonnu ar m\u016bsu datu tipu un ievietosim taj\u0101 daudz sliktu da\u017e\u0101du rindu? Vai m\u016bsu atmi\u0146a neb\u016bs p\u0101rpild\u012bta? N\u0113, ClickHouse ir divi \u012bpa\u0161i iestat\u012bjumi \u0161im nol\u016bkam. Pirmais ir low_cardinality_max_dictionary_size. Tas ir maksim\u0101lais v\u0101rdn\u012bcas lielums, kas var tikt ierakst\u012bts disk\u0101. Ievieto\u0161ana notiek \u0161\u0101di: kad m\u0113s ievietojam datus, pie mums non\u0101k rindu pl\u016bsma, no kur\u0101m m\u0113s veidojam lielu kop\u0113ju v\u0101rdn\u012bcu. Ja v\u0101rdn\u012bca k\u013c\u016bst liel\u0101ka par iestat\u012bjuma v\u0113rt\u012bbu, m\u0113s ierakst\u0101m pa\u0161reiz\u0113jo v\u0101rdn\u012bcu disk\u0101, bet p\u0101r\u0113j\u0101s rindas \u2014 kaut kur \"mal\u0101\", blakus indeksiem. T\u0101d\u0113j\u0101di m\u0113s nekad nep\u0101rskat\u012bsim lielo v\u0101rdn\u012bcu un nesaskarsimies ar atmi\u0146as probl\u0113m\u0101m. <\/p>\n<p>Otrais iestat\u012bjums saucas low_cardinality_use_single_dictionary_for_part. Iedom\u0101jieties, ka iepriek\u0161\u0113j\u0101 sh\u0113m\u0101, kad m\u0113s ievietoj\u0101m datus, m\u016bsu v\u0101rdn\u012bca p\u0101rpild\u012bj\u0101s, un m\u0113s to ierakst\u012bj\u0101m disk\u0101. Rodas jaut\u0101jums, k\u0101p\u0113c \u0161obr\u012bd neizveidot v\u0113l vienu t\u0101du pa\u0161u v\u0101rdn\u012bcu? <\/p>\n<p>Kad t\u0101 p\u0101rpild\u012bsies, m\u0113s atk\u0101rtoti ierakst\u012bsim to disk\u0101 un s\u0101ksim veidot tre\u0161o. \u0160is iestat\u012bjums tie\u0161i atsp\u0113jo \u0161\u0101du iesp\u0113ju p\u0113c noklus\u0113juma. <\/p>\n<p>Paties\u012bb\u0101 daudz v\u0101rdn\u012bcu var b\u016bt noder\u012bgas, ja m\u0113s v\u0113lamies ievietot kaut k\u0101du virkni rindu, bet nejau\u0161i ievietoj\u0101m \"atkritumus\". Teiksim, vispirms m\u0113s ievietoj\u0101m sliktas rindas, un tad ievietoj\u0101m labas. T\u0101d\u0113j\u0101di v\u0101rdn\u012bca tiks sadal\u012bta daudz\u0101s maz\u0101s v\u0101rdn\u012bc\u0101s. Da\u013ca no t\u0101m b\u016bs ar \"atkritumiem\", bet p\u0113d\u0113j\u0101s b\u016bs ar lab\u0101m rind\u0101m. Un, ja m\u0113s las\u012bsim, teiksim, tikai p\u0113d\u0113jo graudu, viss ar\u012b darbosies \u0101tri. <\/p>\n<p><img decoding=\"async\" alt=\"Rindkopu optimiz\u0101cija ClickHouse. Yandex prezent\u0101cija.\" src=\"\/wp-content\/uploads\/2020\/03\/834ba3295ace98870334aa10a78e4781.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nPirms run\u0101jot par LowCardinality priek\u0161roc\u012bb\u0101m, uzreiz teik\u0161u, ka m\u0113s, visticam\u0101k, nesasniegsim datu samazin\u0101jumu disk\u0101 (lai gan tas var notikt), jo ClickHouse saspiesta datus. Ir noklus\u0113juma opcija \u2014 LZ4. T\u0101pat var veikt kompresiju ar ZSTD. Bet abi algoritmi jau \u012bsteno v\u0101rdn\u012bcu kompresiju, t\u0101p\u0113c m\u016bsu \u0101r\u0113j\u0101 v\u0101rdn\u012bca ClickHouse nevienai nepal\u012bdz\u0113s. <\/p>\n<p>Lai neb\u016btu run\u0101\u0161ana tuk\u0161\u0101, es pa\u0146\u0113mu da\u017eus datus no metrikas \u2014 String, LowCardinality(String) un Enum \u2014 un saglab\u0101ju tos da\u017e\u0101d\u0101s datu tip\u0101s. San\u0101ca tr\u012bs kolonnas, kur\u0101s ir ierakst\u012bts viens miljards rindu. Pirmaj\u0101 kolonn\u0101, CodePage, ir tikai 62 v\u0113rt\u012bbas. Un ir redzams, ka LowCardinality(String) mums t\u0101s ir saspiests lab\u0101k. String nedaudz slikt\u0101k, ta\u010du tas, visticam\u0101k, ir t\u0101p\u0113c, ka virknes ir \u012bsas, m\u0113s glab\u0101jam to garumus, un t\u0101s aiz\u0146em daudz vietas, slikti saspie\u017eas. <\/p>\n<p>Ja \u0146em v\u0113r\u0101 PhoneModel, to ir 48 t\u016bksto\u0161i \u2014 jau vair\u0101k, un at\u0161\u0137ir\u012bbas starp String un LowCardinality(String) gandr\u012bz nav. URL m\u0113s ar\u012b ietaup\u012bj\u0101m tikai 2 GB \u2014 dom\u0101ju, uz to nevajadz\u0113tu pa\u013cauties. <\/p>\n<h3>Darba \u0101truma nov\u0113rt\u0113jums<\/h3>\n<p>\n<img decoding=\"async\" alt=\"Rindkopu optimiz\u0101cija ClickHouse. Yandex prezent\u0101cija.\" src=\"\/wp-content\/uploads\/2020\/03\/0b646570639f8f2e29b599473b84e25c.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<b><noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/toddwschneider\/nyc-taxi-data\">Saite no slaida<\/a><\/noindex><\/b><\/p>\n<p>Tagad nov\u0113rt\u0113sim darba \u0101trumu. Lai to nov\u0113rt\u0113tu, es izmantoju datu kopu ar taksometru braucienu aprakstu \u0145ujork\u0101. Tas <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/toddwschneider\/nyc-taxi-data\">pieejams<\/a><\/noindex> ir GitHub. Taj\u0101 ir nedaudz vair\u0101k par miljardu braucienu. Tur ir iek\u013cauta lokaliz\u0101cija, brauciena s\u0101kuma un beig\u0161anas laiks, apmaksas veids, pasa\u017eieru skaits un pat taksometra veids \u2014 za\u013cais, dzeltenais un Uber. <\/p>\n<p><img decoding=\"async\" alt=\"Rindkopu optimiz\u0101cija ClickHouse. Yandex prezent\u0101cija.\" src=\"\/wp-content\/uploads\/2020\/03\/0901669dec01b0f956d6976e3aa8b741.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nPirmo jaut\u0101jumu es veidoju diezgan vienk\u0101r\u0161u \u2014 pajaut\u0101ju, kur visbie\u017e\u0101k pas\u016bta taksometru. Lai to izdar\u012btu, ir j\u0101\u0146em lokaliz\u0101cija, no kuras tika pas\u016bt\u012bts, j\u0101izdara GROUP BY un j\u0101paskaita funkcija count. \u0160eit ClickHouse kaut ko izvada. <\/p>\n<p><img decoding=\"async\" alt=\"Rindkopu optimiz\u0101cija ClickHouse. Yandex prezent\u0101cija.\" src=\"\/wp-content\/uploads\/2020\/03\/50983341e20bcbc1b5adbaed35f8624d.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nLai izm\u0113r\u012btu vaic\u0101juma apstr\u0101des \u0101trumu, es izveidoju tr\u012bs tabulas ar vien\u0101diem datiem, bet izmantoju tr\u012bs da\u017e\u0101dus datu tipus m\u016bsu s\u0101kuma lokaliz\u0101cijai \u2014 String, LowCardinality un Enum. LowCardinality un Enum izr\u0101d\u012bj\u0101s piecas reizes \u0101tr\u0101ki par String. Enum ir \u0101tr\u0101ks, jo darbojas ar skait\u013ciem. LowCardinality \u2014 t\u0101p\u0113c, ka ir ieviesta GROUP BY optimiz\u0101cija. <\/p>\n<p><img decoding=\"async\" alt=\"Rindkopu optimiz\u0101cija ClickHouse. Yandex prezent\u0101cija.\" src=\"\/wp-content\/uploads\/2020\/03\/e2ce4f2127d8728287488c85c92085fc.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nPavisam nedaudz sare\u017e\u0123\u012bsim vaic\u0101jumu \u2014 pajaut\u0101sim, kur atrodas popul\u0101r\u0101kais parks \u0145ujork\u0101. Atkal m\u0113s to m\u0113r\u012bsim p\u0113c t\u0101, kur visbie\u017e\u0101k pas\u016bta taksometru, bet filtr\u0113sim tikai t\u0101s lokaliz\u0101cijas, kur\u0101s ir v\u0101rds \"parks\". Ar\u012b pievienosim funkciju like. <\/p>\n<p><img decoding=\"async\" alt=\"Rindkopu optimiz\u0101cija ClickHouse. Yandex prezent\u0101cija.\" src=\"\/wp-content\/uploads\/2020\/03\/7cf6fd64751f3ae578581caddde17aaf.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nSkat\u0101mies laiku \u2014 redzam, ka Enum p\u0113k\u0161\u0146i s\u0101ka pal\u0113nin\u0101ties. Turkl\u0101t tas darbojas pat l\u0113n\u0101k nek\u0101 standarta datu tips String. Tas notiek t\u0101p\u0113c, ka funkcija like nav visp\u0101r optimiz\u0113ta Enum. Mums n\u0101kas konvert\u0113t m\u016bsu virknes no Enum uz parast\u0101m virk\u0101m \u2014 m\u0113s veicam vair\u0101k darba. LowCardinality(String) ar\u012b p\u0113c noklus\u0113juma nav optimiz\u0113ts, ta\u010du tur like darboj\u0101s p\u0101r v\u0101rdn\u012bcu, t\u0101p\u0113c vaic\u0101jums pa\u0101trin\u0101s sal\u012bdzin\u0101jum\u0101 ar String. <\/p>\n<p>Str\u0101d\u0101jot ar Enum, ir viena glob\u0101l\u0101ka probl\u0113ma. Ja m\u0113s v\u0113lamies to optimiz\u0113t, tas j\u0101dara katr\u0101 koda viet\u0101. Pie\u0146emsim, ka m\u0113s esam uzrakst\u012bju\u0161i jaunu funkciju \u2013 noteikti ir j\u0101izdom\u0101 optimiz\u0101cija Enum. Bet LowCardinality ir optimiz\u0113ts p\u0113c noklus\u0113juma.<\/p>\n<p><img decoding=\"async\" alt=\"Rindkopu optimiz\u0101cija ClickHouse. Yandex prezent\u0101cija.\" src=\"\/wp-content\/uploads\/2020\/03\/459243eb5b3dd993393d0184b224ea09.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nPaskat\u012bsimies uz p\u0113d\u0113jo piepras\u012bjumu, kas ir m\u0101ksl\u012bg\u0101ks. M\u0113s vienk\u0101r\u0161i saskait\u012bsim m\u016bsu atra\u0161an\u0101s vietas he\u0161a funkciju. He\u0161a funkcija ir diezgan l\u0113ns piepras\u012bjums, tas prasa ilgu laiku, t\u0101p\u0113c viss pal\u0113nin\u0101s apm\u0113ram tr\u012bs reizes.<\/p>\n<p><img decoding=\"async\" alt=\"Rindkopu optimiz\u0101cija ClickHouse. Yandex prezent\u0101cija.\" src=\"\/wp-content\/uploads\/2020\/03\/a5f49c22748971646ec358eb52d3feda.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nLowCardinality joproj\u0101m darbojas \u0101tr\u0101k, lai gan \u0161eit nav filtr\u0113\u0161anas. Tas notiek t\u0101p\u0113c, ka m\u016bsu funkcijas darbojas tikai ar v\u0101rdn\u012bcu. He\u0161a apr\u0113\u0137ina funkcijai ir viens arguments \u2013 t\u0101 var apstr\u0101d\u0101t maz\u0101k datus un ar\u012b var atgriezt LowCardinality. <\/p>\n<p><img decoding=\"async\" alt=\"Rindkopu optimiz\u0101cija ClickHouse. Yandex prezent\u0101cija.\" src=\"\/wp-content\/uploads\/2020\/03\/7842b7eb2debb4a4b1d4cbc3488deb32.jpg\" style=\"display:block;margin: 0 auto;\" \/><br \/>\n<br \/>\nM\u016bsu glob\u0101lais pl\u0101ns ir pan\u0101kt \u0101trumu, kas nav zem\u0101ks par String jebkuros gad\u012bjumos, un saglab\u0101t pa\u0101trin\u0101jumu. Un, iesp\u0113jams, m\u0113s k\u0101du dienu aizvietosim String ar LowCardinality, j\u016bs atjaunin\u0101siet ClickHouse, un j\u016bsu darbs nedaudz pa\u0101trin\u0101sies.<br \/>\n<br \/>Avots: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/company\/yandex\/blog\/492868\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0410\u043d\u0430\u043b\u0438\u0442\u0438\u0447\u0435\u0441\u043a\u0430\u044f \u0421\u0423\u0411\u0414 ClickHouse \u043e\u0431\u0440\u0430\u0431\u0430\u0442\u044b\u0432\u0430\u0435\u0442 \u043c\u043d\u043e\u0436\u0435\u0441\u0442\u0432\u043e \u0440\u0430\u0437\u043d\u044b\u0445 \u0441\u0442\u0440\u043e\u043a, \u043f\u043e\u0442\u0440\u0435\u0431\u043b\u044f\u044f \u0440\u0435\u0441\u0443\u0440\u0441\u044b. \u0414\u043b\u044f \u0443\u0441\u043a\u043e\u0440\u0435\u043d\u0438\u044f \u0440\u0430\u0431\u043e\u0442\u044b \u0441\u0438\u0441\u0442\u0435\u043c\u044b \u043f\u043e\u0441\u0442\u043e\u044f\u043d\u043d\u043e \u0434\u043e\u0431\u0430\u0432\u043b\u044f\u044e\u0442\u0441\u044f \u043d\u043e\u0432\u044b\u0435 \u043e\u043f\u0442\u0438\u043c\u0438\u0437\u0430\u0446\u0438\u0438. \u0420\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u0447\u0438\u043a ClickHouse \u041d\u0438\u043a\u043e\u043b\u0430\u0439 \u041a\u043e\u0447\u0435\u0442\u043e\u0432 \u0440\u0430\u0441\u0441\u043a\u0430\u0437\u044b\u0432\u0430\u0435\u0442 \u043e \u0441\u0442\u0440\u043e\u043a\u043e\u0432\u043e\u043c \u0442\u0438\u043f\u0435 \u0434\u0430\u043d\u043d\u044b\u0445, \u0432 \u0442\u043e\u043c \u0447\u0438\u0441\u043b\u0435 \u043e \u043d\u043e\u0432\u043e\u043c \u0442\u0438\u043f\u0435, LowCardinality, \u0438 \u043e\u0431\u044a\u044f\u0441\u043d\u044f\u0435\u0442, \u043a\u0430\u043a \u043c\u043e\u0436\u043d\u043e \u0443\u0441\u043a\u043e\u0440\u0438\u0442\u044c \u0440\u0430\u0431\u043e\u0442\u0443 \u0441\u043e \u0441\u0442\u0440\u043e\u043a\u0430\u043c\u0438. \u2014 \u0421\u043d\u0430\u0447\u0430\u043b\u0430 \u0434\u0430\u0432\u0430\u0439\u0442\u0435 \u0440\u0430\u0437\u0431\u0435\u0440\u0435\u043c\u0441\u044f, \u043a\u0430\u043a \u043c\u043e\u0436\u043d\u043e \u0445\u0440\u0430\u043d\u0438\u0442\u044c \u0441\u0442\u0440\u043e\u043a\u0438. \u0423 \u043d\u0430\u0441 \u0435\u0441\u0442\u044c \u0441\u0442\u0440\u043e\u043a\u043e\u0432\u044b\u0435 \u0442\u0438\u043f\u044b \u0434\u0430\u043d\u043d\u044b\u0445. [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":74738,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-74737","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.5 - aioseo.com -->\n\t<meta name=\"description\" content=\"\u0410\u043d\u0430\u043b\u0438\u0442\u0438\u0447\u0435\u0441\u043a\u0430\u044f \u0421\u0423\u0411\u0414 ClickHouse \u043e\u0431\u0440\u0430\u0431\u0430\u0442\u044b\u0432\u0430\u0435\u0442 \u043c\u043d\u043e\u0436\u0435\u0441\u0442\u0432\u043e \u0440\u0430\u0437\u043d\u044b\u0445 \u0441\u0442\u0440\u043e\u043a, \u043f\u043e\u0442\u0440\u0435\u0431\u043b\u044f\u044f \u0440\u0435\u0441\u0443\u0440\u0441\u044b. \u0414\u043b\u044f \u0443\u0441\u043a\u043e\u0440\u0435\u043d\u0438\u044f \u0440\u0430\u0431\u043e\u0442\u044b \u0441\u0438\u0441\u0442\u0435\u043c\u044b \u043f\u043e\u0441\u0442\u043e\u044f\u043d\u043d\u043e \u0434\u043e\u0431\u0430\u0432\u043b\u044f\u044e\u0442\u0441\u044f \u043d\u043e\u0432\u044b\u0435 \u043e\u043f\u0442\u0438\u043c\u0438\u0437\u0430\u0446\u0438\u0438.\" \/>\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/lv\/blog\/administrirovanie\/optimizacziya-strok-v-clickhouse-doklad-yandeksa\" \/>\n\t\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.5\" \/>\n\t\t<meta property=\"og:locale\" content=\"lv_LV\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u041e\u043f\u0442\u0438\u043c\u0438\u0437\u0430\u0446\u0438\u044f \u0441\u0442\u0440\u043e\u043a \u0432 ClickHouse. \u0414\u043e\u043a\u043b\u0430\u0434 \u042f\u043d\u0434\u0435\u043a\u0441\u0430 | ProHoster\" \/>\n\t\t<meta property=\"og:description\" content=\"\u0410\u043d\u0430\u043b\u0438\u0442\u0438\u0447\u0435\u0441\u043a\u0430\u044f \u0421\u0423\u0411\u0414 ClickHouse \u043e\u0431\u0440\u0430\u0431\u0430\u0442\u044b\u0432\u0430\u0435\u0442 \u043c\u043d\u043e\u0436\u0435\u0441\u0442\u0432\u043e \u0440\u0430\u0437\u043d\u044b\u0445 \u0441\u0442\u0440\u043e\u043a, \u043f\u043e\u0442\u0440\u0435\u0431\u043b\u044f\u044f \u0440\u0435\u0441\u0443\u0440\u0441\u044b. \u0414\u043b\u044f \u0443\u0441\u043a\u043e\u0440\u0435\u043d\u0438\u044f \u0440\u0430\u0431\u043e\u0442\u044b \u0441\u0438\u0441\u0442\u0435\u043c\u044b \u043f\u043e\u0441\u0442\u043e\u044f\u043d\u043d\u043e \u0434\u043e\u0431\u0430\u0432\u043b\u044f\u044e\u0442\u0441\u044f \u043d\u043e\u0432\u044b\u0435 \u043e\u043f\u0442\u0438\u043c\u0438\u0437\u0430\u0446\u0438\u0438.\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/lv\/blog\/administrirovanie\/optimizacziya-strok-v-clickhouse-doklad-yandeksa\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-03-20T05:43:14+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-03-20T05:43:14+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47Rindu optimiz\u0101cija ClickHouse. Yandex zi\u0146ojums | ProHoster","description":"Anal\u012btisk\u0101 DBMS ClickHouse apstr\u0101d\u0101 daudz da\u017e\u0101du rindu, pat\u0113r\u0113jot resursus. Lai pa\u0101trin\u0101tu sist\u0113mas darbu, nep\u0101rtraukti tiek pievienotas jaunas optimiz\u0101cijas.","canonical_url":"https:\/\/prohoster.info\/lv\/blog\/administrirovanie\/optimizacziya-strok-v-clickhouse-doklad-yandeksa","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"lv_LV","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u041e\u043f\u0442\u0438\u043c\u0438\u0437\u0430\u0446\u0438\u044f \u0441\u0442\u0440\u043e\u043a \u0432 ClickHouse. \u0414\u043e\u043a\u043b\u0430\u0434 \u042f\u043d\u0434\u0435\u043a\u0441\u0430 | ProHoster","og:description":"\u0410\u043d\u0430\u043b\u0438\u0442\u0438\u0447\u0435\u0441\u043a\u0430\u044f \u0421\u0423\u0411\u0414 ClickHouse \u043e\u0431\u0440\u0430\u0431\u0430\u0442\u044b\u0432\u0430\u0435\u0442 \u043c\u043d\u043e\u0436\u0435\u0441\u0442\u0432\u043e \u0440\u0430\u0437\u043d\u044b\u0445 \u0441\u0442\u0440\u043e\u043a, \u043f\u043e\u0442\u0440\u0435\u0431\u043b\u044f\u044f \u0440\u0435\u0441\u0443\u0440\u0441\u044b. \u0414\u043b\u044f \u0443\u0441\u043a\u043e\u0440\u0435\u043d\u0438\u044f \u0440\u0430\u0431\u043e\u0442\u044b \u0441\u0438\u0441\u0442\u0435\u043c\u044b \u043f\u043e\u0441\u0442\u043e\u044f\u043d\u043d\u043e \u0434\u043e\u0431\u0430\u0432\u043b\u044f\u044e\u0442\u0441\u044f \u043d\u043e\u0432\u044b\u0435 \u043e\u043f\u0442\u0438\u043c\u0438\u0437\u0430\u0446\u0438\u0438.","og:url":"https:\/\/prohoster.info\/lv\/blog\/administrirovanie\/optimizacziya-strok-v-clickhouse-doklad-yandeksa","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-03-20T05:43:14+00:00","article:modified_time":"2020-03-20T05:43:14+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"74737","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 17:23:10","updated":"2022-09-27 14:48:44","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/posts\/74737","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/comments?post=74737"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/posts\/74737\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/media\/74738"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/media?parent=74737"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/categories?post=74737"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/lv\/wp-json\/wp\/v2\/tags?post=74737"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}