{"id":91438,"date":"2020-08-13T19:42:36","date_gmt":"2020-08-13T17:42:36","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/effektivnoe-ispolzovanie-clickhouse-aleksej-milovidov-yandeks"},"modified":"2020-08-13T19:42:36","modified_gmt":"2020-08-13T17:42:36","slug":"effektivnoe-ispolzovanie-clickhouse-aleksej-milovidov-yandeks","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/effektivnoe-ispolzovanie-clickhouse-aleksej-milovidov-yandeks","title":{"rendered":"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/8e42c1fffffcc964eacef240ea90bbc5.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Duke ClickHouse \u00ebsht\u00eb nj\u00eb sistem i specializuar, prandaj gjat\u00eb p\u00ebrdorimit t\u00eb tij \u00ebsht\u00eb e r\u00ebnd\u00ebsishme t\u00eb merren parasysh ve\u00e7orit\u00eb e arkitektur\u00ebs s\u00eb tij. N\u00eb k\u00ebt\u00eb raport, Aleksei do t\u00eb flas\u00eb p\u00ebr shembuj t\u00eb gabimeve tipike q\u00eb ndodhin gjat\u00eb p\u00ebrdorimit t\u00eb ClickHouse, t\u00eb cilat mund t\u00eb \u00e7ojn\u00eb n\u00eb nj\u00eb funksionim t\u00eb pafavorsh\u00ebm. Do t\u00eb tregohen shembuj nga praktika, se si zgjedhja e nj\u00eb skeme t\u00eb caktuar t\u00eb p\u00ebrpunimit t\u00eb t\u00eb dh\u00ebnave mund t\u00eb ndryshoj\u00eb ndjesh\u00ebm performanc\u00ebn.<\/p>\n<p><noindex><a rel=\"nofollow\" name=\"habracut\"><\/a><\/noindex><\/p>\n<p>P\u00ebrsh\u00ebndetje t\u00eb gjith\u00ebve! M\u00eb quajn\u00eb Aleksei, un\u00eb punoj me ClickHouse.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/7f5fc01663be58d3d25e518a4169b8cf.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>S\u00eb pari, dua t\u00eb ju g\u00ebzoj, nuk do t'ju tregoj sot se \u00e7far\u00eb \u00ebsht\u00eb ClickHouse. Po t\u00eb jem i sinqert\u00eb, kam mjaft disi lodhur nga kjo. \u00c7do her\u00eb flas p\u00ebr t\u00eb, dhe besoj se t\u00eb gjith\u00eb tashm\u00eb e din\u00eb. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/fae6bd07d098200643d591b8e5bf51e8.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>N\u00eb vend t\u00eb k\u00ebsaj, do t\u00eb flas p\u00ebr gropat e mundshme, dometh\u00ebn\u00eb se si mund t\u00eb p\u00ebrdoret gabimisht ClickHouse. N\u00eb t\u00eb v\u00ebrtet\u00eb, nuk ka nevoj\u00eb t\u00eb frik\u00ebsoheni, sepse ne e zhvillojm\u00eb ClickHouse si nj\u00eb sistem q\u00eb \u00ebsht\u00eb i thjesht\u00eb, i p\u00ebrshtatsh\u00ebm dhe funksionon nga kutia. E vendos dhe gjith\u00e7ka, pa asnj\u00eb problem. <\/p>\n<p><\/p>\n<p>Megjithat\u00eb, \u00ebsht\u00eb e nevojshme t\u00eb mbani parasysh se ky \u00ebsht\u00eb nj\u00eb sistem i specializuar dhe \u00ebsht\u00eb e leht\u00eb t\u00eb hasni n\u00eb skenar\u00eb t\u00eb pazakont\u00eb t\u00eb p\u00ebrdorimit q\u00eb mund ta nxjerrin k\u00ebt\u00eb sistem jasht\u00eb zones s\u00eb tij t\u00eb rehatis\u00eb.<\/p>\n<p><\/p>\n<p>Pra, cilat jan\u00eb gropat e mundshme? Kryesisht do t\u00eb flas p\u00ebr gj\u00ebra t\u00eb dukshme. T\u00eb gjith\u00eb e kuptojn\u00eb gjith\u00e7ka dhe mund t\u00eb g\u00ebzohen se jan\u00eb kaq t\u00eb men\u00e7ur, nd\u00ebrsa ata q\u00eb nuk kuptojn\u00eb do t\u00eb m\u00ebsojn\u00eb di\u00e7ka t\u00eb re. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/bffacf353ff31b361460178e911f0e16.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Shembulli i par\u00eb shum\u00eb t\u00eb thjesht\u00eb, i cili, fatkeq\u00ebsisht, shpesh ndodh, \u00ebsht\u00eb nj\u00eb num\u00ebr i madh i insert-eve me grupe t\u00eb vogla, dometh\u00ebn\u00eb nj\u00eb num\u00ebr i madh i insert-eve t\u00eb vogla.<\/p>\n<p><\/p>\n<p>N\u00ebse e shqyrtojm\u00eb se si ClickHouse realizon insert, mund t\u00eb d\u00ebrgoni nj\u00eb fluks t\u00eb dh\u00ebnash deri n\u00eb nj\u00eb terabajt me nj\u00eb k\u00ebrkes\u00eb. Kjo nuk \u00ebsht\u00eb problem. <\/p>\n<p><\/p>\n<p>Le t\u00eb shohim se cila do t\u00eb ishte tipike p\u00ebr performanc\u00ebn. P\u00ebr shembull, tabela jon\u00eb \u00ebsht\u00eb me t\u00eb dh\u00ebnat nga Yandex.Metrika. Hitet. 105 disa kolona. 700 byte n\u00eb form\u00ebn e pa kompresuar. Dhe do t\u00eb insertojm\u00eb n\u00eb m\u00ebnyr\u00eb t\u00eb duhur grupe prej nj\u00eb milioni rreshtash. <\/p>\n<p><\/p>\n<p>Po insertojm\u00eb n\u00eb tabel\u00ebn MergeTree, del gjysm\u00eb milioni rreshta n\u00eb sekond\u00eb. Shum\u00eb mir\u00eb. N\u00eb tabel\u00ebn e replikuar - pak m\u00eb pak, rreth 400,000 rreshta n\u00eb sekond\u00eb. <\/p>\n<p><\/p>\n<p>Dhe n\u00ebse aktivizoni insertimin me kuorum, del pak m\u00eb pak, por p\u00ebrs\u00ebri nj\u00eb performanc\u00eb t\u00eb k\u00ebnaqshme, 250,000 rreshta n\u00eb sekond\u00eb. Insertimi me kuorum \u00ebsht\u00eb nj\u00eb mund\u00ebsi e pa dokumentuar n\u00eb ClickHouse.<\/p>\n<p><\/p>\n<p>* sipas gjendjes n\u00eb vitin 2020, <noindex><a rel=\"nofollow\" href=\"https:\/\/clickhouse.tech\/docs\/en\/operations\/settings\/settings\/#settings-insert_quorum\">tani \u00ebsht\u00eb dokumentuar.<\/a><\/noindex>.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/668498fc82a0f6e851d05e6e1ea67033.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>\u00c7far\u00eb do t\u00eb ndodhte n\u00ebse gjith\u00e7ka b\u00ebhet keq? Shtojm\u00eb nj\u00eb rresht n\u00eb tabel\u00ebn MergeTree dhe p\u00ebrfundojm\u00eb me 59 rreshta n\u00eb sekond\u00eb. Kjo \u00ebsht\u00eb 10,000 her\u00eb m\u00eb e ngadalt\u00eb. N\u00eb ReplicatedMergeTree \u2013 6 rreshta n\u00eb sekond\u00eb. Dhe n\u00ebse aktivizohet edhe quorum-i, at\u00ebher\u00eb b\u00ebhen vet\u00ebm 2 rreshta n\u00eb sekond\u00eb. M\u00eb duket se kjo \u00ebsht\u00eb nj\u00eb d\u00ebshtim i plot\u00eb. Si \u00ebsht\u00eb e mundur q\u00eb t\u00eb ec\u00ebsh kaq ngadal\u00eb? Madje kam nj\u00eb T-shirt q\u00eb thot\u00eb se ClickHouse nuk duhet ngadal\u00ebsuar. Por ndonj\u00ebher\u00eb ndodh. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/4c771eddc4e8453f60ef91123ae5d109.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>N\u00eb t\u00eb v\u00ebrtet\u00eb \u2013 \u00ebsht\u00eb nj\u00eb disavantazh yn\u00eb. Mund t\u00eb kishim b\u00ebr\u00eb q\u00eb gjith\u00e7ka t\u00eb funksiononte normalisht, por nuk e b\u00ebm\u00eb. Dhe nuk e b\u00ebm\u00eb sepse p\u00ebr skenar\u00ebt tan\u00eb \u2013 nuk ishte e nevojshme. Kemi pasur tashm\u00eb batche. Thjesht na vinin batche, dhe nuk kishte asnj\u00eb problem. Shtojm\u00eb dhe gjith\u00e7ka funksionon normal. Por, natyrisht, jan\u00eb t\u00eb mundshme skenar\u00eb t\u00eb ndrysh\u00ebm. P\u00ebr shembull, kur keni shum\u00eb server\u00eb ku gjenerohen t\u00eb dh\u00ebnat. Edhe n\u00ebse ata nuk e shtojn\u00eb t\u00eb dh\u00ebnat shpesh, p\u00ebrs\u00ebri ndodhin shtime t\u00eb shpeshta. Dhe duhet ta shmangim k\u00ebt\u00eb ndonj\u00ebher\u00eb. <\/p>\n<p><\/p>\n<p>Nga nj\u00eb pik\u00ebpamje teknike, thelbi \u00ebsht\u00eb se kur b\u00ebni nj\u00eb insert n\u00eb ClickHouse, t\u00eb dh\u00ebnat nuk shkojn\u00eb n\u00eb asnj\u00eb memtable. As nuk p\u00ebrdorim nj\u00eb MergeTree me struktur\u00eb log, por thjesht MergeTree, sepse nuk kemi as log, as memTable. Thjesht shkruajm\u00eb t\u00eb dh\u00ebnat n\u00eb sistemin e skedar\u00ebve, tashm\u00eb t\u00eb ndara n\u00eb kolona. Dhe n\u00ebse keni 100 coluna, at\u00ebher\u00eb duhet t\u00eb shkruani m\u00eb shum\u00eb se 200 skedar\u00eb n\u00eb nj\u00eb drejtori t\u00eb ve\u00e7ant\u00eb. Gjith\u00e7ka \u00ebsht\u00eb mjaft e ngarkuar. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/24688fe3d39d81b92f12ac1b8b21425b.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Dhe lind pyetja: \"Si t\u00eb b\u00ebjm\u00eb si\u00e7 duhet?\", n\u00ebse ndodhemi n\u00eb nj\u00eb situat\u00eb ku p\u00ebrs\u00ebri duhet t\u00eb regjistroni t\u00eb dh\u00ebnat n\u00eb ClickHouse.<\/p>\n<p><\/p>\n<p>Metoda 1. Kjo \u00ebsht\u00eb m\u00ebnyra m\u00eb e thjesht\u00eb. P\u00ebrdorni nj\u00eb radh\u00eb t\u00eb shp\u00ebrndar\u00eb. P\u00ebr shembull, Kafka. Thjesht nxirrni t\u00eb dh\u00ebnat nga Kafka, gruponi nj\u00eb her\u00eb n\u00eb sekond\u00eb. Dhe gjith\u00e7ka do t\u00eb jet\u00eb n\u00eb rregull, regjistroni, gjith\u00e7ka funksionon mir\u00eb. <\/p>\n<p><\/p>\n<p>Disavantazhi \u00ebsht\u00eb se Kafka \u00ebsht\u00eb nj\u00eb tjet\u00ebr sistem i r\u00ebnd\u00eb t\u00eb shp\u00ebrndar\u00eb. E kuptoj n\u00ebse kompania juaj tashm\u00eb e ka Kafka. Kjo \u00ebsht\u00eb mir\u00eb, kjo \u00ebsht\u00eb e p\u00ebrshtatshme. Por n\u00ebse nuk e keni, duhet t\u00eb mendoni disa her\u00eb para se t\u00eb sillni nj\u00eb tjet\u00ebr sistem t\u00eb shp\u00ebrndar\u00eb n\u00eb projektin tuaj. Prandaj, ka kuptim t\u00eb shqyrtoni alternativa. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/4f687221e8c7443abf02fb002c4eba5b.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Metoda 2. Kjo \u00ebsht\u00eb nj\u00eb alternativ\u00eb retro dhe mjaft e thjesht\u00eb. Keni nj\u00eb server q\u00eb gjeneron logjet tuaja. Ai thjesht i regjistron logjet n\u00eb nj\u00eb skedar. Dhe \u00e7do sekond\u00eb, p\u00ebr shembull, e rinovojm\u00eb at\u00eb skedar, ndaj krijojm\u00eb nj\u00eb t\u00eb ri. Nj\u00eb skript i ve\u00e7ant\u00eb ose p\u00ebrmes cron-it, ose nj\u00eb demon tjet\u00ebr merr skedarin m\u00eb t\u00eb vjet\u00ebr dhe e regjistron n\u00eb ClickHouse. N\u00ebse regjistrohen logjet \u00e7do sekond\u00eb, gjith\u00e7ka do t\u00eb shkoj\u00eb mrekullisht. <\/p>\n<p><\/p>\n<p>Por disavantazhi i k\u00ebsaj metode \u00ebsht\u00eb se n\u00ebse serveri ku gjenerohen logjet zhduket, t\u00eb dh\u00ebnat gjithashtu do t\u00eb zhduken.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/b4c28b6df87386c42b6908bdfccb6b90.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Metoda 3. Ka nj\u00eb tjet\u00ebr m\u00ebnyr\u00eb interesante, e cila \u00ebsht\u00eb krejt\u00ebsisht pa skedar\u00eb p\u00ebrkoh\u00ebsor\u00eb. P\u00ebr shembull, keni ndonj\u00eb aplikacion reklamimi ose ndonj\u00eb demon tjet\u00ebr interesant, i cili gjeneron t\u00eb dh\u00ebna. Dhe ju mund t\u00eb akumuloni nj\u00eb grup t\u00eb dh\u00ebnash direkt n\u00eb RAM, n\u00eb buffer. Kur kalon nj\u00eb sasi e mjaftueshme kohore, e ruheni k\u00ebt\u00eb buffer, krijoni nj\u00eb t\u00eb ri, dhe n\u00eb nj\u00eb thread t\u00eb ve\u00e7ant\u00eb, at\u00eb q\u00eb \u00ebsht\u00eb akumuluar tashm\u00eb, e d\u00ebrgoni n\u00eb ClickHouse.<\/p>\n<p><\/p>\n<p>Nga ana tjet\u00ebr, t\u00eb dh\u00ebnat gjithashtu zhduken kur ndodh kill -9. N\u00ebse serveri juaj bie, at\u00ebher\u00eb do t\u00eb humbni k\u00ebto t\u00eb dh\u00ebna. Nj\u00eb tjet\u00ebr problem \u00ebsht\u00eb se n\u00ebse nuk mund t\u00eb regjistroni n\u00eb baz\u00eb, t\u00eb dh\u00ebnat do t\u00eb akumulohen n\u00eb RAM. Dhe do t'ju mbaroj\u00eb RAM-i ose thjesht do t\u00eb humbni t\u00eb dh\u00ebna. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/dc375b22fcb26cfa111d0bdb4563aa88.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Metoda 4. Nj\u00eb tjet\u00ebr m\u00ebnyr\u00eb interesante. Keni ndonj\u00eb proces serveri. Ai mund t\u00eb d\u00ebrgoj\u00eb t\u00eb dh\u00ebnat n\u00eb ClickHouse menj\u00ebher\u00eb, por ta b\u00ebj\u00eb k\u00ebt\u00eb n\u00eb nj\u00eb lidhje t\u00eb vetme. P\u00ebr shembull, d\u00ebrgoni nj\u00eb k\u00ebrkes\u00eb http me transfer-encoding: chunked me insert-in. Dhe gjeneron chank-e jo shum\u00eb shpesh, mund t\u00eb d\u00ebrgoni edhe \u00e7do rresht, megjithat\u00eb do t\u00eb ket\u00eb overhead mbi framin e k\u00ebtyre t\u00eb dh\u00ebnave. <\/p>\n<p><\/p>\n<p>Megjithat\u00eb, n\u00eb k\u00ebt\u00eb rast, t\u00eb dh\u00ebnat do t\u00eb d\u00ebrgohen menj\u00ebher\u00eb n\u00eb ClickHouse. Dhe ClickHouse vet\u00eb do t'i buferizoj\u00eb ato. <\/p>\n<p><\/p>\n<p>Por gjithashtu lindin probleme. Tani do t\u00eb humbni t\u00eb dh\u00ebnat, p\u00ebrfshir\u00eb n\u00ebse procesi juaj ndalon dhe, n\u00ebse procesi ClickHouse ndalon, sepse do t\u00eb jet\u00eb nj\u00eb insert i pap\u00ebrfunduar. N\u00eb ClickHouse, insert-et jan\u00eb atomike deri n\u00eb nj\u00eb prag t\u00eb caktuar numri rreshtash. N\u00eb parim, kjo \u00ebsht\u00eb nj\u00eb m\u00ebnyr\u00eb interesante. Po ashtu mund t\u00eb p\u00ebrdoret.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/6a18a974b9600ade377a870390e95d3d.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>M\u00ebnyra 5. Ja nj\u00eb tjet\u00ebr m\u00ebnyr\u00eb interesante. Ky \u00ebsht\u00eb nj\u00eb server i zhvilluar nga komuniteti p\u00ebr grumbullimin e t\u00eb dh\u00ebnave. Nuk e kam shqyrtuar vet\u00eb, ndaj nuk mund t\u00eb garantosh asgj\u00eb. Megjithat\u00eb, as p\u00ebr ClickHouse nuk ofrohen garanci. Kjo \u00ebsht\u00eb gjithashtu open source, por nga ana tjet\u00ebr, mund t\u00eb jeni m\u00ebsuar me nj\u00eb standard t\u00eb caktuar cil\u00ebsie q\u00eb ne p\u00ebrpiqemi ta sigurojm\u00eb. P\u00ebr k\u00ebt\u00eb gj\u00eb, nuk e di, hyni n\u00eb GitHub, shikoni kodin. Ndoshta kan\u00eb shkruar di\u00e7ka t\u00eb mir\u00eb. <\/p>\n<p><\/p>\n<p>* n\u00eb gjendje deri n\u00eb vitin 2020, gjithashtu duhet t\u00eb shtohet p\u00ebr shqyrtim <noindex><a rel=\"nofollow\" href=\"https:\/\/habr.com\/en\/company\/vk\/blog\/430168\/\">KittenHouse<\/a><\/noindex>.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/ae6a04af63ae4e7fa160c002fdb6d506.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>M\u00ebnyra 6. Nj\u00eb tjet\u00ebr m\u00ebnyr\u00eb \u00ebsht\u00eb p\u00ebrdorimi i tabelave Buffer. Avantazhi i k\u00ebsaj metode \u00ebsht\u00eb se \u00ebsht\u00eb shum\u00eb e leht\u00eb p\u00ebr t'u filluar. Krijoni nj\u00eb tabel\u00eb Buffer dhe e shtoni atje. <\/p>\n<p><\/p>\n<p>Por disavantazhi \u00ebsht\u00eb se problemi nuk zgjidhet plot\u00ebsisht. N\u00ebse n\u00eb insertimin e llojit MergeTree duhet t\u00eb gruposh t\u00eb dh\u00ebnat p\u00ebr nj\u00eb bat\u00e7 n\u00eb sekond\u00eb, kur e shton n\u00eb tabel\u00ebn buffer, duhet t\u00eb gruposh t\u00eb pakt\u00ebn disa mij\u00ebra n\u00eb sekond\u00eb. N\u00ebse kalon 10,000 n\u00eb sekond\u00eb, ndodhin probleme. Por n\u00ebse e insertoni n\u00eb grupime, do t\u00eb keni qindra mij\u00ebra rreshta n\u00eb sekond\u00eb. Dhe kjo \u00ebsht\u00eb n\u00eb t\u00eb dh\u00ebna mjaft t\u00eb r\u00ebnda. <\/p>\n<p><\/p>\n<p>Po ashtu, tabelat buffer nuk kan\u00eb log. Dhe n\u00ebse me serverin tuaj ndodh ndonj\u00eb gj\u00eb, t\u00eb dh\u00ebnat do t\u00eb humbasin. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/9bb16bf5e8145cc8368b11e54c86c837.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Dhe si nj\u00eb bonus, koh\u00ebt e fundit kemi shtuar n\u00eb ClickHouse mund\u00ebsin\u00eb p\u00ebr t\u00eb marr\u00eb t\u00eb dh\u00ebna nga Kafka. Ekziston nj\u00eb motor tabelash - Kafka. Thjesht krijoni at\u00eb. Dhe mbi t\u00eb mund t\u00eb vendosni pamje t\u00eb materializuara. N\u00eb k\u00ebt\u00eb rast, ai do t\u00eb nxjerr\u00eb t\u00eb dh\u00ebnat nga Kafka dhe t'i vendos\u00eb n\u00eb tabelat q\u00eb ju nevojiten. <\/p>\n<p><\/p>\n<p>Dhe ve\u00e7an\u00ebrisht k\u00ebnaq\u00ebsia e k\u00ebsaj mund\u00ebsie \u00ebsht\u00eb se nuk e kemi b\u00ebr\u00eb ne. Kjo \u00ebsht\u00eb nj\u00eb ve\u00e7ori e komunitetit. Dhe kur them \"ve\u00e7ori e komunitetit\", nuk e them me ndonj\u00eb p\u00ebr\u00e7mim. E kemi lexuar kodin, kemi b\u00ebr\u00eb rishikime, duhet t\u00eb funksionoj\u00eb normalisht. <\/p>\n<p><\/p>\n<p>* n\u00eb gjendje deri n\u00eb vitin 2020, u shfaq mb\u00ebshtetje e ngjashme p\u00ebr <noindex><a rel=\"nofollow\" href=\"https:\/\/clickhouse.tech\/docs\/en\/engines\/table-engines\/integrations\/rabbitmq\/\">RabbitMQ<\/a><\/noindex>.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/15dc61dfd435f5f8640395b7de480fe2.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>\u00c7far\u00eb tjet\u00ebr mund t\u00eb jet\u00eb e pak\u00ebndshme ose e papritur gjat\u00eb insertimit t\u00eb t\u00eb dh\u00ebnave? N\u00ebse b\u00ebni nj\u00eb k\u00ebrkes\u00eb p\u00ebr insertim vlerash dhe n\u00eb vlerat shkruani shprehje t\u00eb kalkuluara. P\u00ebr shembull, now() \u00ebsht\u00eb gjithashtu nj\u00eb shprehje e kalkuluar. N\u00eb k\u00ebt\u00eb rast, ClickHouse \u00ebsht\u00eb i detyruar t\u00eb ekzekutoj\u00eb interpretorin e k\u00ebtyre shprehjeve p\u00ebr \u00e7do rresht, dhe performanca do t\u00eb bjer\u00eb me rregulla. \u00cbsht\u00eb m\u00eb mir\u00eb t\u00eb shmangni k\u00ebt\u00eb.<\/p>\n<p><\/p>\n<p>* p\u00ebr momentin, problemi \u00ebsht\u00eb zgjidhur plot\u00ebsisht, nuk ka m\u00eb regres n\u00eb performanc\u00eb kur p\u00ebrdoren shprehjet n\u00eb VALUES.<\/p>\n<p><\/p>\n<p>Nj\u00eb shembull tjet\u00ebr \u00ebsht\u00eb kur mund t\u00eb ket\u00eb disa probleme kur keni t\u00eb dh\u00ebna p\u00ebr nj\u00eb lot q\u00eb p\u00ebrfshin shum\u00eb particione. N\u00eb m\u00ebnyr\u00eb standarde, n\u00eb ClickHouse particionet jan\u00eb sipas muajve. Dhe n\u00ebse po insertoni nj\u00eb grup prej nj\u00eb milioni rreshtash, dhe aty ka t\u00eb dh\u00ebna p\u00ebr disa vjet, at\u00ebher\u00eb do t\u00eb keni disa dhjet\u00ebra particione. Dhe kjo \u00ebsht\u00eb ekuivalente me faktin se do t\u00eb ekzistojn\u00eb grupe me disa dhjet\u00ebra her\u00eb m\u00eb t\u00eb vogla, sepse brenda tyre gjithmon\u00eb ndahen s\u00eb pari sipas particioneve.<\/p>\n<p><\/p>\n<p>* s\u00eb fundmi, n\u00eb ClickHouse u shtua n\u00eb m\u00ebnyr\u00eb eksperimentale mb\u00ebshtetje p\u00ebr formatin kompakt t\u00eb cop\u00ebzave dhe cop\u00ebzave n\u00eb memorie me regjistrin e shkruar p\u00ebrpara, q\u00eb pothuajse plot\u00ebsisht zgjidh problemin.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/b8c6cecdba53559dad31ed11713a980d.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Tani le t\u00eb shqyrtojm\u00eb llojin e dyt\u00eb t\u00eb problemit - \u00ebsht\u00eb tipizimi i t\u00eb dh\u00ebnave. <\/p>\n<p><\/p>\n<p>Tipizimi i t\u00eb dh\u00ebnave mund t\u00eb jet\u00eb i fort\u00eb ose i varg. I vargu \u00ebsht\u00eb kur thjesht e keni shpallur se t\u00eb gjitha fushat jan\u00eb t\u00eb tipit string. Kjo \u00ebsht\u00eb nj\u00eb katastrof\u00eb. Nuk duhet b\u00ebr\u00eb ashtu. <\/p>\n<p><\/p>\n<p>Le t\u00eb merremi me se si t\u00eb b\u00ebjm\u00eb si\u00e7 duhet n\u00eb ato raste kur d\u00ebshirojm\u00eb t\u00eb themi se nj\u00eb fush\u00eb \u00ebsht\u00eb string, dhe le t\u00eb merret ClickHouse vet\u00eb me k\u00ebt\u00eb, pa u shqet\u00ebsuar. Por prap\u00eb, ia vlen t\u00eb b\u00ebni disa p\u00ebrpjekje. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/a0fdcf6423933293318411242fe2ede7.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>P\u00ebr shembull, kemi nj\u00eb adres\u00eb IP. N\u00eb nj\u00eb rast e kemi ruajtur si string. P\u00ebr shembull, 192.168.1.1. Nd\u00ebrsa n\u00eb rastin tjet\u00ebr - do t\u00eb jet\u00eb nj\u00eb num\u00ebr i tipit UInt32*. 32 bit jan\u00eb t\u00eb mjaftueshme p\u00ebr adres\u00ebn IPv4.<\/p>\n<p><\/p>\n<p>S\u00eb pari, si\u00e7 \u00ebsht\u00eb e habitshme, t\u00eb dh\u00ebnat do t\u00eb kompresohen n\u00eb m\u00ebnyr\u00eb t\u00eb ngjashme. Do t\u00eb ket\u00eb ndonj\u00eb diferenc\u00eb, sigurisht, por jo kaq e madhe. Pra, nuk ka probleme t\u00eb ve\u00e7anta me hyrjen\/daljen n\u00eb disk. <\/p>\n<p><\/p>\n<p>Por ka nj\u00eb ndryshim t\u00eb r\u00ebnd\u00ebsish\u00ebm n\u00eb koh\u00ebn e procesorit dhe n\u00eb koh\u00ebn e ekzekutimit t\u00eb pyetjes. <\/p>\n<p><\/p>\n<p>Le t\u00eb llogarisim numrin e adresave IP unike, n\u00ebse ato ruhen si numra. Rezultati \u00ebsht\u00eb 137 milion rreshta n\u00eb sekond\u00eb. N\u00ebse e nj\u00ebjta n\u00eb form\u00eb string, at\u00ebher\u00eb 37 milion rreshta n\u00eb sekond\u00eb. Nuk e di pse doli nj\u00eb p\u00ebrshtatje e till\u00eb. Un\u00eb vet\u00eb i kam ekzekutuar k\u00ebto pyetje. Megjithat\u00eb, \u00ebsht\u00eb ndoshta rreth 4 her\u00eb m\u00eb e ngadalshme. <\/p>\n<p><\/p>\n<p>Dhe n\u00ebse llogarisim diferenc\u00ebn n\u00eb hap\u00ebsir\u00ebn n\u00eb disk, ka nj\u00eb diferenc\u00eb gjithashtu. Dhe diferenca \u00ebsht\u00eb ndoshta rreth nj\u00eb t\u00eb kat\u00ebrt\u00ebs, sepse ka mjaft adresash IP unike. Dhe n\u00ebse k\u00ebtu do t\u00eb ishin rreshta me nj\u00eb num\u00ebr t\u00eb vog\u00ebl t\u00eb vlerave t\u00eb ndryshme, ato do t\u00eb kompresoheshin qet\u00ebsisht n\u00eb nj\u00eb volum t\u00eb ngjash\u00ebm sipas fjalorit. <\/p>\n<p><\/p>\n<p>Dhe kat\u00ebrfish diferenc\u00eb n\u00eb koh\u00eb n\u00eb rrug\u00eb nuk \u00ebsht\u00eb di\u00e7ka e zakonshme. Mund t\u00eb duket se nuk ka r\u00ebnd\u00ebsi p\u00ebr ju, por kur shoh nj\u00eb diferenc\u00eb t\u00eb till\u00eb, m\u00eb vjen keq.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/f74dbfdab5a9e26fa044a5870a8cc00d.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Le t\u00eb shqyrtojm\u00eb raste t\u00eb ndryshme. <\/p>\n<p><\/p>\n<p>1. Nj\u00eb rast kur keni pak vlera unike t\u00eb ndryshme. N\u00eb k\u00ebt\u00eb rast, p\u00ebrdorim nj\u00eb praktik\u00eb t\u00eb thjesht\u00eb, q\u00eb ndoshta e dini dhe mund ta p\u00ebrdorni p\u00ebr \u00e7do DBMS. Kjo ka kuptim jo vet\u00ebm p\u00ebr ClickHouse. Thjesht regjistroni identifikuesit numerik\u00eb n\u00eb baz\u00ebn e t\u00eb dh\u00ebnave. Nd\u00ebrsa konvertimi n\u00eb vargje dhe anasjelltas mund t\u00eb b\u00ebhet n\u00eb an\u00ebn e aplikacionit tuaj. <\/p>\n<p><\/p>\n<p>Ja, p\u00ebr shembull, keni nj\u00eb rajon. Dhe p\u00ebrpiqeni ta ruani at\u00eb si nj\u00eb varg. Atje do t\u00eb shkruhet: Mosk\u00eb dhe MO. Kur shoh se \u00ebsht\u00eb shkruar \"Mosk\u00eb\", nuk \u00ebsht\u00eb keq, por kur \u00ebsht\u00eb edhe MO, m\u00eb duket shum\u00eb e trishtueshme. Sa byte \u00ebsht\u00eb kjo? <\/p>\n<p><\/p>\n<p>N\u00eb vend t\u00eb k\u00ebsaj, ne thjesht regjistrojm\u00eb numrin Ulnt32 dhe 250. Ne kemi 250 n\u00eb Yandex, ndoshta ju keni ndryshe. P\u00ebr t\u00eb qen\u00eb t\u00eb sigurt, po them se n\u00eb ClickHouse ka nj\u00eb mund\u00ebsi t\u00eb nd\u00ebrtuar p\u00ebr t\u00eb punuar me baz\u00ebn e t\u00eb dh\u00ebnave gjeografike. Thjesht regjistroni nj\u00eb referenc\u00eb me rajonet, p\u00ebrfshir\u00eb hierarkin\u00eb, pra do t\u00eb ket\u00eb dhe Mosk\u00ebn, dhe MO, dhe gjith\u00e7ka q\u00eb ju nevojitet. Dhe mund t\u00eb konvertoni n\u00eb nivelin e k\u00ebrkes\u00ebs. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/cc6e871136dab8f8a35246a5af512cda.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Varianti i dyt\u00eb \u00ebsht\u00eb m\u00eb pak i ngjash\u00ebm, por tashm\u00eb me mb\u00ebshtetje brenda ClickHouse. Ky \u00ebsht\u00eb tipi i t\u00eb dh\u00ebnave Enum. Thjesht brenda Enum regjistroni t\u00eb gjitha vlerat q\u00eb ju nevojiten. P\u00ebr shembull, lloji i pajisjes dhe aty shkruani: desktop, mobil, tablet, televizor. N\u00eb total 4 mund\u00ebsi. <\/p>\n<p><\/p>\n<p>Mungesa \u00ebsht\u00eb se duhet her\u00eb pas here t\u00eb b\u00ebni alter. Shtuat vet\u00ebm nj\u00eb variant. B\u00ebni alter table. N\u00eb fakt, alter table n\u00eb ClickHouse \u00ebsht\u00eb falas. Sidomos falas p\u00ebr Enum, sepse t\u00eb dh\u00ebnat n\u00eb disk nuk ndryshojn\u00eb. Por megjithat\u00eb, alter merr nj\u00eb bllokim* n\u00eb tabel\u00eb dhe duhet t\u00eb pres\u00eb derisa t\u00eb realizohen t\u00eb gjitha selektimet. Dhe vet\u00ebm pas k\u00ebsaj alter do t\u00eb ekzekutohet, pra, megjithat\u00eb disa p\u00ebrshtypje t\u00eb pak\u00ebndshme mbeten.<\/p>\n<p><\/p>\n<p>* n\u00eb versionet e fundit t\u00eb ClickHouse, ALTER \u00ebsht\u00eb b\u00ebr\u00eb plot\u00ebsisht pa bllokim.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/d7284fa5be583ab063bb6de0b60f54d4.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Nj\u00eb variant tjet\u00ebr mjaft unik p\u00ebr ClickHouse \u2013 \u00ebsht\u00eb lidhja e fjalor\u00ebve t\u00eb jasht\u00ebm. Mund t\u00eb shkruani n\u00eb ClickHouse numra, dhe referencat tuaja i mbani n\u00eb ndonj\u00eb sistem q\u00eb ju p\u00ebrshtatet. P\u00ebr shembull, mund t\u00eb p\u00ebrdorni: MySQL, Mongo, Postgres. Madje mund t\u00eb krijoni nj\u00eb mikrosh\u00ebrbim tuaj q\u00eb do t\u00eb d\u00ebrgoj\u00eb k\u00ebto t\u00eb dh\u00ebna p\u00ebrmes http. Dhe n\u00eb nivelin e ClickHouse shkruani nj\u00eb funksion, q\u00eb do t\u00eb konvertoj\u00eb k\u00ebto t\u00eb dh\u00ebna nga numra n\u00eb vargje. <\/p>\n<p><\/p>\n<p>Ky \u00ebsht\u00eb nj\u00eb m\u00ebnyr\u00eb e specializuar, por shum\u00eb efektive p\u00ebr t\u00eb kryer nj\u00eb bashkim me nj\u00eb tabel\u00eb t\u00eb jashtme. Ka dy variante. N\u00eb nj\u00eb variant, t\u00eb dh\u00ebnat do t\u00eb jen\u00eb plot\u00ebsisht t\u00eb ruajtura n\u00eb memorie dhe t\u00eb rinovohen me nj\u00eb frekuenc\u00eb t\u00eb caktuar. N\u00eb variantin tjet\u00ebr, n\u00ebse t\u00eb dh\u00ebnat nuk i p\u00ebrshtaten memoris\u00eb, ato mund t\u00eb ruhen pjes\u00ebrisht. <\/p>\n<p><\/p>\n<p>Ja nj\u00eb shembull. Ekziston Yandex.Direct. Aty ka kompani reklamuese dhe bannera. Ka ndoshta rreth dhjet\u00eb milion kompani reklamuese. Ato mund t\u00eb ruhen n\u00eb memorie. Nd\u00ebrsa p\u00ebr bannerat \u2013 jan\u00eb miliarda, dhe ato nuk i p\u00ebrshtaten. Ne p\u00ebrdorim nj\u00eb fjalor t\u00eb ruajtsh\u00ebm nga MySQL.<\/p>\n<p><\/p>\n<p>Problemi i vet\u00ebm \u00ebsht\u00eb q\u00eb fjalori i ruajtsh\u00ebm do t\u00eb funksionoj\u00eb normalisht n\u00ebse shkalla e goditjes \u00ebsht\u00eb af\u00ebrsisht 100%. N\u00ebse \u00ebsht\u00eb m\u00eb e ul\u00ebt, at\u00ebher\u00eb gjat\u00eb p\u00ebrpunimit t\u00eb k\u00ebrkesave p\u00ebr \u00e7do grup t\u00eb dh\u00ebnash, do t\u00eb duhet t\u00eb merrni realisht \u00e7el\u00ebsat q\u00eb mungojn\u00eb dhe t\u00eb marr\u00ebni t\u00eb dh\u00ebnat nga MySQL. P\u00ebr ClickHouse mund t\u00eb sigurohem se - po, nuk ngadal\u00ebson, p\u00ebr sistemet e tjera nuk do t\u00eb flas.<\/p>\n<p><\/p>\n<p>Si nj\u00eb bonus, fjalor\u00ebt jan\u00eb nj\u00eb m\u00ebnyr\u00eb shum\u00eb e thjesht\u00eb p\u00ebr t\u00eb p\u00ebrdit\u00ebsuar t\u00eb dh\u00ebnat n\u00eb ClickHouse retroaktivisht. Dometh\u00ebn\u00eb, n\u00ebse kishit nj\u00eb raport mbi kompanit\u00eb reklamuese, p\u00ebrdoruesi thjesht nd\u00ebrron kompanin\u00eb reklamuese dhe n\u00eb t\u00eb gjitha t\u00eb dh\u00ebnat e vjetra, n\u00eb t\u00eb gjitha raportet, k\u00ebto t\u00eb dh\u00ebna gjithashtu ndryshojn\u00eb. N\u00ebse shkruani rreshta drejtp\u00ebrdrejt n\u00eb tabel\u00eb, at\u00ebher\u00eb p\u00ebrdit\u00ebsimi i tyre do t\u00eb jet\u00eb i pamundur. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/a5a2919d6be7d00c0ad5875b9a8f0f31.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Nj\u00eb m\u00ebnyr\u00eb tjet\u00ebr, kur nuk e dini nga t\u00eb merrni identifikuesit p\u00ebr rreshtat tuaj, \u00ebsht\u00eb thjesht t\u00eb hash. Dhe varianti m\u00eb i thjesht\u00eb \u00ebsht\u00eb t\u00eb merrni nj\u00eb hash 64-bit\u00ebsh. <\/p>\n<p><\/p>\n<p>Problemi i vet\u00ebm \u00ebsht\u00eb q\u00eb n\u00ebse hash-i \u00ebsht\u00eb 64-bit\u00ebsh, at\u00ebher\u00eb kolidzionet do t\u00eb ndodhin almost me siguri. Sepse n\u00ebse aty ka nj\u00eb miliard rreshtash, probabiliteti tashm\u00eb b\u00ebhet i duksh\u00ebm. <\/p>\n<p><\/p>\n<p>E nuk do t\u00eb ishte shum\u00eb mir\u00eb t\u00eb hash emrat e kompanive reklamuese n\u00eb k\u00ebt\u00eb m\u00ebnyr\u00eb. N\u00ebse kompanit\u00eb reklamuese t\u00eb kompanive t\u00eb ndryshme ngat\u00ebrrohen, at\u00ebher\u00eb do t\u00eb ket\u00eb di\u00e7ka t\u00eb paqart\u00eb. <\/p>\n<p><\/p>\n<p>Dhe \u00ebsht\u00eb nj\u00eb truk i thjesht\u00eb. E v\u00ebrteta, nuk \u00ebsht\u00eb shum\u00eb i p\u00ebrshtatsh\u00ebm p\u00ebr t\u00eb dh\u00ebna serioze, por n\u00ebse keni di\u00e7ka q\u00eb nuk \u00ebsht\u00eb shum\u00eb serioze, thjesht shtoni identifikuesin e klientit n\u00eb \u00e7el\u00ebsin e fjalorit. K\u00ebshtu q\u00eb do t\u00eb keni kolizione, por vet\u00ebm brenda nj\u00eb klienti. Ky metod\u00eb p\u00ebrdoret p\u00ebr kart\u00ebn e lidhjeve n\u00eb Yandex.Metrica. Atje kemi URL, ruajm\u00eb hash-at. E dim\u00eb, sigurisht, q\u00eb kan\u00eb kolizione. Por kur shfaqet faqja, probabiliteti q\u00eb n\u00eb nj\u00eb faqe t\u00eb nj\u00eb p\u00ebrdoruesi disa URL t\u00eb shqiptuara t\u00eb lidhen dhe t\u00eb vihen re \u00ebsht\u00eb aq i vog\u00ebl sa mund ta injoroni. <\/p>\n<p><\/p>\n<p>Si nj\u00eb bonus \u2013 p\u00ebr shum\u00eb operacione, mjaftojn\u00eb vet\u00ebm hash-at dhe vet\u00eb stringat mund t\u00eb mos ruhen askund. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/abbb73aaba29d10e3b7e7650bf68e6a6.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Nj\u00eb rast tjet\u00ebr, n\u00ebse stringat jan\u00eb t\u00eb shkurt\u00ebr, si p\u00ebr shembull, domenet e faqeve. Mund t'i ruani ashtu si jan\u00eb. Ose, p\u00ebr shembull, gjuha e shfletuesit ru \u2013 2 byte. P\u00ebr mua, sigurisht, \u00ebsht\u00eb shum\u00eb keq p\u00ebr byte-at, por mos u shqet\u00ebsoni, 2 byte nuk jan\u00eb shum\u00eb. Ju lutem, ruajtni ashtu si jan\u00eb, mos u shqet\u00ebsoni. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/901eaed0029da6ea59630b57ccf80c25.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Nj\u00eb rast tjet\u00ebr, kur, p\u00ebrkundrazi, ka shum\u00eb stringa dhe n\u00eb to ka shum\u00eb unik\u00eb, edhe shum\u00eb potencialisht t\u00eb pakufizuar. Nj\u00eb shembull tipik \u00ebsht\u00eb frazat e k\u00ebrkimit ose URL-t\u00eb. Fraza t\u00eb k\u00ebrkimit, p\u00ebrfshir\u00eb p\u00ebr shkak t\u00eb gabimeve t\u00eb shkruar. T\u00eb shohim sa fraza unike k\u00ebrkohen n\u00eb nj\u00eb dit\u00eb. Dhe rezulton se gjysma e tyre \u00ebsht\u00eb nga t\u00eb gjitha ngjarjet. Dhe n\u00eb k\u00ebt\u00eb rast mund t\u00eb mendoni se duhet t\u00eb normalizoni t\u00eb dh\u00ebnat, t\u00eb llogaritni identifikuesit, t'i vendosni n\u00eb nj\u00eb tabel\u00eb t\u00eb ve\u00e7ant\u00eb. Por nuk duhet ta b\u00ebni k\u00ebt\u00eb. Thjesht ruani stringat ashtu si jan\u00eb. <\/p>\n<p><\/p>\n<p>M\u00eb mir\u00eb \u2013 mos shpikni asgj\u00eb, sepse n\u00ebse i ruani ve\u00e7mas, do t'ju duhet t\u00eb b\u00ebni join. Dhe ai join \u2013 n\u00eb rastin m\u00eb t\u00eb mir\u00eb, aksesi rast\u00ebsor n\u00eb memorie, n\u00ebse madje futet n\u00eb memorie. N\u00ebse nuk futet, at\u00ebher\u00eb do t\u00eb keni probleme. <\/p>\n<p><\/p>\n<p>Por n\u00ebse t\u00eb dh\u00ebnat ruhen n\u00eb vend, ato thjesht lexohen n\u00eb rendin e duhur nga sistemi i skedar\u00ebve dhe gjith\u00e7ka \u00ebsht\u00eb n\u00eb rregull.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/15f1d632083def16ccbc939579c1cf6b.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>N\u00ebse keni URL ose ndonj\u00eb string t\u00eb gjat\u00eb dhe t\u00eb komplikuar, duhet t\u00eb mendoni p\u00ebr at\u00eb q\u00eb mund t\u00eb llogaritni ndonj\u00eb p\u00ebrmbledhje paraprakisht dhe ta regjistroni n\u00eb nj\u00eb kolon\u00eb t\u00eb ve\u00e7ant\u00eb. <\/p>\n<p><\/p>\n<p>P\u00ebr URL, p\u00ebr shembull, mund t\u00eb ruani ve\u00e7mas domenin. Dhe n\u00ebse ju nevojitet me t\u00eb v\u00ebrtet\u00eb domeni, thjesht p\u00ebrdorni k\u00ebt\u00eb kolon\u00eb, nd\u00ebrsa URL do t\u00eb q\u00ebndrojn\u00eb dhe ju as nuk do t'i prekni. <\/p>\n<p><\/p>\n<p>Le t\u00eb shohim se cila \u00ebsht\u00eb diferenca. N\u00eb ClickHouse ka nj\u00eb funksion t\u00eb specializuar q\u00eb llogarit domenin. Ai \u00ebsht\u00eb shum\u00eb i shpejt\u00eb, ne e kemi optimizuar. Dhe, p\u00ebr ta th\u00ebn\u00eb hapur, ai madje nuk i p\u00ebrputhet RFC, por megjithat\u00eb llogarit gjith\u00e7ka q\u00eb na nevojitet. <\/p>\n<p><\/p>\n<p>N\u00eb nj\u00eb rast, ne do t\u00eb nxjerrim thjesht URL-t\u00eb dhe do t\u00eb llogarisim domenin. Kjo rezulton n\u00eb 166 milisekonda. N\u00ebse e marrim nj\u00eb domen t\u00eb gatsh\u00ebm, rezultati \u00ebsht\u00eb vet\u00ebm 67 milisekonda, pra pothuajse tre her\u00eb m\u00eb shpejt. Dhe kjo \u00ebsht\u00eb m\u00eb e shpejt\u00eb jo p\u00ebr shkak se na duhen llogaritje t\u00eb ndonj\u00eb lloji, por sepse lexojm\u00eb m\u00eb pak t\u00eb dh\u00ebna. <\/p>\n<p><\/p>\n<p>Megjithat\u00eb, p\u00ebr nj\u00eb k\u00ebrkes\u00eb q\u00eb \u00ebsht\u00eb m\u00eb e ngadalshme, rezulton t\u00eb ket\u00eb m\u00eb shum\u00eb shpejt\u00ebsi gigabajt\u00ebsh p\u00ebr sekond\u00eb. Sepse ajo lexon m\u00eb shum\u00eb gigabajt\u00eb. K\u00ebto jan\u00eb t\u00eb dh\u00ebna krejt\u00ebsisht t\u00eb panevojshme. K\u00ebrkesa duket se punon m\u00eb shpejt, por p\u00ebrfundon n\u00eb nj\u00eb koh\u00eb m\u00eb t\u00eb gjat\u00eb.<\/p>\n<p><\/p>\n<p>Dhe kur shohim hap\u00ebsir\u00ebn e t\u00eb dh\u00ebnave n\u00eb disk, rezulton se URL-ja \u00ebsht\u00eb 126 megabajt\u00eb, nd\u00ebrsa domeni vet\u00ebm 5 megabajt\u00eb. Pra, \u00ebsht\u00eb 25 her\u00eb m\u00eb pak. Por megjithat\u00eb, k\u00ebrkesa p\u00ebrfundohet vet\u00ebm 4 her\u00eb m\u00eb shpejt. Kjo \u00ebsht\u00eb p\u00ebr shkak se t\u00eb dh\u00ebnat jan\u00eb t\u00eb ngrohta. N\u00ebse do t\u00eb ishin t\u00eb ftohta, at\u00ebher\u00eb me siguri do t\u00eb ishin 25 her\u00eb m\u00eb shpejt p\u00ebr shkak t\u00eb hyrjes dhe daljes nga disku. <\/p>\n<p><\/p>\n<p>P\u00ebr m\u00eb tep\u00ebr, n\u00ebse vler\u00ebsojm\u00eb se sa m\u00eb i vog\u00ebl \u00ebsht\u00eb domin nga URL-ja, rezulton diku rreth 4 her\u00eb. Por p\u00ebr \u00e7udi, t\u00eb dh\u00ebnat n\u00eb disk z\u00ebn\u00eb 25 her\u00eb m\u00eb pak. Pse? P\u00ebr shkak t\u00eb kompresimit. T\u00eb dy, URL-ja dhe domeni kompresohen. Por shpesh URL-ja p\u00ebrmban shum\u00eb \u2018zhurm\u00eb\u2019. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/c93d2128a41561ed7abb25d77414bc2d.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Dhe, sigurisht, \u00ebsht\u00eb e r\u00ebnd\u00ebsishme t\u00eb p\u00ebrdoren llojet e duhura t\u00eb t\u00eb dh\u00ebnave, t\u00eb cilat jan\u00eb t\u00eb dizajnuara posa\u00e7\u00ebrisht p\u00ebr vlerat e nevojshme ose q\u00eb i p\u00ebrshtaten. N\u00ebse jeni n\u00eb IPv4, ruani UInt32*. N\u00ebse jeni n\u00eb IPv6, at\u00ebher\u00eb FixedString(16), sepse adresa IPv6 \u00ebsht\u00eb 128 bit, pra ruani drejtp\u00ebrdrejt n\u00eb format bina. <\/p>\n<p><\/p>\n<p>\u00c7far\u00eb t\u00eb b\u00ebni n\u00ebse ndonj\u00ebher\u00eb keni adresa IPv4 dhe ndonj\u00ebher\u00eb IPv6? Po, mund t\u00eb ruani t\u00eb dyja. Nj\u00eb kolumn\u00eb p\u00ebr IPv4 dhe nj\u00eb tjet\u00ebr p\u00ebr IPv6. Sigurisht, ka mund\u00ebsin\u00eb p\u00ebr t\u00eb shfaqur IPv4 n\u00eb IPv6. Kjo gjithashtu do t\u00eb funksionoj\u00eb, por n\u00ebse ju nevojitet shpesh adresa IPv4 n\u00eb k\u00ebrkesa, do t\u00eb ishte mir\u00eb ta vendosni n\u00eb nj\u00eb kolumn\u00eb t\u00eb ve\u00e7ant\u00eb. <\/p>\n<p><\/p>\n<p>* tani n\u00eb ClickHouse ekzistojn\u00eb lloje t\u00eb ve\u00e7anta t\u00eb t\u00eb dh\u00ebnave IPv4, IPv6, t\u00eb cilat ruajn\u00eb t\u00eb dh\u00ebnat po aq efektivisht sa numrat, por i paraqesin ato po aq leht\u00ebsisht sa zinxhir\u00eb.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/d7d6342ebd6b8b4f3526b3cd8a06cf26.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>\u00cbsht\u00eb gjithashtu e r\u00ebnd\u00ebsishme t\u00eb theksohet se duhet t\u00eb paraprocesoni t\u00eb dh\u00ebnat paraprakisht. P\u00ebr shembull, n\u00ebse ju vijn\u00eb disa log-e t\u00eb pap\u00ebrpunuara. Dhe, ndoshta, nuk duhet t'i fusni menj\u00ebher\u00eb n\u00eb ClickHouse, megjith\u00ebse \u00ebsht\u00eb shum\u00eb t\u00ebrheq\u00ebse t\u00eb mos b\u00ebni asgj\u00eb dhe gjith\u00e7ka t\u00eb funksionoj\u00eb. Por duhet gjithsesi t\u00eb kryeni ato llogaritje q\u00eb \u00ebsht\u00eb e mundur.<\/p>\n<p><\/p>\n<p>P\u00ebr shembull, versioni i shfletuesit. N\u00eb nj\u00eb departament fqinj, p\u00ebr t\u00eb cilin nuk dua t\u00eb tregoj me gishta, atje versioni i shfletuesit ruhet k\u00ebshtu, dometh\u00ebn\u00eb si nj\u00eb varg: 12.3. Dhe pastaj, p\u00ebr t\u00eb b\u00ebr\u00eb nj\u00eb raport, ata e marrin k\u00ebt\u00eb varg dhe e ndajn\u00eb n\u00eb nj\u00eb masiv, dhe pastaj n\u00eb elementin e par\u00eb t\u00eb masivit. Natyrisht, gjith\u00e7ka ngadal\u00ebsohet. E pyeta se pse e b\u00ebjn\u00eb k\u00ebshtu. Ata m\u00eb than\u00eb se nuk e p\u00eblqejn\u00eb optimizimin e parakohsh\u00ebm. Dhe un\u00eb nuk e p\u00eblqej parakohshm\u00ebrin\u00eb e pesimizmit.<\/p>\n<p><\/p>\n<p>Prandaj, n\u00eb k\u00ebt\u00eb rast, do t\u00eb ishte m\u00eb e sakt\u00eb t\u00eb ndaheshin n\u00eb 4 kolona. Mos kini frik\u00eb k\u00ebtu, sepse kjo \u00ebsht\u00eb ClickHouse. ClickHouse \u00ebsht\u00eb nj\u00eb baz\u00eb t\u00eb dh\u00ebnash kolonore. Dhe sa m\u00eb tep\u00ebr kolona t\u00eb vogla e t\u00eb sakta, aq m\u00eb mir\u00eb. N\u00ebse do t\u00eb keni 5 BrowserVersion, b\u00ebni 5 kolona. Kjo \u00ebsht\u00eb normale. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/f9a67221bf8508933d7bd336b16f9e04.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Tani le t\u00eb shqyrtojm\u00eb se \u00e7far\u00eb t\u00eb b\u00ebni n\u00ebse keni shum\u00eb vargje shum\u00eb t\u00eb gjata, shum\u00eb masa t\u00eb gjata. Nuk \u00ebsht\u00eb e nevojshme t'i ruani ato n\u00eb ClickHouse fare. N\u00eb vend t\u00eb k\u00ebsaj, mund t\u00eb ruani vet\u00ebm disa identifikues n\u00eb ClickHouse. Dhe k\u00ebto vargje t\u00eb gjata vendosini n\u00eb nj\u00eb sistem tjet\u00ebr. <\/p>\n<p><\/p>\n<p>P\u00ebr shembull, n\u00eb nj\u00eb nga sh\u00ebrbimet tona analitike ka disa parametra ngjarjesh. Dhe n\u00ebse n\u00eb ngjarje vijn\u00eb shum\u00eb parametra, ne thjesht ruajm\u00eb 512-t\u00eb e para q\u00eb na bien p\u00ebrpara. Sepse 512 nuk \u00ebsht\u00eb shum\u00eb. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/755271e8788ba4bd638f493bf320c820.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Dhe n\u00ebse nuk mund t\u00eb p\u00ebrcaktoni llojet tuaja t\u00eb t\u00eb dh\u00ebnave, at\u00ebher\u00eb mund ta regjistroni gjithashtu t\u00eb dh\u00ebnat n\u00eb ClickHouse, por n\u00eb nj\u00eb tav\u00eb t\u00eb p\u00ebrkohshme lloj Log, e specializuar p\u00ebr t\u00eb dh\u00ebna t\u00eb p\u00ebrkohshme. Pas k\u00ebsaj, mund t\u00eb analizoni se \u00e7far\u00eb shp\u00ebrndarje vlerash keni aty, se \u00e7far\u00eb ka dhe t\u00eb formoni llojet e sakta.<\/p>\n<p><\/p>\n<p>* tani n\u00eb ClickHouse ka nj\u00eb lloj t\u00eb dh\u00ebnash <noindex><a rel=\"nofollow\" href=\"https:\/\/clickhouse.tech\/docs\/en\/sql-reference\/data-types\/lowcardinality\/\">LowCardinality<\/a><\/noindex> i cili lejon ruajtjen e efektshme t\u00eb vargjeve me m\u00eb pak shpenzime pune.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/7e9e53d86a4189783c28ce56a669198c.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Tani le t\u00eb shqyrtojm\u00eb edhe nj\u00eb rast tjet\u00ebr interesant. Ndodhin ndonj\u00ebher\u00eb q\u00eb gj\u00ebrat t\u00eb funksionojn\u00eb si\u00e7 s'kan\u00eb, un\u00eb hyj brenda dhe shoh k\u00ebshtu. Dhe menj\u00ebher\u00eb paraqitet idea se kjo \u00ebsht\u00eb b\u00ebr\u00eb nga ndonj\u00eb admin shum\u00eb me p\u00ebrvoj\u00eb dhe inteligjent, i cili ka nj\u00eb p\u00ebrvoj\u00eb t\u00eb madhe n\u00eb konfigurimin e MySQL versionit 3.23. <\/p>\n<p><\/p>\n<p>K\u00ebtu ne shohim nj\u00eb mij\u00eb tabela, n\u00eb secil\u00ebn nga t\u00eb cilat \u00ebsht\u00eb regjistruar mbetja nga ndarja e di\u00e7kaje t\u00eb paqart\u00eb me nj\u00eb mij\u00eb. <\/p>\n<p><\/p>\n<p>N\u00eb parim, un\u00eb respektoj p\u00ebrvoj\u00ebn e t\u00eb tjer\u00ebve, duke kuptuar gjithashtu se \u00e7far\u00eb vuajtjesh mund t\u00eb ket\u00eb sjell\u00eb kjo p\u00ebrvoj\u00eb. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/3a5e21ef2076966fcc627670485c5b3a.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Dhe arsyet jan\u00eb m\u00eb shum\u00eb m\u00eb pak t\u00eb qarta. K\u00ebto jan\u00eb stereotype t\u00eb vjetra q\u00eb mund t\u00eb jen\u00eb grumbulluar gjat\u00eb pun\u00ebs me sisteme t\u00eb tjera. P\u00ebr shembull, n\u00eb tabelat MyISAM nuk ka \u00e7el\u00ebs primar klaster. Dhe ky lloj ndarjeje t\u00eb dh\u00ebnash mund t\u00eb jet\u00eb nj\u00eb p\u00ebrpjekje desperate p\u00ebr t\u00eb marr\u00eb funksionalitetin e njejt\u00eb. <\/p>\n<p><\/p>\n<p>Arsye tjet\u00ebr \u00ebsht\u00eb se operacionet si alter mbi tabela t\u00eb m\u00ebdha jan\u00eb t\u00eb v\u00ebshtira. T\u00eb gjitha do t\u00eb bllokohen. Megjithat\u00eb, n\u00eb versionet moderne t\u00eb MySQL, kjo problematik\u00eb nuk \u00ebsht\u00eb m\u00eb aq e r\u00ebnd\u00ebsishme. <\/p>\n<p><\/p>\n<p>Ose, p\u00ebr shembull, mikroshardimi, por p\u00ebr k\u00ebt\u00eb m\u00eb von\u00eb.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/321b1d9caa8db7aca5b96e08c61d05b5.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>N\u00eb ClickHouse nuk \u00ebsht\u00eb e nevojshme t\u00eb b\u00ebni k\u00ebshtu, sepse, n\u00eb radh\u00eb t\u00eb par\u00eb, \u00e7el\u00ebsi primar \u00ebsht\u00eb klaster, t\u00eb dh\u00ebnat jan\u00eb t\u00eb renditura sipas \u00e7el\u00ebsit primar. <\/p>\n<p><\/p>\n<p>Dhe ndonj\u00ebher\u00eb m\u00eb pyesin: \u00abSi ndryshon performanca e k\u00ebrkesave n\u00eb interval n\u00eb ClickHouse nga madh\u00ebsia e tabel\u00ebs?\u00bb. Un\u00eb them se nuk ndryshon fare. P\u00ebr shembull, n\u00ebse keni nj\u00eb tabel\u00eb me nj\u00eb miliard rreshta dhe lexoni nj\u00eb interval nj\u00eb milion rresh, gjith\u00e7ka \u00ebsht\u00eb n\u00eb rregull. N\u00ebse tabela ka nj\u00eb trilion rreshta dhe ju lexoni nj\u00eb milion rreshta, at\u00ebher\u00eb do t\u00eb jet\u00eb pothuajse e nj\u00ebjt\u00eb. <\/p>\n<p><\/p>\n<p>Dhe, n\u00eb radh\u00eb t\u00eb dyt\u00eb, gj\u00ebra si partit\u00eb manuale nuk k\u00ebrkohen. N\u00ebse hyni dhe shikoni \u00e7far\u00eb \u00ebsht\u00eb n\u00eb sistemin e skedar\u00ebve, do t\u00eb shihni se tabela \u00ebsht\u00eb nj\u00eb gj\u00eb mjaft e r\u00ebnd\u00ebsishme. Dhe brenda saj ka di\u00e7ka si partit\u00eb. Pra, ClickHouse e b\u00ebn gjith\u00e7ka p\u00ebr ju dhe nuk keni nevoj\u00eb t\u00eb vuani.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/652f3742ad3a44f172d0d7302abd09f6.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Alter n\u00eb ClickHouse \u00ebsht\u00eb falas, n\u00ebse alteroni shtimin\/zhbjerjen e kolonave. <\/p>\n<p><\/p>\n<p>Dhe nuk ia vlen t\u00eb krijoni tabela t\u00eb vogla, sepse n\u00ebse keni 10 rreshta ose 10,000 rreshta n\u00eb tabel\u00eb, kjo \u00ebsht\u00eb krejt\u00ebsisht e par\u00ebnd\u00ebsishme. ClickHouse \u00ebsht\u00eb nj\u00eb sistem q\u00eb optimizon throughput, jo latency, prandaj nuk ka kuptim t\u00eb p\u00ebrpunoni 10 rreshta. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/144144b4740e9c5daec80c8dadc445f9.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>\u00cbsht\u00eb e drejta t\u00eb p\u00ebrdorni nj\u00eb tabel\u00eb t\u00eb madhe. Largoni stereotype t\u00eb vjetra, gjith\u00e7ka do t\u00eb shkoj\u00eb mir\u00eb. <\/p>\n<p><\/p>\n<p>Si bonus, n\u00eb versionin ton\u00eb t\u00eb fundit kemi shtuar mund\u00ebsin\u00eb p\u00ebr t\u00eb krijuar nj\u00eb \u00e7el\u00ebs t\u00eb rast\u00ebsish\u00ebm t\u00eb ndarjeve p\u00ebr t\u00eb kryer operacione t\u00eb ndryshme mir\u00ebmbajtjeje mbi ndarjet e ve\u00e7anta. <\/p>\n<p><\/p>\n<p>P\u00ebr shembull, ju nevojiten shum\u00eb tabela t\u00eb vogla, p\u00ebr shembull, kur ka nevoj\u00eb p\u00ebr trajtimin e disa t\u00eb dh\u00ebnave nd\u00ebrmjet\u00ebse, ju vijn\u00eb cop\u00ebza dhe ju duhet t\u00eb b\u00ebni transformime mbi to p\u00ebrpara se t'i shkruani n\u00eb tabel\u00ebn p\u00ebrfundimtare. P\u00ebr k\u00ebt\u00eb rast ka nj\u00eb motor t\u00eb shk\u00eblqyer p\u00ebr tabela \u2013 StripeLog. \u00cbsht\u00eb di\u00e7ka si TinyLog, vet\u00ebm m\u00eb e mir\u00eb. <\/p>\n<p><\/p>\n<p>* tani n\u00eb ClickHouse ka edhe <noindex><a rel=\"nofollow\" href=\"https:\/\/clickhouse.tech\/docs\/en\/sql-reference\/table-functions\/input\/\">funksioni tabelar input<\/a><\/noindex>.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/d62f0b3d29876c29c6c0ba236b200186.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Nj\u00eb tjet\u00ebr antipattern \u2013 \u00ebsht\u00eb mikroshardimi. P\u00ebr shembull, ju duhet t\u00eb shardoni t\u00eb dh\u00ebnat dhe keni 5 server\u00eb, kurse nes\u00ebr do t\u00eb keni 6 server\u00eb. Dhe ju mendoni, si ta rishp\u00ebrndani k\u00ebto t\u00eb dh\u00ebna. Dhe n\u00eb vend t\u00eb k\u00ebsaj, ju i ndani jo n\u00eb 5 sharda, por n\u00eb 1,000 sharda. Dhe m\u00eb pas ju tregoni \u00e7do nj\u00eb nga k\u00ebto mikrosharda n\u00eb nj\u00eb server t\u00eb ve\u00e7ant\u00eb. Dhe ju do t\u00eb p\u00ebrfundoni, p\u00ebr shembull, me 200 ClickHouse n\u00eb nj\u00eb server, p\u00ebr shembull. Instance t\u00eb ve\u00e7anta n\u00eb porte t\u00eb ve\u00e7anta ose baza t\u00eb ve\u00e7anta t\u00eb t\u00eb dh\u00ebnave. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/3e550ef7e338b84e4394eb9fa4538769.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Por n\u00eb ClickHouse kjo nuk \u00ebsht\u00eb shum\u00eb mir\u00eb. Sepse edhe nj\u00eb instance ClickHouse p\u00ebrpiqet t\u00eb p\u00ebrdor\u00eb t\u00eb gjitha burimet e disponueshme t\u00eb serverit p\u00ebr t\u00eb trajtuar nj\u00eb k\u00ebrkes\u00eb t\u00eb vetme. Q\u00eb \u00ebsht\u00eb, keni nj\u00eb server ndonj\u00ebher\u00eb dhe atje, p\u00ebr shembull, 56 b\u00ebrtham\u00eb procesori. Ju e ekzekutoni nj\u00eb k\u00ebrkes\u00eb q\u00eb zgjat nj\u00eb sekond\u00eb, dhe ajo do t\u00eb p\u00ebrdor\u00eb 56 b\u00ebrthama. Por n\u00ebse keni vendosur 200 ClickHouse n\u00eb nj\u00eb server, at\u00ebher\u00eb ndodhet q\u00eb do t\u00eb startojn\u00eb 10,000 thread-e. N\u00eb p\u00ebrgjith\u00ebsi, gjith\u00e7ka do t\u00eb shkoj\u00eb shum\u00eb keq.<\/p>\n<p><\/p>\n<p>Arsyet e tjera jan\u00eb se shp\u00ebrndarja e pun\u00ebs midis k\u00ebtyre instanceve do t\u00eb jet\u00eb e pat\u00ebbarabart\u00eb. Disa p\u00ebrfundojn\u00eb m\u00eb her\u00ebt, disa m\u00eb von\u00eb. N\u00ebse gjith\u00eb kjo do t\u00eb ndodhte n\u00eb nj\u00eb instance, ClickHouse do ta merrte vet\u00eb p\u00ebrsip\u00ebr si t\u00eb shp\u00ebrndante t\u00eb dh\u00ebnat n\u00eb m\u00ebnyr\u00eb t\u00eb sakt\u00eb n\u00ebp\u00ebr thread-e. <\/p>\n<p><\/p>\n<p>Dhe nj\u00eb arsye tjet\u00ebr \u00ebsht\u00eb se do t\u00eb keni nd\u00ebrveprim nd\u00ebrprocesor p\u00ebrmes TCP. T\u00eb dh\u00ebnat do t\u00eb duhet t\u00eb serializohen, deserializohen dhe kjo \u00ebsht\u00eb nj\u00eb sasi e madhe mikroshardesh. Do t\u00eb funksionoj\u00eb thjesht n\u00eb m\u00ebnyr\u00eb joefikase.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/18cacc17d22ae1cb2977c6be74769896.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Nj\u00eb tjet\u00ebr antipattern, ndon\u00ebse \u00ebsht\u00eb e v\u00ebshtir\u00eb ta quash t\u00eb till\u00eb. Kjo \u00ebsht\u00eb nj\u00eb sasi e madhe paragrege.<\/p>\n<p><\/p>\n<p>N\u00eb p\u00ebrgjith\u00ebsi, paragrege \u2013 \u00ebsht\u00eb mir\u00eb. Keni pasur nj\u00eb miliard rreshta, e keni agreguar dhe tani keni 1,000 rreshta, dhe tani k\u00ebrkesa ekzekutohet menj\u00ebher\u00eb. Gjith\u00e7ka \u00ebsht\u00eb e shk\u00eblqyer. K\u00ebshtu mund t\u00eb b\u00ebni. Dhe p\u00ebr k\u00ebt\u00eb, madje edhe n\u00eb ClickHouse ka nj\u00eb tip t\u00eb ve\u00e7ant\u00eb tabele AggregatingMergeTree, e cila b\u00ebn agregimin inkremental nd\u00ebrsa ose b\u00ebni inserte t\u00eb dh\u00ebnash. <\/p>\n<p><\/p>\n<p>Por ndodhin raste kur ju mendoni se do t\u00eb agregojm\u00eb t\u00eb dh\u00ebnat k\u00ebshtu dhe p\u00ebrs\u00ebri do t\u00eb agregojm\u00eb t\u00eb dh\u00ebnat. Dhe n\u00eb nj\u00eb departament fqinj, nuk kam d\u00ebshir\u00eb t\u00eb flas se cili, p\u00ebrdorin tabela SummingMergeTree p\u00ebr t\u00eb b\u00ebr\u00eb p\u00ebrllogaritje sipas \u00e7el\u00ebsit primar, dhe si \u00e7el\u00ebs primar p\u00ebrdorin rreth 20 kolona t\u00eb ndryshme. Un\u00eb p\u00ebr hir t\u00eb konspiracionit kam ndryshuar emrat e disa kolonave, por p\u00ebrllogarisni se k\u00ebshtu \u00ebsht\u00eb.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/54e3a02ab4bbce7c63ded595780ae1f6.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Dhe \u00e7far\u00eb probleme shfaqen? S\u00eb pari, volumi i t\u00eb dh\u00ebnave nuk zvog\u00eblohet shum\u00eb. P\u00ebr shembull, zvog\u00eblohet n\u00eb tre her\u00eb. Tre her\u00eb \u2013 do t\u00eb ishte nj\u00eb \u00e7mim i mir\u00eb p\u00ebr t\u00eb lejuar mund\u00ebsi t\u00eb pakufizuara p\u00ebr analiza q\u00eb shfaqen, n\u00ebse t\u00eb dh\u00ebnat tuaja nuk jan\u00eb agreguar. N\u00ebse t\u00eb dh\u00ebnat jan\u00eb t\u00eb agreguara, at\u00ebher\u00eb n\u00eb vend t\u00eb analizave, do t\u00eb merrni vet\u00ebm statistika t\u00eb mjerueshme. <\/p>\n<p><\/p>\n<p>Dhe \u00e7far\u00eb ve\u00e7an\u00ebrisht nervon? Ato q\u00eb njer\u00ebzit nga departamenti fqinj vijn\u00eb dhe k\u00ebrkojn\u00eb ndonj\u00ebher\u00eb t\u00eb shtojn\u00eb nj\u00eb kolon\u00eb tjet\u00ebr n\u00eb \u00e7el\u00ebsin primar. Pra, ne kemi agreguar t\u00eb dh\u00ebnat k\u00ebshtu, por tani duam pak m\u00eb shum\u00eb. Por n\u00eb ClickHouse nuk ka ndryshim t\u00eb \u00e7el\u00ebsit primar. Pra, duhet t\u00eb shkruajm\u00eb disa skripte n\u00eb C++. Dhe un\u00eb nuk i p\u00eblqej skriptet, p\u00ebr shkak se jan\u00eb n\u00eb C++ ose jo.<\/p>\n<p><\/p>\n<p>Dhe n\u00ebse shikoni p\u00ebr \u00e7far\u00eb \u00ebsht\u00eb krijuar ClickHouse, t\u00eb dh\u00ebnat e paagreguara jan\u00eb pik\u00ebrisht skenari p\u00ebr t\u00eb cilin \u00ebsht\u00eb krijuar. N\u00ebse e p\u00ebrdorni ClickHouse p\u00ebr t\u00eb dh\u00ebnat e paagreguara, at\u00ebher\u00eb po b\u00ebni gjith\u00e7ka si\u00e7 duhet. N\u00ebse po agregoni, at\u00ebher\u00eb ndonj\u00ebher\u00eb mund t\u00eb falet. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/09fafd4c8a3ea8c22e586cb7a5cf6564.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Nj\u00eb rast tjet\u00ebr interesant \u00ebsht\u00eb ky i k\u00ebrkesave n\u00eb nj\u00eb cik\u00ebl t\u00eb pafund. Her\u00eb pas here hyj n\u00eb nj\u00eb server prodhimi dhe shoh p\u00ebr proceset. Dhe \u00e7do her\u00eb zbuloj se ndodhin gj\u00ebra t\u00eb tmerrshme. <\/p>\n<p><\/p>\n<p>P\u00ebr shembull, k\u00ebto. K\u00ebtu menj\u00ebher\u00eb \u00ebsht\u00eb e qart\u00eb se mund t\u00eb ishte b\u00ebr\u00eb gjith\u00e7ka n\u00eb nj\u00eb k\u00ebrkes\u00eb. Thjesht shkruani atje url in dhe list\u00ebn.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/ae03b659f178962f50e0640614517296.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Pse shum\u00eb nga k\u00ebto k\u00ebrkesa n\u00eb nj\u00eb cik\u00ebl t\u00eb pafund jan\u00eb t\u00eb k\u00ebqija? N\u00ebse indeksi nuk p\u00ebrdoret, do t\u00eb keni shum\u00eb kalime p\u00ebr t\u00eb nj\u00ebjtat t\u00eb dh\u00ebna. Por n\u00ebse indeksi p\u00ebrdoret, p\u00ebr shembull, keni nj\u00eb \u00e7el\u00ebs primar sipas ru dhe shkruani url = di\u00e7ka. Dhe mendoni se do t\u00eb lexohet sakt\u00ebsisht nga tabela nj\u00eb url, do t\u00eb jet\u00eb gjith\u00e7ka n\u00eb rregull. Por n\u00eb t\u00eb v\u00ebrtet\u00eb nuk \u00ebsht\u00eb. Sepse ClickHouse b\u00ebn gjith\u00e7ka n\u00eb grupe. <\/p>\n<p><\/p>\n<p>Kur ai duhet t\u00eb lexoj\u00eb nj\u00eb gam\u00eb t\u00eb caktuar t\u00eb t\u00eb dh\u00ebnave, lexon pak m\u00eb shum\u00eb, sepse indeksi n\u00eb ClickHouse \u00ebsht\u00eb i rrall\u00eb. Ky indeks nuk lejon t\u00eb gjesh nj\u00eb rresht t\u00eb ve\u00e7ant\u00eb n\u00eb tabel\u00eb, vet\u00ebm nj\u00eb gam\u00eb t\u00eb caktuar. T\u00eb dh\u00ebnat kompresohen n\u00eb blloqe. P\u00ebr t\u00eb lexuar nj\u00eb rresht, duhet t\u00eb marr\u00ebsh t\u00ebr\u00eb nj\u00eb bllok dhe ta dekompresosh. Dhe n\u00ebse b\u00ebn shum\u00eb k\u00ebrkesa, do t\u00eb kesh shum\u00eb p\u00ebrputhje t\u00eb tilla, dhe shum\u00eb pun\u00eb do t\u00eb kryhet p\u00ebrs\u00ebri e p\u00ebrs\u00ebri.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/95eef43c5b869f8b145c61ce3c616cfd.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Dhe si nj\u00eb bonus mund t\u00eb v\u00ebreni se n\u00eb ClickHouse nuk duhet t\u00eb keni frik\u00eb t\u00eb d\u00ebrgoni madje megabajt dhe madje qindra megabajt n\u00eb seksionin IN. E mbaj mend nga praktika jon\u00eb, se n\u00ebse n\u00eb MySQL d\u00ebrgojm\u00eb shum\u00eb vlera n\u00eb seksionin IN, p\u00ebr shembull, d\u00ebrgojm\u00eb aty 100 megabajt t\u00eb disa numrave, MySQL konsumon 10 gigabajt memorie dhe m\u00eb shum\u00eb nuk ndodh asgj\u00eb, gjith\u00e7ka funksionon keq. <\/p>\n<p><\/p>\n<p>Dhe e dyta \u2013 \u00ebsht\u00eb se n\u00eb ClickHouse, n\u00ebse k\u00ebrkesat tuaja p\u00ebrdorin indeksin, at\u00ebher\u00eb kjo \u00ebsht\u00eb gjithmon\u00eb jo m\u00eb ngadal\u00eb se skenimi i plot\u00eb, dometh\u00ebn\u00eb, n\u00ebse duhet t\u00eb lexosh pothuajse t\u00eb gjith\u00eb tabel\u00ebn, do t\u00eb shkoj\u00eb radhazi dhe do t\u00eb lexoj\u00eb t\u00eb gjith\u00eb tabel\u00ebn. N\u00eb p\u00ebrgjith\u00ebsi, ai do t\u00eb merret me t\u00eb vet\u00eb.<\/p>\n<p><\/p>\n<p>Por megjithat\u00eb ka disa v\u00ebshtir\u00ebsi. P\u00ebr shembull, q\u00eb IN me n\u00ebnk\u00ebrkes\u00eb nuk p\u00ebrdor indeksin. Por kjo \u00ebsht\u00eb problemi yn\u00eb dhe duhet ta rregullojm\u00eb. Nuk ka asgj\u00eb themelore k\u00ebtu. Do ta rregullojm\u00eb. <\/p>\n<p><\/p>\n<p>Dhe nj\u00eb gj\u00eb tjet\u00ebr interesante \u2013 \u00ebsht\u00eb se n\u00ebse keni nj\u00eb k\u00ebrkes\u00eb shum\u00eb t\u00eb gjat\u00eb dhe p\u00ebrpunimi i k\u00ebrkesave \u00ebsht\u00eb i shp\u00ebrndar\u00eb, at\u00ebher\u00eb kjo k\u00ebrkes\u00eb shum\u00eb e gjat\u00eb do t\u00eb d\u00ebrgohet n\u00eb \u00e7do server pa kompresim. P\u00ebr shembull, 100 megabajt dhe 500 server\u00eb. Dhe, p\u00ebr pasoj\u00eb, do t\u00eb transmetohen 50 gigabajt p\u00ebr rrjet. Do t\u00eb d\u00ebrgohet dhe pastaj gjith\u00e7ka do t\u00eb ekzekutohet me sukses.<\/p>\n<p><\/p>\n<p>* tashm\u00eb e p\u00ebrdor; gjith\u00e7ka e riparuar, si\u00e7 ishte premtuar.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/5fcde9bf04fff0645be2530daa12743f.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Dhe nj\u00eb rast mjaft i zakonsh\u00ebm \u00ebsht\u00eb n\u00ebse k\u00ebrkesat vijn\u00eb nga API. P\u00ebr shembull, ju keni krijuar nj\u00eb sh\u00ebrbim tuajin. Dhe n\u00ebse sh\u00ebrbimi juaj \u00ebsht\u00eb i nevojsh\u00ebm p\u00ebr dik\u00eb, at\u00ebher\u00eb keni hapur API dhe tashm\u00eb brenda dy dit\u00ebsh shihni se po ndodh di\u00e7ka e paqart\u00eb. T\u00eb gjitha jan\u00eb t\u00eb mbingarkuara dhe disa k\u00ebrkesa t\u00eb tmerrshme po vijn\u00eb, t\u00eb cilat kurr\u00eb nuk duhet t\u00eb ishin. <\/p>\n<p><\/p>\n<p>Dhe zgjidhja \u00ebsht\u00eb nj\u00eb. N\u00ebse keni hapur API, at\u00ebher\u00eb do t'ju duhet ta kufizoni at\u00eb. P\u00ebr shembull, t\u00eb vendosni kuota t\u00eb ndryshme. Nuk ka mund\u00ebsi t\u00eb tjera normale. N\u00ebse jo, menj\u00ebher\u00eb do t\u00eb shkruajn\u00eb nj\u00eb skript dhe do t\u00eb keni probleme. <\/p>\n<p><\/p>\n<p>N\u00eb ClickHouse ka nj\u00eb mund\u00ebsi t\u00eb ve\u00e7ant\u00eb \u2013 \u00ebsht\u00eb num\u00ebrimi i kuotave. Madje, mund t\u00eb kaloni \u00e7el\u00ebsin tuaj t\u00eb kuotave. Kjo, p\u00ebr shembull, \u00ebsht\u00eb identifikuesi i brendsh\u00ebm i p\u00ebrdoruesit. Dhe kuotat do t\u00eb llogariten n\u00eb m\u00ebnyr\u00eb t\u00eb pavarur p\u00ebr secilin prej tyre. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/532438d0f116919e62f04e32ebbd3c45.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Tani ka nj\u00eb gj\u00eb tjet\u00ebr interesante. Kjo \u00ebsht\u00eb replikimi me manual. <\/p>\n<p><\/p>\n<p>Di shum\u00eb raste kur, pavar\u00ebsisht se ClickHouse ka mb\u00ebshtetje t\u00eb integruar p\u00ebr replikimin, njer\u00ebzit replikojn\u00eb ClickHouse manualisht.<\/p>\n<p><\/p>\n<p>Cili \u00ebsht\u00eb parimi? Keni nj\u00eb pipeline p\u00ebr procesimin e t\u00eb dh\u00ebnave. Dhe ai punon n\u00eb m\u00ebnyr\u00eb t\u00eb pavarur, p\u00ebr shembull, n\u00eb qendrat e t\u00eb dh\u00ebnave t\u00eb ndryshme. Regjistroni t\u00eb dh\u00ebna t\u00eb nj\u00ebjta n\u00eb ClickHouse n\u00eb m\u00ebnyr\u00eb t\u00eb nj\u00ebjt\u00eb. Sidoqoft\u00eb, praktika tregon se t\u00eb dh\u00ebnat ende do t\u00eb shp\u00ebrndahen p\u00ebr shkak t\u00eb disa ve\u00e7orive n\u00eb kodin tuaj. Shpresoj q\u00eb jo n\u00eb kodin tuaj. <\/p>\n<p><\/p>\n<p>Dhe periodikisht do t\u00eb duhet ta sinkronizoni manualisht. P\u00ebr shembull, nj\u00eb her\u00eb n\u00eb muaj administratoret b\u00ebjn\u00eb rsync.<\/p>\n<p><\/p>\n<p>N\u00eb t\u00eb v\u00ebrtet\u00eb, \u00ebsht\u00eb shum\u00eb m\u00eb e thjesht\u00eb t\u00eb p\u00ebrdorni replikimin e integruar n\u00eb ClickHouse. Por k\u00ebtu mund t\u00eb ket\u00eb disa kund\u00ebrshtime, sepse p\u00ebr k\u00ebt\u00eb duhet t\u00eb p\u00ebrdorni ZooKeeper. Nuk do t\u00eb flas ndonj\u00eb gj\u00eb t\u00eb keqe p\u00ebr ZooKeeper, n\u00eb parim, sistemi funksionon, por ka raste kur njer\u00ebzit nuk e p\u00ebrdorin p\u00ebr shkak t\u00eb frik\u00ebs nga Java, sepse ClickHouse \u00ebsht\u00eb nj\u00eb sistem i shk\u00eblqyer i shkruar n\u00eb C++, t\u00eb cilin mund ta p\u00ebrdorni dhe gjith\u00e7ka do t\u00eb shkoj\u00eb mrekullisht. Nd\u00ebrsa ZooKeeper \u00ebsht\u00eb n\u00eb Java. Dhe ndonj\u00ebher\u00eb nuk d\u00ebshiron ta shikosh, por mund t\u00eb p\u00ebrdor\u00ebsh replikimin me manual. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/3b5b68996daef6b8920f067b189f25e0.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>ClickHouse \u00ebsht\u00eb nj\u00eb sistem praktik. Ai merr parasysh nevojat tuaja. N\u00ebse keni replikim manual, mund t\u00eb krijoni nj\u00eb tabel\u00eb t\u00eb Distribuar q\u00eb shikon replikat tuaja manuale dhe automatikisht b\u00ebn failover mes tyre. Dhe ka madje nj\u00eb opsion t\u00eb ve\u00e7ant\u00eb q\u00eb lejon shmangien e flukseve, edhe n\u00ebse replikat tuaja shp\u00ebrndahen sistematikisht.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/70f6630f8b14756163923a7db8dd7391.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>M\u00eb pas mund t\u00eb ket\u00eb probleme n\u00ebse p\u00ebrdorni mjete tabelash primitive. ClickHouse \u00ebsht\u00eb nj\u00eb nd\u00ebrtues i till\u00eb, n\u00eb t\u00eb cilin ka shum\u00eb engine tabelash t\u00eb ndryshme. P\u00ebr t\u00eb gjitha rastet serioze, si\u00e7 shkruhet n\u00eb dokumentacion, p\u00ebrdorni tabelat e familjes MergeTree. Nd\u00ebrsa t\u00eb gjitha t\u00eb tjerat \u2013 jan\u00eb k\u00ebshtu, p\u00ebr raste t\u00eb ve\u00e7anta ose p\u00ebr teste.<\/p>\n<p><\/p>\n<p>N\u00eb tabel\u00ebn MergeTree nuk \u00ebsht\u00eb e nevojshme t\u00eb keni ndonj\u00eb dat\u00eb dhe or\u00eb. Mund ta p\u00ebrdorni gjithsesi. N\u00ebse nuk ka dat\u00eb dhe or\u00eb, shkruani se default \u00ebsht\u00eb viti 2000. Kjo do t\u00eb funksionoj\u00eb dhe nuk do t\u00eb k\u00ebrkoj\u00eb burime. <\/p>\n<p><\/p>\n<p>N\u00eb versionin e ri t\u00eb serverit, ju mund t\u00eb specifikoni madje q\u00eb t\u00eb keni nj\u00eb ndarje t\u00eb personalizuar pa \u00e7el\u00ebsin e ndarjes. Kjo do t\u00eb jet\u00eb e nj\u00ebjt\u00eb. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/6d4046d11afdd1e5962d49a1612b666a.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Nga ana tjet\u00ebr, mund t\u00eb p\u00ebrdorni engine t\u00eb thjesht\u00eb tabelash. P\u00ebr shembull, ngarkoni t\u00eb dh\u00ebnat nj\u00eb her\u00eb dhe shihni, provoni dhe fshini. Mund t\u00eb p\u00ebrdorni Log.<\/p>\n<p><\/p>\n<p>Ose ruajtja e volumit t\u00eb vog\u00ebl p\u00ebr p\u00ebrpunim t\u00eb p\u00ebrkohsh\u00ebm \u2013 kjo \u00ebsht\u00eb StripeLog ose TinyLog.<\/p>\n<p><\/p>\n<p>Memory mund t\u00eb p\u00ebrdoret, n\u00ebse volumi i t\u00eb dh\u00ebnave \u00ebsht\u00eb i vog\u00ebl dhe thjesht t\u00eb provoni di\u00e7ka n\u00eb RAM. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/8a3d4e0e3a8c26dd3a2b2f64123e2bf9.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>ClickHouse nuk e p\u00eblqen shum\u00eb t\u00eb dh\u00ebnat e tepruar normalizuar. <\/p>\n<p><\/p>\n<p>Ja nj\u00eb shembull tipik. Kjo \u00ebsht\u00eb nj\u00eb sasi e madhe URL-sh. I keni vendosur ato n\u00eb nj\u00eb tabel\u00eb fqinje. Dhe pastaj vendos\u00ebt t\u00eb b\u00ebni JOIN me to, por kjo nuk do t\u00eb funksionoj\u00eb n\u00eb p\u00ebrgjith\u00ebsi, sepse ClickHouse mb\u00ebshtet vet\u00ebm Hash JOIN. N\u00ebse nuk ka mjaft RAM p\u00ebr shum\u00eb t\u00eb dh\u00ebna q\u00eb duhet t\u00eb bashkohen, at\u00ebher\u00eb JOIN nuk do t\u00eb funksionoj\u00eb. <\/p>\n<p><\/p>\n<p>N\u00ebse t\u00eb dh\u00ebnat kan\u00eb kardinalitet t\u00eb madh, at\u00ebher\u00eb mos u shqet\u00ebsoni, ruani ato n\u00eb nj\u00eb format denormalizuar, URL-t\u00eb direkt n\u00eb tabel\u00ebn kryesore. <\/p>\n<p><\/p>\n<p>* tani n\u00eb ClickHouse ka edhe merge join dhe ai funksionon n\u00eb kushte kur t\u00eb dh\u00ebnat p\u00ebr intermediary nuk p\u00ebrputhen n\u00eb RAM. Por kjo nuk \u00ebsht\u00eb efikase dhe rekomandimi mbetet n\u00eb fuqi.<\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/e8160a8db17c18d48f810092c55b094c.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Edhe disa shembuj t\u00eb tjer\u00eb, por tashm\u00eb dyshoj n\u00ebse ata jan\u00eb antipattern. <\/p>\n<p><\/p>\n<p>N\u00eb ClickHouse ka nj\u00eb disavantazh t\u00eb njohur. Ai nuk mb\u00ebshtet azhurnime. N\u00eb nj\u00eb far\u00eb kuptimi, kjo \u00ebsht\u00eb edhe e mir\u00eb. N\u00ebse keni disa t\u00eb dh\u00ebna t\u00eb r\u00ebnd\u00ebsishme, p\u00ebr shembull, llogaritarin\u00eb, askush nuk do t\u00eb mund t'i d\u00ebrgoj\u00eb, sepse nuk ka azhurnime.<\/p>\n<p><\/p>\n<p>* mb\u00ebshtetje p\u00ebr azhurnim dhe fshirje n\u00eb regjimin batch \u00ebsht\u00eb shtuar prej koh\u00ebsh.<\/p>\n<p><\/p>\n<p>Por ka disa m\u00ebnyra speciale, q\u00eb lejojn\u00eb azhurnimet si\u00e7 do t\u00eb ishin n\u00eb sfond. P\u00ebr shembull, tabelat e tipit ReplaceMergeTree. Ato b\u00ebjn\u00eb azhurnime gjat\u00eb bashkimeve n\u00eb sfond. Mund ta forconi k\u00ebt\u00eb n\u00ebp\u00ebrmjet optimize table. Por mos e b\u00ebni k\u00ebt\u00eb shum\u00eb shpesh, sepse do t\u00eb p\u00ebrb\u00ebj\u00eb rip\u00ebrpunim t\u00eb plot\u00eb t\u00eb ndarjes. <\/p>\n<p><\/p>\n<p>JOIN t\u00eb shp\u00ebrndara n\u00eb ClickHouse \u2013 gjithashtu nuk trajtohen mir\u00eb nga planifikuesi i k\u00ebrkesave. <\/p>\n<p><\/p>\n<p>E keqe, por ndonj\u00ebher\u00eb \u00ebsht\u00eb n\u00eb rregull.<\/p>\n<p><\/p>\n<p>P\u00ebrdorimi i ClickHouse vet\u00ebm p\u00ebr t\u00eb lexuar t\u00eb dh\u00ebnat p\u00ebrmes select.<\/p>\n<p><\/p>\n<p>Nuk e doja t\u00eb rekomandoj t\u00eb p\u00ebrdor\u00ebsh ClickHouse p\u00ebr llogaritje t\u00eb m\u00ebdha. Por nuk \u00ebsht\u00eb pik\u00ebrisht k\u00ebshtu, sepse ne tashm\u00eb po largohemi nga kjo rekomandim. Dhe s\u00eb fundmi, kemi shtuar mund\u00ebsin\u00eb p\u00ebr t\u00eb aplikuar modele t\u00eb m\u00ebsimit t\u00eb makineris\u00eb n\u00eb ClickHouse \u2013 Catboost. Dhe kjo m\u00eb shqet\u00ebson, sepse mendoj: \"\u00c7far\u00eb tmerri. Sa shum\u00eb cikle p\u00ebr bajt del?!\". M\u00eb vjen shum\u00eb keq t\u2019i harxhoj ciklet p\u00ebr bajta. <\/p>\n<p><\/p>\n<p><img decoding=\"async\" alt=\"P\u00ebrdorim efektiv i ClickHouse. Aleksey Milovidov (Yandex)\" src=\"\/wp-content\/uploads\/2020\/08\/f686d5d352ccb444cd06e2ae68897137.jpg\" style=\"display:block;margin: 0 auto;\" \/><\/p>\n<p><\/p>\n<p>Por mos ki frik\u00eb, instaloj ClickHouse, gjith\u00e7ka do t\u00eb shkoj\u00eb mir\u00eb. N\u00ebse ndodh ndonj\u00eb gj\u00eb, kemi nj\u00eb komunitet. Me t\u00eb v\u00ebrtet\u00eb, komuniteti \u2013 jeni ju. Dhe n\u00ebse keni ndonj\u00eb problem, mund t\u00eb hyni n\u00eb bised\u00ebn ton\u00eb, dhe shpresoj q\u00eb do t'ju ndihmojn\u00eb. <\/p>\n<p><\/p>\n<p>Pyetje<\/p>\n<p><\/p>\n<p><em>Faleminderit p\u00ebr prezantimin! Ku mund t\u00eb ankohem p\u00ebr r\u00ebnien e ClickHouse?<\/em><\/p>\n<p><\/p>\n<p>Mund t\u00eb ankohesh personalisht tek un\u00eb n\u00eb k\u00ebt\u00eb moment. <\/p>\n<p><\/p>\n<p><em>Sapo kam filluar t\u00eb p\u00ebrdor ClickHouse. Menj\u00ebher\u00eb e kam rr\u00ebzuar interfacin cli.<\/em><\/p>\n<p><\/p>\n<p>T\u00eb lumt\u00eb. <\/p>\n<p><\/p>\n<p><em>Pak m\u00eb von\u00eb e kam rr\u00ebzuar serverin me nj\u00eb selektor t\u00eb vog\u00ebl.<\/em><\/p>\n<p><\/p>\n<p>Ke talent. <\/p>\n<p><\/p>\n<p><em>Kam hapur nj\u00eb bug n\u00eb GitHub, por e kan\u00eb injoruar.<\/em> <\/p>\n<p><\/p>\n<p>T\u00eb shohim. <\/p>\n<p><\/p>\n<p><em>Aleksej m\u00eb \u00e7oi me mashtrim n\u00eb prezantim, duke m\u00eb premtuar se do t\u00eb tregonte se si e comproni t\u00eb dh\u00ebnat brenda.<\/em><\/p>\n<p><\/p>\n<p>Shum\u00eb thjesht\u00eb. <\/p>\n<p><\/p>\n<p><em>K\u00ebt\u00eb e kuptova q\u00eb dje. M\u00eb shum\u00eb konkret\u00ebsi.<\/em> <\/p>\n<p><\/p>\n<p>Nuk ka ndonj\u00eb hile t\u00eb tmerrshme. Thjesht \u00ebsht\u00eb kompresim me blloqe. N\u00eb m\u00ebnyr\u00eb default p\u00ebrdoret LZ4, mund t\u00eb aktivizosh ZSTD*. Blloqet jan\u00eb nga 64 kilobajt deri n\u00eb 1 megabajt.<\/p>\n<p><\/p>\n<p>*ka gjithashtu mb\u00ebshtetje p\u00ebr kodek\u00ebt e specializuar t\u00eb kompresimit, t\u00eb cilat mund t\u00eb p\u00ebrdoren n\u00eb zinxhir me algoritme t\u00eb tjera.<\/p>\n<p><\/p>\n<p><em>A jan\u00eb t\u00eb dh\u00ebnat n\u00eb blloqe thjesht t\u00eb pap\u00ebrpunuara?<\/em><\/p>\n<p><\/p>\n<p>Jo krejt\u00ebsisht t\u00eb pap\u00ebrpunuara. Ka lista. N\u00ebse ke nj\u00eb kolon\u00eb numerike, atje numrat jan\u00eb renditur radhazi n\u00eb nj\u00eb list\u00eb. <\/p>\n<p><\/p>\n<p><em>Kuptohet.<\/em> <\/p>\n<p><\/p>\n<p><em>Aleksej, shembulli q\u00eb ishte me uniqExact mbi IP-t\u00eb, pra, ajo q\u00eb uniqExact llogaritet m\u00eb ngadal\u00eb mbi stringje sesa mbi numra dhe k\u00ebshtu me radh\u00eb. Dhe n\u00ebse ne p\u00ebrdorim nj\u00eb nd\u00ebrrim dhe do t\u00eb kthejm\u00eb n\u00eb momentin e leximit? Pra, ju duket se that\u00eb se n\u00eb disk nuk dallon shum\u00eb. N\u00ebse ne lexojm\u00eb stringje nga disku, kthejm\u00eb, a do t\u00eb kemi m\u00eb shpejt agregat\u00ebt apo jo? Apo ndoshta gjithsesi do t\u00eb fitojm\u00eb pak k\u00ebtu? M\u00eb duket se e keni testuar k\u00ebt\u00eb, por p\u00ebr ndonj\u00eb arsye nuk e keni p\u00ebrmendur n\u00eb benchmarking.<\/em> <\/p>\n<p><\/p>\n<p>Mendoj se do t\u00eb jet\u00eb m\u00eb i ngadalsh\u00ebm se sa pa kast. N\u00eb k\u00ebt\u00eb rast, duhet t\u00eb shp\u00ebrndajm\u00eb adres\u00ebn IP nga stringu. Sigurisht, n\u00eb ClickHouse, shp\u00ebrndarja e adresave IP \u00ebsht\u00eb optimizuar gjithashtu. Ne kemi punuar shum\u00eb p\u00ebr k\u00ebt\u00eb, por atje ju keni numrat t\u00eb shkruar n\u00eb form\u00ebn e dhjet\u00eb mij\u00ebshit. Shum\u00eb e pak\u00ebndshme. Nga ana tjet\u00ebr, funksioni uniqExact do t\u00eb punoj\u00eb m\u00eb ngadal\u00eb mbi stringje jo vet\u00ebm sepse jan\u00eb stringje, por gjithashtu p\u00ebr shkak se zgjidhet nj\u00eb specializim tjet\u00ebr i algoritmit. Stringjet thjesht p\u00ebrpunohen ndryshe.<\/p>\n<p><\/p>\n<p><em>Dhe n\u00ebse marrim nj\u00eb tip m\u00eb primitiv t\u00eb t\u00eb dh\u00ebnave? P\u00ebr shembull, shkruam user id, q\u00eb kemi brenda, e shkruam si string, dhe pastaj e kastojm\u00eb, do t\u00eb jet\u00eb m\u00eb arg\u00ebtuese apo jo?<\/em><\/p>\n<p><\/p>\n<p>Kam dyshime. Mendoj se do t\u00eb jet\u00eb edhe m\u00eb trishtues, sepse t\u00eb shp\u00ebrndash numrat \u00ebsht\u00eb nj\u00eb problem serioz. M\u00eb duket se ky kolegu kishte madje nj\u00eb referat mbi tem\u00ebn se sa e v\u00ebshtir\u00eb \u00ebsht\u00eb t\u00eb shp\u00ebrndash numrat n\u00eb form\u00ebn e dhjet\u00eb mij\u00ebshit, ndoshta jo. <\/p>\n<p><\/p>\n<p><em>Alexey, faleminderit shum\u00eb p\u00ebr referatin! Dhe gjithashtu faleminderit shum\u00eb p\u00ebr ClickHouse! Kam nj\u00eb pyetje n\u00eb lidhje me planet. A ka ndonj\u00eb plan p\u00ebr nj\u00eb karakteristik\u00eb q\u00eb lejon p\u00ebrdit\u00ebsimin e fjalor\u00ebve jo plot\u00ebsisht?<\/em><\/p>\n<p><\/p>\n<p>Pra, nj\u00eb ri-ngarkim t\u00eb pjessh\u00ebm?<\/p>\n<p><\/p>\n<p><em>Po, po. Nj\u00eb mund\u00ebsi p\u00ebr t\u00eb caktuar aty nj\u00eb fush\u00eb MySQL, pra, p\u00ebr t\u00eb p\u00ebrdit\u00ebsuar pas, q\u00eb t\u00eb ngarkohet vet\u00ebm k\u00ebto t\u00eb dh\u00ebna, n\u00ebse fjalori \u00ebsht\u00eb shum\u00eb i madh.<\/em><\/p>\n<p><\/p>\n<p>Nj\u00eb karakteristik\u00eb shum\u00eb interesante. Dhe, m\u00eb duket, ndonj\u00eb njeri e kishte propozuar at\u00eb n\u00eb bised\u00ebn ton\u00eb. Ndoshta ishit ju. <\/p>\n<p><\/p>\n<p><em>Nuk mendoj se isha un\u00eb.<\/em><\/p>\n<p><\/p>\n<p>Shk\u00eblqyesh\u00ebm, tani del se jan\u00eb dy k\u00ebrkesa. Dhe mund t\u00eb filloni ngadal\u00eb. Por menj\u00ebher\u00eb d\u00ebshiroj t'ju paralajm\u00ebroj se kjo karakteristik\u00eb \u00ebsht\u00eb mjaft e thjesht\u00eb p\u00ebr t'u zbatuar. Pra, n\u00eb ide, duhet thjesht t\u00eb shkruani numrin e versionit n\u00eb tabel\u00eb dhe pastaj t\u00eb shkruani: versioni \u00ebsht\u00eb m\u00eb i vog\u00ebl se kaq. Dhe kjo do t\u00eb thot\u00eb se, shum\u00eb p\u00ebr fat, ne do t'ua propozojm\u00eb k\u00ebt\u00eb entuziast\u00ebve. A jeni entuziast? <\/p>\n<p><\/p>\n<p><em>Po, por fatkeq\u00ebsisht, jo n\u00eb C++.<\/em><\/p>\n<p><\/p>\n<p>A e din\u00eb koleg\u00ebt tuaj t\u00eb shkruajn\u00eb n\u00eb C++? <\/p>\n<p><\/p>\n<p><em>Do t\u00eb gjej dik\u00eb.<\/em><\/p>\n<p><\/p>\n<p>Shk\u00eblqyesh\u00ebm*.<\/p>\n<p><\/p>\n<p>* mund\u00ebsia u shtua dy muaj pas referatit \u2013 e zhvilloi autori i pyetjes dhe d\u00ebrgoi k\u00ebrkes\u00ebn e tij p\u00ebr t\u00ebrheqje <noindex><a rel=\"nofollow\" href=\"https:\/\/github.com\/ClickHouse\/ClickHouse\/pull\/1771\">pull request<\/a><\/noindex>.<\/p>\n<p><\/p>\n<p><em>Faleminderit!<\/em><\/p>\n<p><\/p>\n<p><em>P\u00ebrsh\u00ebndetje! Faleminderit p\u00ebr raportin! Ju p\u00ebrmend\u00ebt se ClickHouse konsumon shum\u00eb mir\u00eb t\u00eb gjitha burimet q\u00eb ka n\u00eb dispozicion. Dhe fol\u00ebsi pran\u00eb Luksoft p\u00ebrmendi zgjidhjen e tij p\u00ebr Post\u00ebn e Rusis\u00eb. Ai tha se u p\u00eblqeu shum\u00eb ClickHouse, por nuk e p\u00ebrdor\u00ebn at\u00eb kund\u00ebr konkurentit t\u00eb tyre kryesor pik\u00ebrisht sepse konsumonte gjith\u00eb procesorin. A ka ndonj\u00eb mund\u00ebsi p\u00ebr ta kufizuar ClickHouse-n q\u00eb t\u00eb mos konsumoj\u00eb gjith\u00e7ka q\u00eb i ofrohet?<\/em><\/p>\n<p><\/p>\n<p>Po, \u00ebsht\u00eb e mundur dhe shum\u00eb e leht\u00eb. N\u00ebse d\u00ebshironi q\u00eb t\u00eb konsumoni m\u00eb pak b\u00ebrthama, thjesht shkruani <code>set max_threads = 1<\/code>. Dhe gjith\u00e7ka, do t\u00eb ekzekutoj\u00eb k\u00ebrkes\u00ebn n\u00eb nj\u00eb b\u00ebrtham\u00eb. Madje mund t\u00eb tregoni p\u00ebrdoruesve t\u00eb ndrysh\u00ebm k\u00ebt\u00eb konfigurim t\u00eb ndrysh\u00ebm. Prandaj, nuk ka probleme. Dhe ju lutem, p\u00ebrcillni koleg\u00ebve nga Luksoft q\u00eb nuk \u00ebsht\u00eb mir\u00eb q\u00eb nuk gjet\u00ebn k\u00ebt\u00eb p\u00ebrb\u00ebr\u00ebs n\u00eb dokumentacion. <\/p>\n<p><\/p>\n<p><em>Aleks, p\u00ebrsh\u00ebndetje! Doja t\u00eb pyesja nj\u00eb pyetje t\u00eb till\u00eb. Nuk \u00ebsht\u00eb hera e par\u00eb q\u00eb d\u00ebgjoj se shum\u00eb fillojn\u00eb ta p\u00ebrdorin ClickHouse si depo p\u00ebr log-et. N\u00eb raportin tuaj that\u00eb se nuk duhet ta b\u00ebjm\u00eb k\u00ebt\u00eb, pra nuk \u00ebsht\u00eb mir\u00eb t\u00eb ruajm\u00eb rreshta t\u00eb gjat\u00eb. \u00c7far\u00eb mendoni p\u00ebr k\u00ebt\u00eb?<\/em><\/p>\n<p><\/p>\n<p>S\u00eb pari, log-et zakonisht nuk jan\u00eb rreshta t\u00eb gjat\u00eb. Natyrisht, ndodhin p\u00ebrjashtime. P\u00ebr shembull, ndonj\u00eb sh\u00ebrbim i shkruar n\u00eb java hedh nj\u00eb exception, dhe ai regjistrohet. Dhe k\u00ebshtu n\u00eb nj\u00eb cik\u00ebl t\u00eb pafund, dhe skadon hap\u00ebsira n\u00eb diskun e fort\u00eb. Zgjidhja \u00ebsht\u00eb shum\u00eb e thjesht\u00eb. N\u00ebse rreshtat jan\u00eb shum\u00eb t\u00eb gjat\u00eb, thjesht prisni ato. Po \u00e7far\u00eb do t\u00eb thot\u00eb t\u00eb gjat\u00eb? Dhjet\u00ebra kilobyte \u2013 \u00ebsht\u00eb e keqe*.<\/p>\n<p><\/p>\n<p>* n\u00eb versionet e reja t\u00eb ClickHouse, \u00ebsht\u00eb aktivizuar &quot;granulimi adaptiv i indeksit&quot;, i cili zgjidh kryesisht problemin e ruajtjes s\u00eb vargjeve t\u00eb gjat\u00eb.<\/p>\n<p><\/p>\n<p><em>Dhe nj\u00eb kilobyte \u2013 \u00ebsht\u00eb n\u00eb rregull?<\/em><\/p>\n<p><\/p>\n<p>N\u00eb rregull. <\/p>\n<p><\/p>\n<p><em>P\u00ebrsh\u00ebndetje! Faleminderit p\u00ebr raportin! Un\u00eb tashm\u00eb e kam pyetur p\u00ebr k\u00ebt\u00eb n\u00eb chat, por nuk e mbaj mend n\u00ebse kam marr\u00eb p\u00ebrgjigje. A \u00ebsht\u00eb planifikuar t\u00eb zgjerohet ndonj\u00ebher\u00eb seksioni WITH n\u00eb m\u00ebnyr\u00ebn e CTE?<\/em><\/p>\n<p><\/p>\n<p>Tani p\u00ebr tani jo. Seksioni WITH p\u00ebr ne \u00ebsht\u00eb paksa jo serioz. Ajo \u00ebsht\u00eb si nj\u00eb funksion i vog\u00ebl.<\/p>\n<p><\/p>\n<p><em>E kuptova. Faleminderit!<\/em><\/p>\n<p><\/p>\n<p><em>Faleminderit p\u00ebr raportin! Shum\u00eb interesante! Nj\u00eb pyetje globale. A \u00ebsht\u00eb planifikuar t\u00eb b\u00ebhet, ndoshta, nj\u00eb modifikim p\u00ebr fshirjen e t\u00eb dh\u00ebnave n\u00eb form\u00ebn e disa z\u00ebvend\u00ebsimeve?<\/em><\/p>\n<p><\/p>\n<p>Patjet\u00ebr. Kjo \u00ebsht\u00eb detyra jon\u00eb e par\u00eb n\u00eb list\u00ebn ton\u00eb. Ne tani jemi duke menduar aktivisht se si t\u00eb b\u00ebjm\u00eb gjith\u00e7ka si\u00e7 duhet. Dhe \u00ebsht\u00eb koha p\u00ebr t\u00eb filluar t\u00eb godasim tastier\u00ebn*.<\/p>\n<p><\/p>\n<p>* keni shtypur butonat n\u00eb tastier\u00eb dhe keni b\u00ebr\u00eb gjith\u00e7ka.<\/p>\n<p><\/p>\n<p><em>A do t\u00eb ket\u00eb ndikim n\u00eb performanc\u00ebn e sistemit apo jo? A do t\u00eb jet\u00eb futja po aq e shpejt\u00eb si tani?<\/em><\/p>\n<p><\/p>\n<p>Ndoshta vet\u00eb deletes, vet\u00eb updates do t\u00eb jen\u00eb shum\u00eb t\u00eb r\u00ebnda, por kjo nuk do t\u00eb ndikoje aspak n\u00eb performanc\u00ebn e selects dhe performanc\u00ebn e inserts.<\/p>\n<p><\/p>\n<p><em>Dhe nj\u00eb pyetje e vog\u00ebl. N\u00eb prezantim p\u00ebrmend\u00ebt p\u00ebr primary key. Pra, kemi partiocionim q\u00eb \u00ebsht\u00eb mujor me default, e sakt\u00eb? Dhe kur vendosim nj\u00eb interval datash q\u00eb bie brenda nj\u00eb muaji, at\u00ebher\u00eb lexojm\u00eb vet\u00ebm k\u00ebt\u00eb partiocion, e sakt\u00eb?<\/em><\/p>\n<p><\/p>\n<p>Po.<\/p>\n<p><\/p>\n<p><em>Nj\u00eb pyetje e till\u00eb. N\u00ebse nuk mund t\u00eb dallojm\u00eb ndonj\u00eb primary key, a \u00ebsht\u00eb e sakt\u00eb t\u00eb b\u00ebhet pik\u00ebrisht n\u00eb fush\u00ebn \"Data\" p\u00ebr t\u00eb pasur nj\u00eb riorganizim m\u00eb t\u00eb vog\u00ebl t\u00eb k\u00ebtyre t\u00eb dh\u00ebnave n\u00eb sfond q\u00eb t\u00eb rregullohen m\u00eb mir\u00eb? N\u00ebse nuk keni k\u00ebrkesa n\u00eb interval dhe as nuk mund t\u00eb zgjidhni ndonj\u00eb \u00e7el\u00ebs primar, a vlen t\u00eb futni dat\u00ebn n\u00eb \u00e7el\u00ebsin primar?<\/em><\/p>\n<p><\/p>\n<p>Po.<\/p>\n<p><\/p>\n<p>Ndoshta ka kuptim t\u00eb vendosni n\u00eb \u00e7el\u00ebsin primar nj\u00eb fush\u00eb p\u00ebr t\u00eb cil\u00ebn t\u00eb dh\u00ebnat do t\u00eb comprimojn\u00eb m\u00eb mir\u00eb, n\u00ebse jan\u00eb t\u00eb renditura sipas k\u00ebsaj fushe. P\u00ebr shembull, identifikuesi i p\u00ebrdoruesit. P\u00ebrdoruesi, p\u00ebr shembull, viziton t\u00eb nj\u00ebjtin website. N\u00eb k\u00ebt\u00eb rast vendosni id-n\u00eb e p\u00ebrdoruesit dhe koh\u00ebn. Dhe k\u00ebshtu t\u00eb dh\u00ebnat tuaja do t\u00eb comprimojn\u00eb m\u00eb mir\u00eb. Sa i p\u00ebrket dates, n\u00ebse me t\u00eb v\u00ebrtet\u00eb nuk keni dhe kurr\u00eb nuk keni k\u00ebrkesa n\u00eb interval p\u00ebr datat, at\u00ebher\u00eb mund t\u00eb mos e vendosni dat\u00ebn n\u00eb \u00e7el\u00ebsin primar. <\/p>\n<p><\/p>\n<p><em>Mir\u00eb, shum\u00eb faleminderit!<\/em><\/p>\n<p>Burimi: <a content=\"nofollow\" rel=\"nofollow\" href=\"https:\/\/habr.com\/ru\/post\/514840\/\">habr.com<\/a> <\/p>","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"excerpt":{"rendered":"<p>\u0422\u0430\u043a \u043a\u0430\u043a ClickHouse \u044f\u0432\u043b\u044f\u0435\u0442\u0441\u044f \u0441\u043f\u0435\u0446\u0438\u0430\u043b\u0438\u0437\u0438\u0440\u043e\u0432\u0430\u043d\u043d\u043e\u0439 \u0441\u0438\u0441\u0442\u0435\u043c\u043e\u0439, \u043f\u0440\u0438 \u0435\u0433\u043e \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u0438 \u0432\u0430\u0436\u043d\u043e \u0443\u0447\u0438\u0442\u044b\u0432\u0430\u0442\u044c \u043e\u0441\u043e\u0431\u0435\u043d\u043d\u043e\u0441\u0442\u0438 \u0435\u0433\u043e \u0430\u0440\u0445\u0438\u0442\u0435\u043a\u0442\u0443\u0440\u044b. \u0412 \u044d\u0442\u043e\u043c \u0434\u043e\u043a\u043b\u0430\u0434\u0435 \u0410\u043b\u0435\u043a\u0441\u0435\u0439 \u0440\u0430\u0441\u0441\u043a\u0430\u0436\u0435\u0442 \u043e \u043f\u0440\u0438\u043c\u0435\u0440\u0430\u0445 \u0442\u0438\u043f\u0438\u0447\u043d\u044b\u0445 \u043e\u0448\u0438\u0431\u043e\u043a \u043f\u0440\u0438 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u0438 ClickHouse, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u043c\u043e\u0433\u0443\u0442 \u043f\u0440\u0438\u0432\u0435\u0441\u0442\u0438 \u043a \u043d\u0435\u044d\u0444\u0444\u0435\u043a\u0442\u0438\u0432\u043d\u043e\u0439 \u0440\u0430\u0431\u043e\u0442\u0435. \u041d\u0430 \u043f\u0440\u0438\u043c\u0435\u0440\u0430\u0445 \u0438\u0437 \u043f\u0440\u0430\u043a\u0442\u0438\u043a\u0438 \u0431\u0443\u0434\u0435\u0442 \u043f\u043e\u043a\u0430\u0437\u0430\u043d\u043e, \u043a\u0430\u043a \u0432\u044b\u0431\u043e\u0440 \u0442\u043e\u0439 \u0438\u043b\u0438 \u0438\u043d\u043e\u0439 \u0441\u0445\u0435\u043c\u044b \u043e\u0431\u0440\u0430\u0431\u043e\u0442\u043a\u0438 \u0434\u0430\u043d\u043d\u044b\u0445 \u043c\u043e\u0436\u0435\u0442 \u0438\u0437\u043c\u0435\u043d\u0438\u0442\u044c \u043f\u0440\u043e\u0438\u0437\u0432\u043e\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0441\u0442\u044c \u043d\u0430 \u043f\u043e\u0440\u044f\u0434\u043a\u0438. \u0412\u0441\u0435\u043c \u043f\u0440\u0438\u0432\u0435\u0442! \u041c\u0435\u043d\u044f \u0437\u043e\u0432\u0443\u0442 [&hellip;]<\/p>\n","protected":false,"gt_translate_keys":[{"key":"rendered","format":"html"}]},"author":1,"featured_media":91439,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[688],"tags":[],"class_list":["post-91438","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-administrirovanie"],"aioseo_notices":[],"aioseo_head":"\n\t\t<!-- All in One SEO 5.0.2 - aioseo.com -->\n\t<meta name=\"robots\" content=\"max-image-preview:large\" \/>\n\t<meta name=\"author\" content=\"Yuri Gagarin\"\/>\n\t<link rel=\"canonical\" href=\"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/effektivnoe-ispolzovanie-clickhouse-aleksej-milovidov-yandeks\" \/>\n\t<meta name=\"generator\" content=\"All in One SEO (AIOSEO) 5.0.2\" \/>\n\t\t<meta property=\"og:locale\" content=\"sq_AL\" \/>\n\t\t<meta property=\"og:site_name\" content=\"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b\" \/>\n\t\t<meta property=\"og:type\" content=\"article\" \/>\n\t\t<meta property=\"og:title\" content=\"\ud83e\udd47\u042d\u0444\u0444\u0435\u043a\u0442\u0438\u0432\u043d\u043e\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u0435 ClickHouse. \u0410\u043b\u0435\u043a\u0441\u0435\u0439 \u041c\u0438\u043b\u043e\u0432\u0438\u0434\u043e\u0432 (\u042f\u043d\u0434\u0435\u043a\u0441) | ProHoster\" \/>\n\t\t<meta property=\"og:url\" content=\"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/effektivnoe-ispolzovanie-clickhouse-aleksej-milovidov-yandeks\" \/>\n\t\t<meta property=\"og:image\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:secure_url\" content=\"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg\" \/>\n\t\t<meta property=\"og:image:width\" content=\"350\" \/>\n\t\t<meta property=\"og:image:height\" content=\"350\" \/>\n\t\t<meta property=\"article:published_time\" content=\"2020-08-13T17:42:36+00:00\" \/>\n\t\t<meta property=\"article:modified_time\" content=\"2020-08-13T17:42:36+00:00\" \/>\n\t\t<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/prohoster\" \/>\n\t\t<!-- All in One SEO -->\n\n","aioseo_head_json":{"title":"\ud83e\udd47P\u00ebrdorimi efektiv i ClickHouse. Aleksei Milovidov (Yandex) | ProHoster","description":"","canonical_url":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/effektivnoe-ispolzovanie-clickhouse-aleksej-milovidov-yandeks","robots":"max-image-preview:large","keywords":"","webmasterTools":{"miscellaneous":""},"schema":null,"og:locale":"sq_AL","og:site_name":"ProHoster | \u041a\u0443\u043f\u0438\u0442\u044c \u043d\u0430\u0434\u0435\u0436\u043d\u044b\u0439 \u0445\u043e\u0441\u0442\u0438\u043d\u0433 \u0434\u043b\u044f \u0441\u0430\u0439\u0442\u043e\u0432 \u0441 \u0437\u0430\u0449\u0438\u0442\u043e\u0439 \u043e\u0442 DDoS, VPS VDS \u0441\u0435\u0440\u0432\u0435\u0440\u044b","og:type":"article","og:title":"\ud83e\udd47\u042d\u0444\u0444\u0435\u043a\u0442\u0438\u0432\u043d\u043e\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u0435 ClickHouse. \u0410\u043b\u0435\u043a\u0441\u0435\u0439 \u041c\u0438\u043b\u043e\u0432\u0438\u0434\u043e\u0432 (\u042f\u043d\u0434\u0435\u043a\u0441) | ProHoster","og:url":"https:\/\/prohoster.info\/sq\/blog\/administrirovanie\/effektivnoe-ispolzovanie-clickhouse-aleksej-milovidov-yandeks","og:image":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:secure_url":"https:\/\/prohoster.info\/wp-content\/uploads\/2021\/11\/logo-350.jpg","og:image:width":350,"og:image:height":350,"article:published_time":"2020-08-13T17:42:36+00:00","article:modified_time":"2020-08-13T17:42:36+00:00","article:publisher":"https:\/\/www.facebook.com\/prohoster","article:author":"https:\/\/www.facebook.com\/prohoster"},"aioseo_meta_data":{"post_id":"91438","title":null,"description":null,"keywords":null,"keyphrases":null,"primary_term":null,"canonical_url":null,"og_title":null,"og_description":null,"og_object_type":"default","og_image_type":"default","og_image_url":null,"og_image_width":null,"og_image_height":null,"og_image_custom_url":null,"og_image_custom_fields":null,"og_video":null,"og_custom_url":null,"og_article_section":null,"og_article_tags":null,"twitter_use_og":false,"twitter_card":"default","twitter_image_type":"default","twitter_image_url":null,"twitter_image_custom_url":null,"twitter_image_custom_fields":null,"twitter_title":null,"twitter_description":null,"schema":{"blockGraphs":[],"customGraphs":[],"default":{"data":{"Article":[],"Course":[],"Dataset":[],"FAQPage":[],"Movie":[],"Person":[],"Product":[],"ProductReview":[],"Car":[],"Recipe":[],"Service":[],"SoftwareApplication":[],"WebPage":[]},"graphName":"","isEnabled":true},"graphs":[]},"schema_type":null,"schema_type_options":null,"pillar_content":false,"robots_default":true,"robots_noindex":false,"robots_noarchive":false,"robots_nosnippet":false,"robots_nofollow":false,"robots_noimageindex":false,"robots_noodp":false,"robots_notranslate":false,"robots_max_snippet":null,"robots_max_videopreview":null,"robots_max_imagepreview":"large","priority":null,"frequency":null,"local_seo":null,"seo_analyzer_scan_date":null,"breadcrumb_settings":null,"limit_modified_date":false,"reviewed_by":null,"ai":null,"created":"2021-02-28 12:27:23","updated":"2022-09-28 17:25:41","focus_keyword":null,"additional_keywords":null,"truseo_locale":null},"gt_translate_keys":[{"key":"link","format":"url"}],"_links":{"self":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts\/91438","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/comments?post=91438"}],"version-history":[{"count":0,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/posts\/91438\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/media\/91439"}],"wp:attachment":[{"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/media?parent=91438"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/categories?post=91438"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/prohoster.info\/sq\/wp-json\/wp\/v2\/tags?post=91438"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}