Ke neʻe nei i ClickHouse: 3 mau makahiki ma hope

ʻEkolu mau makahiki i hala aku nei ʻo Viktor Tarnavsky lāua ʻo Alexey Milovidov mai Yandex ma ke kahua Hoʻouka kiʻekiʻe++ haʻiʻia, pehea ka maikaʻi o ClickHouse, a pehea e lohi ʻole ai. A ma ka pae aʻe aia Alexander Zaitsev с hōʻike e pili ana i ka neʻe ʻana i KaomiHouse mai kekahi DBMS analytical a me ka hopena KaomiHouse, ʻoiaʻiʻo, maikaʻi, akā ʻaʻole kūpono loa. I ka makahiki 2016 ka hui Alanui Ola, kahi i hana ai o Alexander ia manawa, e hoohuli ana i ka multi-petabyte analytical system i KaomiHouse, he "alanui pōhaku melemele" hoihoi ia i piha i nā pōʻino i ʻike ʻole ʻia - KaomiHouse ia manawa me he mala maina la.

ʻEkolu mau makahiki ma hope KaomiHouse ʻoi aku ka maikaʻi - i kēia manawa ua hoʻokumu ʻo Alexander i ka hui ʻo Altinity, ʻaʻole ia e kōkua wale i ka poʻe e neʻe i KaomiHouse nā hanana o nā papahana, akā hoʻomaikaʻi pū i ka huahana ponoʻī me nā hoa hana mai Yandex. I kēia manawa KaomiHouse ʻaʻole naʻe he hele wāwae ʻole, akā ʻaʻole ia he kahua maina.

Ua hana ʻo Alexander me nā ʻōnaehana puʻupuʻu mai 2003, e hoʻomohala ana i nā papahana nui ma MySQL, Oracle и Vertica. Ma ka hope Loaʻa kiʻekiʻe++ 2019 ʻO Alexander, kekahi o nā paionia o ka hoʻohana KaomiHouse, haʻi i ke ʻano o kēia DBMS i kēia manawa. E aʻo mākou e pili ana i nā hiʻohiʻona nui KaomiHouse: pehea ka ʻokoʻa mai nā ʻōnaehana ʻē aʻe a i nā hihia he ʻoi aku ka maikaʻi o ka hoʻohana ʻana. Ke hoʻohana nei i nā laʻana, e nānā mākou i nā hana hou a i hoʻāʻo ʻia e ka papahana no nā ʻōnaehana kūkulu hale e pili ana KaomiHouse.

Nānā hou: ka mea i hana i 3 mau makahiki i hala

ʻEkolu makahiki i hala aku nei ua hoʻoili mākou i ka hui Alanui Ola maluna o KaomiHouse mai kekahi ʻikepili analytical, a ua like ke ʻano o ka neʻe ʻana o ka ʻikepili hoʻolaha hoʻolaha:

• Iune 2016. Ma KahunaHope Ua ikea mai KaomiHouse a ua hoʻomaka kā mākou papahana;
• ʻAukake. Hōʻoiaʻiʻo o ka Manaʻo: pūnaewele hoʻolaha nui, nā mea hana a me 200-300 terabytes o ka ʻikepili;
• ʻOkakopa. ʻikepili hana mua;
• Kekemapa. ʻO ka ukana huahana piha he 10-50 biliona hanana i kēlā me kēia lā.
• Iune 2017. Ua holomua ka neʻe ʻana o nā mea hoʻohana i KaomiHouse, 2,5 petabytes o ka ʻikepili ma kahi hui o 60 mau kikowaena.

I ka wā o ka neʻe ʻana, ua ulu ka ʻike i kēlā KaomiHouse He ʻōnaehana maikaʻi e ʻoluʻolu e hana pū me, akā he papahana kūloko kēia o Yandex. No laila, aia nā nuances: E hana mua ʻo Yandex me kāna mau mea kūʻai aku i loko a laila wale nō me ke kaiāulu a me nā pono o nā mea hoʻohana o waho, a laila ʻaʻole i hiki ʻo ClickHouse i ka pae ʻoihana ma nā wahi hana he nui. ʻO ia ke kumu i hoʻokumu ai mākou iā Altinity i Malaki 2017 e hana KaomiHouse ʻoi aku ka wikiwiki a ʻoi aku ka maʻalahi ʻaʻole wale no Yandex, akā no nā mea hoʻohana ʻē aʻe. A i kēia manawa mākou:

• Hoʻomaʻamaʻa a kōkua mākou i ke kūkulu ʻana i nā haʻina ma muli o KaomiHouse i ʻole e pilikia nā mea kūʻai aku, a no laila e hana ka hopena;
• Hāʻawi mākou i ke kākoʻo 24/7 KaomiHouse- nā mea hoʻonoho;
• Hoʻomohala mākou i kā mākou mau papahana kaiaola;
• Hoʻoikaika ikaika mākou iā mākou iho KaomiHouse, e pane ana i nā noi mai nā mea hoʻohana e makemake e ʻike i kekahi mau hiʻohiʻona.

A ʻoiaʻiʻo, kōkua mākou i ka neʻe ʻana i KaomiHouse с MySQL, Vertica, kahi e'ōlelo ai, Greenplum, ʻO Redshift a me nā ʻōnaehana ʻē aʻe. Ua komo mākou i nā ʻano hana like ʻole, a ua holomua lākou a pau.

No ke aha e neʻe ai i KaomiHouse

ʻAʻole lohi! ʻO kēia ke kumu nui. KaomiHouse - ʻikepili wikiwiki loa no nā hiʻohiʻona like ʻole:

ʻO nā huaʻōlelo maʻamau mai ka poʻe i hana pū me nā kānaka no ka manawa lōʻihi KaomiHouse.

Scalability. Ma kekahi ʻikepili ʻē aʻe hiki iā ʻoe ke hoʻokō i ka hana maikaʻi ma kahi ʻāpana ʻāpana, akā KaomiHouse ʻAʻole hiki iā ʻoe ke hoʻonui wale i ka vertically, akā me ka horizontally, ma ka hoʻohui ʻana i nā kikowaena. ʻAʻole holo pono nā mea a pau e like me kā mākou makemake, akā hana. Hiki iā ʻoe ke hoʻonui i ka ʻōnaehana i ka ulu ʻana o kāu ʻoihana. He mea nui ʻaʻole mākou i kaupalena ʻia e ka hoʻonā i kēia manawa a aia mau ka hiki ke hoʻomohala.

Lawelawe. ʻAʻohe pili i hoʻokahi mea. No ka laʻana, me Amazon RedShift He paʻakikī ke neʻe i kahi. A KaomiHouse hiki iā ʻoe ke hoʻouka iā ia ma kāu kamepiula, kikowaena, e kau i ke ao, e hele i Kubernetes - ʻaʻohe palena i ka hana ʻana o ka ʻoihana. He mea maʻalahi kēia no nā mea a pau, a he pōmaikaʻi nui kēia i hiki ʻole i nā ʻikepili like ʻole ke kaena.

ʻĀkau. KaomiHouse ʻaʻole kū i hoʻokahi mea, no ka laʻana, Yandex.Metrica, akā hoʻomohala a hoʻohana ʻia i nā papahana a me nā ʻoihana like ʻole. Hiki ke hoʻonui ʻia me ka hoʻohui ʻana i nā mana hou e hoʻoponopono i nā pilikia hou. No ka laʻana, ua manaʻo ʻia ʻo ka mālama ʻana i nā lāʻau i loko o kahi waihona he ʻano maikaʻi ʻole, no laila ua hiki mai lākou Elasticsearch. Akā, mahalo i ka maʻalahi KaomiHouse, hiki iā ʻoe ke mālama i nā moʻolelo i loko, a ʻoi aku ka maikaʻi o kēia ma mua Elasticsearch - ma KaomiHouse pono keia i 10 manawa emi hao.

Manuahi Open Source. ʻAʻole pono ʻoe e uku no kekahi mea. ʻAʻohe pono e kūkākūkā i ka ʻae e hoʻokomo i ka ʻōnaehana ma kāu kamepiula a i ʻole kikowaena. ʻAʻohe uku huna. I ka manawa like, ʻaʻohe ʻenehana waihona Open Source e hiki ke hoʻokūkū i ka wikiwiki me KaomiHouse. MySQL, MariaDB, Greenplum - ua lohi lākou a pau.

Kaiāulu, kaʻa a me leʻaleʻa... Loaʻa KaomiHouse kaiaulu maikaʻi loa: hui, kamaʻilio a me Alexey Milovidov, nāna e hoʻopiʻi iā mākou āpau me kona ikaika a me ka manaʻo maikaʻi.

Ke neʻe nei i ClickHouse

E hele i KaomiHouse no kekahi kumu, pono ʻoe i ʻekolu mau mea:

• E hoʻomaopopo i nā palena KaomiHouse a me ka mea kupono ole.
• E lawe pono ʻenehana a me kona ikaika nui loa.
• Hoao. ʻOiai ka hoʻomaopopo ʻana i ka hana ʻana KaomiHouse, ʻaʻole hiki ke wānana i ka wā e ʻoi aku ai ka wikiwiki, i ka wā e lohi ai, i ka wā e ʻoi aku ai ka maikaʻi, a i ka wā e ʻoi aku ka ʻino. No laila e ho'āʻo.

pilikia neʻe

Hoʻokahi wale nō "akā": inā ʻoe e neʻe i KaomiHouse mai kekahi mea ʻē aʻe, a laila hele hewa kekahi mea. Ua maʻa mākou i kekahi mau hana a me nā mea hana i loko o kā mākou waihona punahele. No ka laʻana, kekahi e hana pū me SQE noʻonoʻo nā L-databases i kēia mau hana i koi ʻia:

• nā kālepa;
• kaohi;
• kūlike;
• nā kuhikuhi;
• HOʻOPIʻI/HOPE;
• NULL;
• milliseconds;
• nā hoʻolei ʻano ʻakomi;
• hui nui;
• nā ʻāpana ʻokoʻa;
• nā mea hana hoʻokele hui.

Pono ka hoʻolimalima ʻana, akā ʻekolu mau makahiki i hala ma KaomiHouse ʻAʻohe o kēia mau hana i loaʻa! I kēia manawa, emi iho ma mua o ka hapalua o nā mea i hoʻokō ʻole ʻia e koe: nā hana, nā kaohi, ka kūlike, nā milliseconds a me ka hoʻolei ʻana.

A ʻo ka mea nui i loko KaomiHouse ʻAʻole hana a hana ʻokoʻa paha kekahi mau hana maʻamau a me nā ala e like me kā mākou maʻamau. ʻO nā mea a pau i ʻike ʻia i loko KaomiHouse, pili me "ClickHouse ala", ʻo ia hoʻi. ʻokoʻa nā hana mai nā waihona ʻikepili ʻē aʻe. ʻo kahi laʻana:

• ʻAʻole koho ʻia nā papa kuhikuhi, akā lele ʻia.
• HOʻOPIʻI/HOPE ʻaʻole like, akā ʻaʻole like.
• Nui nā hui, akā ʻaʻohe mea hoʻolālā nīnau. ʻAʻole maopopo loa ke ʻano o ka hana ʻana i nā poʻe mai ka honua waihona.

ClickHouse Scripts

I ka makahiki 1960, he kanaka makemakika ʻAmelika no Hungarian kumu Wigner EP ua kākau i kahi ʻatikala "ʻO ka pono kūpono ʻole o ka makemakika ma ka ʻepekema kūlohelohe” (“The Incomprehensible Effectiveness of Mathematics in the Natural Sciences”) ʻo ka honua a puni mākou no kekahi kumu i wehewehe maikaʻi ʻia e nā kānāwai makemakika. He ʻepekema abstract ka makemakika, a ʻo nā kānāwai kino i hōʻike ʻia ma ke ʻano makemakika, ʻaʻole ia he mea liʻiliʻi. Wigner EP i koʻikoʻi he mea ʻē loa kēia.

Mai ko'u manao, KaomiHouse - ka like ʻole. No ka haʻi hou ʻana iā Wigner, hiki iā mākou ke ʻōlelo i kēia: he mea kupanaha ka hana hiki ʻole ke hoʻomaopopo ʻia KaomiHouse i loko o nā ʻano noiʻi noiʻi!

Eia kekahi laʻana, e lawe kākou Hale Waihona Ikepili Manawa, kahi e hoʻouka ʻia ai ka ʻikepili i loko o kahi kokoke mau. Makemake mākou e loaʻa nā noi mai ia mea me ka lohi ʻelua. E ʻoluʻolu - e hoʻohana KaomiHouse, no ka mea, ʻo kēia ke ʻano i hoʻolālā ʻia. KaomiHouse ʻO kēia ke ʻano o ka hoʻohana ʻia ʻana ʻaʻole wale ma ka pūnaewele, akā i ke kūʻai aku a me ka ʻikepili kālā, AdTech, pela no ma ʻIke hoʻopunipunin. IN Waihona ʻIkepili manawa maoli hoʻohana ʻia kahi ʻōnaehana paʻakikī e like me "star" a i ʻole "snowflake", he nui nā papa me hui (he nui i kekahi manawa), a mālama mau ʻia ka ʻikepili i kekahi mau ʻōnaehana.

E lawe kāua i kekahi hiʻohiʻona - Manawa: ka nānā ʻana i nā mea hana, nā pūnaewele, nā helu hoʻohana, ka Pūnaewele o nā Mea. Maanei mākou e hālāwai ai me nā hanana maʻalahi i kauoha ʻia i ka manawa. KaomiHouse ʻAʻole i hoʻomohala mua ʻia no kēia, akā ua hōʻike ʻo ia e hana maikaʻi, ʻo ia ke kumu e hoʻohana ai nā hui nui KaomiHouse ma ke ʻano he waihona no ka nānā ʻana i ka ʻike. E ʻimi inā kūpono KaomiHouse no ka manawa-series, ua hana mākou i kahi hōʻailona ma muli o ke ala a me nā hopena InfluxDB и TimescaleDB - kūikawā pūʻulu manawa nā waihona. Ua huli mai,ʻo ia KaomiHouse, ʻoiai me ka ʻole o ka hoʻonui ʻia ʻana no ia mau hana, lanakila ma kahi kahua ʻē:

В pūʻulu manawa Hoʻohana pinepine ʻia kahi pākaukau haiki - he mau kolamu liʻiliʻi. Hiki ke loaʻa mai ka nui o ka ʻikepili mai ka nānā ʻana—miliona o nā moʻolelo i kēlā me kēia kekona—a maʻamau lākou e hele mai i nā puʻupuʻu liʻiliʻi (maoli-manawa streaming). No laila, pono kahi palapala hoʻokomo ʻokoʻa, a ʻo nā nīnau ponoʻī iā lākou iho.

Manaʻo Log. ʻO ka hōʻiliʻili ʻana i nā lāʻau i loko o kahi waihona ʻikepili maʻamau he hewa, akā KaomiHouse hiki ke hana me kekahi mau manaʻo e like me ka mea i hōʻike ʻia ma luna. Hoʻohana nui nā ʻoihana KaomiHouse pololei no keia hana. I kēia hihia, hoʻohana mākou i kahi pākaukau ākea ākea kahi e mālama ai mākou i nā lāʻau holoʻokoʻa (no ka laʻana, ma ke ʻano JSON), a ʻokiʻoki i ʻāpana. Hoʻokomo pinepine ʻia ka ʻikepili i nā pūʻulu nui (faila), a ʻimi mākou ma kekahi kahua.

No kēlā me kēia o kēia mau hana, hoʻohana mau ʻia nā ʻikepili kūikawā. KaomiHouse hiki i kekahi ke hana i nā mea a pau a maikaʻi loa e ʻoi aku ka maikaʻi ma mua o lākou. E nānā pono kākou i kēia manawa pūʻulu manawa scenario, a pehea e " kuke ai" pono KaomiHouse no kēia hiʻohiʻona.

Manawa-Pau

I kēia manawa ʻo kēia ke ʻano nui no ka KaomiHouse i noʻonoʻo ʻia ka hopena maʻamau. Hōʻike manawa he pūʻulu hanana i hoʻonohonoho ʻia i ka manawa, e hōʻike ana i nā loli i kekahi kaʻina hana i ka manawa. No ka laʻana, ʻo ia paha ka puʻuwai i kēlā me kēia lā a i ʻole ka helu o nā kaʻina hana i loko o ka ʻōnaehana. ʻO nā mea a pau e hāʻawi i ka manawa ticks me kekahi mau ana pūʻulu manawa:

ʻO ka hapa nui o kēia mau ʻano hanana mai ka nānā ʻana. ʻAʻole hiki i kēia ke nānā wale i ka pūnaewele, akā i nā mea pono maoli: nā kaʻa, nā ʻōnaehana ʻoihana, IoT, nā hale hana a i ʻole nā ​​kaʻa kaʻa ʻole, i loko o ka pahu kahi e waiho nei ʻo Yandex KaomiHouse-server.

Eia kekahi laʻana, aia nā hui e hōʻiliʻili i nāʻikepili mai nā moku. I kēlā me kēia mau kekona, hoʻouna nā mea ʻike ma ka moku pahu i nā haneli o nā ana like ʻole. E aʻo nā ʻenekinia iā lākou, kūkulu i nā hiʻohiʻona a hoʻāʻo e hoʻomaopopo i ka maikaʻi o ka hoʻohana ʻana i ka moku, no ka mea, ʻaʻole pono ka pahu pahu pahu no hoʻokahi kekona. ʻO kēlā me kēia manawa haʻahaʻa he poho ke kālā, no laila he mea nui e wānana i ke ala i mea liʻiliʻi nā pani.

I kēia mau lā ke ulu nei nā ʻikepili kūikawā e ana pūʻulu manawa. Ma ka pūnaewele Nā DB-ʻĀina Hoʻonohonoho ʻia nā ʻikepili like ʻole, a hiki iā ʻoe ke nānā iā lākou ma ke ʻano:

ʻO ke ʻano ulu wikiwiki loa moʻo manawas. Ke ulu nei nā waihona kiʻi, akā moʻo manawas ua ulu wikiwiki i nā makahiki i hala iho nei. ʻO nā ʻelele maʻamau o kēia ʻohana o ka ʻikepili InfluxDB, IHILA, KDB, TimescaleDB (kūkulu ʻia ma luna PostgreSQL), nā hoʻonā mai Amazon. KaomiHouse hiki ke hoʻohana ʻia ma ʻaneʻi, a hoʻohana ʻia. E hāʻawi wau iā ʻoe i kekahi mau hiʻohiʻona lehulehu.

ʻO kekahi o nā paionia ka hui CloudFlare (CDN-mea hoʻolako). Nānā lākou i kā lākou CDN через KaomiHouse (pākuʻina kau-noi, HTTP-queries) me ka ukana nui - 6 miliona mau hanana i kekona. Hele nā ​​mea a pau Kafka, hele i KaomiHouse, e hāʻawi ana i ka manawa kūpono e ʻike i nā dashboard o nā hanana i ka ʻōnaehana i ka manawa maoli.

Comcast - kekahi o nā alakaʻi i ke kelepona ma ʻAmelika: Pūnaewele, kelepona kikohoʻe, kelepona. Ua hana lākou i kahi ʻōnaehana mana like CDN i loko o ke kahua Open Source ka papahana Manaʻo Kaʻahele ʻo Apache e hana me kāu ʻikepili nui. KaomiHouse hoʻohana ʻia ma ke ʻano he backend no ka analytics.

percona kūkulu ʻia i loko KaomiHouse i loko o kou PMMe mālama i ka nānā ʻana i nā ʻano like ʻole MySQL.

Nā Koina Kūikawā

Loaʻa i nā waihona manawa-series ko lākou mau koi ponoʻī.

• Hoʻokomo wikiwiki mai nā ʻelele he nui. Pono mākou e hoʻokomo i ka ʻikepili mai nā kahawai he nui loa. KaomiHouse Hana maikaʻi ʻo ia no ka mea ʻaʻole paʻa nā mea hoʻokomo āpau. Kekahi hookomo he faila hou ma ka diski, a hiki ke hoʻopaʻa ʻia nā mea hoʻokomo liʻiliʻi ma kekahi ʻano a i ʻole. IN KaomiHouse ʻOi aku ka maikaʻi o ka hoʻokomo ʻana i nā ʻikepili i nā pūʻulu nui ma mua o hoʻokahi laina i ka manawa.
• Hoʻolālā maʻalahi. ka pūʻulu manawa ʻAʻole maopopo mākou i ke ʻano o ka ʻikepili. Hiki ke kūkulu i kahi ʻōnaehana nānā no kahi noi kikoʻī, akā paʻakikī ke hoʻohana ʻia no kahi noi ʻē aʻe. Pono kēia i kahi papahana maʻalahi. KaomiHouse, hiki iā ʻoe ke hana i kēia, ʻoiai he kumu paʻa paʻa.
• Mālama maikaʻi a poina i ka ʻikepili. ʻO ka maʻamau i pūʻulu manawa ka nui o ka ʻikepili, no laila pono e mālama ʻia e like me ka hiki. No ka laʻana, ma InfluxDB ʻO ka hoʻopiʻi maikaʻi kona hiʻohiʻona nui. Akā ma waho o ka mālama ʻana, pono ʻoe e "poina" i ka ʻikepili kahiko a hana i kekahi ʻano hoʻohaʻahaʻa - ka helu ʻakomi o nā hōʻuluʻulu.
• Nā nīnau wikiwiki i ka ʻikepili i hui ʻia. I kekahi manawa he mea hoihoi ia ke nānā aku i nā minuke 5 hope loa me ka pololei o nā milliseconds, akā i ka minuke ʻikepili o kēlā me kēia mahina a i ʻole ka lua granularity ʻaʻole pono - ua lawa nā helu helu. Pono ke kākoʻo o kēia ʻano, inā ʻaʻole he noi no 3 mahina e lōʻihi loa ka manawa e hoʻopau ai i loko KaomiHouse.
• Nā noi e like me "helu hope, e like me ka». He mea maʻamau kēia no pūʻulu manawa nā nīnau: e nānā i ke ana hope a i ʻole ke kūlana o ka ʻōnaehana i kekahi manawa t. ʻAʻole kēia mau nīnau ʻoluʻolu no kahi waihona, akā pono ʻoe e hiki ke hana iā lākou.
• "Gluing" pūʻulu manawa. Hōʻike manawa he moolelo manawa. Inā ʻelua pūʻulu manawa, pono pinepine lākou e hoʻopili a hoʻopili ʻia. ʻAʻole maʻalahi ka hana ʻana i kēia ma nā ʻikepili āpau, ʻoi aku hoʻi me nā pūʻulu manawa ʻole: eia kekahi mau manawa, aia kekahi. Hiki iā ʻoe ke noʻonoʻo i ka awelika, akā hikiwawe ke loaʻa kahi lua ma laila, no laila ʻaʻole maopopo.

E ʻike kākou pehea e hoʻokō ʻia ai kēia mau koi KaomiHouse.

ʻO ke kumuhana

В KaomiHouse papahana no pūʻulu manawa hiki ke hana i nā ʻano like ʻole, ma muli o ke ʻano o ka maʻamau o ka ʻikepili. Hiki ke kūkulu i kahi ʻōnaehana ma ka ʻikepili maʻamau ke ʻike mua mākou i nā metric āpau. Eia kekahi laʻana, ua hana wau i kēia CloudFlare me ka nānā ʻana CDN he ʻōnaehana maikaʻi ʻia. Hiki iā ʻoe ke kūkulu i kahi ʻōnaehana maʻamau e nānā i nā ʻōnaehana holoʻokoʻa a me nā lawelawe like ʻole. I ka hihia o ka ʻikepili maʻamau, ʻaʻole mākou ʻike mua i ka mea a mākou e nānā nei - a ʻo kēia paha ka hihia maʻamau.

ʻIke mau. Nā kolamu. He mea maʻalahi ka papahana - nā kolamu me nā ʻano pono:

CREATE TABLE cpu (
  created_date Date DEFAULT today(),  
  created_at DateTime DEFAULT now(),  
  time String,  
  tags_id UInt32,  /* join to dim_tag */
  usage_user Float64,  
  usage_system Float64,  
  usage_idle Float64,  
  usage_nice Float64,  
  usage_iowait Float64,  
  usage_irq Float64,  
  usage_softirq Float64,  
  usage_steal Float64,  
  usage_guest Float64,  
  usage_guest_nice Float64
) ENGINE = MergeTree(created_date, (tags_id, created_at), 8192);

He papaʻaina maʻamau kēia e nānā ana i kekahi ʻano hana hoʻouka ʻōnaehana (mea hoʻohana, nenoaiu, Kūle'a, ʻoluʻolu). Maʻalahi a maʻalahi, akā ʻaʻole maʻalahi. Inā makemake mākou i kahi hoʻolālā maʻalahi, a laila hiki iā mākou ke hoʻohana i nā arrays.

ʻIke pili ʻole. Nā pūʻulu:

CREATE TABLE cpu_alc (
  created_date Date,  
  created_at DateTime,  
  time String,  
  tags_id UInt32,  
  metrics Nested(
    name LowCardinality(String),  
    value Float64
  )
) ENGINE = MergeTree(created_date, (tags_id, created_at), 8192);

SELECT max(metrics.value[indexOf(metrics.name,'usage_user')]) FROM ...

'ole Hoʻopūnana ʻia ʻelua mau papa: metrics.inoa и metrics.value. Ma ʻaneʻi hiki iā ʻoe ke mālama i nā ʻikepili kiaʻi like ʻole e like me ke ʻano o nā inoa a me kahi ʻano o nā ana no kēlā me kēia hanana. No ka hoʻonui hou ʻana, ma kahi o hoʻokahi hale, hiki iā ʻoe ke hana i kekahi. Eia kekahi laʻana, hoʻokahi no ke lana-waiwai, kekahi - no INT-ʻo ia hoʻi no ka mea INT Makemake au e mālama pono.

Akā ʻoi aku ka paʻakikī o kēlā ʻano hale. Pono ʻoe e hoʻohana i kahi kūkulu kūikawā, me ka hoʻohana ʻana i nā hana kūikawā e huki i nā waiwai o ka index mua a laila ka array:

SELECT max(metrics.value[indexOf(metrics.name,'usage_user')]) FROM ...

Akā hikiwawe loa ke hana. ʻO ke ala ʻē aʻe e mālama ai i ka ʻikepili maʻamau ma ka lālani.

ʻIke pili ʻole. Nā kaula. Ma kēia hana kuʻuna, me ka ʻole o nā arrays, mālama ʻia nā inoa a me nā waiwai i ka manawa like. Inā hele mai 5 ana mai hoʻokahi mea hana i ka manawa hoʻokahi, 000 lālani e hana ʻia i loko o ka waihona:

CREATE TABLE cpu_rlc (
  created_date Date,  
  created_at DateTime,  
  time String,  
  tags_id UInt32,  
  metric_name LowCardinality(String),  
  metric_value Float64
) ENGINE = MergeTree(created_date, (metric_name, tags_id, created_at), 8192);


SELECT 
    maxIf(metric_value, metric_name = 'usage_user'),
    ... 
FROM cpu_r
WHERE metric_name IN ('usage_user', ...)

KaomiHouse ke kū nei me kēia - loaʻa iā ia nā hoʻonui kūikawā KaomiHouse SQL. Eia kekahi laʻana maxIf - he hana kūikawā e helu ana i ka palena kiʻekiʻe e ka metric ke hoʻokō ʻia kekahi kūlana. Hiki iā ʻoe ke kākau i nā ʻōlelo like ʻole i hoʻokahi noi a helu koke i ka waiwai no kekahi mau ana.

E hoʻohālikelike kākou i ʻekolu ala:

mea kiko'ī

Maanei ua hoʻohui au i ka "Disk Data Size" no kekahi hoʻonohonoho ʻikepili hoʻāʻo. I ka hihia o nā kolamu, loaʻa iā mākou ka liʻiliʻi loa o ka nui o ka ʻikepili: ka hoʻopaʻa ʻana kiʻekiʻe, ka wikiwiki hulina kiʻekiʻe, akā uku mākou ma ka hoʻopaʻa ʻana i nā mea āpau i ka manawa hoʻokahi.

Ma keʻano o nā arrays,ʻoi aku kaʻino o nā mea a pau. Hoʻopili maikaʻi ʻia ka ʻikepili a hiki ke mālama ʻia kahi ʻano like ʻole. Akā KaomiHouse - kahi waihona columnar, a ke hoʻomaka mākou e mālama i nā mea a pau i loko o kahi ʻano, huli ia i kahi lālani, a uku mākou no ka maʻalahi me ka maikaʻi. No kēlā me kēia hana, pono ʻoe e heluhelu i ke ʻano holoʻokoʻa i ka hoʻomanaʻo, a laila e ʻimi i ka mea i makemake ʻia i loko - a inā ulu ka laha, a laila emi ka wikiwiki.

Ma kekahi o nā hui e hoʻohana nei i kēia ala (no ka laʻana, Uber), ʻoki ʻia nā ʻāpana i ʻāpana o 128 mau mea. ʻAʻole mālama ʻia ka ʻikepili mai kekahi mau kaukani metric me ka nui o 200 TB o ka ʻikepili/lā i hoʻokahi papa, akā ma 10 a i ʻole 30 arrays me ka logic mālama kūikawā.

ʻO ke ala maʻalahi loa me nā kaula. Akā ʻaʻole i hoʻopili maikaʻi ʻia ka ʻikepili, nui ka nui o ka papaʻaina, a ʻoiai ke hoʻokumu ʻia nā nīnau ma nā metric, ʻaʻole hana maikaʻi ʻo ClickHouse.

ʻO ka papahana hybrid

E manaʻo mākou ua koho mākou i kahi kaapuni array. Akā inā ʻike mākou ʻo ka hapa nui o kā mākou dashboard e hōʻike wale ana i nā mea hoʻohana a me nā ʻōnaehana ʻōnaehana, hiki iā mākou ke hoʻohui i kēia mau ana i loko o nā kolamu mai kahi ʻano ma ka pae papa ma kēia ala:

CREATE TABLE cpu_alc (
  created_date Date,  
  created_at DateTime,  
  time String,  
  tags_id UInt32,  
  metrics Nested(
    name LowCardinality(String),  
    value Float64
  ),
  usage_user Float64 
             MATERIALIZED metrics.value[indexOf(metrics.name,'usage_user')],
  usage_system Float64 
             MATERIALIZED metrics.value[indexOf(metrics.name,'usage_system')]
) ENGINE = MergeTree(created_date, (tags_id, created_at), 8192);

Ke hookomo KaomiHouse e helu ʻakomi iā lākou. Ma kēia ala hiki iā ʻoe ke hoʻohui i ka ʻoihana me ka leʻaleʻa: maʻalahi a maʻamau ka hoʻolālā, akā ua huki mākou i nā kolamu i hoʻohana pinepine ʻia. E hoʻomaopopo ʻaʻole pono kēia e hoʻololi i ka hoʻokomo a ETLe hoʻomau ana i ka hoʻokomo ʻana i nā arrays i ka papaʻaina. Ua hana wale mākou KA HANA HANA, hoʻohui i ʻelua mau mea ʻōlelo a loaʻa iā mākou kahi hoʻolālā hybrid a wikiwiki hoʻi hiki iā ʻoe ke hoʻomaka e hoʻohana koke.

Nā codec a me ka hoʻoemi

no ka mea, pūʻulu manawa He mea koʻikoʻi ka maikaʻi o kou hoʻopili ʻana i ka ʻikepili no ka mea hiki ke nui loa ka nui o ka ʻike. IN KaomiHouse Aia kahi pūʻulu o nā mea hana e hoʻokō ai i ka hopena hoʻoemi o 1:10, 1:20, a i kekahi manawa ʻoi aku. ʻO kēia ke ʻano o ka 1 TB o ka ʻikepili i wehe ʻole ʻia ma ka disk e lawe i 50-100 GB. Maikaʻi ka liʻiliʻi liʻiliʻi, hiki ke heluhelu a hoʻoponopono wikiwiki ʻia ka ʻikepili.

No ka loaʻa ʻana o kahi kiʻekiʻe o ka hoʻoemi, KaomiHouse Kākoʻo ia i nā codec penei:

Laʻana papa:

CREATE TABLE benchmark.cpu_codecs_lz4 (
    created_date Date DEFAULT today(), 
    created_at DateTime DEFAULT now() Codec(DoubleDelta, LZ4), 
    tags_id UInt32, 
    usage_user Float64 Codec(Gorilla, LZ4), 
    usage_system Float64 Codec(Gorilla, LZ4), 
    usage_idle Float64 Codec(Gorilla, LZ4), 
    usage_nice Float64 Codec(Gorilla, LZ4), 
    usage_iowait Float64 Codec(Gorilla, LZ4), 
    usage_irq Float64 Codec(Gorilla, LZ4), 
    usage_softirq Float64 Codec(Gorilla, LZ4), 
    usage_steal Float64 Codec(Gorilla, LZ4), 
    usage_guest Float64 Codec(Gorilla, LZ4), 
    usage_guest_nice Float64 Codec(Gorilla, LZ4), 
    additional_tags String DEFAULT ''
)
ENGINE = MergeTree(created_date, (tags_id, created_at), 8192);

Maanei mākou e wehewehe ai i ka codec DoubleDelta i kekahi hihia, i ka lua - ʻO Gorila, a e hoʻohui hou mākou LZ4 kaomi ʻana. ʻO ka hopena, ua hoʻemi nui ʻia ka nui o ka ʻikepili ma ka disk:

Hōʻike kēia i ka nui o ka nui o ka ʻikepili like, akā me ka hoʻohana ʻana i nā codec a me nā kaomi ʻokoʻa:

• i loko o kahi faila GZIP ma ka disk;
• i ClickHouse me ka ʻole o nā codecs, akā me ka ZSTD compression;
• i ClickHouse me nā codecs a me ka hoʻopili LZ4 a me ZSTD.

Hiki ke ʻike ʻia ʻo nā papa me nā codecs ʻoi aku ka liʻiliʻi o ka lumi.

Nui mea

ʻAʻole i emi iki ka mea nui e koho ʻano ʻikepili pololei:

Ma nā laʻana a pau ma luna nei aʻu i hoʻohana ai Lanakila64. Akā inā mākou i koho Lanakila32, a laila ʻoi aku ka maikaʻi. Ua hōʻike maikaʻi ʻia kēia e nā kāne mai Perkona ma ka ʻatikala i hoʻopili ʻia ma luna. He mea nui ka hoʻohana ʻana i ke ʻano paʻa loa i kūpono i ka hana: ʻoi aku ka liʻiliʻi no ka nui disk ma mua o ka wikiwiki o ka nīnau. KaomiHouse paʻakikī loa i kēia.

Inā hiki iā ʻoe ke hoʻohana int32 ma kahi o int64, a laila e manaʻo i ka piʻi ʻana o ka hana. Hoʻemi liʻiliʻi ka hoʻomanaʻo ʻana o ka ʻikepili, a ʻoi aku ka wikiwiki o ka hana ʻana o nā "arithmetic". KaomiHouse ma loko, he ʻōnaehana paʻa loa ia, hoʻohana nui ʻia i nā mea hiki ke hāʻawi ʻia e nā ʻōnaehana hou.

Huihui a Nānā Mea Pono

ʻO ka hui ʻana a me nā hiʻohiʻona materialized hiki iā ʻoe ke hana i nā aggregates no nā manawa like ʻole:

No ka laʻana, loaʻa paha iā ʻoe ka ʻikepili kumu ʻaʻole i hui pū ʻia, a hiki iā ʻoe ke hoʻopili i nā hiʻohiʻona materialized iā lākou me ka hōʻuluʻulu ʻakomi ma o kahi ʻenekini kūikawā. SummingMergeTree (SMT). SMT ʻO kahi hoʻonohonoho ʻikepili hōʻuluʻulu kūikawā e helu maʻalahi i nā aggregates. Hoʻokomo ʻia ka ʻikepili maka i loko o ka waihona, hoʻohui ʻia ia, a hiki ke hoʻohana koke ʻia nā dashboards ma luna.

TTL - "poina" ʻikepili kahiko

Pehea e "poina" i ka ʻikepili i pono ʻole? KaomiHouse ʻike pehea e hana ai i kēia. I ka hana ʻana i nā papa, hiki iā ʻoe ke kuhikuhi TTL nā ʻōlelo: no ka laʻana, mālama mākou i ka ʻikepili minuke no ka lā hoʻokahi, ka ʻikepili i kēlā me kēia lā no 30 mau lā, a ʻaʻole e hoʻopā i ka ʻikepili o kēlā me kēia pule a i ʻole ka mahina:

CREATE TABLE aggr_by_minute
…
TTL time + interval 1 day

CREATE TABLE aggr_by_day
…
TTL time + interval 30 day

CREATE TABLE aggr_by_week
…
/* no TTL */

ʻĀpana nui - hoʻokaʻawale i ka ʻikepili ma nā disks

Ke lawe hou aku i kēia manaʻo, hiki ke mālama ʻia ka ʻikepili i loko KaomiHouse ma na wahi like ole. Inā makemake mākou e mālama i ka ʻikepili wela no ka pule hope loa ma kahi kūloko wikiwiki loa SSD, a waiho mākou i nā ʻikepili mōʻaukala hou aku ma kahi ʻē aʻe. IN KaomiHouse hiki i kēia manawa:

Hiki iā ʻoe ke hoʻonohonoho i kahi kulekele mālama (kulekele mālama) Pela KaomiHouse e hoʻoili ʻakomi i ka ʻikepili ke hiki i kekahi mau kūlana i kahi waihona ʻē aʻe.

Akā ʻaʻole ʻo ia wale nō. Ma ke kiʻekiʻe o kahi papaʻaina kikoʻī, hiki iā ʻoe ke wehewehe i nā lula no ka manawa e hele ai ka ʻikepili i loko o kahi waihona anu. No ka laʻana, mālama ʻia ka ʻikepili ma kahi diski wikiwiki loa no 7 mau lā, a hoʻololi ʻia nā mea kahiko i kahi lohi. He maikaʻi kēia no ka mea hiki iā ʻoe ke mālama i ka ʻōnaehana i ka hana kiʻekiʻe loa, ʻoiai e kāohi ana i nā kumukūʻai a ʻaʻole hoʻopau kālā i ka ʻikepili anu:

CREATE TABLE 
... 
TTL date + INTERVAL 7 DAY TO VOLUME 'cold_volume', 
    date + INTERVAL 180 DAY DELETE

Nā manawa kūikawā KaomiHouse

Ma kahi kokoke i nā mea a pau KaomiHouse Aia nā "mea koʻikoʻi", akā ua hoʻopau ʻia lākou e ka exclusivity - kahi mea ʻaʻole i loko o nā ʻikepili ʻē aʻe. Eia kekahi laʻana, eia kekahi mau hiʻohiʻona kūikawā KaomiHouse:

• Huina. ka KaomiHouse kākoʻo maikaʻi loa no nā arrays, a me ka hiki ke hana i nā helu paʻakikī ma luna o lākou.
• Hoʻohui ʻana i nā hale ʻikepili. ʻO kēia kekahi o nā "killer hiʻohiʻona" KaomiHouse. ʻOiai ke ʻōlelo nei nā kāne mai Yandex ʻaʻole mākou makemake e hōʻuluʻulu i ka ʻikepili, ua hōʻuluʻulu ʻia nā mea a pau. KaomiHouse, no ka mea wikiwiki a maʻalahi.
• Nā Manaʻo i hoʻokumu ʻia. Me ka hui pū ʻana i nā hoʻolālā ʻikepili, nā hiʻohiʻona materialized e ʻae iā ʻoe e hana maʻalahi maoli-manawa ka hui ʻana.
• ClickHouse SQL. He hoʻonui ʻōlelo kēia SQL me kekahi mau hiʻohiʻona hou a kūʻokoʻa i loaʻa wale i loko KaomiHouse. Ma mua, ua like ia me ka hoʻonui ʻana ma kekahi ʻaoʻao, a me kahi hemahema ma kekahi. I kēia manawa kokoke i nā hemahema a pau i hoʻohālikelike ʻia SQL 92 ua wehe mākou, i kēia manawa he hoʻonui wale nō.
• ʻo Lambda–nā ʻōlelo. Aia lākou i loko o kekahi waihona?
• ML-kākoʻo. Loaʻa kēia ma nā ʻikepili like ʻole, ʻoi aku ka maikaʻi o kekahi, ʻoi aku ka maikaʻi o kekahi.
• puna hāmama. Hiki iā mākou ke hoʻonui KaomiHouse hui pu. I kēia manawa i loko KaomiHouse ma kahi o 500 mau mea kōkua, a ke ulu mau nei kēia helu.

Nā nīnau paʻakikī

В KaomiHouse nui nā ʻano like ʻole e hana ai i ka mea like. No ka laʻana, hiki iā ʻoe ke hoʻihoʻi i ka waiwai hope mai kahi pākaukau ma ʻekolu mau ala like ʻole no CPU (he hāhā nō hoʻi, akā ʻoi aku ka ʻoi aku o ka ʻokoʻa).

Hōʻike ka mea mua i ka maʻalahi o ka hana KaomiHouse nā nīnau inā makemake ʻoe e nānā i kēlā tuple i loko o ka subquery. He mea kēia aʻu i nalo maoli ai ma nā waihona ʻikepili ʻē aʻe. Inā makemake wau e hoʻohālikelike i kekahi mea me kahi subquery, a laila ma nā ʻikepili ʻē aʻe hiki ke hoʻohālikelike ʻia kahi scalar me ia, akā no kekahi mau kolamu pono wau e kākau. hui. ka KaomiHouse hiki iā ʻoe ke hoʻohana i ka tuple:

SELECT *
  FROM cpu 
 WHERE (tags_id, created_at) IN 
    (SELECT tags_id, max(created_at)
        FROM cpu 
        GROUP BY tags_id)

Hana like ka hana ʻelua akā hoʻohana i kahi hana aggregate argMax:

SELECT 
    argMax(usage_user), created_at),
    argMax(usage_system), created_at),
...
 FROM cpu 

В KaomiHouse he mau ʻumi mau hana aggregate, a inā ʻoe e hoʻohana i nā combinators, a laila e like me nā kānāwai o combinatorics e loaʻa iā ʻoe ma kahi o hoʻokahi tausani o lākou. ArgMax - kekahi o nā hana e helu ai i ka waiwai nui: hoʻihoʻi ka noi i ka waiwai hoʻohana_mea hoʻohana, kahi i hiki ai ka waiwai nui haku_i:

SELECT now() as created_at,
       cpu.*
  FROM (SELECT DISTINCT tags_id from cpu) base 
  ASOF LEFT JOIN cpu USING (tags_id, created_at)

ASOF HUI - "gluing" lālani me nā manawa like ʻole. He hiʻohiʻona kūikawā kēia no nā ʻikepili i loaʻa wale i loko kdb+. Inā ʻelua pūʻulu manawa me nā manawa like ʻole, ASOF HUI hiki iā ʻoe ke neʻe a hoʻohui iā lākou i hoʻokahi noi. No kēlā me kēia waiwai i ka pūʻulu manawa hoʻokahi, ʻike ʻia ka waiwai kokoke loa i kekahi, a ua hoʻihoʻi ʻia lākou ma ka laina like:

Hana Analytic

I ka maʻamau SQL-2003 hiki iā ʻoe ke kākau penei:

SELECT origin,
       timestamp,
       timestamp -LAG(timestamp, 1) OVER (PARTITION BY origin ORDER BY timestamp) AS duration,
       timestamp -MIN(timestamp) OVER (PARTITION BY origin ORDER BY timestamp) AS startseq_duration,
       ROW_NUMBER() OVER (PARTITION BY origin ORDER BY timestamp) AS sequence,
       COUNT() OVER (PARTITION BY origin ORDER BY timestamp) AS nb
  FROM mytable
ORDER BY origin, timestamp;

В KaomiHouse ʻAʻole hiki iā ʻoe ke hana i kēlā - ʻaʻole ia e kākoʻo i ka maʻamau SQL-2003 a ʻaʻole paha e hana. Akā, ma KaomiHouse He mea mau ke kakau penei:

Ua hoʻohiki au i nā lambdas - eia lākou!

He analogue keia o ka ninau analytical i ka ma'amau SQL-2003: helu ʻo ia i ka ʻokoʻa ma waena o lāua kau manawa, lōʻihi, helu ordinal - nā mea a pau a mākou e noʻonoʻo ai i nā hana analytical. IN KaomiHouse Helu mākou iā lākou ma o nā arrays: mua mākou e hāʻule i ka ʻikepili i kahi array, a laila hana mākou i nā mea a pau a mākou e makemake ai ma ka array, a laila hoʻonui mākou iā ia. ʻAʻole maʻalahi loa, pono ia i ke aloha i ka hoʻolālā hana ma ka liʻiliʻi, akā hiki ke maʻalahi.

Nā hiʻohiʻona kūikawā

Eia kekahi, ma KaomiHouse nui nā hana kūikawā. No ka laʻana, pehea e hoʻoholo ai i ka nui o nā kau e hana nei i ka manawa like? ʻO kahi hana nānā maʻamau ka hoʻoholo ʻana i ka ukana kiʻekiʻe me hoʻokahi noi. IN KaomiHouse Aia kekahi hana kūikawā no kēia kumu:

Ma ka laulā, he mau hana kūikawā ko ClickHouse no nā kumu he nui:

• runningDifference, runningAccumulate, hoalauna;
• sumMap(kī, waiwai);
• timeSeriesGroupSum(uid, timestamp, waiwai);
• timeSeriesGroupRateSum(uid, timestamp, waiwai);
• skewPop, skewSamp, kurtPop, kurtSamp;
• ME KA PIHA / ME NA KILI;
• SimpleLinearRegression, stochasticLinearRegression.

ʻAʻole kēia he papa inoa piha o nā hana, aia ka 500-600 i ka huina. Manaʻo: nā hana a pau i loko KaomiHouse aia ma ka papa ʻōnaehana (ʻaʻole i kākau ʻia nā mea a pau, akā hoihoi nā mea a pau):

select * from system.functions order by name

KaomiHouse mālama ia i nā ʻike he nui e pili ana iā ia iho, me nā papa lāʻau, query_log, trace log, log of operations with data blocks (part_log), log metrics, a me ka log system, ka mea maʻamau e kākau ai i ka disk. ʻO nā metric log pūʻulu manawa в KaomiHouse i ka 'oiaʻiʻo KaomiHouse: Hiki i ka waihona waihona pono'ī ke pāʻani pūʻulu manawa nā waihona, no laila "ʻai" iā ia iho.

He mea ʻokoʻa nō hoʻi kēia - ʻoiai he hana maikaʻi kā mākou no pūʻulu manawa, no ke aha e hiki ʻole ai iā mākou ke mālama i nā mea āpau a mākou e pono ai i loko o mākou iho? ʻAʻole pono mākou IHILA, mālama mākou i nā mea a pau iā mākou iho. Hoʻopili ʻia grafana a nānā mākou iā mākou iho. Eia naʻe, inā KaomiHouse hāʻule, ʻaʻole mākou e ʻike i ke kumu, no laila ʻaʻole lākou e hana i kēlā.

Hui nui a i ʻole nā ​​mea liʻiliʻi KaomiHouse

He aha ka maikaʻi - hoʻokahi hui nui a i ʻole nā ​​ClickHouse liʻiliʻi? Kūʻai kuʻuna i DWH he pūʻulu nui i hoʻokaʻawale ʻia nā kaʻapuni no kēlā me kēia noi. Ua hele mai mākou i ka luna waihona waihona - hāʻawi mai iā mākou i kahi kiʻi, a hāʻawi lākou iā mākou i hoʻokahi:

В KaomiHouse hiki iā ʻoe ke hana ʻokoʻa. Hiki iā ʻoe ke hana i kēlā me kēia noi iā ʻoe iho KaomiHouse:

ʻAʻole pono mākou i ka mea weliweli nui DWH a me nā luna hoʻomalu. Hiki iā mākou ke hāʻawi i kēlā me kēia noi iā ia iho KaomiHouse, a hiki i ka mea hoʻomohala ke hana iā ia iho, ʻoiai KaomiHouse maʻalahi loa e hoʻouka a ʻaʻole koi i ka hoʻokele paʻakikī:

Akā inā he nui kā mākou KaomiHouse, a pono ʻoe e hoʻouka pinepine, a laila makemake ʻoe e hoʻokaʻawale i kēia kaʻina hana. No kēia hiki iā mākou, no ka laʻana, hoʻohana Kubernetes и hale kaomi-mea hoʻohana. IN Kubernetes ClickHouse hiki iā ʻoe ke hoʻokomo i ka "on-click": Hiki iaʻu ke kaomi i kahi pihi, holo i ka hōʻike a mākaukau ka waihona. Hiki iaʻu ke hana koke i kiʻi, hoʻomaka i ka hoʻouka ʻana i nā metric ma laila, a i loko o 5 mau minuke ua mākaukau kahi dashboard. grafana. He maʻalahi loa!

He aha ka hopena?

A pēlā, KaomiHouse - ʻO kēia:

• Kolaha. Ua ʻike nā kānaka a pau i kēia.
• ʻAe wale nō. He mea hoʻopaʻapaʻa iki, akā ke manaʻoʻiʻo nei au he paʻakikī i ke aʻo ʻana, maʻalahi i ka hakakā. Inā maopopo iā ʻoe pehea KaomiHouse hana ia, a laila maʻalahi loa nā mea a pau.
• Kulanui. He kūpono ia no nā hiʻohiʻona like ʻole: DWH, Manawa Manawa, Waihona Log. ʻAʻole naʻe OLTP ʻikepili, no laila mai hoʻāʻo e hana i nā hoʻokomo pōkole a heluhelu ma laila.
• Pīhoihoi. ʻO ka mea hana pū paha KaomiHouse, ua ʻike i nā manawa hoihoi he nui ma ke ʻano maikaʻi a maikaʻi ʻole. Eia kekahi laʻana, ua puka mai kahi hoʻokuʻu hou, ua pau nā mea a pau i ka hana. A i ʻole ʻoe i hakakā me kahi hana no ʻelua mau lā, akā ma hope o ka nīnau ʻana i kahi nīnau ma Telegram chat, ua hoʻopau ʻia ka hana i ʻelua mau minuke. A i ʻole e like me ka ʻaha kūkā ma ka hōʻike a Lesha Milovidov, kahi kiʻi kiʻi mai KaomiHouse uhaki i ka hoolaha Hoʻouka kiʻekiʻe++. Hana ʻia kēia ʻano mea i nā manawa a pau a paʻakikī ko mākou ola. KaomiHouse nani a hoihoi!

Hiki iā ʻoe ke nānā i ka hōʻike maanei.

ʻO ka hālāwai lōʻihi o nā mea hoʻomohala o nā ʻōnaehana kiʻekiʻe ma Hoʻouka kiʻekiʻe++ e hana ʻia ma ka lā 9 a me 10 Nowemapa ma Skolkovo. ʻO ka mea hope loa, he hālāwai kūkā waho kēia (ʻoiai me ka mālama ʻana i nā wahi āpau), ʻoiai ʻaʻole hiki ke hoʻopili ʻia ka ikehu o HighLoad++ ma ka pūnaewele.

No ka ʻaha kūkā, ʻike mākou a hōʻike iā ʻoe i nā hihia e pili ana i ka hiki o ka ʻenehana: HighLoad ++, ʻo ia a ʻo ia wale nō kahi e hiki ai iā ʻoe ke aʻo i nā lā ʻelua pehea e hana ai ʻo Facebook, Yandex, VKontakte, Google a me Amazon.

Ma hope o ka mālama ʻana i kā mākou mau hālāwai me ke keakea ʻole mai ka makahiki 2007, e hui mākou i kēia makahiki no ka manawa 14. I kēia manawa, ua ulu ka ʻaha kūkā he 10 mau manawa; i ka makahiki i hala, ua hui pū ka hanana ʻoihana koʻikoʻi i 3339 mau mea komo, 165 mau mea haʻiʻōlelo, hōʻike a me nā hui, a me 16 mau mele e holo like ana.
I ka makahiki i hala aku nei he 20 kaʻa kaʻa, 5280 lita o ke kī a me ke kofe, 1650 lita o nā mea inu hua a me 10200 mau hue wai. A he 2640 kilokani ai, 16 papa a me 000 kiaha. Ma ke ala, me ke kālā i hōʻiliʻili ʻia mai ka pepa i hana hou ʻia, kanu mākou i 25 mau hua ʻoka :)

Hiki iā ʻoe ke kūʻai i nā tiketi maanei, loaʻa ka nūhou e pili ana i ka ʻaha kūkā - maanei, a kamaʻilio ma nā pūnaewele kaiapili a pau: Telegram, Facebook, Vkontakte и Twitter.

Source: www.habr.com