ʻO ke kumumanaʻo a me ka hana o ka hoʻohana ʻana i ClickHouse i nā noi maoli. ʻO Alexander Zaitsev (2018)

ʻOiai ʻo ka nui o nā ʻikepili i kēia manawa kokoke i nā wahi āpau, ʻoi aku ka nui o nā ʻikepili analytical. ʻAʻole ʻike maikaʻi ʻia lākou a ʻoi aku ka maikaʻi o ka hiki ke hoʻohana pono iā lākou. Nui ka poʻe e hoʻomau i ka "ʻai i ka cactus" me MySQL a i ʻole PostgreSQL, i hoʻolālā ʻia no nā hiʻohiʻona ʻē aʻe, e ʻeha me NoSQL, a i ʻole ka uku nui no nā hoʻonā pāʻoihana. Hoʻololi ʻo ClickHouse i nā lula o ka pāʻani a hoʻohaʻahaʻa loa i ka paepae no ke komo ʻana i ke ao o ka DBMS analytical.

Hōʻike mai BackEnd Conf 2018 a paʻi ʻia me ka ʻae o ka mea haʻiʻōlelo.

ʻO wai wau a no ke aha wau e kamaʻilio nei e pili ana i ClickHouse? He luna hoʻomohala au ma LifeStreet, nāna e hoʻohana iā ClickHouse. Eia kekahi, ʻo wau ka mea nāna i hoʻokumu iā Altinity. He hoa hana ʻo Yandex e hoʻolaha ana iā ClickHouse a kōkua iā Yandex e hoʻomaikaʻi i ka ClickHouse. Mākaukau pū e kaʻana like i ka ʻike e pili ana iā ClickHouse.

ʻAʻole wau ke kaikunāne o Petya Zaitsev. Ninau pinepine ia au no keia. ʻAʻole, ʻaʻole mākou he kaikunāne.

"Ua ʻike nā kānaka a pau" i kēlā ClickHouse:

• wikiwiki loa,
• ʻoluʻolu loa
• Hoʻohana ʻia ma Yandex.

ʻIke ʻia ka liʻiliʻi liʻiliʻi i nā ʻoihana a pehea e hoʻohana ʻia ai.

E haʻi aku wau iā ʻoe i ke kumu, kahi a pehea e hoʻohana ʻia ai ʻo ClickHouse, koe wale no Yandex.

E haʻi wau iā ʻoe pehea e hoʻopau ʻia ai nā hana kikoʻī me ke kōkua o ClickHouse i nā ʻoihana like ʻole, nā mea hana ClickHouse hiki iā ʻoe ke hoʻohana no kāu mau hana, a pehea i hoʻohana ʻia ai i nā hui like ʻole.

Ua ʻohi au i ʻekolu mau hiʻohiʻona e hōʻike ana i ka ClickHouse mai nā kihi like ʻole. Manaʻo wau he mea hoihoi ia.

ʻO ka nīnau mua: "No ke aha mākou e pono ai i ClickHouse?". Me he mea lā he nīnau maopopo loa ia, akā ʻoi aku ma mua o hoʻokahi pane.

• ʻO ka pane mua no ka hana. He wikiwiki loa ʻo ClickHouse. He wikiwiki loa ka Analytics ma ClickHouse. Hiki ke hoʻohana pinepine ʻia i kahi mea ʻē aʻe lohi a ʻino loa.
• ʻO ka pane ʻelua he kumukūʻai. A ʻo ka mea mua, ʻo ke kumukūʻai o ka scaling. No ka laʻana, ʻo Vertica kahi waihona ʻikepili maikaʻi loa. He hana maikaʻi ia inā ʻaʻole nui kāu terabytes o ka ʻikepili. Akā i ka hiki ʻana mai i nā haneli terabytes a i ʻole petabytes, hele ke kumukūʻai o kahi laikini a me ke kākoʻo i kahi nui nui. A he pipiʻi. A manuahi ʻo ClickHouse.
• ʻO ke kolu o ka pane ʻo ke kumukūʻai hana. He ʻano ʻokoʻa iki kēia. ʻO RedShift kahi analog maikaʻi loa. Ma RedShift, hiki iā ʻoe ke hoʻoholo wikiwiki loa. E hana maikaʻi ia, akā i ka manawa like, i kēlā me kēia hola, i kēlā me kēia lā, a i kēlā me kēia mahina, e uku nui ʻoe iā Amazon, no ka mea, he lawelawe nui kēia. ʻO Google BigQuery kekahi. Inā hoʻohana kekahi iā ia, a laila ʻike ʻo ia ma laila hiki iā ʻoe ke holo i kekahi mau noi a loaʻa i kahi bila no nā haneli kālā i ka manawa koke.

ʻAʻohe pilikia o ClickHouse.

Ma hea kahi i hoʻohana ʻia ai ʻo ClickHouse i kēia manawa? Ma waho aʻe o Yandex, hoʻohana ʻia ʻo ClickHouse i kahi hui o nā ʻoihana like ʻole a me nā ʻoihana.

• ʻO ka mea mua, ʻo kēia ka loiloi noi pūnaewele, ʻo ia hoʻi kahi hihia hoʻohana i hele mai mai Yandex.
• Nui nā hui AdTech e hoʻohana i ka ClickHouse.
• Nui nā ʻoihana e pono e kālailai i nā log transaction mai nā kumu like ʻole.
• Hoʻohana kekahi mau hui i ClickHouse e nānā i nā lāʻau palekana. Hoʻouka lākou iā ClickHouse, hana i nā hōʻike, a loaʻa nā hopena e pono ai lākou.
• Ke hoʻomaka nei nā ʻoihana e hoʻohana iā ia i ka loiloi kālā, ʻo ia hoʻi ke hoʻokokoke nei nā ʻoihana nui i ClickHouse.
• cloudflare. Inā hahai kekahi iā ClickHouse, a laila ua lohe paha lākou i ka inoa o kēia hui. ʻO kēia kekahi o nā mea kōkua nui mai ke kaiāulu. A loaʻa iā lākou kahi hoʻonohonoho ClickHouse koʻikoʻi. No ka laʻana, hana lākou iā Kafka Engine no ClickHouse.
• Ua hoʻomaka nā hui kelepona e hoʻohana. Hoʻohana kekahi mau hui i ClickHouse ma ke ʻano he hōʻoia i ka manaʻo a i ʻole i ka hana ʻana.
• Hoʻohana kekahi hui i ka ClickHouse e nānā i nā kaʻina hana. Hoʻāʻo lākou i nā microcircuits, kākau i kahi hui o nā ʻāpana, aia ma kahi o 2 mau hiʻohiʻona. A laila hoʻoponopono lākou inā maikaʻi a maikaʻi ʻole paha ka pāʻani.
• ʻIkepili blockchain. Aia kekahi hui Lūkini e like me Bloxy.info. He loiloi kēia o ka pūnaewele ethereum. Ua hana pū lākou i kēia ma ClickHouse.

A ʻaʻole pili ka nui. Nui nā hui e hoʻohana i hoʻokahi kikowaena liʻiliʻi. A ʻae ʻo ia iā lākou e hoʻoponopono i ko lākou mau pilikia. A ʻoi aku ka nui o nā ʻoihana e hoʻohana i nā pūʻulu nui o nā kikowaena he nui a i ʻole ka nui o nā kikowaena.

A inā ʻoe e nānā i nā moʻolelo, a laila:

• Yandex: 500+ mau kikowaena, mālama lākou i 25 biliona moʻolelo i ka lā ma laila.
• LifeStreet: 60 mau kikowaena, ma kahi o 75 biliona moʻolelo i kēlā me kēia lā. He liʻiliʻi nā kikowaena, ʻoi aku ka nui o nā moʻolelo ma mua o Yandex.
• CloudFlare: 36 mau kikowaena, mālama lākou i 200 billion mau moʻolelo i ka lā. ʻOi aku ka liʻiliʻi o nā kikowaena a mālama i nā ʻikepili hou aʻe.
• Bloomberg: 102 mau kikowaena, ma kahi o ka trillion komo i kēlā me kēia lā. Mea paʻa moʻolelo.

Ma ka ʻāina, he nui nō hoʻi kēia. Hōʻike kēia palapala ʻāina i kahi palapala wela o kahi e hoʻohana ʻia ai ʻo ClickHouse ma ka honua. ʻO Rusia, Kina, ʻAmelika ke kū aʻe nei ma ʻaneʻi. He kakaikahi na aina o Europa. A he 4 pūʻulu.

He hoʻohālikelike hoʻohālikelike kēia, ʻaʻohe pono e ʻimi i nā helu piha. He loiloi kēia o nā malihini e heluhelu ana i nā mea ʻōlelo Pelekania ma ka pūnaewele Altinity, no ka mea, ʻaʻohe mea ʻōlelo Lūkini ma laila. A ʻo Russia, Ukraine, Belarus, ʻo ia hoʻi ka ʻōlelo Lūkini o ke kaiāulu, ʻo ia ka mea hoʻohana nui loa. A laila hele mai ʻo US a me Kanada. Ke hoʻomau nui nei ʻo Kina. Aneane ʻaʻohe Kina ma laila ʻeono mahina i hala aku nei, i kēia manawa ua lanakila ʻo Kina iā ʻEulopa a hoʻomau i ka ulu. ʻAʻole mamao loa ʻo ʻEulopa kahiko, a ʻo ke alakaʻi i ka hoʻohana ʻana iā ClickHouse, ʻo ia hoʻi, ʻo Farani.

No ke aha wau e haʻi nei i kēia mau mea a pau? No ka hōʻike ʻana ua lilo ʻo ClickHouse i mea hoʻonā maʻamau no ka ʻikepili nui a ua hoʻohana ʻia ma nā wahi he nui. Inā ʻoe e hoʻohana iā ia, aia ʻoe i ke ʻano kūpono. Inā ʻaʻole ʻoe e hoʻohana nei, a laila ʻaʻole hiki iā ʻoe ke makaʻu e waiho wale ʻia ʻoe a ʻaʻohe mea e kōkua iā ʻoe, no ka mea he nui nā mea e hana nei.

He mau hiʻohiʻona kēia o ka hoʻohana maoli ʻana o ClickHouse i kekahi mau hui.

• ʻO ka laʻana mua he pūnaewele hoʻolaha: neʻe mai Vertica i ClickHouse. A ʻike wau i kekahi mau hui i hoʻololi ʻia mai Vertica a i ʻole i ke kaʻina o ka hoʻololi.
• ʻO ka lua o ka laʻana, ʻo ia ka mālama kālā ma ClickHouse. He laʻana kēia i kūkulu ʻia ma luna o nā antipatterns. ʻO nā mea a pau ʻaʻole pono e hana ʻia ma ClickHouse ma ka ʻōlelo aʻo a nā mea hoʻomohala e hana ʻia ma aneʻi. A ua hana maikaʻi ʻia e hana. A ʻoi aku ka maikaʻi ma mua o ka hoʻonā transactional maʻamau.
• ʻO ke kolu o nā hiʻohiʻona e hāʻawi ʻia i ka helu helu ma ClickHouse. Aia kekahi nīnau e pili ana i ka hiki ke hoʻohui ʻia ʻo ClickHouse i loko o ke kaiaola Hadoop. E hōʻike wau i kahi hiʻohiʻona o ka hana ʻana o kahi ʻoihana i kahi mea e like me kahi pahu hōʻemi i ka palapala ʻāina ma ClickHouse, ka mālama ʻana i ka localization data, etc., e helu i kahi hana koʻikoʻi ʻole.

• ʻO LifeStreet kahi hui Ad Tech nona nā ʻenehana āpau e hele mai me kahi pūnaewele hoʻolaha.
• Ke komo nei ʻo ia i ka hoʻolaha hoʻolaha hoʻolaha, koho papahana.
• Nui nā ʻikepili: ma kahi o 10 biliona hanana i kēlā me kēia lā. I ka manawa like, hiki ke hoʻokaʻawale ʻia nā hanana ma laila i mau sub-events.
• Nui nā mea kūʻai aku o kēia ʻikepili, a ʻaʻole kēia he poʻe wale nō, ʻoi aku ka nui - he mau algorithms like ʻole kēia e komo nei i ka bidding programmatic.

Ua hele mai ka ʻoihana i kahi ala lōʻihi a ʻoi. A kamaʻilio wau e pili ana iā HighLoad. ʻO ka mua, ua neʻe ʻo LifeStreet mai MySQL (me kahi kū pōkole ma Oracle) i Vertica. A hiki iā ʻoe ke loaʻa kahi moʻolelo e pili ana iā ia.

A maikaʻi loa nā mea a pau, akā ua maopopo koke ka ulu ʻana o ka ʻikepili a he pipiʻi ʻo Vertica. No laila, ua ʻimi ʻia nā ʻano ʻokoʻa like ʻole. Ua helu ʻia kekahi o lākou ma aneʻi. A ʻo ka mea ʻoiaʻiʻo, ua hana mākou i ka hōʻoia o ka manaʻo a i ʻole i kekahi manawa ka hoʻāʻo ʻana i ka hana kokoke i nā ʻikepili āpau i loaʻa ma ka mākeke mai ka makahiki 13 a hiki i ka makahiki 16 a ua kūpono i nā ʻano o ka hana. A ua kamaʻilio pū wau e pili ana i kekahi o lākou ma HighLoad.

ʻO ka hana ka neʻe ʻana mai Vertica ma kahi mua, no ka mea ua ulu ka ʻikepili. A ua ulu nui lākou i nā makahiki. A laila hele lākou i ka papa, akā naʻe. A me ka wānana ʻana i kēia ulu ʻana, nā koi ʻoihana no ka nui o ka ʻikepili kahi e pono ai ke hana ʻana i kekahi ʻano loiloi, ua maopopo e kūkākūkā koke ʻia nā petabytes. A ʻo ka uku ʻana i nā petabytes ua nui loa ke kumukūʻai, no laila ke ʻimi nei mākou i kahi ʻē aʻe kahi e hele ai.

Ma hea e hele ai? A no ka manawa lōʻihi ʻaʻole maopopo loa i kahi e hele ai, no ka mea ma kekahi ʻaoʻao aia nā waihona ʻikepili kalepa, ʻike lākou e hana maikaʻi. Hana like kekahi me Vertica, ʻoi aku ka maikaʻi o kekahi. Akā, he pipiʻi lākou a pau, ʻaʻohe mea ʻoi aku ka maikaʻi a ʻaʻole hiki ke loaʻa.

Ma ka ʻaoʻao ʻē aʻe, aia nā haʻina hāmama, ʻaʻole nui loa, ʻo ia hoʻi no ka analytics, hiki ke helu ʻia ma nā manamana lima. A manuahi a kūʻai ʻole paha lākou, akā lohi. A nele pinepine lākou i ka hana pono a me ka pono.

A ʻaʻohe mea e hoʻohui i ka maikaʻi i loko o nā waihona ʻikepili kalepa a me nā manuahi āpau i loaʻa i ka open source.

ʻAʻohe mea a hiki i ka wā i manaʻo ʻole ʻia, ua huki ʻo Yandex iā ClickHouse, e like me ka mea kilokilo mai kahi pāpale, e like me ka rabbit. A he hoʻoholo i manaʻo ʻole ʻia, nīnau mau lākou i ka nīnau: "No ke aha?", Akā naʻe.

A i kēia manawa i ke kauwela o 2016, hoʻomaka mākou e nānā i ke ʻano o ClickHouse. A ua ʻike ʻia i kekahi manawa hiki ke ʻoi aku ka wikiwiki ma mua o Vertica. Ua hoʻāʻo mākou i nā hiʻohiʻona like ʻole ma nā nīnau like ʻole. A inā hoʻohana ka nīnau i hoʻokahi papaʻaina, ʻo ia hoʻi, me ka ʻole o kahi hui (hui), a laila ua ʻoi aku ka wikiwiki o ClickHouse ma mua o Vertica.

ʻAʻole wau i palaualelo a nānā i nā hoʻokolohua Yandex i kekahi lā. Ua like ia ma laila: ʻO ClickHouse ʻelua ʻoi aku ka wikiwiki o Vertica, no laila ke kamaʻilio pinepine nei lākou no ia mea.

Akā inā loaʻa nā hui i nā nīnau, a laila huli nā mea āpau ʻaʻole maopopo loa. A hiki i ka ClickHouse ke lohi ʻelua e like me Vertica. A inā hoʻoponopono iki ʻoe i ka noi a kākau hou iā ia, a laila ua like lākou. ʻaʻole maikaʻi ʻole. A manuahi.

A i ka loaʻa ʻana o nā hopena hōʻike, a nānā iā ia mai nā kihi like ʻole, hele ʻo LifeStreet i ClickHouse.

ʻO kēia ka makahiki 16, hoʻomanaʻo wau iā ʻoe. Ua like ia me ka hoʻohenehene e pili ana i nā ʻiole e uē ana a e ʻoki iā lākou iho, akā hoʻomau i ka ʻai ʻana i ka cactus. A ua wehewehe kikoʻī ʻia kēia, aia kahi wikiō e pili ana i kēia, etc.

No laila, ʻaʻole wau e kamaʻilio kikoʻī, e kamaʻilio wale wau e pili ana i nā hopena a me kekahi mau mea hoihoi aʻu i kamaʻilio ʻole ai ia manawa.

ʻO nā hopena:

• Ua holomua ka neʻe ʻana a ʻoi aku ma mua o hoʻokahi makahiki ke hana nei ka ʻōnaehana i ka hana.
• Ua hoʻonui ʻia ka huahana a me ka maʻalahi. ʻO nā moʻolelo 10 biliona i hiki iā mākou ke mālama i kēlā me kēia lā a no ka manawa pōkole, mālama ʻo LifeStreet i kēia manawa he 75 biliona moʻolelo i kēlā me kēia lā a hiki ke hana i kēia no 3 mau mahina a ʻoi aku paha. Inā helu ʻoe i ke kiʻekiʻe, a laila hiki kēia i kahi miliona hanana i kēlā me kēia kekona. ʻOi aku ma mua o hoʻokahi miliona mau nīnau SQL i ka lā i hiki mai i kēia ʻōnaehana, ka hapa nui mai nā robots like ʻole.
• ʻOiai ʻoi aku ka nui o nā kikowaena i hoʻohana ʻia no ClickHouse ma mua o Vertica, ua mālama pū lākou i nā lako, no ka mea, ua hoʻohana ʻia nā disks SAS ma Vertica. Ua hoʻohana ʻo ClickHouse iā SATA. A no ke aha? No ka mea i loko o Vertica hoʻokomo i synchronous. A koi ʻia ka synchronization ʻaʻole e lohi nui nā disks, a ʻaʻole hoʻi e lohi nui ka pūnaewele, ʻo ia hoʻi, he hana ʻoi aku ka pipiʻi. A i loko o ClickHouse hoʻokomo ʻia asynchronous. Eia kekahi, hiki iā ʻoe ke kākau mau i nā mea āpau ma ka ʻāina, ʻaʻohe kumu kūʻai hou no kēia, no laila hiki ke hoʻokomo ʻia ka ʻikepili i ClickHouse ʻoi aku ka wikiwiki ma mua o Vertika, ʻoiai ma nā kaʻa lohi. A like ka heluhelu ʻana. Heluhelu ma SATA, inā aia lākou ma RAID, a laila wikiwiki kēia.
• ʻAʻole kaupalena ʻia e ka laikini, ʻo ia hoʻi 3 petabytes o ka ʻikepili ma 60 mau kikowaena (20 mau kikowaena hoʻokahi kope) a me 6 trillion moʻolelo i nā ʻoiaʻiʻo a me nā hōʻuluʻulu. ʻAʻohe mea like me kēia i hiki ke loaʻa ma Vertica.

Ke huli nei au i nā mea kūpono i kēia laʻana.

• ʻO ka mea mua he papahana kūpono. Pili ka nui i ka schema.
• ʻO ka lua, ʻo ia ka hana SQL maikaʻi.

ʻO kahi nīnau OLAP maʻamau he koho. Hele kekahi o nā kolamu i ka hui ʻana, hele kekahi o nā kolamu i nā hana aggregate. Aia ma kahi, hiki ke hōʻike ʻia ma ke ʻano he ʻāpana o kahi cube. Hiki ke noʻonoʻo ʻia ka hui holoʻokoʻa ma ke ʻano he projection. A ʻo ia ke kumu i kapa ʻia ai ʻo multivariate data analysis.

A pinepine hoʻohālikelike ʻia kēia ma ke ʻano o ka hōkū-scheme, ke loaʻa kahi ʻoiaʻiʻo a me nā hiʻohiʻona o kēia ʻoiaʻiʻo ma nā ʻaoʻao, ma nā kukuna.

A ma ke ʻano o ka hoʻolālā kino, pehea e kūpono ai ma ka papaʻaina, hana maʻamau lākou i kahi hōʻike maʻamau. Hiki iā ʻoe ke denormalize, akā he pipiʻi ma ka disk a ʻaʻole maikaʻi loa i nā nīnau. No laila, hana maʻamau lākou i kahi hōʻike maʻamau, ʻo ia hoʻi he papa ʻoiaʻiʻo a me nā papa ʻāpana he nui.

Akā ʻaʻole maikaʻi ka hana ma ClickHouse. ʻElua kumu:

• ʻO ka mea mua no ka mea ʻaʻole maikaʻi loa ka hui ʻana o ClickHouse, ʻo ia hoʻi, aia nā hui, akā ʻino lākou. ʻOiai ʻino.
• ʻO ka lua, ʻaʻole i hōʻano hou ʻia nā papa. ʻO ka mea maʻamau i loko o kēia mau papa, e puni ana i ka hōkū hōkū, pono e hoʻololi ʻia kekahi mea. No ka laʻana, inoa mea kūʻai aku, inoa ʻoihana, etc. A ʻaʻole pono.

A aia kahi ala i waho o kēia ma ClickHouse. ʻelua ʻelua:

• ʻO ka mua ka hoʻohana ʻana i nā puke wehewehe. ʻO nā puke wehewehe'ōlelo waho ka mea e kōkua iā 99% e hoʻoponopono i ka pilikia me ka hōkū-schema, me nā mea hou a pēlā aku.
• ʻO ka lua ka hoʻohana ʻana i nā arrays. Kōkua pū nā Arrays e hoʻopau i nā hui a me nā pilikia me ka normalization.

• ʻAʻole pono e hui.
• Hiki ke hoʻonui. Mai Malaki 2018, ua puka mai kahi manawa i palapala ʻole ʻia (ʻaʻole ʻoe e ʻike i kēia ma ka palapala) e hoʻonui i nā puke wehewehe ʻōlelo, ʻo ia hoʻi kēlā mau mea i loli. ʻO ka maʻamau, ua like ia me ka papaʻaina.
• Ma ka hoʻomanaʻo mau, no laila e hui pū me kahi puke wehewehe ʻōlelo e hana wikiwiki ma mua o ka papa ʻaina ma ka disk a ʻaʻole ia he ʻoiaʻiʻo aia i loko o ka cache, ʻaʻole paha.

• ʻAʻole pono ʻoe e hui pū.
• He hōʻike paʻa kēia 1-a-nui.
• A i koʻu manaʻo, ua hana ʻia nā arrays no nā geeks. ʻO kēia nā hana lambda a pēlā aku.

ʻAʻole kēia no nā huaʻōlelo ʻulaʻula. He hana ikaika loa kēia e hiki ai iā ʻoe ke hana i nā mea he nui i kahi ala maʻalahi a nani.

Nā laʻana maʻamau e kōkua i ka hoʻoponopono ʻana i nā arrays. He maʻalahi kēia mau hiʻohiʻona:

• Huli ma nā hōʻailona. Inā loaʻa iā ʻoe nā hashtags ma laila a makemake ʻoe e ʻimi i kekahi mau pou ma ka hashtag.
• Huli ma nā hui waiwai kī. Aia kekahi mau ʻano me ka waiwai.
• E mālama i nā papa inoa o nā kī āu e pono ai e unuhi i kekahi mea ʻē aʻe.

Hiki ke hoʻopau ʻia kēia mau hana a pau me ka ʻole o nā arrays. Hiki ke hoʻokomo ʻia nā tag i kekahi laina a koho ʻia me kahi ʻōlelo maʻamau a i ʻole i kahi papa ʻokoʻa, akā pono ʻoe e hana i nā hui.

A ma ClickHouse, ʻaʻole pono ʻoe e hana i kekahi mea, ua lawa ia e wehewehe i ke kaula string no nā hashtags a i ʻole e hana i kahi hoʻolālā nested no nā ʻōnaehana waiwai kī.

ʻAʻole paha ka inoa ʻoi aku ka maikaʻi. ʻO kēia nā papa ʻelua i loaʻa kahi ʻāpana maʻamau i ka inoa a me kekahi mau hiʻohiʻona pili.

A he mea maʻalahi loa ka huli ʻana ma ka hōʻailona. Loaʻa i kahi hana has, ka mea e nānā i ka array i loaʻa i kahi mea. Loaʻa nā mea a pau e pili ana i kā mākou ʻaha kūkā.

ʻOi aku ka paʻakikī o ka huli ʻana ma ka subid. Pono mākou e ʻimi mua i ka index o ke kī, a laila lawe i ka mea me kēia index a nānā i kēia waiwai ka mea a mākou e pono ai. Eia naʻe, maʻalahi loa a paʻa.

ʻO ka ʻōlelo maʻamau āu e makemake ai e kākau inā mālama ʻoe ia mau mea a pau i ka laina hoʻokahi, ʻo ia ka mea, ʻo ka mea mua, ʻaʻole. A ʻo ka lua, ʻoi aku ka lōʻihi o ka hana ma mua o ʻelua arrays.

ʻO kekahi laʻana. Loaʻa iā ʻoe kahi ʻāpana kahi āu e mālama ai i ka ID. A hiki iā ʻoe ke unuhi iā lākou i nā inoa. Hana arrayMap. He hana lambda maʻamau kēia. Hele ʻoe i nā ʻōlelo lambda ma laila. A huki ʻo ia i ka waiwai o ka inoa no kēlā me kēia ID mai ka puke wehewehe.

Hiki ke hana like me ka huli. Hoʻoholo ʻia kahi hana predicate e nānā i nā mea e pili ana.

Hoʻomaʻamaʻa nui kēia mau mea i ke kaapuni a hoʻoponopono i nā pilikia.

Akā ʻo ka pilikia aʻe a mākou e kū nei, a makemake wau e haʻi aku, ʻo ia nā nīnau kūpono.

• ʻAʻohe mea hoʻolālā nīnau ʻo ClickHouse. ʻAʻole loa.
• Eia nō naʻe, pono e hoʻolālā ʻia nā nīnau paʻakikī. Ma na hihia hea?
• Inā nui nā hui i loko o ka nīnau, e hoʻopili ʻoe iā lākou i nā koho. A ʻo ke ʻano o ka hoʻokō ʻia ʻana he mea nui ia.
• Aʻo ka lua - inā e māheleʻia ka noi. No ka mea, i loko o kahi nīnau i hāʻawi ʻia, ʻo ka subselect i loko wale nō i hoʻokō ʻia, a hāʻawi ʻia nā mea ʻē aʻe i kahi kikowaena āu i hoʻopili ai a hoʻokō ʻia ma laila. No laila, inā ua hāʻawi ʻoe i nā nīnau me nā hui he nui (hui), pono ʻoe e koho i ke kauoha.

A i nā hihia maʻalahi, i kekahi manawa pono e hana i ka hana a ka mea hoʻonohonoho a kākau hou i nā nīnau liʻiliʻi.

Eia kekahi laʻana. Aia ma ka ʻaoʻao hema kahi nīnau e hōʻike ana i nā ʻāina 5 kiʻekiʻe. A he 2,5 kekona, i ko'u manao. A ma ka ʻaoʻao ʻākau, ka nīnau like, akā kākau hou ʻia. Ma kahi o ka hui ʻana ma ke kaula, ua hoʻomaka mākou e hui ma ke kī (int). A ʻoi aku ka wikiwiki. A laila hoʻohui mākou i kahi puke wehewehe ʻōlelo i ka hopena. Ma kahi o 2,5 kekona, lawe ka noi i 1,5 kekona. Maikaʻi kēia.

ʻO kahi laʻana like me ke kākau hou ʻana i nā kānana. Eia ke noi no Rusia. Holo no 5 kekona. Inā mākou e kākau hou iā ia ma ke ʻano e hoʻohālikelike hou mākou ʻaʻole i kahi kaula, akā ʻo nā helu me kekahi pūʻulu o kēlā mau kī e pili ana iā Rusia, a laila e ʻoi aku ka wikiwiki.

Nui nā ʻano hoʻopunipuni. A ʻae lākou iā ʻoe e wikiwiki wikiwiki i nā nīnau āu e manaʻo nei e holo wikiwiki nei, a i ʻole, holo mālie. Hiki iā lākou ke hana wikiwiki.

• ʻO ka hana kiʻekiʻe ma ke ʻano puʻunaue.
• Hoʻokaʻawale ʻia e nā ʻano liʻiliʻi, e like me kaʻu i hana ai ma nā ints.
• Inā loaʻa kekahi hui (hui), nā puke wehewehe ʻōlelo, a laila ʻoi aku ka maikaʻi o ka hana ʻana iā lākou ma ke ʻano he hopena hope loa, inā loaʻa iā ʻoe ka ʻikepili ma ka liʻiliʻi loa i hui pū ʻia, a laila e kapa ʻia ka hana hui ʻana a i ʻole ke kāhea ʻana i ka puke wehewehe ʻana i nā manawa liʻiliʻi a ʻoi aku ka wikiwiki. .
• Ke pani nei i nā kānana.

Aia nā ʻenehana ʻē aʻe, ʻaʻole wale nā ​​​​mea aʻu i hōʻike ai. A hiki iā lākou a pau ke wikiwiki i ka hoʻokō ʻana i nā nīnau.

E neʻe kākou i ka laʻana aʻe. Hui X mai USA. He aha kona hana?

Aia kekahi hana:

• Hoʻopili pahemo o nā hana hoʻolaha.
• Hoʻohālike i nā ʻano hoʻohālike like ʻole.

He aha ke kūlana?

Hele mai kahi malihini maʻamau i ka pūnaewele, no ka laʻana, 20 mau manawa o ka mahina mai nā hoʻolaha like ʻole, a i ʻole like me ia i kekahi manawa hele mai me ka ʻole o nā hoʻolaha, no ka mea, hoʻomanaʻo ʻo ia i kēia pūnaewele. Nānā i kekahi mau huahana, hoʻokomo i loko o ka hīnaʻi, lawe i waho o ka hīnaʻi. A, i ka hopena, kūʻai kekahi mea.

Nā nīnau kūpono: "ʻO wai ka mea e uku no ka hoʻolaha, inā pono?" a "He aha ka hoʻolaha i hoʻoikaika iā ia, inā he?". ʻO ia, no ke aha ʻo ia i kūʻai ai a pehea e kiʻi ai i nā poʻe e like me kēia kanaka e kūʻai pū?

I mea e hoʻoponopono ai i kēia pilikia, pono ʻoe e hoʻopili i nā hanana e kū nei ma ka pūnaewele ma ke ala kūpono, ʻo ia hoʻi, e kūkulu i kahi pilina ma waena o lākou. A laila hoʻouna ʻia lākou no ka nānā ʻana iā DWH. A ma muli o kēia loiloi, kūkulu i nā hiʻohiʻona o ka mea a me nā hoʻolaha e hōʻike ai.

ʻO ke kālepa hoʻolaha he pūʻulu o nā hanana mea hoʻohana pili e hoʻomaka ana mai ka hōʻike ʻana i kahi hoʻolaha, a laila hiki mai kekahi mea, a laila kūʻai paha, a laila aia paha nā kūʻai i loko o kahi kūʻai. No ka laʻana, inā he polokalamu kelepona kēia a he pāʻani kelepona paha, a laila maʻamau ka hoʻokomo ʻana o ka noi no ka manuahi, a inā hana ʻia kekahi mea ma laila, a laila pono ke kālā no kēia. A ʻoi aku ka nui o ka hoʻolilo ʻana o ke kanaka i ka noi, ʻoi aku ka waiwai. Akā no kēia pono ʻoe e hoʻohui i nā mea āpau.

Nui nā hiʻohiʻona paʻa.

ʻO nā mea kaulana loa:

• ʻO ka Interaction hope, kahi o ka launa pū ʻana he kaomi a i ʻole ka manaʻo.
• ʻO ka pilina mua, ʻo ia ka mea mua i lawe mai i ke kanaka i ka pūnaewele.
• Hui laina laina - like nā mea a pau.
• Hoʻohaʻahaʻa.
• A laila.

A pehea i hana mua ai? Aia ʻo Runtime a me Cassandra. Ua hoʻohana ʻia ʻo Cassandra ma ke ʻano he mālama kālā, ʻo ia hoʻi, ua mālama ʻia nā hana pili āpau i loko. A i ka hiki ʻana mai o kekahi hanana i Runtime, no ka laʻana, e hōʻike ana i kekahi ʻaoʻao a i ʻole kekahi mea ʻē aʻe, a laila ua noi ʻia iā Cassandra - aia kekahi kanaka a ʻaʻole paha. A laila ua loaʻa nā hana e pili ana iā ia. A ua hana ʻia ka pilina.

A inā he laki ka loaʻa ʻana o ka id kālepa i ka noi, a laila maʻalahi. Akā, ʻaʻohe laki. No laila, pono e ʻimi i ke kālepa hope loa a i ʻole ke kālepa me ke kaomi hope, etc.

A ua hana maikaʻi loa ia e like me ka lōʻihi o ka hoʻopaʻa ʻana i ke kaomi hope loa. No ka mea, he 10 miliona kaomi i kēlā me kēia lā, 300 miliona i kēlā me kēia mahina, inā mākou e hoʻonoho i kahi puka makani no hoʻokahi mahina. A no ka mea ma Cassandra pono ia e hoʻomanaʻo i mea e holo wikiwiki ai, no ka mea, pono e pane koke ka Runtime, ua lawe ma kahi o 10-15 mau kikowaena.

A i ka wā i makemake ai lākou e hoʻopili i kahi kālepa i ka hōʻike, ʻaʻole ia i leʻaleʻa. A no ke aha? Hiki ke ʻike ʻia he 30 mau manawa hou aku e pono ke mālama ʻia. A, no laila, pono ʻoe i 30 mau manawa hou aku i nā kikowaena. A ua ʻike ʻia he ʻano kiʻi hōkū kēia. No ka mālama ʻana i nā kikowaena 500 i mea e hana ai i ka loulou, ʻoiai ʻo ka liʻiliʻi o nā kikowaena i Runtime, a laila he ʻano kuhi hewa kēia. A hoʻomaka lākou e noʻonoʻo i ka mea e hana ai.

A hele mākou i ClickHouse. A pehea e hana ai ma ClickHouse? I ka nānā mua ʻana, me he mea lā he hoʻonohonoho kēia o nā ʻano anti-pattern.

• Ke ulu nei ke kālepa, hoʻopili mākou i nā hanana hou aʻe iā ia, ʻo ia hoʻi, hiki ke loli, a ʻaʻole hana maikaʻi ʻo ClickHouse me nā mea hiki ke hoʻololi.
• Ke hele mai kahi malihini iā ​​mākou, pono mākou e huki i kāna mau hana ma ke kī, ma kāna kipa id. He nīnau helu kēia, ʻaʻole lākou e hana i kēlā ma ClickHouse. ʻO ka maʻamau ka ClickHouse he nui ... nā scans, akā pono mākou e kiʻi i kekahi mau moʻolelo. He antipattern kekahi.
• Eia kekahi, aia ka hana ma json, akā ʻaʻole makemake lākou e kākau hou, no laila makemake lākou e mālama iā json i kahi ala ʻole, a inā pono, e huki i kahi mea mai ia mea. A he antipattern nō hoʻi kēia.

ʻO ia hoʻi, he set of antipatterns.

Akā naʻe, ua lilo ia i ʻōnaehana i hana maikaʻi loa.

He aha ka mea i hana ʻia? Ua ʻike ʻia ʻo ClickHouse, kahi i hoʻolei ʻia ai nā lāʻau, i māhele ʻia i nā moʻolelo. Ua ʻike ʻia kahi lawelawe i loaʻa i nā lāʻau mai ClickHouse. Ma hope o kēlā, no kēlā me kēia komo ʻana, ma ke kipa id, ua loaʻa iaʻu nā hana i hiki ʻole ke hana ʻia a me nā kiʻi paʻi kiʻi, ʻo ia hoʻi nā hana i hoʻopili ʻia, ʻo ia ka hopena o ka hana mua. Ua hana mua wau i ka loiloi mai loko mai o lākou, koho i ke kālepa kūpono, pili i nā hanana hou. Kakau hou. Ua hoʻi ka lāʻau i ClickHouse, ʻo ia hoʻi he ʻōnaehana pōʻaiapuni mau. A eia kekahi, ua hele au i DWH e kālailai ma laila.

Ma kēia ʻano ʻaʻole i hana maikaʻi loa. A i mea e maʻalahi ai ka ClickHouse, i ka wā i loaʻa ai kahi noi ma ka kipa id, ua hui lākou i kēia mau noi i loko o nā poloka o 1-000 kipa kipa a huki i nā hana āpau no 2-000 poʻe. A laila ua hana nā mea a pau.

Inā ʻoe e nānā i loko o ClickHouse, aia he 3 mau papa nui e lawelawe nei i kēia mau mea a pau.

ʻO ka papa mua kahi i hoʻoili ʻia ai nā lāʻau, a ua hoʻouka ʻia nā lāʻau me ka ʻole o ka hana ʻana.

Papa ʻelua. Ma o ka ʻike pilikino, ua nahu ʻia nā hanana i ʻike ʻole ʻia, ʻo ia hoʻi, nā mea pili ʻole. A ma o ka ʻike maoli ʻana, ua huki ʻia nā kālepa mai kēia mau lāʻau e kūkulu i kahi kiʻi. ʻO ia hoʻi, ua kūkulu ʻia kahi hiʻohiʻona materialized kūikawā, ʻo ia hoʻi ka mokuʻāina hōʻiliʻili hope loa o ke kālepa.

Eia ka kikokikona i kākau ʻia ma SQL. Makemake au e ʻōlelo i kekahi mau mea nui i loko.

ʻO ka mea nui mua ka hiki ke huki i nā kolamu a me nā māla mai json ma ClickHouse. ʻO ia hoʻi, loaʻa iā ClickHouse kekahi mau ala no ka hana ʻana me json. He kahiko loa lakou.

kipaParamExtractInt hiki iā ʻoe ke unuhi i nā ʻano mai json, ʻo ia hoʻi ka hana mua. A ma kēia ala hiki iā ʻoe ke huki i ka id kālepa a i ʻole kipa id. ʻO kēia manawa.

ʻO ka lua, ua hoʻohana ʻia kahi māla maʻalahi ma aneʻi. He aha ka manaʻo? ʻO ia hoʻi, ʻaʻole hiki iā ʻoe ke hoʻokomo i loko o ka papaʻaina, ʻo ia hoʻi, ʻaʻole i hoʻokomo ʻia, helu ʻia a mālama ʻia ma ka hoʻokomo ʻana. Ke hoʻopili nei, hana ʻo ClickHouse i ka hana no ʻoe. A ʻo ka mea āu e pono ai ma hope ua huki ʻia i waho o json.

I kēia hihia, ʻike ʻia ka ʻike no nā lālani maka. A hoʻohana wale ʻia ka papa ʻaina mua me nā lāʻau maʻamau. A he aha kāna hana? ʻO ka mea mua, hoʻololi ia i ka hoʻokaʻawale ʻana, ʻo ia hoʻi, ʻo ka hoʻokaʻawale ʻana i kēia manawa e hele i ka id kipa, no ka mea pono mākou e huki koke i kāna kālepa no kahi kanaka kikoʻī.

ʻO ka lua o ka mea nui ʻo index_granularity. Inā ʻoe i ʻike iā MergeTree, ʻo ia ka mea maʻamau 8 e ka index_granularity paʻamau. He aha ia? ʻO kēia ka ʻāpana sparseness index. Ma ClickHouse he kakaikahi ka index, ʻaʻole ia e kuhikuhi i kēlā me kēia komo. Hana ʻo ia i kēlā me kēia 192. A maikaʻi kēia ke koi ʻia ka nui o ka ʻikepili e helu ʻia, akā ʻino ke liʻiliʻi, no ka mea, aia kahi ma luna. A inā e hoʻemi mākou i ka granularity index, a laila e hoʻemi mākou i ka overhead. ʻAʻole hiki ke hoʻemi ʻia i hoʻokahi, no ka mea ʻaʻole lawa ka hoʻomanaʻo. Hoʻopaʻa mau ʻia ka papa kuhikuhi i ka hoʻomanaʻo.

Hoʻohana pū ʻo Snapshot i kekahi mau hiʻohiʻona ClickHouse hoihoi.

ʻO ka mea mua, ʻo ia ʻo AggregatingMergeTree. A ʻo AggregatingMergeTree nā hale kūʻai argMax, ʻo ia hoʻi ke kūlana o ke kālepa e pili ana i ka manawa hope. Hana ʻia nā hana i nā manawa a pau no kahi malihini i hāʻawi ʻia. A i ke kūlana hope loa o kēia kālepa, ua hoʻohui mākou i kahi hanana a loaʻa iā mākou kahi mokuʻāina hou. Ua paʻi hou iā ClickHouse. A ma o argMax i kēia hiʻohiʻona materialized, hiki iā mākou ke loaʻa i ke kūlana o kēia manawa.

• Hoʻokaʻawale ʻia ka paʻa ʻana mai ka Runtime.
• E mālama ʻia a mālama ʻia a hiki i 3 biliona mau hana i kēlā me kēia mahina. ʻO kēia kahi kauoha o ka nui ma mua o Cassandra, ʻo ia hoʻi i kahi ʻōnaehana transactional maʻamau.
• Pūʻulu o nā kikowaena ClickHouse 2x5. 5 mau kikowaena a he kope ko kēlā me kēia kikowaena. ʻOi aku ka liʻiliʻi o kēia ma mua o Cassandra i mea e hana ai i ka attribution ma muli o ka kaomi, a eia mākou e pili ana i ka manaʻo. ʻO ia hoʻi, ma kahi o ka hoʻonui ʻana i ka helu o nā kikowaena i nā manawa 30, ua hoʻokō lākou e hōʻemi iā lākou.

A ʻo ka hiʻohiʻona hope loa ʻo ia ka hui kālā Y, nāna i nānā i ka hoʻoponopono ʻana o nā loli i nā kumukūʻai kumukūʻai.

A ʻo ka hana:

• Aia ma kahi o 5 mau mahele.
• ʻIke ʻia nā huaʻōlelo i kēlā me kēia 100 milliseconds.
• Ua hōʻiliʻili ʻia ka ʻikepili ma luna o 10 mau makahiki. ʻIke ʻia, no kekahi mau ʻoihana ʻoi aku ka nui, no ka mea liʻiliʻi.
• Aia ma kahi o 100 biliona lālani i ka huina.

A ua pono e helu i ka pilina o nā loli.

Eia nā waihona ʻelua a me kā lākou mau ʻōlelo. Inā piʻi kekahi a piʻi kekahi, a laila he pilina maikaʻi kēia, ʻo ia hoʻi, piʻi kekahi a piʻi kekahi. Inā piʻi kekahi, e like me ka hopena o ka pakuhi, a iho kekahi i lalo, a laila, he hoʻopili ʻino kēia, ʻo ia hoʻi, ke piʻi kekahi, hāʻule kekahi.

Ke nānā nei i kēia mau hoʻololi like, hiki i kekahi ke hana i nā wānana i ka mākeke kālā.

Akā paʻakikī ka hana. He aha ka hana ʻia no kēia? Loaʻa iā mākou he 100 biliona moʻolelo i loaʻa: ka manawa, ka waiwai, a me ke kumukūʻai. Pono mākou e helu mua i ka 100 biliona manawa o ka holo ʻokoʻa mai ke kumukūʻai algorithm. ʻO RunningDifference kahi hana ma ClickHouse e helu helu i ka ʻokoʻa ma waena o nā kaula ʻelua.

A ma hope o kēlā, pono ʻoe e helu i ka pilina, a pono e helu ʻia ka pilina no kēlā me kēia pālua. No 5 kaʻana, ʻo 000 miliona mau pālua. A he nui kēia, ʻo ia hoʻi, he 12,5 mau manawa pono e helu i kahi hana hoʻoponopono.

A inā poina kekahi, ʻo ͞x a me ͞y he hoa pili. manaʻolana laʻana. ʻO ia hoʻi, ʻaʻole pono e helu wale i nā kumu a me nā huina, akā hoʻokahi pūʻulu hou i loko o kēia mau huina. Pono e hana ʻia kahi hui helu 12,5 miliona mau manawa, a hui pū ʻia e nā hola. Nui nō hoʻi kā mākou mau hola. A pono ʻoe e hana i loko o 60 kekona. He ʻakaʻaka.

Pono e loaʻa ka manawa ma ka liʻiliʻi loa, no ka mea, ua hana nui kēia mau mea a pau ma mua o ka hiki ʻana mai o ClickHouse.

Ua hoʻāʻo lākou e helu ma Hadoop, ma Spark, ma Greenplum. A he lohi a pipiʻi paha kēia mau mea a pau. ʻO ia hoʻi, hiki ke helu i kekahi ʻano, akā ua pipiʻi.

A laila hele mai ʻo ClickHouse a ʻoi aku ka maikaʻi o nā mea.

Ke hoʻomanaʻo nei au iā ʻoe he pilikia ko mākou me ka locality data, no ka mea ʻaʻole hiki ke hoʻopili ʻia nā correlations. ʻAʻole hiki iā mākou ke kau i kekahi o ka ʻikepili ma kahi kikowaena, kekahi ma kekahi a helu, pono mākou e loaʻa nā ʻikepili āpau ma nā wahi āpau.

He aha kā lākou i hana ai? I ka hoʻomaka ʻana, ʻike ʻia ka ʻikepili. Mālama kēlā me kēia kikowaena i ka ʻikepili e pili ana i ke kumu kūʻai o kekahi pūʻulu o nā ʻāpana. A ʻaʻole lākou e uhi. No laila, hiki ke helu i ka logReturn ma ke ʻano like a me ke kūʻokoʻa, hiki i kēia mau mea a pau i ka parallel a puʻunaue ʻia.

A laila ua hoʻoholo mākou e hōʻemi i kēia mau ʻikepili, ʻoiai ʻaʻole e nalowale i ka expressiveness. E hōʻemi i ka hoʻohana ʻana i nā arrays, ʻo ia hoʻi no kēlā me kēia manawa, e hana i kahi ʻano o nā waihona a me nā kumukūʻai. No laila, ʻoi aku ka liʻiliʻi o ka ʻikepili. A ua maʻalahi lākou e hana. He mau hana like like kēia, ʻo ia hoʻi, heluhelu hapa mākou i ka like a kākau i ke kikowaena.

Ma hope o kēlā, hiki ke hana hou. 'O ka hua'ōlelo "r" 'o ia ho'i, ua hana hou mākou i kēia 'ikepili. ʻO ia hoʻi, loaʻa iā mākou ka ʻikepili like ma nā kikowaena ʻekolu - ʻo ia nā arrays.

A laila me kahi palapala kūikawā mai kēia pūʻulu o 12,5 miliona correlations pono e helu ʻia, hiki iā ʻoe ke hana i nā pūʻulu. ʻO ia hoʻi, 2 mau hana me 500 mau pai o ka pilina. A e helu ʻia kēia hana ma kahi kikowaena ClickHouse kikoʻī. Loaʻa iā ia nā ʻikepili a pau, no ka mea, ua like ka ʻikepili a hiki iā ia ke helu ma ke kaʻina.

Eia hou ke ano o keia. ʻO ka mea mua, loaʻa iā mākou nā ʻikepili āpau i kēia ʻano: ka manawa, nā ʻāpana, ke kumu kūʻai. A laila helu mākou i ka logReturn, ʻo ia hoʻi ka ʻikepili o ka hale like, akā ma kahi o ke kumukūʻai ua loaʻa iā mākou ka logReturn. A laila ua hana hou ʻia lākou, ʻo ia hoʻi, loaʻa iā mākou ka manawa a me ka groupArray no nā kumukūʻai a me nā kumukūʻai. Hoʻopili ʻia. A ma hope o kēlā, ua hana mākou i nā hana he nui a hānai iā lākou i ClickHouse i helu ʻia lākou. A hana ia.

Ma ka hōʻoia o ka manaʻo, he subtask ka hana, ʻo ia hoʻi, liʻiliʻi ka ʻikepili i lawe ʻia. A ʻekolu mau kikowaena wale nō.

ʻO nā pae mua ʻelua: ʻo ka helu ʻana i ka Log_return a me ka hoʻopili ʻana i nā arrays ma kahi o hoʻokahi hola.

A ʻo ka helu ʻana o ka pilina ma kahi o 50 mau hola. Akā ʻaʻole lawa nā hola he 50, no ka mea ua hana lākou no nā pule. He lanakila nui. A inā helu ʻoe, a laila, helu ʻia nā manawa he 70 i kēlā me kēia kekona ma kēia pūʻulu.

Akā ʻo ka mea koʻikoʻi, ʻo kēia ʻōnaehana maʻamau me ka ʻole o ka bottlenecks, ʻo ia hoʻi, ʻaneʻane ʻaneʻane laina. A nānā lākou. Ua hoʻonui maikaʻi ʻia.

• ʻO ka papahana kūpono ka hapalua o ke kaua. A ʻo ka hoʻolālā kūpono ka hoʻohana ʻana i nā ʻenehana ClickHouse pono āpau.
• ʻO Summing/AggregatingMergeTrees nā ʻenehana e ʻae iā ʻoe e hōʻuluʻulu a noʻonoʻo paha i kahi kiʻi mokuʻāina ma ke ʻano he hihia kūikawā. A maʻalahi loa ia i nā mea he nui.
• Hiki iā ʻoe ke kāʻalo i ka palena kuhikuhi hoʻokahi. Malia paha ʻaʻole wau i ʻōlelo maopopo loa, akā i ka wā i hoʻouka ai mākou i nā lāʻau, aia nā lāʻau maka i ka papaʻaina me hoʻokahi papa kuhikuhi, a aia nā logs attribute i ka papaʻaina, ʻo ia hoʻi ka ʻikepili like, kānana wale ʻia, akā ua piha ka index. nā mea ʻē aʻe. Me he mea lā ʻo ia ka ʻikepili like, akā ʻokoʻa ka hoʻokaʻawale ʻana. A ʻo Materialized Views e ʻae iā ʻoe, inā pono ʻoe, e kāpae i kahi palena ClickHouse.
• E hoemi i ka index granularity no na ninau kiko.
• A e puʻunaue i ka ʻikepili me ke akamai, e hoʻāʻo e hoʻokaʻawale i ka ʻikepili i loko o ke kikowaena e like me ka hiki. A e hoʻāʻo e hōʻoia i ka hoʻohana ʻana o nā noi i ka localization inā hiki ke hiki.

A i ka hōʻuluʻulu ʻana i kēia ʻōlelo pōkole, hiki iā mākou ke ʻōlelo ua paʻa paʻa ʻo ClickHouse i ka ʻāina o nā ʻikepili pāʻoihana a me nā ʻikepili open source, ʻo ia hoʻi, kūikawā no ka analytics. Ua kūpono ʻo ia i kēia ʻāina. A ʻo ka mea ʻē aʻe, hoʻomaka mālie ia e hoʻopili i nā poʻe ʻē aʻe, no ka mea inā loaʻa iā ʻoe ka ClickHouse, ʻaʻole pono ʻoe iā InfiniDB. ʻAʻole makemake koke ʻo Vertika inā hana lākou i ke kākoʻo SQL maʻamau. Nanea!

-Mahalo no ka hōʻike! Hoihoi loa! Aia kekahi mau hoʻohālikelike me Apache Phoenix?

ʻAʻole, ʻaʻole au i lohe i ka hoʻohālikelike ʻana. Ke hoʻāʻo nei mākou a me Yandex e mālama i nā hoʻohālikelike ClickHouse āpau me nā ʻikepili like ʻole. No ka mea, inā ʻoi aku ka wikiwiki o kekahi mea ma mua o ClickHouse, a laila ʻaʻole hiki iā Lesha Milovidov ke hiamoe i ka pō a hoʻomaka koke i ka wikiwiki. ʻAʻole au i lohe i kēlā ʻano hoʻohālikelike.

• (Aleksey Milovidov) ʻO Apache Phoenix kahi mīkini SQL i hoʻoikaika ʻia e Hbase. ʻO ka Hbase ka mea nui no ke kūlana hana waiwai nui. Ma laila, i kēlā me kēia laina, hiki ke loaʻa ka helu kūʻokoʻa o nā kolamu me nā inoa ʻole. Hiki ke ʻōlelo ʻia e pili ana i nā ʻōnaehana e like me Hbase, Cassandra. A he mau nīnau noiʻi koʻikoʻi koʻikoʻi ʻaʻole e hana maʻamau no lākou. A i ʻole manaʻo paha ʻoe e hana maikaʻi lākou inā ʻaʻole ʻoe i ʻike me ClickHouse.

• Спасибо

  • Aloha awakea, Aloha Auinalā Ua makemake nui au i kēia kumuhana, no ka mea, he subsystem analytical kaʻu. Akā i koʻu nānā ʻana iā ClickHouse, loaʻa iaʻu ka manaʻo ua kūpono loa ʻo ClickHouse no ka nānā ʻana i nā hanana, mutable. A inā pono iaʻu e kālailai i ka nui o ka ʻikepili ʻoihana me ka pūʻulu o nā papa nui, a laila ʻo ClickHouse, i koʻu hoʻomaopopo ʻana, ʻaʻole kūpono loa iaʻu? ʻOi loa inā hoʻololi lākou. He pololei paha kēia a i ʻole he mau laʻana e hiki ke hōʻole i kēia?

  • Ua pololei kēia. A he ʻoiaʻiʻo kēia i ka hapa nui o nā ʻikepili analytical kūikawā. Hoʻopili ʻia lākou no ka ʻoiaʻiʻo he hoʻokahi a ʻoi aku paha nā papa nui i hiki ke hoʻololi ʻia, a no nā mea liʻiliʻi e loli mālie. ʻO ia, ʻaʻole like ʻo ClickHouse me Oracle, kahi hiki iā ʻoe ke kau i nā mea āpau a kūkulu i kekahi mau nīnau paʻakikī. I mea e hoʻohana pono ai i ka ClickHouse, pono ʻoe e kūkulu i kahi hoʻolālā ma kahi ala e hana maikaʻi ai ma ClickHouse. ʻO ia hoʻi, e pale i ka hana maʻamau, hoʻohana i nā puke wehewehe'ōlelo, e ho'āʻo e hana i nā loulou lōʻihi liʻiliʻi. A inā i kūkulu ʻia ka schema ma kēia ʻano, a laila hiki ke hoʻopau ʻia nā hana ʻoihana like ma ClickHouse ʻoi aku ka maikaʻi ma mua o kahi waihona pili kuʻuna.

Mahalo no ka hōʻike! He nīnau kaʻu e pili ana i ka hihia kālā hou loa. Loaʻa iā lākou ka analytics. Pono e hoʻohālikelike i ko lākou piʻi ʻana a me lalo. A maopopo iaʻu ua kūkulu ʻoe i ka ʻōnaehana no kēia ʻikepili? Inā ʻapōpō, no ka laʻana, pono lākou i kahi hōʻike ʻē aʻe e pili ana i kēia ʻikepili, pono lākou e kūkulu hou i ka papahana a hoʻouka i ka ʻikepili? ʻO ia hoʻi, e hana i kekahi ʻano preprocessing e kiʻi i ka noi?

ʻOiaʻiʻo, ʻo kēia ka hoʻohana ʻana o ClickHouse no kahi hana kikoʻī. Hiki ke hoʻoponopono maʻamau i loko o Hadoop. No Hadoop, he hana kūpono kēia. Akā ma Hadoop ua lohi loa. A ʻo kaʻu pahuhopu ʻo ia ka hōʻike ʻana hiki iā ClickHouse ke hoʻoponopono i nā hana i hoʻoponopono maʻamau ʻia e nā ʻano ʻokoʻa like ʻole, akā i ka manawa like e hana ʻoi aku ka maikaʻi. Hoʻonohonoho ʻia no kahi hana kikoʻī. Ua maopopo inā loaʻa kahi pilikia me kahi mea like, a laila hiki ke hoʻoponopono ʻia ma ke ʻano like.

Ua maopopo. Ua ʻōlelo ʻoe ua hana ʻia nā hola 50. Mai ka hoʻomaka ʻana, i ka manawa hea ʻoe i hoʻouka ai i ka ʻikepili a loaʻa paha nā hopena?

Ae 'ae.

OK mahalo nui loa.

Aia kēia ma kahi pūʻulu kikowaena 3.

Aloha mai! Mahalo no ka hōʻike! He mea hoihoi loa nā mea a pau. ʻAʻole wau e nīnau iki e pili ana i ka hana, akā e pili ana i ka hoʻohana ʻana o ClickHouse ma ke ʻano o ka paʻa. ʻO ia hoʻi, ua loaʻa iā ʻoe kekahi, pono ʻoe e hoʻihoʻi? Pehea e hana ai ʻo ClickHouse i kēia hihia? A ua loaʻa iā ʻoe kahi kope? No ka laʻana, ua loaʻa iā mākou kahi pilikia me ClickHouse i ka wā e puka ai i waho o kona palena a hāʻule.

ʻOiaʻiʻo,ʻaʻohe'ōnaehana kūpono. A loaʻa iā ClickHouse kona mau pilikia ponoʻī. Akā ua lohe paha ʻoe e pili ana iā Yandex.Metrica ʻaʻole hana no ka manawa lōʻihi? ʻAʻole paha. Ua hana hilinaʻi ia mai 2012-2013 ma ClickHouse. Hiki iaʻu ke ʻōlelo like e pili ana i koʻu ʻike. ʻAʻole i loaʻa iā mākou nā hemahema piha. Hiki ke hana ʻia kekahi mau mea, akā ʻaʻole lākou i koʻikoʻi e hoʻopilikia koʻikoʻi i ka ʻoihana. ʻAʻole i hiki. Hiki ke hilinaʻi ʻia ʻo ClickHouse a ʻaʻole hāʻule wale. ʻAʻole pono ʻoe e hopohopo no ia mea. ʻAʻole ia he mea maka. Ua hōʻoia kēia e nā hui he nui.

Aloha! Ua ʻōlelo ʻoe, pono ʻoe e noʻonoʻo koke i ka schema data. Pehea inā ua hiki mai? Ke ninini nei kaʻu ʻikepili. ʻEono mahina i hala, a maopopo iaʻu ʻaʻole hiki ke ola e like me kēia, pono wau e hoʻouka hou i ka ʻikepili a hana i kekahi mea me lākou.

Pili kēia i kāu ʻōnaehana. Nui nā ala e hana ai i kēia me ka pau ʻole. No ka laʻana, hiki iā ʻoe ke hana i kahi Materialized View kahi e hana ai i kahi hoʻolālā ʻikepili ʻē aʻe inā hiki ke kahakaha ʻia. ʻO ia hoʻi, inā e ʻae ʻo ia i ka palapala ʻāina me ka hoʻohana ʻana i ClickHouse, ʻo ia hoʻi e unuhi i kekahi mau mea, hoʻololi i ke kī mua, hoʻololi i ka partitioning, a laila hiki iā ʻoe ke hana i kahi Materialized View. Hoʻopau i kāu ʻikepili kahiko ma laila, e kākau aunoa ʻia nā mea hou. A laila hoʻololi wale i ka hoʻohana ʻana i ka Materialized View, a laila hoʻololi i ka moʻolelo a pepehi i ka papaʻaina kahiko. ʻO kēia ka mea maʻamau he ala pau ʻole.

Спасибо.

Source: www.habr.com