Kev xav thiab kev xyaum siv ClickHouse hauv kev siv tiag tiag. Alexander Zaitsev (2018)

Txawm hais tias qhov tseeb tias tam sim no muaj ntau cov ntaub ntawv yuav luag txhua qhov chaw, cov ntaub ntawv txheeb xyuas tseem yog qhov txawv heev. Lawv paub tsis zoo thiab tseem tuaj yeem siv tau zoo. Ntau tus mus txuas ntxiv "noj cactus" nrog MySQL lossis PostgreSQL, uas yog tsim los rau lwm yam xwm txheej, raug kev txom nyem nrog NoSQL, lossis them nyiaj ntau dhau rau kev lag luam daws teeb meem. ClickHouse hloov cov cai ntawm qhov kev ua si thiab txo qis qhov pib nkag mus rau hauv lub ntiaj teb ntawm kev tshuaj ntsuam DBMS.

Qhia los ntawm BackEnd Conf 2018 thiab nws tau luam tawm nrog kev tso cai los ntawm tus hais lus.

Kuv yog leej twg thiab vim li cas kuv thiaj tham txog ClickHouse? Kuv yog tus thawj coj txhim kho ntawm LifeStreet, uas siv ClickHouse. Tsis tas li ntawd, kuv yog tus tsim ntawm Altinity. Nws yog Yandex tus khub uas txhawb nqa ClickHouse thiab pab Yandex ua ClickHouse ua tiav. Kuj npaj los qhia kev paub txog ClickHouse.

Thiab kuv tsis yog tus tij laug ntawm Petya Zaitsev. Kuv feem ntau nug txog qhov no. Tsis yog, peb tsis yog kwv tij.

"Txhua tus paub" uas ClickHouse:

• ceev heev,
• Yooj yim heev
• Siv hauv Yandex.

Tsawg tsawg yog paub nyob rau hauv cov tuam txhab uas muag thiab yuav siv li cas.

Kuv yuav qhia koj vim li cas, qhov twg thiab yuav siv ClickHouse li cas, tshwj tsis yog Yandex.

Kuv yuav qhia koj tias cov haujlwm tshwj xeeb raug daws li cas nrog kev pab ntawm ClickHouse hauv cov tuam txhab sib txawv, cov cuab yeej ClickHouse koj tuaj yeem siv rau koj cov dej num, thiab lawv tau siv li cas hauv cov tuam txhab sib txawv.

Kuv khaws peb qhov piv txwv uas qhia ClickHouse los ntawm ntau lub ces kaum. Kuv xav tias nws yuav nthuav.

Thawj lo lus nug yog: "Vim li cas peb thiaj xav tau ClickHouse?". Nws zoo nkaus li yog ib lo lus nug pom tseeb, tab sis muaj ntau dua ib lo lus teb rau nws.

• Thawj cov lus teb yog rau kev ua tau zoo. ClickHouse ceev heev. Analytics ntawm ClickHouse kuj ceev heev. Nws tuaj yeem siv tau qhov twg lwm yam qeeb qeeb lossis phem heev.
• Qhov thib ob teb yog tus nqi. Thiab ua ntej ntawm tag nrho cov, tus nqi ntawm scaling. Piv txwv li, Vertica yog cov ntaub ntawv zoo heev. Nws ua haujlwm zoo heev yog tias koj tsis muaj ntau cov ntaub ntawv terabytes. Tab sis thaum nws los txog rau pua pua ntawm terabytes lossis petabytes, tus nqi ntawm daim ntawv tso cai thiab kev txhawb nqa mus rau ib qho tseem ceeb heev. Thiab nws kim heev. Thiab ClickHouse yog dawb.
• Cov lus teb thib peb yog tus nqi khiav haujlwm. Qhov no yog ib qho txawv me ntsis. RedShift yog qhov zoo tshaj plaws analog. Ntawm RedShift, koj tuaj yeem txiav txim siab sai heev. Nws yuav ua haujlwm zoo, tab sis tib lub sijhawm, txhua teev, txhua hnub, thiab txhua lub hlis, koj yuav them Amazon heev, vim tias qhov no yog qhov kev pabcuam kim heev. Google BigQuery ib yam nkaus. Yog tias ib tug neeg siv nws, ces nws paub tias nyob ntawd koj tuaj yeem khiav ntau qhov kev thov thiab tau txais ib daim nqi rau ntau pua daus las tam sim ntawd.

ClickHouse tsis muaj cov teeb meem no.

ClickHouse siv tam sim no nyob qhov twg? Ntxiv nrog rau Yandex, ClickHouse tau siv ntau lub lag luam thiab tuam txhab sib txawv.

• Ua ntej tshaj plaws, qhov no yog web application analytics, i.e. qhov no yog ib qho kev siv uas tuaj ntawm Yandex.
• Ntau lub tuam txhab AdTech siv ClickHouse.
• Ntau lub tuam txhab uas xav tau los txheeb xyuas cov ntaub ntawv sib pauv los ntawm ntau qhov chaw.
• Ntau lub tuam txhab siv ClickHouse los saib xyuas kev ruaj ntseg cav. Lawv xa lawv mus rau ClickHouse, ua cov ntawv ceeb toom, thiab tau txais cov txiaj ntsig lawv xav tau.
• Cov tuam txhab lag luam tau pib siv nws hauv kev txheeb xyuas nyiaj txiag, piv txwv li maj mam ua lag luam loj tseem tab tom mus txog ClickHouse.
• cloudflare. Yog tias ib tug neeg ua raws ClickHouse, ces lawv tau hnov ​​​​lub npe ntawm lub tuam txhab no. Qhov no yog ib qho tseem ceeb ntawm kev koom tes los ntawm cov zej zog. Thiab lawv muaj qhov loj heev ClickHouse installation. Piv txwv li, lawv ua Kafka Cav rau ClickHouse.
• Cov tuam txhab kev sib txuas lus tau pib siv. Ntau lub tuam txhab siv ClickHouse los ua pov thawj ntawm lub tswv yim lossis twb tau tsim lawm.
• Ib lub tuam txhab siv ClickHouse los saib xyuas cov txheej txheem ntau lawm. Lawv sim microcircuits, sau tawm ib pawg ntawm cov tsis muaj, muaj txog 2 tus yam ntxwv. Thiab tom qab ntawd lawv txheeb xyuas seb qhov kev ua si zoo lossis phem.
• Blockchain analytics. Muaj xws li ib lub tuam txhab Lavxias teb sab li Bloxy.info. Qhov no yog kev txheeb xyuas ntawm ethereum network. Lawv kuj tau ua qhov no ntawm ClickHouse.

Thiab qhov loj me tsis muaj teeb meem. Muaj ntau lub tuam txhab uas siv ib lub server me me. Thiab nws cia lawv daws lawv tej teeb meem. Thiab ntau lub tuam txhab siv cov pawg loj ntawm ntau lub servers lossis ntau lub servers.

Thiab yog tias koj saib cov ntaub ntawv, ces:

• Yandex: 500+ servers, lawv khaws 25 billion cov ntaub ntawv ib hnub nyob rau ntawd.
• LifeStreet: 60 servers, kwv yees li 75 billion cov ntaub ntawv ib hnub. Muaj tsawg servers, cov ntaub ntawv ntau dua li hauv Yandex.
• CloudFlare: 36 servers, lawv khaws 200 billion cov ntaub ntawv ib hnub. Lawv muaj tsawg dua servers thiab khaws cov ntaub ntawv ntau dua.
• Bloomberg: 102 servers, txog ib trillion nkag hauv ib hnub. Tus tuav ntaub ntawv.

Geographically, qhov no kuj muaj ntau. Daim ntawv qhia no qhia txog lub tshav kub ntawm qhov twg ClickHouse tau siv hauv ntiaj teb. Russia, Tuam Tshoj, America sawv tawm kom meej ntawm no. Muaj ob peb lub tebchaws nyob sab Europe. Thiab muaj 4 pawg.

Qhov no yog ib qho kev sib piv kev tsom xam, tsis tas yuav mus nrhiav cov nuj nqis kiag li. Qhov no yog ib qho kev soj ntsuam ntawm cov neeg tuaj saib uas nyeem cov ntaub ntawv lus Askiv ntawm lub vev xaib Altinity, vim tias tsis muaj cov neeg hais lus Lavxias nyob ntawd. Thiab Russia, Ukraine, Belarus, piv txwv li ib feem ntawm Lavxias teb sab-hais lus ntawm zej zog, cov no yog cov neeg siv ntau tshaj plaws. Tom qab ntawd los ntawm Teb Chaws Asmeskas thiab Canada. Suav teb coob heev. Yuav luag tsis muaj Tuam Tshoj nyob rau lub hlis dhau los, tam sim no Tuam Tshoj twb dhau Europe lawm thiab tseem loj hlob ntxiv. Cov teb chaws Europe qub kuj tsis nyob deb, thiab tus thawj coj hauv kev siv ClickHouse yog, oddly txaus, Fabkis.

Vim li cas kuv thiaj qhia tag nrho cov no? Los qhia tias ClickHouse tau dhau los ua cov qauv kev daws teeb meem rau kev txheeb xyuas cov ntaub ntawv loj thiab twb tau siv ntau qhov chaw. Yog tias koj siv nws, koj nyob rau hauv txoj kev sib. Yog koj tseem tsis tau siv ces koj yuav tsis ntshai tsam koj yuav tso tseg tsis muaj leej twg pab koj, vim muaj coob tus twb ua li no lawm.

Cov no yog cov piv txwv ntawm ClickHouse tiag tiag siv hauv ntau lub tuam txhab.

• Thawj qhov piv txwv yog kev tshaj tawm xov xwm: kev tsiv teb tsaws ntawm Vertica mus rau ClickHouse. Thiab kuv paub ob peb lub tuam txhab uas tau hloov pauv los ntawm Vertica lossis tab tom tab tom hloov pauv.
• Qhov thib ob piv txwv yog kev lag luam cia ntawm ClickHouse. Qhov no yog ib qho piv txwv ua los ntawm antipatterns. Txhua yam uas yuav tsum tsis txhob ua hauv ClickHouse ntawm cov lus qhia ntawm cov neeg tsim khoom tau ua tiav ntawm no. Thiab nws ua tau zoo heev uas nws ua haujlwm. Thiab nws ua haujlwm zoo dua li cov kev hloov pauv kev daws teeb meem.
• Qhov thib peb piv txwv yog faib xam ntawm ClickHouse. Muaj lus nug txog yuav ua li cas ClickHouse tuaj yeem koom ua ke hauv Hadoop ecosystem. Kuv yuav qhia ib qho piv txwv ntawm yuav ua li cas ib lub tuam txhab tau ua ib yam dab tsi zoo ib yam li daim ntawv qhia txo cov thawv ntawm ClickHouse, khaws cov ntaub ntawv hauv zos, thiab lwm yam, los xam cov hauj lwm tsis tseem ceeb heev.

• LifeStreet yog lub tuam txhab Ad Tech uas muaj tag nrho cov thev naus laus zis uas los nrog kev tshaj tawm.
• Nws koom nrog kev ua kom zoo tshaj plaws, kev sib tw programmatic.
• Ntau cov ntaub ntawv: txog 10 billion txheej xwm ib hnub twg. Nyob rau tib lub sijhawm, cov xwm txheej muaj peev xwm muab faib ua ob peb ntu.
• Muaj ntau tus neeg siv ntawm cov ntaub ntawv no, thiab cov no tsis yog tib neeg nkaus xwb, ntau ntxiv - cov no yog ntau yam algorithms uas koom nrog hauv kev sib tw programmatic.

Lub tuam txhab tau los ntev thiab pos. Thiab kuv tau tham txog nws ntawm HighLoad. Ua ntej, LifeStreet tau tsiv los ntawm MySQL (nrog rau luv luv ntawm Oracle) mus rau Vertica. Thiab koj tuaj yeem nrhiav tau ib zaj dab neeg txog nws.

Thiab txhua yam tau zoo heev, tab sis nws sai sai tau pom tseeb tias cov ntaub ntawv loj hlob thiab Vertica yog kim. Yog li ntawd, ntau yam kev xaiv tau nrhiav. Ib txhia ntawm lawv muaj nyob rau ntawm no. Thiab qhov tseeb, peb tau ua pov thawj ntawm lub tswv yim lossis qee zaum kev sim ua haujlwm ntawm yuav luag txhua cov ntaub ntawv uas muaj nyob rau hauv kev ua lag luam txij li xyoo 13th txog rau xyoo 16th thiab tau kwv yees li qhov tsim nyog ntawm kev ua haujlwm. Thiab kuv kuj tau tham txog qee qhov ntawm lawv ntawm HighLoad.

Txoj haujlwm yog tsiv los ntawm Vertica thawj qhov chaw, vim tias cov ntaub ntawv loj hlob tuaj. Thiab lawv tau loj hlob exponentially ntau xyoo. Tom qab ntawd lawv tau mus rau ntawm lub txee, tab sis txawm li cas los xij. Thiab kwv yees qhov kev loj hlob no, kev lag luam yuav tsum muaj rau cov ntaub ntawv uas qee yam kev tshuaj xyuas yuav tsum tau ua, nws tau pom tseeb tias petabytes yuav tau tham sai sai. Thiab them rau petabytes twb kim heev, yog li peb tau nrhiav lwm txoj hauv kev mus.

mus qhov twg? Thiab ntev dhau los nws tsis paub meej tias yuav mus qhov twg, vim hais tias ntawm ib sab muaj cov ntaub ntawv lag luam, lawv zoo li ua haujlwm zoo. Qee qhov ua haujlwm yuav luag zoo li Vertica, qee qhov phem dua. Tab sis lawv tag nrho cov kim, tsis muaj dab tsi pheej yig dua thiab zoo dua nrhiav tsis tau.

Ntawm qhov tod tes, muaj cov kev daws teeb meem qhib, uas tsis muaj ntau heev, piv txwv li rau kev txheeb xyuas, lawv tuaj yeem suav rau ntawm tus ntiv tes. Thiab lawv yog dawb los yog pheej yig, tab sis qeeb. Thiab lawv feem ntau tsis muaj qhov tsim nyog thiab muaj txiaj ntsig ua haujlwm.

Thiab tsis muaj dab tsi los muab qhov zoo uas yog nyob rau hauv kev lag luam databases thiab tag nrho cov dawb uas yog nyob rau hauv qhib qhov chaw.

Tsis muaj ib yam dab tsi kom txog thaum, poob nthav, Yandex rub tawm ClickHouse, zoo li tus kws yees duab los ntawm lub kaus mom, zoo li luav. Thiab nws yog qhov kev txiav txim siab tsis xav txog, lawv tseem nug cov lus nug: "Vim li cas?", Txawm li cas los xij.

Thiab tam sim ntawd thaum lub caij ntuj sov 2016, peb pib saib dab tsi ClickHouse. Thiab nws tau muab tawm tias qee zaum nws tuaj yeem nrawm dua Vertica. Peb sim cov xwm txheej sib txawv ntawm kev thov sib txawv. Thiab yog tias cov lus nug siv tsuas yog ib lub rooj, uas yog, tsis muaj kev koom nrog (sib koom), ces ClickHouse yog ob zaug sai li Vertica.

Kuv tsis tub nkeeg dhau lawm thiab saib Yandex xeem rau lwm hnub. Nws yog tib yam nyob ntawd: ClickHouse yog ob zaug ceev li Vertica, yog li lawv feem ntau tham txog nws.

Tab sis yog hais tias muaj kev koom nrog hauv cov lus nug, ces txhua yam hloov tawm tsis heev unambiguously. Thiab ClickHouse tuaj yeem ua ob zaug qeeb li Vertica. Thiab yog tias koj kho qhov kev thov me ntsis thiab rov sau dua, ces lawv yog kwv yees li sib npaug. Tsis phem. Thiab dawb.

Thiab tau txais cov txiaj ntsig kev xeem, thiab saib nws los ntawm ntau lub ces kaum, LifeStreet tau mus rau ClickHouse.

Nov yog 16 xyoo, kuv nco koj. Nws zoo li ib qho lus tso dag hais txog cov nas uas quaj thiab pricked lawv tus kheej, tab sis txuas ntxiv noj cactus. Thiab qhov no tau piav qhia meej, muaj video txog qhov no, thiab lwm yam.

Yog li ntawd, kuv yuav tsis tham txog nws kom meej, Kuv tsuas yog tham txog cov txiaj ntsig thiab ob peb yam nthuav uas kuv tsis tau tham txog thaum ntawd.

Cov txiaj ntsig yog:

• Kev vam meej kev tsiv teb tsaws chaw thiab ntau tshaj li ib xyoos ntawm qhov system twb ua haujlwm hauv kev tsim khoom.
• Productivity thiab yooj tau nce. Ntawm 10 billion cov ntaub ntawv uas peb tuaj yeem them taus khaws cia ib hnub thiab tom qab ntawd rau lub sijhawm luv luv, LifeStreet tam sim no khaws 75 billion cov ntaub ntawv ib hnub thiab tuaj yeem ua qhov no rau 3 lub hlis lossis ntau dua. Yog tias koj suav ntawm lub ncov, ces qhov no yog mus txog ib lab txheej xwm ib ob. Ntau tshaj li ib lab SQL lus nug ib hnub tuaj txog hauv qhov system no, feem ntau los ntawm cov neeg hlau sib txawv.
• Txawm tias muaj tseeb tias ntau cov servers tau siv rau ClickHouse dua rau Vertica, lawv kuj tau txais kev cawmdim ntawm cov khoom siv, vim tias kim heev SAS disks tau siv hauv Vertica. ClickHouse siv SATA. Thiab yog vim li cas? Vim hais tias nyob rau hauv Vertica insert yog synchronous. Thiab synchronization xav kom cov disks tsis txhob qeeb ntau dhau, thiab tseem tias lub network tsis qeeb ntau dhau, uas yog, kev ua haujlwm kim heev. Thiab hauv ClickHouse insert yog asynchronous. Ntxiv mus, koj tuaj yeem sau txhua yam hauv zos, tsis muaj nqi ntxiv rau qhov no, yog li cov ntaub ntawv tuaj yeem muab tso rau hauv ClickHouse sai dua rau hauv Vertika, txawm tias qeeb qeeb. Thiab kev nyeem ntawv yog hais txog tib yam. Nyeem ntawm SATA, yog tias lawv nyob hauv RAID, ces qhov no yog txhua yam nrawm txaus.
• Tsis txwv los ntawm daim ntawv tso cai, piv txwv li 3 petabytes ntawm cov ntaub ntawv nyob rau hauv 60 servers (20 servers yog ib tug replica) thiab 6 trillion cov ntaub ntawv nyob rau hauv qhov tseeb thiab aggregations. Tsis muaj dab tsi zoo li no tuaj yeem them taus ntawm Vertica.

Tam sim no kuv tig mus rau qhov ua tau zoo hauv qhov piv txwv no.

• Qhov thib ib yog ib txoj kev ua tau zoo. Muaj ntau nyob ntawm schema.
• Qhov thib ob yog qhov ua tau zoo SQL tiam.

Ib qho lus nug OLAP ib txwm yog xaiv. Ib txhia ntawm cov kab mus rau pawg los ntawm, ib co ntawm cov kab mus rau aggregate functions. Muaj qhov twg, uas tuaj yeem sawv cev ua ib daim ntawm lub voos xwmfab. Tag nrho cov pab pawg neeg tuaj yeem xav tias yog qhov projection. Thiab yog vim li cas nws thiaj li hu ua multivariate data analysis.

Thiab feem ntau qhov no yog ua qauv nyob rau hauv daim ntawv ntawm lub hnub qub-scheme, thaum muaj ib tug tseem ceeb qhov tseeb thiab cov yam ntxwv ntawm qhov tseeb nyob rau hauv lub sab, nyob rau hauv lub rays.

Thiab nyob rau hauv cov nqe lus ntawm lub cev tsim, yuav ua li cas nws haum rau ntawm lub rooj, lawv feem ntau ua ib tug normalized sawv cev. Koj tuaj yeem denormalize, tab sis nws kim heev ntawm disk thiab tsis ua haujlwm zoo ntawm cov lus nug. Yog li ntawd, lawv feem ntau ua ib tug normalized sawv cev, piv txwv li ib lub rooj tseeb thiab ntau, ntau qhov ntev.

Tab sis nws tsis ua haujlwm zoo hauv ClickHouse. Muaj ob qho laj thawj:

• Thawj yog vim ClickHouse tsis tau zoo heev koom nrog, piv txwv li muaj kev koom nrog, tab sis lawv tsis zoo. Thaum phem.
• Qhov thib ob yog tias cov ntxhuav tsis hloov tshiab. Feem ntau nyob rau hauv cov phiaj no, uas nyob ib ncig ntawm lub hnub qub-circuit, ib yam dab tsi yuav tsum tau hloov. Piv txwv li, tus neeg siv khoom lub npe, lub tuam txhab npe, thiab lwm yam. Thiab nws tsis ua haujlwm.

Thiab muaj txoj hauv kev tawm ntawm qhov no hauv ClickHouse. txawm ob:

• Qhov thib ib yog siv cov phau ntawv txhais lus. Cov phau ntawv txhais lus sab nraud yog dab tsi pab 99% daws qhov teeb meem nrog lub hnub qub-schema, nrog kev hloov tshiab thiab lwm yam.
• Qhov thib ob yog kev siv cov arrays. Arrays kuj pab kom tshem tawm cov kev koom nrog thiab teeb meem nrog normalization.

• Tsis xav koom.
• Upgradable. Txij li thaum Lub Peb Hlis 2018, lub sijhawm tsis muaj ntaub ntawv tau tshwm sim (koj yuav tsis pom qhov no hauv cov ntaub ntawv) los hloov cov phau ntawv txhais lus ib nrab, piv txwv li cov ntawv nkag uas tau hloov pauv. Tswv yim, nws zoo li lub rooj.
• Ib txwm nyob hauv lub cim xeeb, yog li koom nrog phau ntawv txhais lus ua haujlwm sai dua yog tias nws yog lub rooj uas nyob hauv disk thiab nws tseem tsis tau muaj qhov tseeb tias nws nyob hauv cache, feem ntau yuav tsis yog.

• Koj tsis tas yuav koom nrog.
• Qhov no yog ib qho kev sib cog lus 1-rau-ntau tus sawv cev.
• Thiab hauv kuv lub tswv yim, arrays yog tsim rau geeks. Cov no yog cov haujlwm lambda thiab lwm yam.

Qhov no tsis yog rau cov lus liab. Qhov no yog qhov ua haujlwm muaj zog heev uas tso cai rau koj ua ntau yam hauv txoj kev yooj yim heev thiab muaj kuab heev.

Cov piv txwv uas pab daws cov arrays. Cov piv txwv no yooj yim thiab meej txaus:

• Nrhiav los ntawm tag. Yog tias koj muaj hashtags nyob ntawd thiab xav nrhiav qee cov lus los ntawm hashtag.
• Nrhiav los ntawm tus nqi tseem ceeb. Kuj tseem muaj qee tus cwj pwm nrog tus nqi.
• Khaws cov npe ntawm cov yuam sij uas koj xav tau los txhais ua lwm yam.

Tag nrho cov haujlwm no tuaj yeem daws tau yam tsis muaj arrays. Cim npe tuaj yeem muab tso rau hauv qee kab thiab xaiv nrog cov lus qhia tsis tu ncua lossis hauv ib lub rooj sib cais, tab sis tom qab ntawd koj yuav tsum tau koom nrog.

Thiab hauv ClickHouse, koj tsis tas yuav ua dab tsi, nws txaus los piav txog txoj hlua array rau hashtags lossis ua ib qho qauv tsim rau cov txheej txheem tseem ceeb.

Nested qauv yuav tsis yog lub npe zoo tshaj plaws. Cov no yog ob lub arrays uas muaj feem ntau hauv lub npe thiab qee yam muaj feem cuam tshuam.

Thiab nws yooj yim heev rau kev tshawb nrhiav los ntawm tag. Ua haujlwm has, uas xyuas tias cov array muaj cov khoom. Txhua tus, pom tag nrho cov kev nkag uas cuam tshuam nrog peb lub rooj sib tham.

Tshawb nrhiav los ntawm subid yog qhov nyuaj me ntsis. Peb yuav tsum xub nrhiav qhov Performance index ntawm tus yuam sij, thiab tom qab ntawd coj lub caij nrog qhov ntsuas no thiab xyuas tias tus nqi no yog qhov peb xav tau. Txawm li cas los, nws yog heev yooj yim thiab compact.

Cov lus qhia tsis tu ncua uas koj xav sau yog tias koj khaws nws tag nrho hauv ib kab, nws yuav yog, ua ntej, clumsy. Thiab, qhov thib ob, nws ua haujlwm ntev dua li ob lub arrays.

Lwm qhov piv txwv. Koj muaj ib qho chaw uas koj khaws tus ID. Thiab koj tuaj yeem txhais lawv ua cov npe. Muaj nuj nqi arrayMap. Qhov no yog ib txwm lambda muaj nuj nqi. Koj dhau lambda kab lus muaj. Thiab nws rub tawm tus nqi ntawm lub npe rau txhua tus ID los ntawm phau ntawv txhais lus.

Kev tshawb nrhiav tuaj yeem ua tau tib yam. Ib qho kev ua haujlwm predicate tau dhau los uas kuaj xyuas cov ntsiab lus zoo li cas.

Tej yam no ua rau lub voj voog yooj yim heev thiab daws tau ntau yam teeb meem.

Tab sis qhov teeb meem tom ntej peb tab tom ntsib, thiab qhov kuv xav hais, yog cov lus nug tau zoo.

• ClickHouse tsis muaj cov lus nug npaj. Tsis yog kiag li.
• Txawm li cas los xij, cov lus nug nyuaj tseem yuav tsum tau npaj. Thaum twg?
• Yog tias muaj ntau qhov kev koom nrog hauv cov lus nug, koj qhwv lawv hauv subselects. Thiab qhov kev txiav txim uas lawv raug tua tseem ceeb.
• Thiab qhov thib ob - yog tias qhov kev thov raug faib. Vim hais tias nyob rau hauv cov lus nug faib, tsuas yog cov innermost subselect raug tua faib, thiab txhua yam yog kis mus rau ib tug neeg rau zaub mov uas koj txuas nrog thiab tua muaj. Yog li ntawd, yog tias koj tau faib cov lus nug nrog ntau cov koom nrog (sib koom), ces koj yuav tsum xaiv qhov kev txiav txim.

Thiab txawm tias nyob rau hauv cov ntaub ntawv yooj yim dua, qee zaum kuj tseem yuav tsum tau ua haujlwm ntawm tus teem sijhawm thiab rov sau cov lus nug me ntsis.

Ntawm no yog ib qho piv txwv. Ntawm sab laug yog cov lus nug uas qhia txog 5 lub teb chaws sab saum toj. Thiab nws yuav siv sij hawm 2,5 vib nas this, hauv kuv lub tswv yim. Thiab ntawm sab xis, tib cov lus nug, tab sis me ntsis rov sau dua. Hloov chaw los ntawm txoj hlua, peb pib ua pawg los ntawm qhov tseem ceeb (int). Thiab nws nrawm dua. Thiab ces peb txuas ib phau ntawv txhais lus rau qhov tshwm sim. Hloov 2,5 vib nas this, qhov kev thov yuav siv 1,5 vib nas this. Qhov no yog qhov zoo.

Ib qho piv txwv zoo sib xws nrog rov sau cov lim dej. Ntawm no yog ib qho kev thov rau Russia. Nws khiav rau 5 vib nas this. Yog hais tias peb rov sau dua nws nyob rau hauv ib txoj kev uas peb piv dua tsis yog ib txoj hlua, tab sis cov lej nrog qee cov yuam sij uas cuam tshuam rau Russia, ces nws yuav nrawm dua.

Muaj ntau yam tricks. Thiab lawv tso cai rau koj kom ceev cov lus nug uas koj xav tias twb tau khiav ceev, lossis, hloov pauv, khiav qeeb. Lawv tuaj yeem ua tau sai dua.

• Ua haujlwm siab tshaj plaws hauv kev faib tawm.
• Kev txheeb xyuas los ntawm yam tsawg kawg nkaus, raws li kuv tau ua los ntawm ints.
• Yog tias muaj kev koom nrog (sib koom), phau ntawv txhais lus, ces nws yog qhov zoo dua los ua lawv raws li qhov chaw kawg, thaum koj twb muaj cov ntaub ntawv tsawg kawg yog ib feem ntawm pawg, ces kev koom tes lossis phau ntawv txhais lus hu yuav raug hu tsawg zaus thiab nws yuav nrawm dua. .
• Hloov cov lim dej.

Muaj lwm yam txuj ci, thiab tsis yog cov uas kuv tau qhia xwb. Thiab txhua tus ntawm lawv qee zaum kuj tseem tuaj yeem ua kom tiav cov lus nug.

Cia peb mus rau qhov piv txwv tom ntej. Tuam txhab X los ntawm teb chaws USA. Nws ua dab tsi?

Muaj ib txoj hauj lwm:

• Offline txuas ntawm kev tshaj tawm kev lag luam.
• Ua qauv sib txawv khi qauv.

Dab tsi yog qhov scenario?

Ib tug neeg tuaj saib zoo tib yam tuaj rau ntawm lub xaib, piv txwv li, 20 zaug hauv ib hlis los ntawm kev tshaj tawm txawv, lossis ib yam li ntawd qee zaum tuaj tsis muaj kev tshaj tawm, vim nws nco txog qhov chaw no. Saib qee cov khoom, muab tso rau hauv lub pob tawb, coj lawv tawm ntawm lub pob tawb. Thiab, thaum kawg, ib yam dab tsi yuav.

Cov lus nug tsim nyog: "Leej twg yuav tsum them rau kev tshaj tawm, yog tias tsim nyog?" thiab "Dab tsi kev tshaj tawm cuam tshuam nws, yog tias muaj?". Qhov ntawd yog vim li cas nws thiaj yuav thiab yuav ua li cas kom cov neeg zoo li tus neeg no yuav thiab?

Yuav kom daws tau qhov teeb meem no, koj yuav tsum txuas cov xwm txheej uas tshwm sim hauv lub vev xaib kom raug, uas yog, qee qhov tsim kev sib txuas ntawm lawv. Tom qab ntawd lawv raug xa mus rau kev tshuaj xyuas rau DWH. Thiab raws li qhov kev tshuaj ntsuam no, tsim cov qauv ntawm leej twg thiab dab tsi tshaj tawm los qhia.

Kev lag luam kev tshaj tawm yog ib txheej txheej ntawm cov neeg siv khoom sib xws uas pib los ntawm kev qhia ib qho kev tshaj tawm, tom qab ntawd ib yam dab tsi tshwm sim, ces tej zaum yuav yuav, thiab tom qab ntawd tej zaum yuav muaj kev yuav khoom hauv kev yuav khoom. Piv txwv li, yog tias qhov no yog daim ntawv thov mobile lossis kev ua si hauv xov tooj, feem ntau qhov kev teeb tsa ntawm daim ntawv thov yuav siv qhov chaw dawb, thiab yog tias muaj qee yam ua tiav, ces yuav tsum tau nyiaj rau qhov no. Thiab qhov ntau tus neeg siv nyiaj hauv daim ntawv thov, nws muaj txiaj ntsig ntau dua. Tab sis rau qhov no koj yuav tsum tau txuas txhua yam.

Muaj ntau yam kev sib khi ua qauv.

Qhov nrov tshaj plaws yog:

• Kev Sib Tham Kawg, qhov twg kev sib cuam tshuam yog nias los yog kev xav.
• Thawj Kev Sib Tham, piv txwv li thawj yam uas coj tus neeg mus rau qhov chaw.
• Linear ua ke - txhua yam sib npaug.
• Attenuation.
• Thiab li ntawd.

Thiab nws ua haujlwm li cas hauv thawj qhov chaw? Muaj Runtime thiab Cassandra. Cassandra tau siv los ua kev lag luam cia, piv txwv li tag nrho cov kev lag luam cuam tshuam tau muab khaws cia rau hauv nws. Thiab thaum qee qhov kev tshwm sim los hauv Runtime, piv txwv li, qhia qee nplooj ntawv lossis lwm yam, tom qab ntawv thov rau Cassandra - puas muaj tus neeg zoo li no lossis tsis yog. Tom qab ntawd cov kev pauv uas cuam tshuam rau nws tau txais. Thiab qhov kev sib txuas tau ua.

Thiab yog tias nws muaj hmoo tias qhov kev thov muaj tus lej hloov pauv, ces nws yooj yim. Tab sis feem ntau tsis muaj hmoo. Yog li ntawd, nws yog ib qho tsim nyog yuav tsum nrhiav qhov kev hloov pauv zaum kawg lossis kev hloov pauv nrog qhov nyem kawg, thiab lwm yam.

Thiab nws tag nrho ua haujlwm zoo heev ntev npaum li qhov kev khi yog mus rau qhov kawg nyem. Vim tias muaj, hais, 10 lab clicks ib hnub, 300 lab ib hlis, yog tias peb teeb lub qhov rais rau ib hlis. Thiab txij li thaum nyob rau hauv Cassandra nws yuav tsum tau tag nrho nyob rau hauv lub cim xeeb nyob rau hauv thiaj li yuav khiav ceev, vim hais tias lub Runtime yuav tsum tau teb sai sai, nws siv li ntawm 10-15 servers.

Thiab thaum lawv xav txuas kev lag luam rau cov zaub, nws tam sim ntawd tig tawm tsis lom zem heev. Thiab yog vim li cas? Nws tuaj yeem pom tias 30 lub sijhawm ntau qhov xwm txheej yuav tsum tau khaws cia. Thiab, raws li, koj xav tau 30 npaug ntxiv servers. Thiab nws hloov tawm tias qhov no yog qee yam ntawm astronomical daim duab. Txhawm rau kom mus txog 500 servers txhawm rau ua qhov txuas, txawm hais tias muaj cov servers tsawg dua hauv Runtime, ces qhov no yog qee yam tsis raug. Thiab lawv pib xav tias yuav ua li cas.

Thiab peb mus rau ClickHouse. Thiab yuav ua li cas rau ClickHouse? Thaum xub thawj siab ib muag, nws zoo nkaus li tias qhov no yog ib txheej ntawm kev tiv thaiv.

• Kev lag luam loj hlob, peb sib txuas ntau thiab ntau cov xwm txheej rau nws, piv txwv li nws yog qhov hloov tau, thiab ClickHouse tsis ua haujlwm zoo nrog cov khoom sib hloov.
• Thaum tus qhua tuaj rau peb, peb yuav tsum rub tawm nws cov kev lag luam los ntawm qhov tseem ceeb, los ntawm nws tus ID nkag mus. Qhov no kuj yog ib qho lus nug, lawv tsis ua li ntawd hauv ClickHouse. Feem ntau ClickHouse muaj qhov loj ... scans, tab sis ntawm no peb yuav tsum tau txais qee cov ntaub ntawv. Kuj tseem muaj antipattern.
• Tsis tas li ntawd, kev lag luam nyob hauv json, tab sis lawv tsis xav rov sau dua, yog li lawv xav khaws json hauv txoj kev tsis tsim nyog, thiab yog tias tsim nyog, rub qee yam tawm ntawm nws. Thiab qhov no kuj yog ib qho antipattern.

Ntawd yog, txheej ntawm antipatterns.

Tab sis txawm li cas los xij nws tau tig los ua ib qho system uas ua haujlwm zoo heev.

Ua li cas? ClickHouse tau tshwm sim, uas tau muab pov tseg, muab faib ua cov ntaub ntawv. Ib qho kev pab cuam tshwm sim uas tau txais cov cav los ntawm ClickHouse. Tom qab ntawd, rau txhua qhov nkag, los ntawm kev mus ntsib id, kuv tau txais kev lag luam uas tej zaum yuav tsis tau ua tiav thiab ntxiv rau snapshots, piv txwv li kev lag luam twb tau txuas nrog, uas yog qhov tshwm sim ntawm kev ua haujlwm dhau los. Kuv twb ua logic tawm ntawm lawv, xaiv qhov kev sib pauv kom raug, txuas cov xwm txheej tshiab. Sau dua. Lub cav rov qab mus rau ClickHouse, piv txwv li nws yog ib qho kev ua haujlwm tsis tu ncua. Thiab ib qho ntxiv, kuv tau mus rau DWH txhawm rau txheeb xyuas qhov ntawd.

Nws yog nyob rau hauv daim ntawv no uas nws tsis ua hauj lwm zoo heev. Thiab kom nws yooj yim dua rau ClickHouse, thaum muaj kev thov los ntawm kev mus ntsib id, lawv tau muab cov kev thov no rau hauv cov blocks ntawm 1-000 mus ntsib ids thiab rub tawm tag nrho cov kev lag luam rau 2-000 tus neeg. Thiab tom qab ntawd nws tag nrho ua haujlwm.

Yog tias koj saib hauv ClickHouse, ces tsuas muaj 3 lub rooj loj uas ua haujlwm rau txhua qhov no.

Thawj lub rooj nyob rau hauv uas cav yog uploaded, thiab cov cav yog uploaded yuav luag tsis ua.

Ob lub rooj. Los ntawm cov khoom siv pom, cov xwm txheej uas tseem tsis tau raug ntaus nqi, piv txwv li, cov tsis muaj feem cuam tshuam, raug tom tawm ntawm cov cav no. Thiab los ntawm cov khoom siv pom, kev hloov pauv tau rub tawm ntawm cov cav no los tsim cov duab thaij duab. Ntawd yog, qhov tshwj xeeb materialized saib tau ua ib qho snapshot, uas yog lub xeev cov khoom sib sau kawg ntawm kev sib pauv.

Nov yog cov ntawv sau hauv SQL. Kuv xav tawm tswv yim txog ob peb yam tseem ceeb hauv nws.

Thawj qhov tseem ceeb yog lub peev xwm rub tawm txhua kab thiab teb los ntawm json hauv ClickHouse. Ntawd yog, ClickHouse muaj qee txoj hauv kev ua haujlwm nrog json. Lawv yog cov heev, heev primitive.

visitParamExtractInt tso cai rau koj rho tawm cov cwj pwm los ntawm json, piv txwv li thawj ntaus ua haujlwm. Thiab txoj kev no koj tuaj yeem rub tawm tus lej hloov pauv lossis mus ntsib id. Lub sijhawm no.

Qhov thib ob, ib qho khoom siv tsis yooj yim yog siv ntawm no. Nws txhais li cas? Qhov no txhais tau hais tias koj tsis tuaj yeem ntxig nws rau hauv lub rooj, piv txwv li nws tsis tau ntxig, nws raug suav thiab khaws cia thaum ntxig. Thaum pasting, ClickHouse ua haujlwm rau koj. Thiab qhov koj xav tau tom qab yog twb rub tawm ntawm json.

Hauv qhov no, cov ntaub ntawv pom yog rau cov kab nyoos. Thiab thawj lub rooj nrog xyaum ua raw cov ntaub ntawv tsuas yog siv. Thiab nws ua li cas? Ua ntej, nws hloov qhov kev txheeb xyuas, piv txwv li kev txheeb xyuas tam sim no mus los ntawm kev mus ntsib id, vim tias peb yuav tsum rub tawm sai sai rau kev hloov pauv rau ib tus neeg tshwj xeeb.

Qhov thib ob tseem ceeb tshaj plaws yog index_granularity. Yog tias koj tau pom MergeTree, nws feem ntau yog 8 los ntawm lub neej ntawd index_granularity. Nws yog dab tsi? Qhov no yog qhov index sparseness parameter. Hauv ClickHouse qhov Performance index yog sparse, nws yeej tsis indexes txhua qhov nkag. Nws ua qhov no txhua txhua 192. Thiab qhov no yog qhov zoo thaum cov ntaub ntawv ntau yuav tsum tau muab xam, tab sis phem thaum me ntsis, vim tias muaj cov nyiaj siv ua haujlwm loj. Thiab yog hais tias peb txo qhov index granularity, ces peb txo cov nyiaj siv ua haujlwm. Nws tsis tuaj yeem txo mus rau ib qho, vim tias tej zaum yuav tsis muaj lub cim xeeb txaus. Qhov Performance index yeej ib txwm muab cia rau hauv lub cim xeeb.

Snapshot kuj siv qee qhov nthuav ClickHouse nta.

Ua ntej, nws yog AggregatingMergeTree. Thiab AggregatingMergeTree khw muag khoom argMax, piv txwv li qhov no yog lub xeev ntawm kev sib pauv hloov pauv mus rau lub sijhawm kawg. Kev ua lag luam yog tsim txhua lub sijhawm rau tus neeg tuaj saib. Thiab nyob rau hauv lub xeev kawg ntawm qhov kev pauv no, peb tau ntxiv ib qho kev tshwm sim thiab peb muaj lub xeev tshiab. Nws ntaus ClickHouse dua. Thiab dhau ntawm argMax hauv qhov kev pom no, peb tuaj yeem tau txais lub xeev tam sim no.

• Kev khi yog "decoupled" los ntawm Runtime.
• Txog li 3 billion kev lag luam hauv ib hlis yog khaws cia thiab ua tiav. Qhov no yog qhov kev txiav txim siab ntau dua li nws tau nyob hauv Cassandra, piv txwv li hauv kev sib pauv hloov pauv.
• Pawg ntawm 2x5 ClickHouse servers. 5 servers thiab txhua tus neeg rau zaub mov muaj qhov hloov pauv. Qhov no tseem tsawg dua li nws nyob hauv Cassandra thiaj li ua tau nyem raws li qhov muaj nqis, thiab ntawm no peb muaj kev xav raws li. Ntawd yog, es tsis txhob nce tus naj npawb ntawm servers los ntawm 30 zaug, lawv tau tswj kom txo lawv.

Thiab qhov piv txwv kawg yog lub tuam txhab nyiaj txiag Y, uas tau txheeb xyuas qhov sib cuam tshuam ntawm kev hloov pauv ntawm cov nqi Tshuag.

Thiab txoj haujlwm yog:

• Muaj kwv yees li 5 shares.
• Quotes txhua 100 milliseconds paub.
• Cov ntaub ntawv tau sau ntau tshaj 10 xyoo. Thaj, rau qee lub tuam txhab ntau dua, rau qee qhov tsawg.
• Muaj kwv yees li 100 billion kab hauv tag nrho.

Thiab nws yog qhov tsim nyog los xam qhov sib txheeb ntawm kev hloov pauv.

Ntawm no yog ob lub Tshuag thiab lawv cov quotes. Yog hais tias ib tug mus thiab lwm tus mus, ces qhov no yog ib tug zoo correlation, piv txwv li ib tug mus thiab lwm yam mus. Yog hais tias ib tug nce mus, raws li qhov kawg ntawm daim duab, thiab lwm yam mus down, qhov no yog ib tug tsis zoo correlation, piv txwv li thaum ib tug nce, lwm yam ntog.

Txheeb xyuas cov kev hloov pauv no, ib tus tuaj yeem ua rau kev kwv yees hauv kev lag luam nyiaj txiag.

Tab sis txoj hauj lwm nyuaj. Yuav ua li cas rau qhov no? Peb muaj 100 billion cov ntaub ntawv uas muaj: sijhawm, Tshuag, thiab nqi. Peb yuav tsum xam thawj 100 billion lub sij hawm khiav qhov sib txawv ntawm tus nqi algorithm. RunningDifference yog ib qho kev ua haujlwm hauv ClickHouse uas suav nrog qhov sib txawv ntawm ob txoj hlua.

Thiab tom qab ntawd, koj yuav tsum xam qhov kev sib raug zoo, thiab kev sib raug zoo yuav tsum raug xam rau txhua khub. Rau 5 shares, khub yog 000 lab. Thiab qhov no yog ntau heev, piv txwv li 12,5 zaug nws yog ib qho tsim nyog los xam qhov kev sib raug zoo li no.

Thiab yog hais tias ib tug neeg tsis nco qab, ces ͞x thiab ͞y yog ib tug checkmate. kev cia siab ua piv txwv. Ntawd yog, nws yog ib qho tsim nyog tsis tsuas yog suav cov hauv paus thiab cov lej, tab sis kuj tseem muaj ib qho ntxiv hauv cov lej no. Ib pawg ntawm kev suav yuav tsum tau ua tiav 12,5 lab lub sijhawm, thiab txawm tias ua pawg los ntawm teev. Peb kuj muaj ntau teev. Thiab koj yuav tsum ua nws hauv 60 vib nas this. Nws yog ib tug tso dag.

Nws yog ib qho tsim nyog yuav tsum muaj sijhawm tsawg kawg yog qee yam, vim tias txhua yam no ua haujlwm heev, maj mam ua ntej ClickHouse tuaj.

Lawv sim xam nws ntawm Hadoop, ntawm Spark, ntawm Greenplum. Thiab tag nrho cov no tau qeeb heev los yog kim heev. Ntawd yog, nws tau ua tau rau qee qhov suav, tab sis tom qab ntawd nws kim heev.

Thiab ces ClickHouse tuaj nrog thiab tej yam tau zoo dua.

Kuv ceeb toom rau koj tias peb muaj teeb meem nrog cov ntaub ntawv hauv cheeb tsam, vim tias kev sib raug zoo tsis tuaj yeem nyob hauv cheeb tsam. Peb tsis tuaj yeem tso qee cov ntaub ntawv ntawm ib lub server, qee qhov ntawm lwm tus thiab suav, peb yuav tsum muaj tag nrho cov ntaub ntawv nyob txhua qhov chaw.

Lawv ua dab tsi? Thaum xub thawj, cov ntaub ntawv yog localized. Txhua tus neeg rau zaub mov khaws cov ntaub ntawv ntawm tus nqi ntawm qee qhov sib koom. Thiab lawv tsis sib tshooj. Yog li ntawd, nws muaj peev xwm xam logReturn nyob rau hauv parallel thiab ntawm nws tus kheej, tag nrho cov no tshwm sim kom deb li deb nyob rau hauv parallel thiab faib.

Tom qab ntawd peb txiav txim siab los txo cov ntaub ntawv no, thaum tsis poob kev nthuav qhia. Txo siv arrays, piv txwv li rau txhua lub sijhawm, ua ib qho array ntawm cov khoom lag luam thiab cov nqi ntawm cov nqi. Yog li ntawd, nws yuav siv ntau qhov chaw cov ntaub ntawv tsawg. Thiab lawv yooj yim me ntsis ua haujlwm nrog. Cov no yog yuav luag ua haujlwm sib luag, piv txwv li peb ib nrab nyeem nyob rau hauv parallel thiab tom qab ntawd sau rau hauv server.

Tom qab ntawd, nws tuaj yeem rov ua dua. Tsab ntawv "r" txhais tau tias peb rov ua cov ntaub ntawv no. Ntawd yog, peb muaj tib cov ntaub ntawv ntawm tag nrho peb lub servers - cov no yog cov arrays.

Thiab tom qab ntawd nrog cov ntawv tshwj xeeb los ntawm cov txheej txheem no ntawm 12,5 lab kev sib raug zoo uas yuav tsum tau muab xam, koj tuaj yeem ua pob. Ntawd yog, 2 txoj haujlwm nrog 500 khub ntawm kev sib raug zoo. Thiab txoj haujlwm no yuav tsum suav rau ntawm ib qho tshwj xeeb ClickHouse server. Nws muaj tag nrho cov ntaub ntawv, vim hais tias cov ntaub ntawv yog tib yam thiab nws muaj peev xwm xam lawv sequentially.

Ib zaug ntxiv, qhov no yog qhov nws zoo li. Ua ntej, peb muaj tag nrho cov ntaub ntawv nyob rau hauv cov qauv no: lub sij hawm, shares, nqi. Tom qab ntawd peb suav logReturn, piv txwv li cov ntaub ntawv ntawm tib lub qauv, tab sis tsis yog tus nqi peb twb muaj logReturn. Tom qab ntawd lawv tau rov ua dua, piv txwv li peb tau txais lub sijhawm thiab groupArray rau cov khoom lag luam thiab cov nqi. Sreplicated. Thiab tom qab ntawd, peb tsim ib pawg ntawm cov haujlwm thiab pub rau ClickHouse kom nws suav lawv. Thiab nws ua haujlwm.

Los ntawm cov ntaub ntawv pov thawj ntawm lub tswv yim, txoj haujlwm yog subtask, i.e., cov ntaub ntawv tsawg dua. Thiab tsuas yog peb lub servers.

Thawj ob theem no: xam Log_return thiab qhwv hauv arrays siv sijhawm li ib teev.

Thiab qhov kev suav ntawm kev sib raug zoo yog li 50 teev. Tab sis 50 teev tsis txaus, vim lawv tau siv ua haujlwm ntau lub lis piam. Nws yog ib qho kev vam meej loj. Thiab yog tias koj suav, ces 70 zaug ib ob txhua yam raug suav rau ntawm pawg no.

Tab sis qhov tseem ceeb tshaj plaws yog tias qhov system no xyaum tsis muaj lub hauv paus, piv txwv li, nws ntsuas yuav luag linearly. Thiab lawv tau kuaj xyuas nws. Ua tiav scaled nws.

• Txoj kev cai yog ib nrab ntawm kev sib ntaus sib tua. Thiab txoj cai txoj cai yog siv tag nrho cov tsim nyog ClickHouse technologies.
• Summing/AggregatingMergeTrees yog cov thev naus laus zis uas tso cai rau koj los sib sau ua ke lossis xav txog lub xeev snapshot ua qhov tshwj xeeb. Thiab nws zoo heev simplifies ntau yam.
• Materialized Views tso cai rau koj hla dhau qhov kev txwv ib qho kev txwv. Tej zaum kuv tsis tau hais kom meej meej, tab sis thaum peb thauj cov cav, cov cav nyoos tau nyob rau hauv lub rooj nrog ib qho Performance index, thiab tus cwj pwm cav tau nyob hauv lub rooj, piv txwv li cov ntaub ntawv tib yam, tsuas yog lim, tab sis qhov ntsuas tau tag nrho. lwm tus. Nws zoo li cov ntaub ntawv tib yam, tab sis sib txawv. Thiab Materialized Views tso cai rau koj, yog tias koj xav tau, kom hla dhau qhov kev txwv ClickHouse.
• Txo qhov ntsuas granularity rau cov lus nug.
• Thiab faib cov ntaub ntawv ntse, sim ua kom cov ntaub ntawv nyob hauv lub server kom ntau li ntau tau. Thiab sim ua kom ntseeg tau tias cov lus thov kuj siv qhov chaw nyob qhov twg ntau li ntau tau.

Thiab xaus cov lus luv luv no, peb tuaj yeem hais tias ClickHouse tam sim no tau ruaj khov nyob hauv thaj chaw ntawm ob qho tib si kev lag luam databases thiab qhib qhov chaw databases, piv txwv li, tshwj xeeb rau kev txheeb xyuas. Nws haum zoo kawg nkaus rau hauv toj roob hauv pes no. Thiab dab tsi ntxiv, nws maj mam pib tawm ntawm lwm tus, vim tias thaum koj muaj ClickHouse, koj tsis xav tau InfiniDB. Vertika tej zaum yuav tsis xav tau sai sai yog tias lawv ua ib qho kev txhawb nqa SQL. Txaus siab rau!

-Ua tsaug rau daim ntawv qhia! Txaus siab heev! Puas muaj kev sib piv nrog Apache Phoenix?

Tsis yog, kuv tsis tau hnov ​​leej twg piv. Peb thiab Yandex sim taug qab txhua qhov kev sib piv ClickHouse nrog cov databases sib txawv. Vim hais tias yog dheev ib yam dab tsi hloov mus sai dua ClickHouse, ces Lesha Milovidov pw tsis taus thaum hmo ntuj thiab pib sai sai. Kuv tsis tau hnov ​​txog qhov kev sib piv.

• (Aleksey Milovidov) Apache Phoenix yog SQL engine powered by Hbase. Hbase feem ntau yog rau qhov tseem ceeb-tus nqi ua haujlwm scenario. Nyob ntawd, nyob rau hauv txhua kab, muaj peev xwm muaj ib tug arbitrary naj npawb ntawm kab nrog arbitrary npe. Qhov no tuaj yeem hais txog cov tshuab xws li Hbase, Cassandra. Thiab nws yog precisely hnyav analytical queries uas yuav tsis ua hauj lwm ib txwm rau lawv. Lossis koj yuav xav tias lawv ua haujlwm zoo yog tias koj tsis tau muaj kev paub nrog ClickHouse.

• Спасибо

  • Nyob zoo tav su Kuv twb txaus siab rau lub ntsiab lus no, vim kuv muaj ib qho kev txheeb xyuas subsystem. Tab sis thaum kuv saib ClickHouse, Kuv tau txais qhov kev xav tias ClickHouse yog qhov zoo heev rau kev txheeb xyuas qhov xwm txheej, hloov tau. Thiab yog tias kuv yuav tsum tau txheeb xyuas ntau cov ntaub ntawv lag luam nrog cov rooj loj, ces ClickHouse, raws li kuv nkag siab, tsis haum rau kuv? Tshwj xeeb yog lawv hloov. Qhov no puas yog lossis puas muaj cov piv txwv uas tuaj yeem thim qhov no?

  • Qhov no yog lawm. Thiab qhov no yog qhov tseeb ntawm feem ntau tshwj xeeb analytical databases. Lawv raug kho rau qhov tseeb tias muaj ib lossis ntau lub rooj loj uas hloov tau, thiab rau ntau qhov me me uas hloov maj mam. Ntawd yog, ClickHouse tsis zoo li Oracle, qhov twg koj tuaj yeem tso txhua yam thiab tsim qee cov lus nug nyuaj heev. Txhawm rau siv ClickHouse kom zoo, koj yuav tsum tsim kom muaj lub tswv yim ua haujlwm zoo hauv ClickHouse. Ntawd yog, zam dhau qhov normalization, siv phau ntawv txhais lus, sim ua kom tsawg dua qhov txuas ntev. Thiab yog hais tias lub tswv yim yog tsim nyob rau hauv no txoj kev, ces zoo xws li cov hauj lwm ua lag luam yuav daws tau ntawm ClickHouse ntau npaum li cas zoo dua nyob rau hauv ib txwm relational database.

Ua tsaug rau daim ntawv qhia! Kuv muaj lus nug txog cov ntaub ntawv nyiaj txiag zaum kawg. Lawv muaj analytics. Nws yog ib qho tsim nyog los sib piv seb lawv nce thiab nqis. Thiab kuv nkag siab tias koj tau tsim qhov system tshwj xeeb rau qhov kev tshuaj ntsuam no? Yog tias tag kis, piv txwv li, lawv xav tau lwm daim ntawv tshaj tawm txog cov ntaub ntawv no, lawv puas yuav tsum rov tsim cov schema thiab upload cov ntaub ntawv? Ntawd yog, ua qee yam ua ntej kom tau txais qhov kev thov?

Tau kawg, qhov no yog kev siv ClickHouse rau ib txoj haujlwm tshwj xeeb. Nws tuaj yeem ua ntau txoj kev daws teeb meem hauv Hadoop. Rau Hadoop, qhov no yog txoj haujlwm zoo tshaj plaws. Tab sis ntawm Hadoop nws qeeb heev. Thiab kuv lub hom phiaj yog los ua kom pom tias ClickHouse tuaj yeem daws cov haujlwm uas feem ntau daws tau los ntawm kev sib txawv kiag li, tab sis tib lub sijhawm ua nws tau zoo dua. Qhov no yog tsim los rau ib txoj haujlwm tshwj xeeb. Nws yog qhov tseeb tias yog tias muaj teeb meem nrog qee yam zoo sib xws, ces nws tuaj yeem daws tau zoo ib yam.

Nws yog qhov tseeb. Koj hais tias 50 teev tau ua tiav. Puas yog txij thaum pib, thaum twg koj thauj cov ntaub ntawv lossis tau txais cov txiaj ntsig?

Yog lawm.

OK ua tsaug ntau.

Qhov no yog nyob rau ntawm 3 server pawg.

Nyob zoo! Ua tsaug rau daim ntawv qhia! Txhua yam yog nthuav heev. Kuv yuav tsis nug me ntsis txog kev ua haujlwm, tab sis hais txog kev siv ClickHouse hais txog kev ruaj ntseg. Qhov ntawd yog, koj puas muaj, koj puas tau rov qab? ClickHouse coj li cas hauv qhov no? Thiab nws puas tau tshwm sim tias koj muaj ib qho replica thiab? Peb, piv txwv li, ntsib teeb meem nrog ClickHouse thaum nws tseem tawm ntawm nws qhov kev txwv thiab poob.

Tau kawg, tsis muaj cov txheej txheem zoo tagnrho. Thiab ClickHouse kuj muaj nws tus kheej cov teeb meem. Tab sis koj puas tau hnov ​​​​txog Yandex.Metrica tsis ua haujlwm ntev? Tej zaum tsis yog. Nws tau ua haujlwm zoo txij li xyoo 2012-2013 ntawm ClickHouse. Kuv tuaj yeem hais tib yam txog kuv qhov kev paub dhau los. Peb yeej tsis muaj qhov ua tsis tiav. Qee qhov ib nrab tuaj yeem tshwm sim, tab sis lawv yeej tsis tseem ceeb txaus los cuam tshuam rau kev lag luam. Nws yeej tsis tshwm sim. ClickHouse yog qhov txhim khu kev qha heev thiab tsis poob qis. Koj tsis tas yuav txhawj txog nws. Nws tsis yog khoom nyoos. Qhov no tau raug pov thawj los ntawm ntau lub tuam txhab.

Nyob zoo! Koj hais tias koj yuav tsum xav txog cov ntaub ntawv schema tam sim ntawd. Yuav ua li cas yog nws tshwm sim? Kuv cov ntaub ntawv yog pouring thiab pouring. Rau lub hlis dhau los, thiab kuv nkag siab tias nws tsis tuaj yeem ua neej nyob zoo li no, kuv yuav tsum rov muab cov ntaub ntawv rov qab thiab ua qee yam nrog lawv.

Qhov no nyob ntawm qhov tseeb ntawm koj qhov system. Muaj ntau txoj hauv kev los ua qhov no nrog zoo tsis muaj nres. Piv txwv li, koj tuaj yeem tsim Cov Khoom Siv Saib Xyuas uas yuav ua rau cov ntaub ntawv sib txawv yog tias nws tuaj yeem tsim tshwj xeeb. Ntawd yog, yog tias nws tso cai rau kev siv ClickHouse, piv txwv li rho tawm qee yam, hloov tus yuam sij tseem ceeb, hloov kev faib, ces koj tuaj yeem ua Cov Khoom Siv Saib. Overwrite koj cov ntaub ntawv qub nyob ntawd, cov tshiab yuav raug sau cia. Thiab tom qab ntawd tsuas yog hloov mus rau siv Materialized View, ces hloov cov ntaub ntawv thiab tua lub rooj qub. Qhov no feem ntau yog ib txoj kev tsis nres.

Ua tsaug.

Tau qhov twg los: www.hab.com