Sannu duka! Sunana Golov Nikolay. A baya, na yi aiki a Avito kuma na gudanar da Data Platform na tsawon shekaru shida, wato, na yi aiki a kan dukkanin bayanai: nazari (Vertica, ClickHouse), streaming da OLTP (Redis, Tarantool, VoltDB, MongoDB, PostgreSQL). A wannan lokacin, na yi hulɗa da adadi mai yawa na bayanan bayanai - daban-daban kuma ba a saba ba, kuma tare da lokuta marasa daidaitattun amfani da su.
A halin yanzu ina aiki a ManyChat. A zahiri, wannan farawa ne - sabo, mai buri da girma cikin sauri. Kuma lokacin da na fara shiga kamfanin, wata tambaya mai mahimmanci ta taso: "Menene ya kamata matashin farawa yanzu ya ɗauka daga DBMS da kasuwar bayanai?"
A cikin wannan labarin, bisa ga rahoton na a , Zan amsa wannan tambayar. Ana samun sigar bidiyo na rahoton a .
Sanannen bayanan bayanai 2020
Shekarar 2020 ce, na duba sai na ga nau'ikan bayanan bayanai iri uku.
Nau'in farko - classic OLTP databases: PostgreSQL, SQL Server, Oracle, MySQL. An rubuta su tuntuni, amma har yanzu suna da dacewa saboda sun saba da al'ummar haɓakawa.
Nau'i na biyu shine tushe daga "zero". Sun yi ƙoƙarin ƙaura daga ƙirar gargajiya ta hanyar barin SQL, tsarin al'ada da ACID, ta ƙara ginanniyar sharding da sauran abubuwan ban sha'awa. Misali, wannan shine Cassandra, MongoDB, Redis ko Tarantool. Duk waɗannan mafita sun so ba da kasuwa wani sabon abu na asali kuma sun shagaltar da su saboda sun zama masu dacewa da wasu ayyuka. Zan nuna waɗannan bayanan bayanai tare da kalmar laima NOSQL.
"Sifili" sun ƙare, mun saba da bayanan NOSQL, kuma duniya, daga ra'ayi na, ta ɗauki mataki na gaba - zuwa. sarrafa bayanai. Waɗannan ma'ajin bayanai suna da asali iri ɗaya da bayanan bayanan OLTP na yau da kullun ko sabbin NoSQL. Amma ba su da buƙatar DBA da DevOps kuma suna aiki akan kayan aikin sarrafawa a cikin gajimare. Ga mai haɓakawa, wannan "tushe ne kawai" wanda ke aiki a wani wuri, amma babu wanda ya damu da yadda aka shigar da shi a kan uwar garke, wanda ya tsara uwar garken kuma wanda ya sabunta shi.
Misalan irin waɗannan bayanan bayanai:
- AWS RDS babban abin rufewa ne don PostgreSQL/MySQL.
- DynamoDB analog ne na AWS na tushen bayanan daftarin aiki, mai kama da Redis da MongoDB.
- Amazon Redshift babban bayanan nazari ne da ake gudanarwa.
Waɗannan su ne ainihin tsofaffin bayanan bayanai, amma an tashe su a cikin yanayin sarrafawa, ba tare da buƙatar yin aiki da kayan aiki ba.
Lura. Ana ɗaukar misalan don yanayin AWS, amma analogues ɗin su kuma suna cikin Microsoft Azure, Google Cloud, ko Yandex.Cloud.
Menene sabo game da wannan? A cikin 2020, babu ɗayan wannan.
Ra'ayi mara amfani
Abin da ke da gaske sabo akan kasuwa a cikin 2020 shine mafita mara sabar ko uwar garken.
Zan yi ƙoƙarin bayyana abin da wannan ke nufi ta amfani da misalin sabis na yau da kullun ko aikace-aikacen baya.
Don tura aikace-aikacen baya na yau da kullun, muna siya ko hayan sabar, kwafi lambar akansa, buga ƙarshen ƙarshen waje kuma a kai a kai muna biyan kuɗin haya, wutar lantarki da sabis na cibiyar bayanai. Wannan shi ne ma'auni tsarin.
Akwai wata hanya kuma? Tare da sabis maras sabar zaka iya.
Menene manufar wannan hanya: babu uwar garken, babu ma hayar wani misali mai kama-da-wane a cikin gajimare. Don tura sabis ɗin, kwafi lambar (ayyukan) zuwa ma'ajiya kuma buga ta zuwa ƙarshen ƙarshen. Sannan muna biyan kowane kira zuwa wannan aikin, muna watsi da kayan aikin gaba daya inda ake aiwatar da shi.
Zan yi ƙoƙarin kwatanta wannan hanyar da hotuna.
Kayan aiki na gargajiya. Muna da sabis tare da takamaiman kaya. Muna ɗaga misalai guda biyu: sabar jiki ko lokuta a cikin AWS. Ana aika buƙatun waje zuwa waɗannan abubuwan kuma ana sarrafa su a can.
Kamar yadda kuke gani a hoton, ba a zubar da sabar daidai gwargwado ba. Ana amfani da ɗayan 100%, akwai buƙatu guda biyu, ɗayan kuma 50% kawai - wani bangare mara aiki. Idan ba buƙatun uku sun zo ba, amma 30, to, duk tsarin ba zai iya jurewa da kaya ba kuma zai fara raguwa.
Aiwatar da mara amfani. A cikin mahalli mara sabar, irin wannan sabis ɗin ba shi da misalai ko sabobin. Akwai takamaiman wurin tafki na albarkatu masu zafi - ƙananan kwantenan Docker da aka shirya tare da lambar aikin turawa. Tsarin yana karɓar buƙatun waje kuma ga kowane ɗayansu tsarin tsarin mara amfani yana ɗaga ƙaramin akwati tare da lamba: yana aiwatar da wannan buƙatar ta musamman kuma yana kashe akwati.
Buƙata ɗaya - akwati ɗaya tashe, buƙatun 1000 - kwantena 1000. Kuma turawa akan sabar kayan masarufi ya riga ya zama aikin mai samar da girgije. An ɓoye gaba ɗaya ta tsarin tsarin mara amfani. A cikin wannan ra'ayi muna biyan kowane kira. Misali, kira daya ya zo a rana - mun biya kira daya, miliyan daya ya zo a minti daya - mun biya miliyan. Ko a cikin dakika guda, wannan ma yana faruwa.
Manufar buga aikin mara sabar ya dace da sabis mara jiha. Kuma idan kuna buƙatar sabis ɗin (jihar) cikakke, to muna ƙara bayanan bayanai zuwa sabis ɗin. A wannan yanayin, idan ya zo ga aiki tare da jihar, kowane aikin jaha kawai yana rubutu da karantawa daga bayanan bayanai. Bugu da ƙari, daga bayanan kowane nau'in nau'i uku da aka kwatanta a farkon labarin.
Menene iyakance gama gari na duk waɗannan bayanan bayanai? Waɗannan su ne farashin girgije ko uwar garken hardware da ake amfani da su akai-akai (ko sabar da yawa). Ba kome ko muna amfani da bayanan gargajiya ko sarrafa bayanai, ko muna da Devops da admin ko a'a, har yanzu muna biyan kuɗin kayan aiki, wutar lantarki da hayar cibiyar bayanai 24/7. Idan muna da tushe na gargajiya, muna biya don master da bawa. Idan bayanai ne masu kayatarwa sosai, muna biyan sabobin 10, 20 ko 30, kuma muna biya akai-akai.
Kasancewar sabobin da aka tanada na dindindin a cikin tsarin farashi an riga an gane shi azaman mugunyar da ta dace. Har ila yau, ma'ajin bayanai na al'ada suna da wasu matsaloli, kamar iyaka akan adadin haɗin yanar gizo, ƙuntatawa mai ƙima, ra'ayi mai rarraba geo-rarraba ko ta yaya za a iya warware su a cikin wasu bayanan bayanai, amma ba gaba ɗaya ba kuma ba daidai ba.
Database marar uwar garken - ka'idar
Tambayar 2020: shin zai yiwu a yi rashin uwar garken bayanai kuma? Kowa ya ji labarin baya-bayan da ba shi da uwar garken.
Wannan yana da ban mamaki, saboda ma'ajin bayanai sabis ne na jiha, bai dace da ababen more rayuwa mara sabar ba. A lokaci guda, yanayin bayanan yana da girma sosai: gigabytes, terabytes, kuma a cikin ma'ajin bayanai har ma da petabytes. Ba shi da sauƙi a ɗaga shi a cikin kwantena Docker masu nauyi.
A gefe guda, kusan dukkanin ɗakunan bayanai na zamani sun ƙunshi adadi mai yawa na dabaru da abubuwan haɗin gwiwa: ma'amaloli, daidaita daidaito, hanyoyin, dogaro da alaƙa da dabaru da yawa. Don cikakkun dabaru na bayanan bayanai, ƙaramin jihar ya isa. Gigabytes da Terabytes ana amfani da su kai tsaye ta hanyar ƙaramin yanki na dabaru na bayanan da ke cikin aiwatar da tambayoyin kai tsaye.
Saboda haka, ra'ayin shine: idan wani ɓangare na tunani ya ba da izinin aiwatar da kisa ba tare da ƙasa ba, me zai hana a raba tushe zuwa sassa na Jiha da marasa Jiha.
Marasa uwar garke don mafita na OLAP
Bari mu ga yadda yanke bayanai zuwa sassa na Jiha da marasa Jiha zai yi kama da amfani da misalai masu amfani.
Misali, muna da bayanan nazari: bayanan waje (jan silinda a hagu), tsarin ETL wanda ke loda bayanai a cikin ma'ajin bayanai, da kuma manazarci wanda ke aika tambayoyin SQL zuwa ma'adanin. Wannan tsari ne na aikin ajiyar bayanai na gargajiya.
A cikin wannan makirci, ETL ana yin shi ta sharadi sau ɗaya. Sannan kuna buƙatar biyan kuɗi akai-akai don sabar da ma'aunin bayanai ke gudana tare da bayanan da ke cike da ETL, ta yadda za a sami abin da za a aika da tambaya.
Bari mu kalli wata hanyar da aka aiwatar a cikin AWS Athena Serverless. Babu kayan aikin da aka keɓe na dindindin wanda aka adana bayanan da aka sauke akansa. Maimakon wannan:
- Mai amfani ya ƙaddamar da tambayar SQL zuwa Athena. Mai ingantawa Athena yana nazarin tambayar SQL kuma yana bincika ma'ajin metadata (Metadata) don takamaiman bayanan da ake buƙata don aiwatar da tambayar.
- Mai ingantawa, dangane da bayanan da aka tattara, yana zazzage mahimman bayanai daga kafofin waje zuwa ma'ajiyar wucin gadi (bayanan bayanai na wucin gadi).
- Ana aiwatar da tambayar SQL daga mai amfani a cikin ma'ajiya ta wucin gadi kuma ana mayar da sakamakon ga mai amfani.
- An share maajiyar ɗan lokaci kuma an fitar da albarkatu.
A cikin wannan gine-gine, muna biya kawai don tsarin aiwatar da buƙatar. Babu buƙatu - babu farashi.
Wannan tsarin aiki ne kuma ana aiwatar da shi ba kawai a cikin Athena Serverless ba, har ma a cikin Redshift Spectrum (a cikin AWS).
Misalin Athena ya nuna cewa Databaseless Server yana aiki akan tambayoyi na gaske tare da dubun da ɗaruruwan Terabytes na bayanai. Daruruwan Terabytes za su buƙaci ɗaruruwan sabobin, amma ba dole ne mu biya su ba - muna biyan buƙatun. Gudun kowane buƙatu yana da ƙananan (sosai) idan aka kwatanta da ƙwararrun bayanan ƙididdiga kamar Vertica, amma ba ma biyan kuɗi don lokutan raguwa.
Irin wannan ma'ajin bayanai yana aiki don tambayoyin ad-hoc na nazari da ba kasafai ba. Misali, lokacin da muka yanke shawarar gwada hasashe akan wasu manyan adadin bayanai. Athena cikakke ne ga waɗannan lokuta. Don buƙatun yau da kullun, irin wannan tsarin yana da tsada. A wannan yanayin, cache bayanan a cikin wasu mafita na musamman.
Marasa uwar garke don mafita na OLTP
Misalin da ya gabata ya kalli ayyukan OLAP (nazari). Yanzu bari mu kalli ayyukan OLTP.
Bari mu yi tunanin PostgreSQL ko MySQL mai daidaitawa. Bari mu ɗaga misali na yau da kullun na PostgreSQL ko MySQL tare da albarkatun ƙasa kaɗan. Lokacin da misalin ya sami ƙarin kaya, za mu haɗa ƙarin kwafi waɗanda za mu rarraba ɓangaren nauyin karatun zuwa gare su. Idan babu buƙatu ko kaya, muna kashe kwafin. Misali na farko shine maigidan, sauran kuma kwafi ne.
Ana aiwatar da wannan ra'ayin a cikin rumbun adana bayanai da ake kira Aurora Serverless AWS. Ƙa'idar mai sauƙi ce: buƙatun daga aikace-aikacen waje ana karɓar su ta hanyar wakilai. Ganin karuwar lodi, yana keɓance albarkatun ƙididdiga daga ƙananan yanayin da aka riga aka yi dumi - ana haɗa haɗin da sauri. Abubuwan kashewa suna faruwa ta hanya ɗaya.
A cikin Aurora akwai manufar Aurora Capacity Unit, ACU. Wannan shi ne (sharadi) misali (server). Kowane takamaiman ACU na iya zama maigida ko bawa. Kowacce Unit yana da nasa RAM, processor da ƙaramin faifai. Saboda haka, daya ne master, sauran ana karanta kawai kwafi.
Adadin waɗannan Raka'o'in Ƙarfin Aurora da ke gudana siga ce mai daidaitawa. Matsakaicin adadin zai iya zama ɗaya ko sifili (a wannan yanayin, bayanan ba ya aiki idan babu buƙatun).
Lokacin da tushe ya karɓi buƙatun, rundunar wakilai ta ɗaga Aurora CapacityUnits, yana haɓaka albarkatun aikin tsarin. Ƙarfin haɓakawa da rage albarkatu yana ba da damar tsarin don "juggle" albarkatun: nuna ACUs guda ɗaya ta atomatik (maye gurbin su da sababbi) kuma ya fitar da duk abubuwan sabuntawa na yanzu zuwa albarkatun da aka cire.
Aurora Serverless tushe na iya auna nauyin karatun. Amma takardun ba su faɗi wannan kai tsaye ba. Yana iya jin kamar za su iya ɗaga mashawarci da yawa. Babu sihiri.
Wannan rumbun adana bayanai ya dace sosai don gujewa kashe makudan kudade akan tsarin tare da samun damar da ba za a iya faɗi ba. Alal misali, lokacin ƙirƙirar MVP ko shafukan katin kasuwanci na tallace-tallace, yawanci ba ma tsammanin kaya mai tsayi. Dangane da haka, idan babu damar shiga, ba mu biya ga misalai. Lokacin da nauyin da ba zato ba tsammani ya faru, misali bayan taron taro ko tallan tallace-tallace, taron jama'a sun ziyarci shafin kuma nauyin yana ƙaruwa sosai, Aurora Serverless yana ɗaukar wannan nauyin ta atomatik kuma ya haɗa albarkatun da suka ɓace (ACU). Sa'an nan taron ya wuce, kowa ya manta game da samfurin, sabobin (ACU) sun yi duhu, kuma farashin ya ragu zuwa sifili - dace.
Wannan bayani bai dace da tsayin daka mai tsayi ba saboda baya auna nauyin rubutu. Duk waɗannan haɗin gwiwa da katsewar albarkatu suna faruwa ne a abin da ake kira "ma'aunin sikelin" - lokaci a lokacin da ma'amaloli ko tebur na wucin gadi ba su goyan bayan bayanan. Alal misali, a cikin mako guda ma'aunin ma'auni bazai faru ba, kuma tushe yana aiki akan albarkatun guda ɗaya kuma kawai ba zai iya fadadawa ko kwangila ba.
Babu sihiri - PostgreSQL ne na yau da kullun. Amma tsarin ƙara injina da kuma cire haɗin su yana aiki ne da wani bangare.
Marasa uwar garke ta ƙira
Aurora Serverless tsohon rumbun adana bayanai ne da aka sake rubutawa don girgijen don cin gajiyar wasu fa'idodin Serverless. Kuma yanzu zan gaya muku game da tushe, wanda aka rubuta da farko don gajimare, don tsarin uwar garke - Serverless-by-design. Nan da nan an haɓaka shi ba tare da tsammanin cewa zai gudana akan sabobin jiki ba.
Ana kiran wannan tushe Snowflake. Yana da maɓalli guda uku.
Na farko shine toshe metadata. Wannan sabis ɗin ƙwaƙwalwar ajiya ce mai sauri wanda ke warware batutuwa tare da tsaro, metadata, ma'amaloli, da haɓaka tambaya (wanda aka nuna a cikin hoton hagu).
Toshe na biyu shine saitin gungu na kwamfuta mai kama-da-wane don ƙididdigewa (a cikin misalin akwai saitin da'ira mai shuɗi).
Kashi na uku shine tsarin adana bayanai akan S3. S3 shine ma'ajin abu mara girma a cikin AWS, nau'in Dropbox mara girma don kasuwanci.
Bari mu ga yadda Snowflake ke aiki, muna ɗaukan farawa mai sanyi. Wato akwai rumbun adana bayanai, ana loda bayanan a cikinsa, babu tambayoyin da ke gudana. Saboda haka, idan babu buƙatun zuwa bayanan bayanai, to mun ɗaga sabis ɗin Metadata na ƙwaƙwalwar ajiya mai sauri (tushe na farko). Kuma muna da ajiyar S3, inda ake adana bayanan tebur, zuwa abin da ake kira micropartitions. Don sauƙi: idan tebur ya ƙunshi ma'amaloli, to, micropartitions sune kwanakin ma'amaloli. Kowace rana keɓaɓɓen micropartition, fayil ɗin daban. Kuma lokacin da ma'ajin bayanai ke aiki a wannan yanayin, kawai kuna biyan kuɗin sararin da bayanan ke ciki. Bugu da ƙari, ƙimar kowane wurin zama yana da ƙasa sosai (musamman la'akari da mahimmancin matsawa). Sabis ɗin metadata kuma yana aiki koyaushe, amma ba kwa buƙatar albarkatu masu yawa don haɓaka tambayoyin, kuma ana iya ɗaukar sabis ɗin shareware.
Yanzu bari mu yi tunanin cewa mai amfani ya zo wurin bayanan mu kuma ya aika da tambayar SQL. Ana aika tambayar SQL nan take zuwa sabis na Metadata don sarrafawa. Saboda haka, lokacin karɓar buƙatu, wannan sabis ɗin yana nazarin buƙatar, bayanan da ake da su, izinin mai amfani kuma, idan komai ya yi kyau, zana tsari don sarrafa buƙatar.
Bayan haka, sabis ɗin yana fara ƙaddamar da gungu na kwamfuta. Tarin kwamfuta wani gungu ne na sabobin da ke yin lissafi. Wato wannan cluster ne wanda zai iya ƙunsar uwar garken 1, sabar 2, 4, 8, 16, 32 - gwargwadon yadda kuke so. Kuna jefa buƙatu kuma ƙaddamar da wannan tari nan da nan ya fara. Yana ɗaukar daƙiƙa sosai.
Bayan haka, bayan gungu ya fara, ƙananan ɓangarorin da ake buƙata don aiwatar da buƙatarku za su fara kwafi cikin gungu daga S3. Wato, bari mu yi tunanin cewa don aiwatar da tambayar SQL kuna buƙatar ɓangarori biyu daga tebur ɗaya kuma ɗaya daga na biyu. A wannan yanayin, kawai sassa uku masu mahimmanci za a kwafi zuwa gungu, kuma ba duka tebur ɗin gaba ɗaya ba. Abin da ya sa, kuma daidai saboda duk abin da ke cikin cibiyar bayanai guda ɗaya kuma an haɗa shi ta hanyar tashoshi masu sauri, duk tsarin canja wurin yana faruwa da sauri: a cikin daƙiƙa, da wuya a cikin mintuna, sai dai idan muna magana ne game da wasu buƙatun ban tsoro. Don haka, ana kwafin ƙananan ɓangarorin zuwa gungu na kwamfuta, kuma, bayan kammalawa, ana aiwatar da tambayar SQL akan wannan gungu na kwamfuta. Sakamakon wannan buƙatar na iya zama layi ɗaya, layi ɗaya ko tebur - ana aika su a waje ga mai amfani don ya sauke shi, ya nuna shi a cikin kayan aikin BI, ko amfani da shi ta wata hanya dabam.
Kowace tambaya ta SQL ba za ta iya karanta tarawa kawai daga bayanan da aka ɗora a baya ba, amma kuma za ta iya ɗaukar / ƙirƙira sabbin bayanai a cikin bayanan. Wato, yana iya zama tambaya, alal misali, shigar da sabbin bayanai a cikin wani tebur, wanda ke haifar da bayyanar sabon bangare akan cluster na kwamfuta, wanda, bi da bi, ana adana ta atomatik a cikin ma'ajin S3 guda ɗaya.
Yanayin da aka bayyana a sama, daga zuwan mai amfani zuwa haɓaka tari, ɗora bayanai, aiwatar da tambayoyi, samun sakamako, ana biyan kuɗi na tsawon mintuna na yin amfani da gungu na ƙira mai ƙima, rumbun ajiya. Adadin ya bambanta dangane da yankin AWS da girman tari, amma a matsakaita 'yan daloli ne a cikin awa ɗaya. Rukunin injuna guda hudu ya ninka na injuna biyu, kuma gungu na injuna takwas yana da ninki biyu. Zaɓuɓɓuka na injuna 16, 32 suna samuwa, dangane da rikitaccen buƙatun. Amma kuna biyan waɗannan mintuna ne kawai lokacin da gungu ke gudana a zahiri, saboda lokacin da babu buƙatu, kuna ɗaukar hannun ku, kuma bayan mintuna 5-10 na jira (daidaitaccen sigina) zai tafi da kansa. 'yantar da albarkatun kuma ku zama 'yanci.
Halin da ya dace gaba ɗaya shine lokacin da kuka aika buƙatu, gungu ya tashi, in mun gwada da magana, a cikin minti ɗaya, yana ƙidaya minti ɗaya, sannan mintuna biyar don rufewa, kuma kun ƙare biyan kuɗi na mintuna bakwai na aiki na wannan rukunin, kuma ba na watanni da shekaru ba.
Halin farko da aka kwatanta ta amfani da Snowflake a cikin saitin mai amfani guda ɗaya. Yanzu bari mu yi tunanin cewa akwai masu amfani da yawa, wanda ya fi kusa da ainihin yanayin.
Bari mu ce muna da manazarta da yawa da rahoton Tableau waɗanda koyaushe suna yin boma-bomai tare da ɗimbin tambayoyin SQL masu sauƙi na nazari.
Bugu da ƙari, bari mu ce muna da ƙwararrun Masana Kimiyya na Data waɗanda ke ƙoƙarin yin abubuwa masu ban mamaki da bayanai, suna aiki da dubun Terabyte, suna nazarin biliyoyin da tiriliyan na layuka na bayanai.
Don nau'ikan nauyin aiki guda biyu da aka kwatanta a sama, Snowflake yana ba ku damar haɓaka gungu na kwamfuta masu zaman kansu da yawa na iyakoki daban-daban. Haka kuma, waɗannan gungu na kwamfuta suna aiki da kansu, amma tare da daidaitattun bayanai gama gari.
Don ɗimbin tambayoyin haske, zaku iya ɗaga ƙananan gungu 2-3, kusan injuna 2 kowanne. Ana iya aiwatar da wannan hali, a tsakanin wasu abubuwa, ta amfani da saitunan atomatik. Don haka sai ku ce, “Gwargwadon dusar ƙanƙara, ɗaga ƙaramin gungu. Idan nauyin da ke kan sa ya ƙaru sama da wani ma'auni, ɗaga irin wannan na biyu, na uku. Lokacin da kaya ya fara raguwa, kashe abin da ya wuce gona da iri. Ta yadda komi nawa manazarta suka zo su fara duba rahotanni, kowa yana da isassun kayan aiki.
A lokaci guda, idan manazarta suna barci kuma babu wanda ya kalli rahotannin, rukunin na iya yin duhu gaba ɗaya, kuma ku daina biyan su.
A lokaci guda, don tambayoyi masu nauyi (daga masana kimiyyar bayanai), zaku iya haɓaka babban gungu guda ɗaya don injuna 32. Hakanan za'a biya wannan gungu na waɗannan mintuna da sa'o'i lokacin da babbar buƙatarku ke gudana a can.
Damar da aka bayyana a sama tana ba ku damar rarraba ba kawai 2 ba, har ma da ƙarin nau'ikan nauyin aiki zuwa gungu (ETL, saka idanu, samar da rahoto, ...).
Bari mu taƙaita Snowflake. Tushen ya haɗu da kyakkyawan ra'ayi da aiwatarwa mai aiki. A ManyChat, muna amfani da Snowflake don tantance duk bayanan da muke da su. Ba mu da gungu uku, kamar a cikin misali, amma daga 5 zuwa 9, masu girma dabam. Muna da na'ura 16 na al'ada, na'ura 2, da kuma manyan ƙananan na'urori 1 don wasu ayyuka. Sun yi nasarar rarraba kaya kuma sun ba mu damar adana da yawa.
Rubutun bayanai sun yi nasarar daidaita nauyin karatu da rubutu. Wannan babban bambanci ne da babban ci gaba idan aka kwatanta da "Aurora", wanda kawai ya ɗauki nauyin karatun. Snowflake yana ba ku damar haɓaka aikin rubutunku tare da waɗannan gungu na kwamfuta. Wato, kamar yadda na ambata, muna amfani da gungu da yawa a cikin ManyChat, ƙanana da manyan ƙananan gungun ana amfani da su don ETL, don loda bayanai. Kuma manazarta sun riga sun rayu akan matsakaicin gungu, waɗanda nauyin ETL bai shafe su ba, don haka suna aiki da sauri.
Saboda haka, bayanan bayanan sun dace sosai don ayyukan OLAP. Koyaya, abin takaici, har yanzu ba a aiwatar da nauyin aikin OLTP ba. Da fari dai, wannan ma'adanin bayanai shine columnar, tare da duk sakamakon da ya biyo baya. Abu na biyu, hanyar da kanta, lokacin da kowane buƙatu, idan ya cancanta, kuna ɗaga gungu na kwamfuta kuma ku cika shi da bayanai, abin takaici, bai yi saurin isa ga lodin OLTP ba. Jiran daƙiƙa don ayyukan OLAP al'ada ne, amma ga ayyukan OLTP ba abin yarda bane; 100 ms zai fi kyau, ko 10 ms zai fi kyau.
Sakamakon
Ma'ajin bayanai mara uwar garken yana yiwuwa ta hanyar rarraba bayanan zuwa sassa marasa Jiha da Jiha. Wataƙila kun lura cewa a cikin duk misalan da ke sama, ɓangaren Jiha shine, in mun gwada da magana, adana ƙananan ɓangarori a cikin S3, kuma Mara Jiha shine ingantawa, aiki tare da metadata, magance matsalolin tsaro waɗanda za a iya ɗaga su azaman sabis marasa nauyi masu zaman kansu.
Ana iya ganin aiwatar da tambayoyin SQL a matsayin sabis na yanayi mai haske wanda zai iya tashi cikin yanayin maras sabar, kamar gungu na kwamfuta na Snowflake, zazzage bayanan da suka dace kawai, aiwatar da tambayar kuma "fita."
Tabbatattun matakan samar da bayanai sun riga sun kasance don amfani, suna aiki. Waɗannan bayanan bayanai marasa sabar sun riga sun shirya don gudanar da ayyukan OLAP. Abin takaici, don ayyukan OLTP ana amfani da su ... tare da nuances, tun da akwai iyakoki. A gefe guda, wannan ragi ne. Amma, a daya bangaren, wannan dama ce. Wataƙila ɗaya daga cikin masu karatu zai sami hanyar yin bayanan OLTP gabaɗaya mara amfani, ba tare da iyakancewar Aurora ba.
Ina fatan kun sami abin ban sha'awa. Mara amfani shine gaba :)
source: www.habr.com