Yandex mupiro kune anotevera dhatabhesi ichaongororwa.
- DzvanyaImba
- Odyssey
- Kudzoreredza kune imwe nguva panguva (WAL-G)
- PostgreSQL (kusanganisira logerrrors, Amcheck, heapcheck)
- Greenplum
Vhidhiyo:
Mhoro nyika! Zita rangu ndiAndrey Borodin. Uye zvandinoita paYandex.Cloud kugadzira dhatabhesi rakavhurika rehukama mune zvido zveYandex.Cloud uye Yandex.Cloud vatengi.
Muhurukuro iyi, tichataura nezve matambudziko akatarisana akavhurika dhatabhesi pachiyero. Nei zvichikosha? Nekuti matambudziko madiki, madiki ayo, seumhutu, anobva aita nzou. Vanokura kana uine masumbu akawanda.
Asi handicho chinhu chikuru. Zvinhu zvisingadaviriki zvinoitika. Zvinhu zvinoitika mune imwe mumiriyoni zviitiko. Uye munzvimbo yegore, iwe unofanirwa kuve wakagadzirira izvo, nekuti zvinhu zvinoshamisa zvinova zvakanyanya kuitika kana chimwe chinhu chiripo pachiyero.
Asi! Chii chakanakira dhatabhesi rakavhurika? Icho chokwadi ndechekuti iwe une mukana wedzidziso wekubata nechero dambudziko. Iwe une kodhi kodhi, iwe une ruzivo rwekuronga. Tinozvibatanidza uye zvinoshanda.
Ndedzipi nzira dziripo mukushanda pane yakavhurika sosi software?
- Iyo yakanyanya kujeka nzira ndeye kushandisa software. Kana iwe ukashandisa maprotocol, kana ukashandisa zviyero, kana ukashandisa mafomati, kana iwe ukanyora mibvunzo mune yakavhurika sosi software, saka unotoitsigira.
- Uri kuita kuti ecosystem yawo ikure. Iwe unoita kuti mukana wekukurumidza kuonekwa kwebug uwedzere. Iwe unowedzera kuvimbika kwegadziriro ino. Iwe unowedzera kuwanikwa kwevagadziri mumusika. Unovandudza iyi software. Iwe watova mubatsiri kana wakangoita up kuwana style uye tinkered nechimwe chinhu ipapo.
- Imwe nzira inonzwisisika ndeyekutsigira yakavhurika sosi software. Semuenzaniso, chirongwa chinozivikanwa cheGoogle Summer of Code, apo Google inobhadhara nhamba huru yevadzidzi vanobva kumativi ose enyika mari inonzwisisika kuitira kuti vagadzire mapurogiramu epurogiramu akazaruka anozadzisa zvimwe zvinodikanwa zverezinesi.
- Iyi inzira inonakidza nekuti inobvumira software kuti ishanduke pasina kushandura kutarisa kure nenharaunda. Google, sehofori yetekinoroji, haitaure kuti isu tinoda chimiro ichi, tinoda kugadzirisa bug uye apa ndipo patinofanira kuchera. Google inoti: βIta zvaunoita. Ingoramba uchishanda nenzira yawanga uchishanda uye zvese zvichanaka. "
- Nzira inotevera yekutora chikamu mune yakavhurika sosi kutora chikamu. Paunenge uine dambudziko mune yakavhurika sosi software uye paine vanogadzira, vagadziri vako vanotanga kugadzirisa matambudziko. Vanotanga kuita kuti zvivakwa zvako zvinyatsoshanda, zvirongwa zvako nekukurumidza uye zvakavimbika.
Imwe yeanonyanya kuzivikanwa mapurojekiti eYandex mumunda wekuvhura sosi software ndeye ClickHouse. Iyi dhatabhesi yakazvarwa semhinduro kumatambudziko akatarisana neYandex.Metrica.
Uye sedhatabhesi, yakagadzirwa munzvimbo yakavhurika kuitira kugadzira ecosystem uye kuivandudza pamwe chete nevamwe vanogadzira (kwete mukati meYandex chete). Uye zvino iyi ipurojekiti hombe umo makambani mazhinji akasiyana anobatanidzwa.
MuYandex.Cloud, takagadzira ClickHouse pamusoro peYandex Object Storage, i.e. pamusoro pekuchengetwa kwegore.
Sei izvi zvakakosha mugore? Nekuti chero dhatabhesi inoshanda mune iyi katatu, mupiramidhi iyi, mune iyi hierarchy yemhando dzendangariro. Une marejista anokurumidza asi madiki uye akachipa akakura asi anononoka maSSD, madhiraivha akaomarara uye mamwe mablock madivayiri. Uye kana iwe uchibudirira kumusoro kwepiramidhi, saka une database yekukurumidza. kana iwe uchibudirira pazasi pepiramidhi iyi, saka une dhatabhesi yakasimudzwa. Uye mune izvi, kuwedzera imwe layer kubva pazasi inzira inonzwisisika yekuwedzera scalability yedatabase.
Zvaigona kuitwa sei? Ichi chinhu chakakosha mumushumo uyu.
- Tinogona kushandisa ClickHouse pamusoro peMDS. MDS ndeyemukati Yandex cloud storage interface. Iyo yakanyanya kuomarara kupfuura yakajairwa S3 protocol, asi yakanyanya kukodzera kune block mudziyo. Zviri nani kurekodha data. Zvinoda mamwe mapurogiramu. Vagadziri vanozoronga, zvakatonaka, zvinonakidza.
- S3 ndiyo nzira yakajairika inoita kuti iyo interface ive nyore pamutengo wekushomeka kune mamwe marudzi emabasa.
Nomuzvarirwo, tichida kupa mashandiro kuClickHouse ecosystem yese uye kuita basa rinodiwa mukati meYandex.Cloud, takasarudza kuve nechokwadi chekuti nharaunda yese yeClickHouse ichabatsirwa nayo. Isu takaita ClickHouse pamusoro peS3, kwete ClickHouse pamusoro peMDS. Uye iri ibasa rakawanda.
Mareferensi:
"Filesystem abstraction layer"
"AWS SDK S3 kubatanidzwa"
"Base kuitiswa kweIDisk interafce yeS3"
"Kubatanidzwa kweinjini dzekuchengetedza log neIDisk interface"
"Log injini rutsigiro rweS3 uye SeekableReadBuffer"
"Kuchengeta Stripe Log S3 rutsigiro"
"Storage MergeTree yekutanga rutsigiro rweS3"
"MergeTree izere rutsigiro rweS3"
"Support ReplicatedMergeTree pamusoro peS3"
"Wedzera default zvitupa uye tsika misoro ye s3 kuchengetedza"
"S3 ine dynamic proxy kumisikidza"
"S3 ine proxy solver"
Iri irondedzero yekudhonza yekukumbira kuita chaiyo faira system muClickHouse. Iyi nhamba huru yekudhonza zvikumbiro.
Mareferensi:
"DiskS3 hardlinks yakakwana kuita"
"S3 HTTP mutengi - Dzivisa kutevedzera rwizi rwemhinduro mundangariro"
"Dzivisa kutevedzera rwizi rwemhinduro yese mundangariro muS3 HTTP
mutengi"
"Kugona cache mark uye index mafaera eS3 disk"
"Fambisa zvikamu kubva kuDiskLocal kuenda kuDiskS3 zvakafanana"
Asi basa harina kuperera ipapo. Mushure mekunge chimiro chaitwa, rimwe basa raidiwa kuti rikwanise kuita basa iri.
Mareferensi:
"Wedzera SelectedRows uye SelectedBytes zviitiko"
"Wedzera zviitiko zveprofile kubva kuS3 chikumbiro kune system.events"
"Wedzera QueryTimeMicroseconds, SelectQueryTimeMicroseconds uye InsertQueryTimeMicroseconds"
Uye zvadaro zvaive zvakafanira kuita kuti zvionekwe, kumisikidza yekutarisa uye kuita kuti igoneke.
Uye zvese izvi zvakaitwa kuti nharaunda yese, iyo yese ClickHouse ecosystem, igamuchire mhedzisiro yebasa iri.
Ngatienderere mberi kune zvekutengeserana dhatabhesi, kune OLTP dhatabhesi, ari padyo neni pachangu.
Iyi ndiyo yakavhurika sosi DBMS yekuvandudza chikamu. Ava vakomana vari kuita mashiripiti emumugwagwa kuti vavandudze transactional open database.
Imwe yemapurojekiti, tichishandisa muenzaniso watinogona kutaura nezve maitiro uye zvatinoita, ndeye Connection Pooler muPostgres.
Postgres inzvimbo yedatabase. Izvi zvinoreva kuti dhatabhesi inofanirwa kunge iine mashoma network yekubatanidza sezvinobvira inobata matransaction.
Kune rimwe divi, munzvimbo yemakore, mamiriro akajairika ndeye apo churu chekubatanidza chinouya kune rimwe sumbu kamwechete. Uye basa rekubatanidza pooler ndere kurongedza churu chekubatanidza muhuwandu hudiki hwesevha yekubatanidza.
Isu tinogona kutaura kuti yekubatanidza pooler ndiye nhare opareta anorongazve mabheti kuitira kuti asvike padhatabhesi.
Nehurombo, hapana izwi rakanaka reRussia rekubatanidza pooler. Dzimwe nguva inonzi multiplexer connections. Kana iwe uchiziva zvekudaidza iyo yekubatanidza pooler, saka ita chokwadi chekundiudza, ini ndichafara kwazvo kutaura iyo chaiyo Russian hunyanzvi mutauro.
Isu takaongorora mapoinzi ekubatanidza ayo aive akakodzera kune inotungamirwa postgres cluster. Uye PgBouncer yaive sarudzo yakanakira isu. Asi takasangana nematambudziko akati wandei nePgBouncer. Makore mazhinji apfuura, Volodya Borodin akapa mishumo yekuti tinoshandisa PgBouncer, tinoda zvese, asi pane nuances, pane chimwe chinhu chekushanda.
Uye takashanda. Isu takagadzirisa matambudziko atakasangana nawo, takabata Bouncer, uye kuyedza kusundira zvikumbiro zvekudhonza kumusoro. Asi yakakosha tambo imwe chete yaive yakaoma kushanda nayo.
Taifanira kuunganidza macascade kubva kumaBouncer ane zvigamba. Kana isu tine akawanda-akarukwa maBouncer, zvinongedzo zviri pamusoro-soro zvinotamirwa kune yemukati layer yeBouncers. Iyi ihurongwa husina kufambiswa zvakanaka iyo yakaoma kuvaka uye kuyera kumashure nekudzoka.
Takasvika pakugumisa kuti takagadzira yedu yekubatanidza pooler, inonzi Odyssey. Takazvinyora kubva pakutanga.
Muna 2019, pamusangano wePgCon, ndakapa iyi pooler kunharaunda yekuvandudza. Iye zvino tine zvishoma zvishoma pane 2 nyeredzi paGitHub, i.e. chirongwa chipenyu, chirongwa chacho chakakurumbira.
Uye kana iwe ukagadzira Postgres cluster muYandex.Cloud, zvino ichave sumbu ine yakavakirwa-muOdyssey, iyo inogadziridzwa zvakare pakuyera sumbu kumashure kana mberi.
Takadzidzei pachirongwa ichi? Kutangisa purojekiti inokwikwidza inogara iri nhanho ine hukasha, inhanho yakanyanyisa kana tichiti kune matambudziko asiri kugadziriswa nekukurumidza zvakakwana, asiri kugadziriswa munguva dzenguva dzinoenderana nesu. Asi ichi chiyero chinoshanda.
PgBouncer yakatanga kukura nekukurumidza.
Uye zvino mamwe mapurojekiti akaonekwa. Semuenzaniso, pgagroal, iyo inogadzirwa nevagadziri veRed Hat. Vanotevera zvinangwa zvakafanana uye vanoshandisa mazano akafanana, asi, hongu, neavo chaiwo, ari padyo nevagadziri vepgagroal.
Imwe nyaya yekushanda nenharaunda yepostgres iri kudzorera kune imwe nguva nenguva. Uku kupora mushure mekutadza, uku kupora kubva kune backup.
Kune akawanda backups uye ese akasiyana. Anenge wese mutengesi wePostgres ane mhinduro yake yekuchengetedza.
Kana iwe ukatora ese backup masisitimu, gadzira chimiro matrix uye nejee kuverenga chinotemesa mune iyi matrix, ichave zero. Izvi zvinorevei? Ko kana iwe ukatora chaiyo backup faira, saka haigone kuunganidzwa kubva kuzvidimbu zvevamwe vese. Iyo yakasiyana mukushandiswa kwayo, yakasiyana muchinangwa chayo, yakasiyana mune mazano akaiswa mairi. Uye dzose dzakananga.
Patakanga tichishanda panyaya iyi, CitusData yakatanga chirongwa cheWAL-G. Iyi igadziriro yekuchengetedza iyo yakagadzirwa neziso kune nharaunda yegore. Iye zvino CitusData yatove chikamu cheMicrosoft. Uye panguva iyoyo, isu takanyanya kufarira mazano akaiswa pasi mukutanga kuburitswa kweWAL-G. Uye isu takatanga kupa mupurojekiti iyi.
Ikozvino kune akawanda akawanda evagadziri muchirongwa ichi, asi vepamusoro gumi vanopa WAL-G vanosanganisira 10 Yandexoids. Takaunza pfungwa dzedu dzakawanda ikoko. Uye, hongu, takazvishandisa isu pachedu, takazviedza isu pachedu, takazvitenderedza mukugadzira isu pachedu, isu tinozvishandisa isu pachedu, isu pachedu tinoona kwatingaendera mberi, tichidyidzana nenharaunda huru yeWAL-G.
Uye kubva pamaonero edu, ikozvino iyi backup system, kusanganisira kufunga nezve kuedza kwedu, yave yakanakira nzvimbo yegore. Iyi ndiyo yakanakisa mutengo wekutsigira Postgres mugore.
Zvinorevei? Isu taisimudzira zano rakakura: backup inofanirwa kuve yakachengeteka, yakachipa kushanda uye nekukurumidza sezvinobvira kudzoreredza.
Nei zvichifanira kuva zvakachipa kushanda? Kana pasina chakaputswa, haufanirwe kuziva kuti une backups. Zvose zvinoshanda zvakanaka, unoparadza seCPU shoma sezvinobvira, unoshandisa zvishoma zvedhisiki yako zviwanikwa sezvinobvira, uye unotumira semabheti mashoma kunetiweki sezvinobvira kuti usakanganise nemubhadharo webasa rako rakakosha.
Uye kana zvese zvaputsika, semuenzaniso, maneja akadonhedza data, chimwe chinhu chakashata, uye iwe unofanirwa kukurumidza kudzokera kune yakapfuura, unopora nemari yese, nekuti iwe unoda kuti data rako ridzoke nekukurumidza uye rakasimba.
Uye isu takasimudzira iri nyore zano. Uye, zvinoratidzika kwatiri, takakwanisa kuzviita.
Asi handizvo zvoga. Taida chimwezve chinhu chidiki. Taida akawanda akasiyana dhatabhesi. Havasi vese vatengi vedu vanoshandisa Postgres. Vamwe vanhu vanoshandisa MySQL, MongoDB. Munharaunda, vamwe vanogadzira vakatsigira FoundationDB. Uye rondedzero iyi iri kuramba ichiwedzera.
Nharaunda inoda zano reiyo dhatabhesi iri kuitiswa munzvimbo inochengetedzwa mugore. Uye vanogadzira vanochengetedza dhatabhesi yavo, iyo inogona kutsigirwa zvakafanana pamwe chete nePostgres neyedu backup system.
Tadzidzei kubva munyaya iyi? Chigadzirwa chedu, sechikamu chekusimudzira, hachisi mitsara yekodhi, haisi zvirevo, haisi mafaera. Chigadzirwa chedu hachisi chekudhonza zvikumbiro. Aya ndiwo mazano atinosvitsa kunharaunda. Uhwu hunyanzvi hwetekinoroji uye kufamba kwehunyanzvi kuenda kunzvimbo yemakore.
Kune dhatabhesi yakadai sePostgres. Ini ndinoda iyo Postgres musimboti zvakanyanya. Ini ndinopedza nguva yakawanda ndichigadzira iyo Postgres musimboti nenharaunda.
Asi pano panofanira kutaurwa kuti Yandex.Cloud ine yekumisikidzwa kwemukati kwemadatabase akachengetwa. Uye yakatanga kare kare muYandex.Mail. Hunyanzvi hwave kuita kuti Postgres igadziriswe yakaunganidzwa apo tsamba yaida kuenda kuPostgres.
Tsamba ine zvinodiwa zvakafanana kune gore. Zvinoda kuti iwe ugone kukwira kune isingatarisirwe exponential kukura chero nguva mune yako data. Uye iyo tsamba yaive yatove nemutoro nemamwe mazana emamiriyoni emabhokisi emabhokisi ehuwandu hukuru hwevashandisi vanogara vachiita zvikumbiro zvakawanda.
Uye iri raive dambudziko rakakura kuchikwata chaigadzira Postgres. Kare ikako, chero matambudziko atakasangana nawo aishumwa kunharaunda. Uye matambudziko aya akagadziriswa, uye akagadziriswa nenharaunda mune dzimwe nzvimbo kunyangwe padanho rekubhadharwa kwerutsigiro kune mamwe dhatabhesi uye zvirinani. Ndiko kuti, unogona kutumira tsamba kuPgSQL hacker uye ugamuchire mhinduro mukati memaminitsi makumi mana. Rutsigiro rwakabhadharwa mune mamwe dhatabhesi inogona kufunga kuti pane zvinhu zvinonyanya kukosha kupfuura bug yako.
Ikozvino kuiswa kwemukati kwePostgres kune mamwe mapetabytes edata. Aya mamwe mamiriyoni ezvikumbiro pasekondi. Izvi zviuru zvemasumbu. Yakakura zvikuru-mwero.
Asi pane nuance. Iyo hairarami pane fancy network madhiraivha, asi pane zviri nyore hardware. Uye kune nzvimbo yekuyedza yakanangana nezvinhu zvitsva zvinonakidza.
Uye pane imwe nguva munzvimbo yekuyedzwa takagamuchira meseji inoratidza kuti mukati memukati meiyo database indexes yakatyorwa.
Chisingachinji rudzi rwehukama hwatinotarisira kugara takabata.
Mamiriro ezvinhu akaoma zvikuru kwatiri. Zvinoratidza kuti imwe data inogona kunge yakarasika. Uye kurasikirwa nedata chimwe chinhu chakashata.
Pfungwa yakajairika yatinotevera mumadhatabhesi akachengetwa ndeyekuti kunyangwe nekuedza, zvichanetsa kurasikirwa nedata. Kunyangwe ukadzibvisa nemaune, iwe uchazofanira kufuratira kusavapo kwavo kwenguva yakareba. Kuchengetedzwa kwedata chitendero chatinotevera nemoyo wose.
Uye pano panomuka mamiriro ezvinhu anoratidza kuti pangava nemamiriro ezvinhu atingave tisina kugadzirira. Uye takatanga kugadzirira mamiriro ezvinhu aya.
Chinhu chekutanga chatakaita kuviga matanda kubva muzviuru zvemasumbu aya. Isu takawana kuti ndeapi emasumbu aive ari pamadisiki ane dambudziko firmware aive achirasikirwa nedata peji inogadziridza. Yakamaka ese maPostgres data kodhi. Uye isu takamaka iwo mameseji anoratidza kutyorwa kwemukati invariants nekodhi iyo yakagadzirirwa kuona huwori hwe data.
Ichi chigamba chakagamuchirwa nenharaunda pasina hurukuro yakawanda, nokuti muchiitiko chega chega zvaive pachena kuti pane chakaipa chakanga chaitika uye chaida kuudzwa kurogi.
Mushure meizvi, takasvika padanho rekuti tine monitoring inoongorora matanda. Uye kana ari mashoko anonyumwira, anomutsa mukuru webasa, uye mukuru webasa anorigadzira.
Asi! Kuongorora matanda ibasa rakachipa pane rimwe sumbu uye rinodhura zvine njodzi kumasumbu ane chiuru.
Takanyora kuwedzera kunonzi . Iyo inogadzira maonero eiyo dhatabhesi maunogona zvakachipa uye nekukurumidza kusarudza zviverengero pane zvakambokanganisa. Uye kana tichida kumutsa mukuru webasa, ipapo tichazoziva pamusoro peizvi pasina kuongorora mafaira egigabyte, asi nekubvisa mabheti mashomanana kubva patafura yehashi.
Iyi yekuwedzera yakagamuchirwa, semuenzaniso, mune repository ye . Kana iwe uchida kuishandisa, unogona kuiisa iwe pachako. Chokwadi inzvimbo yakavhurika.
Asi handizvo zvoga. Takatanga kushandisa Amcheck, yekuwedzera-yakavakirwa nharaunda, kuti tiwane kutyora kusingaite muma indexes.
Uye takaona kuti kana ukaishandisa pachiyero, kune tsikidzi. Takatanga kuvagadzirisa. Kururamisa kwedu kwakagamuchirwa.
Isu takaona kuti iyi yekuwedzera haigone kuongorora GiST & GIT indexes. Takavaita kuti vatsigire. Asi tsigiro iyi ichiri kukurukurwa nenharaunda, nekuti iyi ibasa idzva uye kune zvakawanda zvakawanda ipapo.
Uye isu takaonawo kuti kana tichitarisa ma indexes ekutyorwa kwemutungamiri wekudzokorora, pane tenzi, zvese zvinoshanda nemazvo, asi pane replicas, pamuteveri, kutsvaga huwori hakuna kunyatso. Havasi vese vasingachinji vanotariswa. Uye mumwe asingachinji akatinetsa zvikuru. Uye takaita gore nechidimbu tichitaurirana nenharaunda kuitira kuti cheki iyi igone kutevedzera.
Takanyora kodeti inofanira kutevera zvose zvinogona ... zvibvumirano. Takakurukura chigamba ichi kwenguva yakati rebei naPeter Gaghan anobva kuCrunchy Data. Aifanira kushandura zvishoma iripo B-muti muPostgres kuti agamuchire chigamba ichi. Akagamuchirwa. Uye ikozvino kutarisa indexes pane replicas zvakare kwave kushanda zvakakwana kuti tione kutyorwa kwatakasangana nako. Kureva, uku ndiko kukanganisa kunogona kukonzerwa nezvikanganiso mu disk firmware, tsikidzi muPostgres, tsikidzi muLinux kernel, uye matambudziko ehardware. Runyoro rwakadzama rwemanyuko ezvinetso zvatakanga tichigadzirira.
Asi kunze kwema indexes, kune chikamu chakadai semurwi, i.e. nzvimbo inochengeterwa data. Uye hapana akawanda akasiyana-siyana anogona kuongororwa.
Tine yekuwedzera inonzi Heapcheck. Takatanga kuigadzira. Uye zvakafanana, pamwe nesu, iyo EnterpriseDB kambani yakatangawo kunyora module, yavakadaidza kuti Heapcheck nenzira imwecheteyo. Isu chete isu takaidaidza kuti PgHeapcheck, uye ivo vakangoidaidza kuti Heapcheck. Ivo vanayo nemabasa akafanana, siginicha yakati siyanei, asi ine pfungwa dzakafanana. Vakazvishandisa zviri nani mune dzimwe nzvimbo. Uye vakaiisa mune yakavhurika sosi kare.
Uye ikozvino tiri kukudziridza kuwedzera kwavo, nokuti haisisiri kuwedzera kwavo, asi kuwedzera kwenharaunda. Uye mune ramangwana, ichi chikamu chekernel chinozopihwa kune wese munhu kuti vazive nezve ramangwana rematambudziko pamberi.
Mune dzimwe nzvimbo, takatosvika pakufunga kuti tine manyepo muhurongwa hwedu hwekutarisisa. Semuenzaniso, iyo 1C system. Paunenge uchishandisa dhatabhesi, Postgres dzimwe nguva inonyora dhata mairi kuti inogona kuverenga, asi pg_dump haigone kuverenga.
Mamiriro ezvinhu aya aiita sehuwori kuhurongwa hwedu hwekuona dambudziko. Mupurisa webasa akabva amutswa. Mukuru webasa akatarisa zvaiitika. Kwapera nguva yakati, mumwe munhu aiuya kuzondishandira akauya akati ndakanga ndine dambudziko. Mushandi akatsanangura dambudziko raivapo. Asi dambudziko riri muPostgres musimboti.
Ndakawana hurukuro pamusoro pechinhu ichi. Uye akanyora kuti takasangana nechinhu ichi uye chakanga chisingafadzi, munhu aimuka usiku kuti aone kuti chii.
Nharaunda yakapindura, "Ah, isu tinofanirwa kuzvigadzirisa."
Ndine fananidzo iri nyore. Kana iwe uchifamba neshangu ine tsanga yejecha mairi, saka, mumutemo, iwe unogona kuenderera mberi - hapana dambudziko. Kana iwe uchitengesa bhutsu kuzviuru zvevanhu, saka ngatigadzire mabhutsu asina jecha zvachose. Uye kana mumwe wevashandisi veshangu dzako achizomhanya marathon, saka iwe unoda kugadzira shangu dzakanaka kwazvo, wobva wazviyera kune vese vashandisi vako. Uye vashandisi vasingatarisirwi vakadaro vanogara vari munzvimbo yegore. Kune nguva dzose vashandisi vanoshandisa cluster neimwe nzira yekutanga. Unofanira kugara uchigadzirira izvi.
Tadzidzei pano? Takadzidza chinhu chakapfava: chinhu chinonyanya kukosha kutsanangurira nharaunda kuti pane dambudziko. Kana nharaunda yakaziva dambudziko, ipapo makwikwi echisikigo anomuka kuti agadzirise dambudziko. Nokuti munhu wose anoda kugadzirisa dambudziko rinokosha. Vese vatengesi, vese vanobira vanonzwisisa kuti ivo pachavo vanogona kutsika pane iyi reki, saka ivo vanoda kuvabvisa.
Kana iwe uri kushanda pane dambudziko, asi rinoshungurudza hapana munhu kunze kwako, asi iwe unoshanda pairi zvakarongeka uye rinozoonekwa sedambudziko, ipapo chikumbiro chako chekudhonza chichagamuchirwa. Chigamba chako chichagamuchirwa, kuvandudzwa kwako kana zvikumbiro zvekuvandudzwa zvichaongororwa nenharaunda. Pakupera kwezuva, tinoita dhatabhesi zviri nani kune mumwe nemumwe.
Iyo inonakidza dhatabhesi ndeyeGreenplum. Iro dhatabhesi rakanyanya kufanana rakavakirwa paPostgres codebase, yandinonyatsoziva nayo.
Uye Greenplum ine inonakidza mashandiro - wedzera optimized matafura. Aya ndiwo matafura aunogona kukurumidza kuwedzera. Vanogona kunge vari columnar kana mutsara.
Asi pakanga pasina kubatanidza, kureva kuti pakanga pasina kushanda kwaunogona kuronga data iri mutafura maererano nehurongwa huri mune imwe yezvinyorwa.
Vakomana vomutekisi vakauya kwandiri ndokuti: βAndrey, unoziva Postgres. Uye pano zvinenge zvakafanana. Chinja kumaminitsi makumi maviri. Chitora wozviita.β Ndakafunga kuti hongu, ndinoziva Postgres, kuchinja kwemaminetsi makumi maviri - ndinofanira kuita izvi.
Asi kwete, yakanga isiri maminitsi makumi maviri, ndakainyora kwemwedzi yakawanda. Pamusangano wePgConf.Russia, ndakaenda kuna Heikki Linakangas wokuPivotal ndokumubvunza kuti: βPane matambudziko here neizvi? Nei pasina append yakakwenenzverwa tafura kubatanidza? " Anoti: βUnotora data racho. Unoronga, unoronga patsva. Ibasa chete." Ini : "Ehe, iwe unongoda kuitora wozviita." Anoti: "Ehe, tinoda maoko emahara kuti tiite izvi." Ndakafunga kuti zvechokwadi ndaifanira kuita izvi.
Uye mwedzi mishoma gare gare ndakaendesa chikumbiro chekudhonza icho chakaita basa iri. Ichi chikumbiro chekudhonza chakaongororwa nePivotal pamwe chete nenharaunda. Chokwadi, paiva netsikidzi.
Asi chinonyanya kufadza ndechekuti kana ichi chikumbiro chekudhonza chabatanidzwa, tsikidzi dzakawanikwa muGreenplum pachayo. Isu takaona kuti mirwi matafura dzimwe nguva anotyora transaction kana akaungana. Uye ichi chinhu chinoda kugadziriswa. Uye ari panzvimbo yandangobata. Uye maitiro angu epanyama aive - zvakanaka, regai ndiite izvi zvakare.
Ndakagadzirisa bug iri. Akatumira chikumbiro chekudhonza kune vanogadzirisa. Akaurayiwa.
Mushure mezvo zvakazoitika kuti basa iri rinoda kuwanikwa muGreenplum version yePostgreSQL 12. Ndiko kuti, 20-minute adventure inoenderera mberi nezviitiko zvitsva zvinonakidza. Zvainakidza kubata budiriro yazvino, uko nharaunda iri kucheka zvitsva uye zvakanyanya kukosha maficha. Kwaoma nechando.
Asi hazvina kuperera ipapo. Mushure mezvose, zvakazoitika kuti taifanira kunyora zvinyorwa zvezvose izvi.
Ndakatanga kunyora magwaro. Sezvineiwo, madocumentarians kubva kuPivotal akauya. Chirungu ndiwo mutauro wavo wekuzvarwa. Vakandibatsira nemagwaro. Kutaura zvazviri, ivo pachavo vakanyora patsva zvandakakumbira muchirungu chaicho.
Uye pano, zvinoita sekunge, mutambo wacho wakapera. Uye unoziva here zvakaitika ipapo? Vakomana vomutekisi vakauya kwandiri ndokuti: βPachine nhambo mbiri, imwe neimwe yemaminiti gumi.β Uye ndovaudzei? Ndakati ikozvino ini ndichapa mushumo pachiyero, zvino tichaona adventures yako, nokuti iri ibasa rinonakidza.
Takadzidzei panyaya iyi? Nekuti kushanda neyakavhurika sosi kunogara kuchishanda nemumwe munhu, kunogara kuchishanda nenharaunda. Nekuti padanho rega rega ndakashanda nemumwe mugadziri, mumwe muedzi, mumwe hacker, mumwe documentarian, mumwe mugadziri. Ini handina kushanda neGreenplum, ndaishanda nevanhu vakatenderedza Greenplum.
Asi! Pane imwe pfungwa inokosha - ibasa chete. Ndiko kuti, unouya, inwa kofi, nyora kodhi. Mhando dzose dzezvipingamupinyi zviri nyore zvinoshanda. Ita zvakajairika - zvichave zvakanaka! Uye ibasa rinonakidza. Pane chikumbiro chebasa iri kubva kuYandex.Cloud vatengi, vashandisi vemapoka edu mukati meYandex uye kunze. Uye ndinofunga kuti huwandu hwemapurojekiti atinotora chikamu huchawedzera uye kudzika kwekubatanidzwa kwedu kuchawedzerawo.
Ndizvo zvose. Ngatienderei kumibvunzo.
Chikamu chemibvunzo
Mhoro! Tine chimwe chikamu chemibvunzo nemhinduro. Uye mu studio Andrei Borodin. Uyu ndiye munhu achangokuudza nezve mupiro weYandex.Cloud uye Yandex kuvhura sosi. Chirevo chedu ikozvino hachisi zvachose nezve Cloud, asi panguva imwechete isu takavakirwa pamatekinoroji akadaro. Pasina izvo zvawakaita mukati meYandex, pangadai pasina sevhisi muYandex.Cloud, saka ndinokutendai kubva kwandiri pachangu. Uye mubvunzo wekutanga kubva kunhepfenyuro: "Ndeipi imwe neimwe yemapurojekiti yawakataura yakanyorwa pairi?"
Iyo backup system muWAL-G yakanyorwa muGo. Ichi chimwe chezvirongwa zvitsva zvatakashanda pazviri. Anongova nemakore matatu chete. Uye dhatabhesi kazhinji ndeyekuvimbika. Uye izvi zvinoreva kuti dhatabhesi dzakasakara uye dzinowanzonyorwa muC. Iyo Postgres chirongwa chakatanga makore makumi matatu apfuura. Ipapo iyo C3 yaive sarudzo chaiyo. Uye Postgres yakanyorwa pairi. Mamwe madhatabheti emazuva ano akadai seClickHouse anowanzo kunyorwa muC ++. Yese kuvandudza sisitimu yakavakirwa kutenderedza C uye C ++.
Mubvunzo kubva kuna maneja wedu wezvemari, uyo ane basa rekushandisa paCloud: "Nei Cloud ichishandisa mari kutsigira yakavhurika sosi?"
Pane mhinduro iri nyore kune maneja wezvemari pano. Tinoita izvi kuti masevhisi edu aite nani. Tingaita zviri nani munzira dzipi? Tinogona kuita zvinhu nemazvo, nekukurumidza, uye kuita kuti zvinhu zviwedzere. Asi kwatiri, nyaya iyi inonyanya kutaura nezvekuvimbika. Semuenzaniso, mune backup system tinoongorora 100% yezvigamba zvinoshanda pairi. Isu tinoziva kuti kodhi chii. Uye isu takasununguka kuburitsa mavhezheni matsva mukugadzira. Ndiko kuti, kutanga kwezvose, ndezvekuvimba, nezvekugadzirira kwekusimudzira uye nezvekuvimbika
Mumwe mubvunzo: "Izvo zvinodiwa nevashandisi vekunze vanogara muYandex.Cloud zvakasiyana nevashandisi vemukati vanogara mukati me Cloud?"
Iyo yekutakura profiles, hongu, yakasiyana. Asi kubva pakuona kwedhipatimendi rangu, ese akakosha uye anonakidza makesi anogadzirwa pane isiri-yakajairika mutoro. Vagadziri vane fungidziro, vanogadzira vanoita zvisingatarisirwi, vanogona kunge vachiwanikwa mukati uye kunze. Panyaya iyi, tose takafanana. Uye, zvichida, chinhu chinokosha chete mukati mekushanda kweYandex ye databases ichava iyo mukati meYandex tine dzidziso. Pane imwe nguva, imwe nzvimbo inowanikwa inoenda zvachose mumumvuri, uye ese Yandex masevhisi anofanira neimwe nzira kuramba achishanda kunyangwe izvi. Uyu mutsauko mudiki. Asi inogadzira yakawanda yekuvandudzwa kwekutsvagisa pane interface yedatabase uye network stack. Zvikasadaro, kuisirwa kwekunze nemukati kunoburitsa zvikumbiro zvakafanana zvezvimiro uye zvikumbiro zvakafanana zvekuvandudza kuvimbika uye kuita.
Mubvunzo unotevera: "Iwe pachako unonzwa sei nezve chokwadi chekuti zvakawanda zvaunoita zvinoshandiswa nemamwe Makore?" Hatingatauri mazita chaiwo, asi mapurojekiti mazhinji akaitwa muYandex.Cloud anoshandiswa mumakore evamwe vanhu.
Izvi zvakanaka. Kutanga, chiratidzo chokuti taita chimwe chinhu zvakanaka. Uye zvinokwenya ego. Uye tine chivimbo chakawedzerwa chokuti takaita chisarudzo chakanaka. Kune rumwe rutivi, iyi ndiyo tariro yokuti mune ramangwana izvi zvichatiunzira mazano matsva, zvikumbiro zvitsva kubva kuvashandisi vechitatu. Nyaya zhinji paGitHub dzinogadzirwa nevanoronga masystem ega ega, maDBA ega ega, vagadziri vezvivakwa, mainjiniya ega, asi dzimwe nguva vanhu vane ruzivo rwakarongeka vanouya vachiti mu30% yezvimwe zviitiko tine dambudziko iri uye ngatifungei kuti tingarigadzirisa sei. Izvi ndizvo zvatiri kutarisira zvakanyanya. Tinotarisira kugovera zviitiko nemamwe mapuratifomu emakore.
Wakataura zvakawanda nezve marathon. Ndinoziva kuti wakamhanya marathon muMoscow. Semagumo? Wakatora vakomana kubva kuPostgreSQL?
Kwete, Oleg Bartunov anomhanya zvikuru. Akapedza awa akanditangira. Pakazara, ndinofara nekusvika kwandakaita. Kwandiri, kungopedza kwaive kubudirira. Pakazara, zvinoshamisa kuti kune vakawanda vanomhanya munharaunda yepostgres. Zvinoratidzika kwandiri kuti pane imwe mhando yehukama pakati pemitambo yeaerobic uye chishuwo chehurongwa hwehurongwa.
Iwe urikuti hapana vanomhanya paClickHouse?
Ndinonyatsoziva kuti varipo. ClickHouse zvakare dhatabhesi. Sezvineiwo, Oleg ave kundinyorera kuti: "Tomhanya here mushure memushumo?" Iyi ipfungwa huru.
Mumwe mubvunzo kubva kutepfenyuro kubva kuna Nikita: "Sei wakagadzirisa bug muGreenplum iwe pachako uye usingape kuvadiki?" Ichokwadi, hazvina kunyatsojeka kuti chii chinonzi bug uye mune sevhisi ipi, asi pamwe zvinoreva iyo yawakataura nezvayo.
Hongu, pamusimboti, yaigona kunge yakapihwa kune mumwe munhu. Yaingove code yandakangochinja. Uye zvaiva zvomuzvarirwo kuramba uchizviita pakarepo. Muchokwadi, iyo pfungwa yekugovana hunyanzvi nechikwata ipfungwa yakanaka. Tichazogovera mabasa eGreenplum pakati penhengo dzese dzechikamu chedu.
Sezvo tiri kutaura nezvevadiki, heino mubvunzo. Iye munhu akafunga kugadzira yekutanga kuzvipira muPostgres. Chii chaanofanira kuita kuti aite kuzvipira kwekutanga?
Uyu mubvunzo unonakidza: "Kutangira kupi?" Zvinowanzonetsa kutanga nechimwe chinhu chiri mu kernel. MuPostgres, semuenzaniso, pane yekuita runyorwa. Asi chaizvoizvo, iyi ipepa yezvavakaedza kuita, asi hazvina kubudirira. Izvi zvinhu zvakaoma kunzwisisa. Uye kazhinji iwe unogona kuwana zvimwe zvinoshandiswa mu ecosystem, zvimwe zvekuwedzera zvinogona kuvandudzwa, izvo zvinokwezva kushoma kutarisa kubva kune vanogadzira kernel. Uye, maererano, kune mamwe mapoinzi ekukura ikoko. PaGoogle Zhizha rekodhi chirongwa, gore rega rega nharaunda yepostgres inoisa pamberi akawanda akasiyana misoro inogona kugadziriswa. Gore rino takanga tine, ndinofunga, vadzidzi vatatu. Mumwe akatonyora muWAL-G pamisoro yakakosha kuYandex. MuGreenplum, zvese zviri nyore kupfuura munharaunda yePostgres, nekuti Greenplum hackers vanobata zvikumbiro zvekudhonza zvakanaka uye vanotanga kuongorora ipapo ipapo. Kutumira chigamba kuPostgres inyaya yemwedzi, asi Greenplum ichauya muzuva uye kuona zvawaita. Chimwe chinhu ndechekuti Greenplum inoda kugadzirisa matambudziko aripo. Greenplum haisi kushandiswa zvakanyanya, saka kuwana dambudziko rako kwakaoma. Uye chekutanga pane zvese, isu tinofanirwa kugadzirisa matambudziko, hongu.
Source: www.habr.com