Igalelo leYandex kwezi nkcukacha zilandelayo ziya kuhlaziywa.
- Cofa indlu
- Odyssey
- Ukufumana kwakhona kwinqanaba lexesha (WAL-G)
- I-PostgreSQL (kubandakanywa iilogers, i-Amcheck, i-heapcheck)
- Greenplum
Ividiyo:
Molo Lizwe! Igama lam nguAndrey Borodin. Kwaye into endiyenzayo kwi-Yandex.Cloud kukuphuhlisa i-database yobudlelwane obuvulekileyo kwizinto ezinomdla kwi-Yandex.Cloud kunye nabaxhasi be-Yandex.Cloud.
Kule ntetho, siza kuthetha ngemingeni ejongene nogcino-lwazi oluvulekileyo ngokomlinganiselo. Kutheni ibalulekile nje? Kuba zincinci, iingxaki ezincinci ezithi, njengeengcongconi, zibe ziindlovu. Ziba nkulu xa unamaqela amaninzi.
Kodwa ayisiyiyo eyona nto iphambili leyo. Kwenzeka izinto ezimangalisayo. Izinto ezenzeka kwimeko enye kwisigidi. Kwaye kwindawo yelifu, kuya kufuneka uyilungiselele loo nto, kuba izinto ezimangalisayo ziba nokwenzeka kakhulu xa kukho into ekhoyo kwinqanaba.
Kodwa! Yintoni inzuzo yogcino-lwazi oluvulekileyo? Inyani yeyokuba unethuba lethiyori yokujongana nayo nayiphi na ingxaki. Unayo ikhowudi yomthombo, unolwazi lwenkqubo. Siyayidibanisa kwaye iyasebenza.
Zeziphi iindlela ezikhoyo ekusebenzeni kwisoftware yomthombo ovulekileyo?
- Eyona ndlela icacileyo kukusebenzisa isoftware. Ukuba usebenzisa iiprothokholi, ukuba usebenzisa imigangatho, ukuba usebenzisa iifomati, ukuba ubhala imibuzo kwisoftware yomthombo ovulekileyo, ngoko sele uyixhasa.
- Uyenza ibenkulu inkqubo yayo yendalo. Wenza ukuba amathuba okufunyanwa kwangoko kwebug abe mkhulu. Ukwandisa ukuthembeka kwale nkqubo. Uyakwandisa ukufumaneka kwabaphuhlisi kwimarike. Uyayiphucula le software. Sele unegalelo ukuba uye waphinda wafumana isitayile kwaye wajonga into ethile apho.
- Enye indlela eqondakalayo kukuxhasa isoftware yomthombo ovulekileyo. Ngokomzekelo, inkqubo eyaziwayo ye-Google yeHlobo yeKhowudi, xa i-Google ihlawula inani elikhulu labafundi abavela kwihlabathi lonke imali eqondakalayo ukuze baphuhlise iiprojekthi zesofthiwe ezivulekileyo ezihlangabezana neemfuno ezithile zelayisensi.
- Le yindlela enomdla kakhulu kuba ivumela isoftware ukuba ivele ngaphandle kokususa ujoliso kude noluntu. UGoogle, njengesigebenga setekhnoloji, ayitsho ukuba sifuna eli nqaku, sifuna ukulungisa le bug kwaye kulapho kufuneka simbe khona. UGoogle uthi: “Yenza le nto uyenzayo. Hlala nje usebenza ngendlela obusebenza ngayo kwaye yonke into izakulunga. ”
- Indlela elandelayo yokuthatha inxaxheba kumthombo ovulekileyo kukuthatha inxaxheba. Xa unengxaki kwisoftware yomthombo ovulekileyo kwaye kukho abaphuhlisi, abaphuhlisi bakho baqala ukusombulula iingxaki. Baqala ukwenza amaziko akho asebenze ngakumbi, iinkqubo zakho zikhawuleza kwaye zithembeke ngakumbi.
Enye yeeprojekthi ezidumileyo zeYandex kwinkalo yesoftware evulekileyo yiClickHouse. Le datha yedatha eyazalwa njengempendulo kwimingeni ejongene neYandex.Metrica.
Kwaye njengesiseko sedatha, yenziwe kumthombo ovulekileyo ukwenzela ukudala i-ecosystem kwaye iphuhlise kunye nabanye abaphuhlisi (kungekhona ngaphakathi kweYandex kuphela). Kwaye ngoku le yiprojekthi enkulu apho iinkampani ezininzi ezahlukeneyo zibandakanyeka.
Kwi-Yandex.Cloud, senze i-ClickHouse phezulu kwe-Yandex Object Storage, i.e. ngaphezulu kwendawo yokugcina ifu.
Kutheni oku kubalulekile efini? Ngenxa yokuba nayiphi na isiseko sedatha sisebenza kulo nxantathu, kule phiramidi, kule hierarchy yeentlobo zeememori. Uneerejista ezikhawulezayo kodwa ezincinci kunye nee-SSD ezinkulu nezicothayo, ii-hard drives kunye nezinye izixhobo zebhloko. Kwaye ukuba uyasebenza phezulu kwiphiramidi, ngoko unesiseko sedatha esikhawulezayo. ukuba uyasebenza ezantsi kwale phiramidi, ngoko unesiseko sedata esikalisiweyo. Kwaye kulo mba, ukongeza omnye umaleko ukusuka ngezantsi kuyindlela enengqiqo yokwandisa i-scalability ye-database.
Yayinokwenziwa njani? Le ngongoma ebalulekileyo kule ngxelo.
- Sinokusebenzisa i-ClickHouse phezu kwe-MDS. I-MDS yindawo yangaphakathi yokugcina ifu yeYandex. Inzima ngakumbi kuneprotocol ye-S3 eqhelekileyo, kodwa ifaneleke ngakumbi kwisixhobo sebhloko. Kungcono ukurekhoda idatha. Ifuna inkqubo engaphezulu. Abadwelisi benkqubo baya kucwangcisa, ilungile, inomdla.
- I-S3 yindlela eqhelekileyo eyenza i-interface ibe lula kwiindleko zokulungiswa okuncinci kwiintlobo ezithile zemithwalo yomsebenzi.
Ngokwemvelo, sifuna ukubonelela ngokusebenza kuyo yonke iClickHouse ecosystem kwaye wenze umsebenzi ofunekayo ngaphakathi kweYandex.Cloud, sagqiba ekubeni siqinisekise ukuba lonke uluntu lweClickHouse luya kuzuza kuyo. Siphumeze iClickHouse ngaphezulu kweS3, hayi iClickHouse ngaphezulu kweMDS. Kwaye lo ngumsebenzi omninzi.
Izalathiso:
"Umaleko wokuthatha inkqubo yefayile"
"Ukudityaniswa kwe-AWS SDK S3"
"Ukuphunyezwa kwesiseko se-IDisk interafce ye-S3"
"Ukudityaniswa kweenjini zokugcina log kunye ne-IDisk interface"
"Inkxaso ye-injini ye-Log ye-S3 kunye ne-SeekableReadBuffer"
"Ugcino Stripe Log S3 inkxaso"
"Storage MergeTree inkxaso yokuqala ye-S3"
"MergeTree inkxaso epheleleyo ye-S3"
"Inkxaso yeReplicatedMergeTree ngaphezulu kwe-S3"
"Yongeza iziqinisekiso ezingagqibekanga kunye neentloko zesiko lokugcinwa kwe-s3"
"I-S3 enolungelelwaniso lwe-proxy eguqukayo"
"I-S3 ene-proxy resolution"
Olu luluhlu lwesicelo sokutsala ukuphumeza inkqubo yefayile yenyani kwiClickHouse. Eli linani elikhulu lezicelo zokutsalwa.
Izalathiso:
"DiskS3 hardlinks ukuphunyezwa ngokufanelekileyo"
"Umthengi we-S3 HTTP-Kuphephe ukukopa umjelo wempendulo kwimemori"
"Kuphephe ukukopa yonke impendulo kwimemori kwi-S3 HTTP
umxhasi"
"Ukwazi ukugcina i-cache kunye neefayile zesalathisi kwidiski ye-S3"
"Hambisa iindawo ukusuka kwiDiskiLocal ukuya kwiDiskS3 ngokufanayo"
Kodwa umsebenzi awuzange uphelele apho. Emva kokwenziwa kolu phawu, kwafuneka omnye umsebenzi ongakumbi ukuze kunyuswe lo msebenzi.
Izalathiso:
"Yongeza i-SelectedRows kunye ne-SelectedBytes iziganeko"
"Yongeza iziganeko zeprofayile ukusuka kwisicelo se-S3 ukuya kwi-system.events"
"Yongeza i-QueryTimeMicroseconds, khetha i-QueryTimeMicroseconds kunye ne-InsertQueryTimeMicroseconds"
Kwaye ke kwakuyimfuneko ukuyenza ukuba ixilongwe, ibeke iliso kwaye iyenze ilawuleke.
Kwaye konke oku kwenziwa ukuze lonke uluntu, yonke iClickHouse ecosystem, ifumane isiphumo salo msebenzi.
Masiqhubele phambili koovimba beenkcukacha zentengiselwano, siye koovimba beenkcukacha ze-OLTP, ezikufutshane kum ngokobuqu.
Eli licandelo elivulekileyo lophuhliso lweDBMS. Aba bafana benza umlingo wesitrato ukuphucula i-database evulekileyo yentengiselwano.
Enye yeeprojekthi, sisebenzisa umzekelo apho sinokuthetha malunga nendlela kunye nento esiyenzayo, yi-Connection Pooler kwi-Postgres.
I-Postgres yinkqubo yedatha. Oku kuthetha ukuba uvimba weenkcukacha kufuneka ube noqhagamshelo lwenethiwekhi olumbalwa kangangoko kunokwenzeka oluphatha iitransekshini.
Ngakolunye uhlangothi, kwindawo yefu, imeko eqhelekileyo xa udibaniso lwewaka lufika kwiqela elinye kanye. Kwaye umsebenzi we-pooler yoqhagamshelwano kukupakisha uqhagamshelo lwewaka kwinani elincinci lonxibelelwano lweseva.
Sinokuthi i-pooler yoqhagamshelwano ngumqhubi wefowuni olungisa kwakhona ii-bytes ukuze zifikelele ngokufanelekileyo kwisiseko sedatha.
Ngelishwa, akukho gama lilungileyo lesiRashiya lokudibanisa i-pooler. Ngamanye amaxesha ibizwa ngokuba yi-multiplexer connections. Ukuba uyazi ukuba ubize ntoni i-pooler yoxhulumaniso, ke qiniseka ukuba undixelele, ndiya kuvuya kakhulu ukuthetha ulwimi oluchanekileyo lwesiRashiya.
Siye saphanda iipowuli zoqhagamshelo ezazifanele i-postgres cluster elawulwayo. Kwaye iPgBouncer yayilolona khetho lulungileyo kuthi. Kodwa siye sadibana nenani leengxaki ngePgBouncer. Kwiminyaka emininzi edlulileyo, uVolodya Borodin wanika iingxelo ukuba sisebenzisa iPgBouncer, sithanda yonke into, kodwa kukho ama-nuances, kukho into yokusebenza.
Kwaye sasebenza. Sizilungisile iingxaki esiye sadibana nazo, sabopha iBouncer, kwaye sazama ukutyhala izicelo zokutsalela phezulu. Kodwa umsonto omnye osisiseko kwakunzima ukusebenza nawo.
Kwafuneka siqokelele ii-cascades kwiiBouncers ezipeyiweyo. Xa sinee-Bouncers ezininzi ezinemisonto enye, imidibaniso ekumgangatho ophezulu idluliselwa kumaleko angaphakathi e-Bouncers. Le yinkqubo elawulwa kakubi ekunzima ukuyakha kunye nokunyusa umva naphambili.
Siye safikelela kwisigqibo sokuba senze eyethu i-pooler yokudibanisa, ebizwa ngokuba yi-Odyssey. Sayibhala ukusuka ekuqaleni.
Ngo-2019, kwinkomfa ye-PgCon, ndibonise le pooler kuluntu oluphuhlisayo. Ngoku sineenkwenkwezi ezingaphantsi kwe-2 kwi-GitHub, oko kukuthi iprojekthi iyaphila, iprojekthi iyathandwa.
Kwaye ukuba udala i-Postgres cluster kwi-Yandex.Cloud, ngoko iya kuba yimbumba ene-Odyssey eyakhelweyo, ephinda iqwalaselwe xa ulinganisa iqela emva okanye ngaphambili.
Sifunde ntoni kule projekthi? Ukusungula iprojekthi ekhuphisanayo kusoloko kulinyathelo elinobundlobongela, ngumlinganiselo ogqithisileyo xa sisithi kukho iingxaki ezingasonjululwanga ngokukhawuleza ngokwaneleyo, azisonjululwa ngamaxesha athile anokusifanela. Kodwa lo ngumlinganiselo osebenzayo.
I-PgBouncer yaqala ukukhula ngokukhawuleza.
Kwaye ngoku ezinye iiprojekthi ziye zavela. Ngokomzekelo, i-pgagroal, ephuhliswa ngabaphuhlisi be-Red Hat. Baphishekela iinjongo ezifanayo kwaye basebenzise iingcamango ezifanayo, kodwa, ngokuqinisekileyo, kunye neenkcukacha zabo, ezikufutshane nabaphuhlisi be-pgagroal.
Enye imeko yokusebenza kunye noluntu lwe-postgres lubuyisela kwinqanaba ngexesha. Oku kukubuyisela emva kokusilela, oku kukubuyisela kwi-backup.
Kukho ii-backups ezininzi kwaye zonke zahlukile. Phantse wonke umthengisi wePostgres unesisombululo sakhe sokugcina.
Ukuba uthatha zonke iisistim zogcino, yenza i-matrix ebonakalayo kwaye ngokuqhula ubale isilawuli kule matrix, iya kuba ngu-zero. Ithetha ntoni le nto? Kuthekani ukuba uthatha ifayile ethile yogcino, ngoko ayinakudityaniswa kwiziqwenga zabo bonke abanye. Iyingqayizivele ekuphunyezweni kwayo, iyingqayizivele kwinjongo yayo, iyingqayizivele kwiingcamango ezifakwe kuyo. Kwaye zonke zithe ngqo.
Ngelixa sasisebenza kulo mbandela, iCitusData yasungula iprojekthi ye-WAL-G. Le yinkqubo yogcino eyenziwe ngeliso kwindawo yelifu. Ngoku iCitusData sele iyinxalenye yeMicrosoft. Kwaye ngelo xesha, sasiyithanda kakhulu imibono eyayibekwe phantsi kokukhutshwa kokuqala kwe-WAL-G. Kwaye saqala ukufaka isandla kule projekthi.
Ngoku kukho ezininzi ezininzi zabaphuhlisi kule projekthi, kodwa abaxhasi be-10 abaphezulu kwi-WAL-G baquka i-Yandexoids eyi-6. Size neengcinga zethu ezininzi apho. Kwaye, ngokuqinisekileyo, siziphumeze ngokwethu, sizivavanye ngokwethu, saziqengqela ukuba zibe yimveliso ngokwethu, sizisebenzisa ngokwethu, thina ngokwethu sifumanisa ukuba masihambe phi ngokulandelayo, ngelixa sinxibelelana noluntu olukhulu lwe-WAL-G.
Kwaye ngokwembono yethu, ngoku le nkqubo yogcino, kubandakanywa nokuthathela ingqalelo iinzame zethu, ibe yeyona ilungileyo kwindawo yelifu. Eli lelona xabiso lilungileyo lokuxhasa iPostgres efini.
Ingaba ithetha ntoni? Besikhuthaza umbono omkhulu: i-backup kufuneka ikhuseleke, ingabizi mali isebenze kwaye ngokukhawuleza kangangoko kunokwenzeka ukubuyisela.
Kutheni le nto kufuneka kutshiphu ukusebenza? Xa kungekho nto yaphukileyo, akufanele wazi ukuba une-backups. Yonke into isebenza kakuhle, uchitha i-CPU encinci ngokusemandleni, usebenzisa i-disk yakho encinci kangangoko kunokwenzeka, kwaye uthumela ii-bytes ezimbalwa kwinethiwekhi ngokusemandleni ukuze ungaphazamisi umthwalo weenkonzo zakho ezixabisekileyo.
Kwaye xa yonke into iphuka, umzekelo, umlawuli wehla idatha, kukho into engalunganga, kwaye kufuneka ngokukhawuleza ubuyele kwixesha elidlulileyo, ululame ngayo yonke imali, kuba ufuna idatha yakho ibuye ngokukhawuleza kwaye iphelele.
Kwaye sakhuthaza le ngcamango ilula. Kwaye, kubonakala ngathi, sikwazile ukuyiphumeza.
Kodwa akuphelelanga apho. Sasifuna enye into encinci ngakumbi. Besifuna oovimba beenkcukacha ezininzi ezahlukeneyo. Ayingabo bonke abathengi bethu abasebenzisa iiPostgres. Abanye abantu basebenzisa iMySQL, iMongoDB. Kuluntu, abanye abaphuhlisi baye baxhasa i-FoundationDB. Kwaye olu luhlu lukhula rhoqo.
Uluntu luyawuthanda umbono wokuba isiseko sedatha siqhutywa kwindawo elawulwayo efini. Kwaye abaphuhlisi bagcina i-database yabo, enokuthi ixhaswe ngokufanayo kunye ne-Postgres kunye nenkqubo yethu yokugcina.
Sifunde ntoni kweli bali? Imveliso yethu, njengecandelo lophuhliso, ayikho imigca yekhowudi, ayiyiyo inkcazo, ayiyiyo iifayile. Imveliso yethu ayizizo izicelo zokutsalwa. Ezi ziimbono esizidlulisela eluntwini. Obu bubuchule bobuchwepheshe kunye nokuhamba kwethekhnoloji ukuya kwindawo yelifu.
Kukho isiseko sedatha esinjengePostgres. Ndiyithanda kakhulu iPostgres core. Ndichitha ixesha elininzi ndiphuhlisa undoqo wePostgres noluntu.
Kodwa apha kufuneka kuthiwe iYandex.Cloud inofakelo lwangaphakathi lwedatha elawulwayo. Kwaye yaqala ixesha elidlulileyo kwiYandex.Mail. Ubuchwephesha obuthe ngoku bukhokelela ekulawuleni i-Postgres yaqokelelwa xa i-imeyile ifuna ukufudukela kwi-Postgres.
Imeyile ineemfuno ezifanayo kakhulu kwilifu. Ifuna ukuba ukwazi ukukhulisa ukukhula okungalindelekanga nakweliphi na ixesha kwidatha yakho. Kwaye iposi sele inomthwalo kunye namakhulu ezigidi zeebhokisi zeposi zenani elikhulu labasebenzisi abahlala besenza izicelo ezininzi.
Kwaye lo yayingumngeni omkhulu kwiqela elaliphuhlisa iPostgres. Emva phaya, naziphi na iingxaki esidibana nazo zazixelwa kuluntu. Kwaye ezi ngxaki zalungiswa, kwaye zalungiswa luluntu kwezinye iindawo nakwinqanaba lenkxaso ehlawulwayo kwezinye iindawo zogcino-lwazi nangaphezulu. Oko kukuthi, ungathumela ileta kuPgSQL hacker kwaye ufumane impendulo ngaphakathi kwemizuzu engama-40. Inkxaso ehlawulweyo kwezinye ii-database inokucinga ukuba kukho izinto eziphambili kakhulu kunebug yakho.
Ngoku ukufakwa kwangaphakathi kwePostgres yenye yeepetabytes zedatha. Ezi zizigidi zezicelo ngomzuzwana. La ngamawaka amaqela. Inkulu kakhulu.
Kodwa kukho i-nuance. Ayihlali kwiidrive zenethiwekhi ezithandekayo, kodwa kwihardware elula. Kwaye kukho indawo yovavanyo ngokukodwa kwizinto ezintsha ezinomdla.
Kwaye ngexesha elithile kwindawo yovavanyo sifumene umyalezo obonisa ukuba ukungaguquguquki kwangaphakathi kwezalathisi zedatha zaphulwa.
Ukungaguquguquki luhlobo oluthile lobudlelwane esilindele ukuba sihlale sibambe.
Imeko enzima kakhulu kuthi. Ibonisa ukuba enye idatha inokuba ilahlekile. Kwaye ilahleko yedatha yinto eyintlekele.
Ingcamango eqhelekileyo esiyilandelayo kwiinkcukacha ezilawulwayo kukuba nangona umzamo, kuya kuba nzima ukulahlekelwa idatha. Nokuba uzisuse ngabom, kuya kufuneka ungabuhoyi ukungabikho kwazo kangangexesha elide. Ukhuseleko lwedatha yinkolo esiyilandelayo ngenkuthalo.
Kwaye apha kuvela imeko ebonisa ukuba kusenokubakho imeko esisenokungayilungiseleli. Kwaye saqala ukulungiselela le meko.
Into yokuqala esayenzayo kukungcwaba izigodo kula maqela angamawaka. Sifumene ukuba yeyiphi na amaqela abekwe kwiidiski ezine-firmware eyingxaki ephulukene nohlaziyo lwephepha ledatha. Imakishwe yonke ikhowudi yedatha yePostgres. Kwaye siphawule loo miyalezo ebonisa ukunyhashwa kokungaguquguquki kwangaphakathi ngekhowudi eyenzelwe ukufumanisa urhwaphilizo lwedatha.
Esi siqwenga samkelwa luluntu ngaphandle kweengxoxo ezininzi, kuba kwimeko nganye ethile kwakucacile ukuba kukho into embi eyenzekileyo kwaye kufuneka ixelwe kwilog.
Emva koku, sifikelele kwinqanaba lokuba sibeke iliso esiskena izingodo. Kwaye kwimeko yemiyalezo ekrokrisayo, uvusa igosa elijongene nomsebenzi, kwaye igosa elijongene nalo liyayilungisa.
Kodwa! Ukuskena izigodo ngumsebenzi onexabiso eliphantsi kwiqela elinye kwaye zibiza imali eninzi kakhulu kwiiklasta eziliwaka.
Sabhala isandiso esibizwa . Idala umbono wesiseko sedatha apho ungancinci kwaye ukhethe ngokukhawuleza izibalo kwiimpazamo ezidlulileyo. Kwaye ukuba sifuna ukuvusa igosa lomsebenzi, ngoko siya kufumanisa ngale nto ngaphandle kokuskena iifayile zegigabyte, kodwa ngokukhupha ama-byte ambalwa kwitafile ye-hash.
Olu lwandiso lwamkelwe, umzekelo, kwindawo yokugcina . Ukuba ufuna ukuyisebenzisa, ungayifaka ngokwakho. Ngokuqinisekileyo ngumthombo ovulekileyo.
Kodwa akuphelelanga apho. Saqala ukusebenzisa i-Amcheck, ulwandiso olwakhiwe luluntu, ukufumana ukwaphulwa okungaguqukiyo kwizalathisi.
Kwaye sifumanise ukuba ukuba uyayisebenzisa kwisikali, kukho iibugs. Saqalisa ukuzilungisa. Izilungiso zethu zamkelwe.
Sifumanise ukuba olu lwandiso alunakuhlalutya izalathisi ze-GiST kunye ne-GIT. Sabenza inkxaso. Kodwa le nkxaso isaxoxwa luluntu, kuba lo ngumsebenzi omtsha kwaye kukho iinkcukacha ezininzi apho.
Kwaye siphinde safumanisa ukuba xa sijonga izalathisi zokuphulwa kwenkokeli yokuphindaphinda, kwinkosi, yonke into isebenza kakuhle, kodwa kwii-replicas, kumlandeli, ukukhangela urhwaphilizo alusebenzi kangako. Ayizizo zonke iinguqulelo ezijongisiweyo. Kwaye omnye ongaguqukiyo wasikhathaza kakhulu. Kwaye sichithe unyaka onesiqingatha sinxibelelana noluntu ukuze sikwazi ukwenza oku kuhlolwa kwiikopi.
Sabhala ikhowudi emele ilandele yonke into ... iiprothokholi. Sixoxe ngesi siqwenga ixesha elide kunye noPeter Gaghan ovela kwiDatha yeCrunchy. Kwafuneka aguqule kancinane i-B-tree ekhoyo kwi-Postgres ukuze amkele le patch. Wamkelwa. Kwaye ngoku ukujonga izalathisi kwii-replicas kuye kwasebenza ngokwaneleyo ukubona ulwaphulo-mthetho esidibene nalo. Oko kukuthi, olu lulwaphulo-mthetho olunokuthi lubangelwe ziimpazamo kwidiski firmware, bugs kwi Postgres, bugs kwi Linux kernel, kunye neengxaki hardware. Uluhlu olubanzi kakhulu lwemithombo yeengxaki ebesizilungiselela.
Kodwa ngaphandle kwezalathisi, kukho indawo efana nemfumba, o.k.t. indawo apho idatha igcinwa khona. Kwaye akukho ziguquko ezininzi ezinokujongwa.
Sinolwandiso olubizwa ngokuba yi-Heapcheck. Saqalisa ukuyiphuhlisa. Kwaye ngokunxuseneyo, kunye nathi, inkampani ye-EnterpriseDB nayo yaqala ukubhala imodyuli, abayibiza ngokuba yiHeapcheck ngendlela efanayo. Kuphela siyibiza ngokuba yiPgHeapcheck, kwaye bayibiza nje ngokuba yiHeapcheck. Banayo ngemisebenzi efanayo, umsayino owahluke kancinane, kodwa oneengcamango ezifanayo. Bawaphumeza ngcono kancinci kwezinye iindawo. Kwaye bayithumela kwindawo evulekileyo ngaphambili.
Kwaye ngoku siphuhlisa ukwanda kwabo, kuba akusekho ukwanda kwabo, kodwa ukwandiswa koluntu. Kwaye kwixesha elizayo, le yinxalenye ye-kernel eya kunikwa wonke umntu ukuze azi malunga neengxaki ezizayo kwangaphambili.
Kwezinye iindawo, sade safikelela kwisigqibo sokuba sinemibono engeyonyani kwiinkqubo zethu zokubeka iliso. Umzekelo, inkqubo ye-1C. Xa usebenzisa i-database, i-Postgres ngamanye amaxesha ibhala idatha kuyo ukuba ifunde, kodwa i-pg_dump ayikwazi ukufunda.
Le meko yayijongeka njengorhwaphilizo kwinkqubo yethu yokufumanisa ingxaki. Igosa elijongene nomsebenzi lavuswa. Igosa elijongene nomsebenzi lajonga okwenzekayo. Emva kwexesha elithile, kwafika umthengi wathi ndineengxaki. Umlindi wachaza ingxaki. Kodwa ingxaki ikwiPostgres core.
Ndifumene ingxoxo malunga neli nqaku. Kwaye wabhala ukuba sidibene nalo mboniso kwaye kwakungekho mnandi, umntu wayevuka ebusuku ukuze aqonde ukuba yintoni.
Uluntu lwaphendula, "Owu, kufuneka siyilungise ngokwenene."
Ndinesifaniso esilula. Ukuba uhamba ngesicathulo esinokhozo lwesanti kuyo, ngoko, ngokomgaqo, unokuqhubeka - akukho ngxaki. Ukuba uthengisa iibhutsi kumawaka abantu, ke masenze iibhutsi ngaphandle kwesanti kwaphela. Kwaye ukuba omnye wabasebenzisi bezihlangu zakho uya kubaleka i-marathon, ngoko ufuna ukwenza izicathulo ezintle kakhulu, kwaye emva koko uzilinganise kubo bonke abasebenzisi bakho. Kwaye abasebenzisi abanjalo abangalindelekanga bahlala bekwimeko yelifu. Kusoloko kukho abasebenzisi abaxhaphaza iqela ngendlela ethile yoqobo. Kufuneka uhlale ulungiselela oku.
Sifunde ntoni apha? Sifunde into elula: eyona nto ibalulekileyo kukuchazela uluntu ukuba kukho ingxaki. Ukuba uluntu luye lwaqaphela ingxaki, ngoko ke ukhuphiswano lwendalo luvela ukusombulula ingxaki. Kuba wonke umntu ufuna ukucombulula ingxaki ebalulekileyo. Bonke abathengisi, bonke abahlaseli bayaqonda ukuba bona ngokwabo banokunyathela kule raki, ngoko bafuna ukubaphelisa.
Ukuba usebenza kwingxaki, kodwa ayikhathazi mntu ngaphandle kwakho, kodwa usebenza kuyo ngokucwangcisiweyo kwaye ekugqibeleni ithathwa njengengxaki, ngoko isicelo sakho sokutsala siya kwamkelwa ngokuqinisekileyo. Ibala lakho liya kwamkelwa, uphuculo lwakho okanye izicelo zophuculo ziya kuhlolwa luluntu. Ekupheleni kosuku, senza i-database ibe ngcono omnye komnye.
Isiseko sedatha esinomdla yiGreenplum. Yisiseko sedatha esihambelana kakhulu esekwe kwikhowudi yePostgres, endiqhelene nayo kakhulu.
Kwaye iGreenplum inomsebenzi onomdla- yongeza iitafile eziphuculweyo. Ezi zitafile onokuzongeza ngokukhawuleza kuzo. Ziyakwazi nokuba yi-columnar okanye umqolo.
Kodwa kwakungekho kuhlanganiswa, oko kukuthi, kwakungekho msebenzi apho unokucwangcisa idatha ebekwe kwitafile ngokuhambelana nomyalelo okwisinye sezalathiso.
Abafana ababekhwele eteksini beza kum baza bathi: “Andrey, uyamazi uPostgres. Kwaye apha iphantse ifane. Tshintshela kwimizuzu engama-20. Yithathe uyenze.” Ndicinge ukuba ewe, ndiyazazi iPostgres, nditshintshela imizuzu engama-20- kufuneka ndenze oku.
Kodwa hayi, ibingekho imizuzu engama-20, ndiyibhale kwiinyanga. Kwinkomfa yePgConf.Russia, ndaya kuHeikki Linakangas wasePivotal ndaza ndambuza: “Ngaba kukho iingxaki kule nto? Kutheni kungekho ludibaniso lwetafile olulungiselelweyo?” Uthi: “Uthatha idatha. Uyahlela, uhlengahlengisa kwakhona. Ngumsebenzi nje. " Me: "Ewe, kufuneka uyithathe kwaye uyenze." Uthi: “Ewe, sifuna izandla zasimahla ukwenza oku.” Ndacinga ukuba ngokuqinisekileyo kufuneka ndiyenze le nto.
Kwaye kwiinyanga ezimbalwa kamva ndangenisa isicelo sokutsala esiphumeza lo msebenzi. Esi sicelo sokutsalwa sajongwa kwakhona nguPivotal kunye noluntu. Kakade ke, zazikho iincukuthu.
Kodwa eyona nto inika umdla kukuba xa esi sicelo sokutsalwa sidityanisiwe, iibugs zafunyanwa kwiGreenplum ngokwayo. Sifumanise ukuba iitafile zemfumba ngamanye amaxesha ziyayophula intengiselwano xa zidityanisiwe. Kwaye le yinto efuna ukulungiswa. Kwaye ukwindawo endisandul’ ukuyibamba. Kwaye impendulo yam yendalo yaba - kulungile, nam mandiyenze le nto.
Ndiyilungisile le bug. Kuthunyelwe isicelo sokutsala kubalungisi. Wabulawa.
Emva koko kwavela ukuba lo msebenzi kufuneka ufunyanwe kwi-Greenplum version ye-PostgreSQL 12. Oko kukuthi, i-adventure ye-20-minute iyaqhubeka kunye nezinto ezintsha ezinomdla. Kwakunomdla ukuchukumisa uphuhliso lwangoku, apho uluntu lusika izinto ezintsha kunye nezona zibalulekileyo. Ingumkhenkce.
Kodwa ayizange iphelele apho. Emva kwayo yonke into, kwavela ukuba kufuneka sibhale amaxwebhu kuyo yonke le nto.
Ndaqalisa ukubhala amaxwebhu. Ngethamsanqa, ii-documentarians ezivela kwi-Pivotal zeza. IsiNgesi lulwimi lwabo lweenkobe. Bandincedile ngamaxwebhu. Enyanisweni, bona bayibhala kwakhona into endiyicelileyo kwisiNgesi sokwenene.
Kwaye apha, kubonakala ngathi, i-adventure iphelile. Ngaba uyazi ukuba kwenzeka ntoni emva koko? Abafana ababevela eteksini beza kum baza bathi: “Kusekho izinto ezimbini zokukhenketha, ngasinye yimizuzu eli-10.” Ndithini ke kubo? Ndathi ngoku ndiza kunika ingxelo kwisikali, ngoko siza kubona i-adventures yakho, kuba lo ngumsebenzi onomdla.
Sifunde ntoni kweli tyala? Ngenxa yokuba ukusebenza nomthombo ovulekileyo uhlala usebenza nomntu othile, uhlala usebenza noluntu. Kuba kulo lonke inqanaba ndisebenze kunye nomphuhlisi othile, omnye umvavanyi, omnye umkhangeli, omnye umculi, umyili othile. Khange ndisebenze neGreenplum, bendisebenza nabantu baseGreenplum.
Kodwa! Kukho enye into ebalulekileyo - ngumsebenzi nje. Oko kukuthi, uze, usele ikofu, ubhale ikhowudi. Zonke iintlobo zezinto eziguquguqukayo ezilula zisebenza. Yenza ngokuqhelekileyo - kuya kulunga! Kwaye ngumsebenzi onomdla kakhulu. Kukho isicelo salo msebenzi kubathengi beYandex.Cloud, abasebenzisi bamaqela ethu ngaphakathi kweYandex nangaphandle. Kwaye ndicinga ukuba inani leeprojekthi esithatha inxaxheba kuzo ziya kwanda kwaye ubunzulu bokubandakanyeka kwethu buya kwanda.
Kuko konke. Masiqhubele phambili kwimibuzo.
Iseshoni yemibuzo
Mholo! Sinenye iseshoni yemibuzo neempendulo. Kwaye kwi studio uAndrei Borodin. Lo ngumntu osandul 'ukuxelela malunga negalelo leYandex.Cloud kunye neYandex ukuvula umthombo. Ingxelo yethu ngoku ayikho ngokupheleleyo malunga nefu, kodwa kwangaxeshanye sisekwe kubuchwephesha obunjalo. Ngaphandle kwento oyenzileyo ngaphakathi kweYandex, bekungayi kubakho nkonzo kwiYandex.Cloud, ngoko ndiyabulela kum ngokobuqu. Kwaye umbuzo wokuqala ovela kusasazo: "Yeyiphi iprojekthi nganye okhankanyiweyo ebhalwe kuyo?"
Inkqubo yogcino kwi-WAL-G ibhalwe kwi-Go. Le yenye yeeprojekthi ezintsha esisebenze kuzo. Ngokwenene uneminyaka emi-3 kuphela ubudala. Kwaye i-database isoloko imalunga nokuthembeka. Kwaye oku kuthetha ukuba oovimba bedatha bandala kakhulu kwaye badla ngokubhalwa kwi C. Iprojekthi yePostgres yaqala malunga ne-30 kwiminyaka eyadlulayo. Emva koko i-C89 yayilukhetho oluchanekileyo. Kwaye i-Postgres ibhaliwe kuyo. Oovimba bolwazi banamhlanje abaninzi njengeClickHouse bahlala bebhalwa kwiC ++. Lonke uphuhliso lwenkqubo lusekwe malunga neC kunye neC ++.
Umbuzo ovela kumphathi wethu wezezimali, onoxanduva lweendleko e-Cloud: "Kutheni i-Cloud ichitha imali ekuxhaseni umthombo ovulekileyo?"
Kukho impendulo elula yomphathi wezezimali apha. Senza oku ukwenza iinkonzo zethu zibengcono. Zeziphi iindlela esinokwenza ngcono ngazo? Sinokwenza izinto ngobuchule, ngokukhawuleza, kwaye senze izinto zibe lula ngakumbi. Kodwa kuthi, eli bali ngokuyintloko limalunga nokuthembeka. Umzekelo, kwinkqubo yogcino sihlaziya i-100% yeepetshi ezisebenza kuyo. Siyazi ukuba yintoni ikhowudi. Kwaye sikhululekile ngakumbi ukukhupha iinguqulelo ezintsha kwimveliso. Oko kukuthi, okokuqala, malunga nokuzithemba, malunga nokulungela uphuhliso kunye nokuthembeka
Omnye umbuzo: "Ngaba iimfuno zabasebenzisi bangaphandle abahlala kwi-Yandex.Cloud bahluke kubasebenzisi bangaphakathi abahlala kwi-Cloud yangaphakathi?"
Iprofayile yomthwalo, ngokuqinisekileyo, yahlukile. Kodwa ukusuka kwindawo yokujonga isebe lam, zonke iimeko ezikhethekileyo kunye nezinomdla zenziwe kumthwalo ongekho umgangatho. Abaphuhlisi abaneentelekelelo, abaphuhlisi abenza into engalindelekanga, banokufumaneka ngaphakathi nangaphandle. Ngokumalunga noku, sonke siyafana. Kwaye, mhlawumbi, into ebalulekileyo kuphela ngaphakathi komsebenzi we-Yandex we-database iya kuba ngaphakathi kweYandex sinemfundiso. Ngexesha elithile, indawo ethile yokufumaneka ingena ngokupheleleyo emthunzini, kwaye zonke iinkonzo zeYandex kufuneka ngandlela-thile ziqhubeke nokusebenza nangona kunjalo. Lo ngumahluko omncinci. Kodwa idala uphuhliso oluninzi lophando kwi-interface yedatabase kunye ne-network stack. Kungenjalo, ufakelo lwangaphandle nolwangaphakathi luvelisa izicelo ezifanayo zeempawu kunye nezicelo ezifanayo zokuphucula ukuthembeka nokusebenza.
Umbuzo olandelayo: "Uziva njani wena buqu malunga nento yokuba uninzi lwento oyenzayo isetyenziswa ngamanye amafu?" Asiyi kukhankanya amagama athile, kodwa ezininzi iiprojekthi ezenziwe kwiYandex.Cloud zisetyenziselwa amafu abanye abantu.
Oku kupholile. Okokuqala, luphawu lokuba senze into elungileyo. Kwaye izonwaya i-ego. Kwaye siqiniseke ngakumbi ukuba senze isigqibo esifanelekileyo. Ngakolunye uhlangothi, eli lithemba lokuba kwixesha elizayo oku kuya kusizisela iingcamango ezintsha, izicelo ezintsha ezivela kubasebenzisi beqela lesithathu. Imiba emininzi kwi-GitHub yenziwe ngabalawuli benkqubo ngabanye, ii-DBAs ngabanye, abakhi bezakhiwo, iinjineli ngabanye, kodwa ngamanye amaxesha abantu abanamava acwangcisiweyo beza bathi kwi-30% yeemeko ezithile sinale ngxaki kwaye makhe sicinge malunga nendlela yokusombulula ngayo. Le yeyona nto sijonge phambili kuyo. Sijonge phambili ukwabelana ngamava namanye amaqonga elifu.
Uthethe kakhulu malunga nomdyarho. Ndiyazi ukuba ubaleke umdyarho eMoscow. Nje ngeziphumo? Ugqithise abafana abavela kwi-PostgreSQL?
Hayi, uOleg Bartunov ubaleka kakhulu. Ugqibe iyure phambi kwam. Lilonke, ndonwabile yindlela endiye ndafikelela kuyo. Kum, ukugqiba kwaba yimpumelelo. Ngokubanzi, kuyamangalisa ukuba kukho iimbaleki ezininzi kwiindawo ze-postgres. Kubonakala kum ukuba kukho uhlobo lobudlelwane phakathi kwemidlalo ye-aerobic kunye nomnqweno weenkqubo zeenkqubo.
Ngaba uthi akukho mbaleki eClickHouse?
Ndiyazi ngokuqinisekileyo ukuba bekhona. I-ClickHouse ikwayi-database. Kakade ke, u-Oleg ngoku uyandibhalela: “Ngaba siza kubaleka emva kwengxelo?” Lo ngumbono omkhulu.
Omnye umbuzo ovela kusasazo oluvela kuNikita: "Kutheni ulungise i-bug eGreenplum ngokwakho kwaye ungayiniki abancinci?" Enyanisweni, akucaci kakuhle ukuba yintoni i-bug kwaye yeyiphi inkonzo, kodwa mhlawumbi ithetha le nto uthetha ngayo.
Ewe, ngokomgaqo, yayinokunikwa umntu. Yayiyikhowudi nje endisandula ukuyitshintsha. Kwaye kwakungokwemvelo ukuqhubeka usenza ngoko nangoko. Ngokomgaqo, umbono wokwabelana ngobungcali kunye neqela ngumbono olungileyo. Ngokuqinisekileyo siya kwabelana ngemisebenzi yeGreenplum phakathi kwawo onke amalungu ecandelo lethu.
Kuba sithetha ngabancinci, nanku umbuzo. Umntu wagqiba ekubeni enze isibophelelo sokuqala kwi-Postgres. Yintoni ekufuneka ayenze ukuze enze isibophelelo sokuqala?
Lo ngumbuzo onomdla: "Uqala phi?" Kudla ngokuba nzima kakhulu ukuqala ngento ekwi kernel. Kwi-Postgres, umzekelo, kukho uluhlu lokwenza. Kodwa eneneni, eli liphepha koko bazama ukukwenza, kodwa abazange baphumelele. Ezi zizinto ezintsonkothileyo. Kwaye ngokwesiqhelo ungafumana izinto eziluncedo kwi-ecosystem, ezinye izandiso ezinokuphuculwa, ezitsala ingqalelo encinci kubaphuhlisi bekernel. Kwaye, ngokufanelekileyo, kukho amanqaku amaninzi okukhula apho. Kwihlobo likaGoogle lenkqubo yekhowudi, minyaka le uluntu lwe-postgres lubeka phambili izihloko ezininzi ezahlukeneyo ezinokuqwalaselwa. Kulo nyaka, ndicinga ukuba sinabafundi abathathu. Omnye wada wabhala kwi-WAL-G kwizihloko ezibalulekileyo kwiYandex. EGreenplum, yonke into ilula kunoluntu lwasePostgres, kuba abahlaseli baseGreenplum baphatha izicelo zokutsala kakuhle kwaye baqale ukuphonononga kwangoko. Ukuthumela isiqwenga kwi-Postgres ngumcimbi weenyanga, kodwa iGreenplum iya kuza ngosuku kwaye ibone into oyenzileyo. Enye into kukuba iGreenplum idinga ukusombulula iingxaki ezikhoyo. IGreenplum ayisetyenziswa kakhulu, ke ukufumana ingxaki yakho kunzima kakhulu. Kwaye okokuqala, kufuneka sisombulule iingxaki, kunjalo.
umthombo: www.habr.com