Umnikelo we-Yandex kuzinqolobane ezilandelayo uzobuyekezwa.
- ChofozaHouse
- Odyssey
- Ukuthola kabusha kuze kufike ephuzwini ngesikhathi (WAL-G)
- I-PostgreSQL (kufaka phakathi ama-loger, i-Amcheck, i-heapcheck)
- I-Greenplum
Ividiyo:
Sawubona Mhlaba! Igama lami ngingu-Andrey Borodin. Futhi engikwenzayo kwa-Yandex.Cloud ukuthuthukisa imininingo egciniwe yobudlelwano evulekile ngezintshisakalo zamakhasimende e-Yandex.Cloud kanye ne-Yandex.Cloud.
Kule nkulumo, sizokhuluma ngezinselele ezibhekene nesizindalwazi esivulekile esikalini. Kungani ibalulekile? Ngoba izinkinga ezincane, ezinjengomiyane, zibe izindlovu. Ziba zikhulu uma unamaqoqo amaningi.
Kodwa lokho akuyona into eyinhloko. Kwenzeka izinto ezimangalisayo. Izinto ezenzeka esimweni esisodwa esigidini. Futhi endaweni yamafu, kufanele ukulungiselele lokho, ngoba izinto ezimangalisayo ziba nokwenzeka kakhulu uma kukhona okuthile esikalini.
Kodwa! Iyini inzuzo yolwazi oluvulekile? Iqiniso liwukuthi unethuba lokubhekana nanoma iyiphi inkinga. Unayo ikhodi yomthombo, unolwazi lokuhlela. Siyayihlanganisa futhi iyasebenza.
Yiziphi izindlela ezikhona ekusebenzeni ngesoftware yomthombo ovulekile?
- Indlela eqondile kakhulu ukusebenzisa isoftware. Uma usebenzisa amaphrothokholi, uma usebenzisa izindinganiso, uma usebenzisa amafomethi, uma ubhala imibuzo kusofthiwe yomthombo ovulekile, khona-ke usuvele uyayisekela.
- Wenza i-ecosystem yayo ibe nkulu. Wenza amathuba okutholwa kusenesikhathi kwesiphazamisi abe mkhulu. Ukhulisa ukuthembeka kwalolu hlelo. Ukhulisa ukutholakala konjiniyela emakethe. Uthuthukisa le softhiwe. Usuvele ungumuntu onikelayo uma usanda kwenza isitayela futhi wathinta okuthile lapho.
- Enye indlela eqondakalayo ukuxhasa isofthiwe yomthombo ovulekile. Isibonelo, uhlelo oludumile lwe-Google Summer of Code, lapho i-Google ikhokha inani elikhulu labafundi abavela emhlabeni wonke imali eqondakalayo ukuze bathuthukise amaphrojekthi esofthiwe avulekile ahlangabezana nezidingo ezithile zelayisensi.
- Lena indlela ethakazelisa kakhulu ngoba ivumela isoftware ukuthi iguquke ngaphandle kokususa ukugxila emphakathini. I-Google, njengesiqhwaga sezobuchwepheshe, ayisho ukuthi sifuna lesi sici, sifuna ukulungisa lesi siphazamisi futhi sidinga ukumba lapho. I-Google ithi: “Yenza okwenzayo. Qhubeka usebenze ngendlela obusebenza ngayo futhi konke kuzolunga."
- Indlela elandelayo yokubamba iqhaza kumthombo ovulekile ukubamba iqhaza. Uma unenkinga ku-software yomthombo ovulekile futhi kukhona onjiniyela, abathuthukisi bakho baqala ukuxazulula izinkinga. Baqala ukwenza ingqalasizinda yakho isebenze kahle, izinhlelo zakho zisheshe futhi zithembeke kakhulu.
Enye yamaphrojekthi e-Yandex adume kakhulu emkhakheni wesoftware yomthombo ovulekile yi-ClickHouse. Lena isizindalwazi esazalwa njengempendulo ezinseleleni ezibhekene ne-Yandex.Metrica.
Futhi njengesizindalwazi, yenziwe kumthombo ovulekile ukuze kwakhiwe i-ecosystem futhi ithuthukiswe kanye nabanye abathuthukisi (hhayi ngaphakathi kwe-Yandex kuphela). Futhi manje lena iphrojekthi enkulu lapho izinkampani eziningi ezahlukene zibandakanyeka.
Ku-Yandex.Cloud, sidale i-ClickHouse phezu kwe-Yandex Object Storage, i.e. ngaphezulu kwesitoreji samafu.
Kungani lokhu kubalulekile emafini? Ngoba noma iyiphi isizindalwazi sisebenza kulonxantathu, kule phiramidi, kulesi sigaba sezinhlobo zenkumbulo. Unamarejista asheshayo kodwa amancane nama-SSD amakhulu ashibhile kodwa ahamba kancane, ama-hard drive kanye namanye ama-block device. Futhi uma usebenza kahle phezulu kwephiramidi, une-database esheshayo. uma usebenza kahle ngaphansi kwale phiramidi, khona-ke unesizindalwazi esikalini. Futhi mayelana nalokhu, ukwengeza esinye isendlalelo kusuka ngezansi kuyindlela enengqondo yokwandisa ukuqina kwesizindalwazi.
Kwakungenziwa kanjani? Leli yiphuzu elibalulekile kulo mbiko.
- Singasebenzisa i-ClickHouse phezu kwe-MDS. I-MDS iyisixhumi esibonakalayo sesitoreji samafu sangaphakathi se-Yandex. Iyinkimbinkimbi kakhulu kune-protocol evamile ye-S3, kodwa ifaneleka kakhulu idivayisi ye-block. Kungcono ukurekhoda idatha. Idinga ukuhlela okwengeziwe. Abahleli bazohlela, kuhle kakhulu, kuyathakazelisa.
- I-S3 iyindlela evamile kakhulu eyenza isixhumi esibonakalayo sibe lula ngezindleko zokuzivumelanisa kancane nezinhlobo ezithile zemithwalo yomsebenzi.
Ngokwemvelo, sifuna ukunikeza ukusebenza kuyo yonke i-ClickHouse ecosystem futhi senze umsebenzi odingekayo ngaphakathi kwe-Yandex.Cloud, sinqume ukwenza isiqiniseko sokuthi wonke umphakathi we-ClickHouse uthola inzuzo kuyo. Senze i-ClickHouse ngaphezulu kwe-S3, hhayi i-ClickHouse phezu kwe-MDS. Futhi lona umsebenzi omningi.
Izinkomba:
"Isendlalelo se-Filesystem abstraction"
"Ukuhlanganiswa kwe-AWS SDK S3"
"Ukuqaliswa okuyisisekelo kwe-IDisk interafce ye-S3"
"Ukuhlanganiswa kwezinjini zokugcina log ne-IDisk interface"
"Ukusekelwa kwenjini yokungena ye-S3 ne-SeekableReadBuffer"
"Ukusekelwa kwe-Stripe Log S3"
"Isitoreji MergeTree ukwesekwa kokuqala kwe-S3"
"MergeTree ukwesekwa okugcwele kwe-S3"
"Sekela i-ReplicatedMergeTree phezu kwe-S3"
"Engeza imininingwane ezenzakalelayo kanye nezihloko zangokwezifiso zesitoreji se-s3"
"I-S3 enokumiswa kommeleli oguqukayo"
"I-S3 ene-proxy solver"
Lolu uhlu lwezicelo zokudonsa ukuze kusetshenziswe uhlelo lwamafayela olubonakalayo ku-ClickHouse. Leli inani elikhulu lezicelo zokudonsa.
Izinkomba:
"DiskS3 hardlinks ukuqaliswa ukusebenza kahle"
"Iklayenti le-S3 HTTP - Gwema ukukopisha ukusakazwa kwempendulo kumemori"
"Gwema ukukopisha ukusakazwa kwempendulo yonke kumemori ku-S3 HTTP
iklayenti"
"Ikhono lokubeka uphawu lwe-cache kanye nokukhomba amafayela we-S3 disk"
"Hambisa izingxenye zisuka ku-DiskLocal ziye ku-DiskS3 ngokuhambisana"
Kodwa umsebenzi awugcinanga lapho. Ngemva kokwenziwa kwesici, kwadingeka omunye umsebenzi owengeziwe ukuze kuthuthukiswe lo msebenzi.
Izinkomba:
"Engeza i-SelectedRows kanye nemicimbi ye-SelectedBytes"
"Engeza imicimbi yephrofayela kusuka esicelweni se-S3 kuya ku-system.events"
"Engeza QueryTimeMicrosecond, SelectQueryTimeMicroseconds kanye InsertQueryTimeMicrosecond"
Futhi-ke kwakudingeka ukuthi kwenziwe ukuthi kuhlolwe, ukusetha ukuqapha nokwenza kube lula.
Futhi konke lokhu kwenziwa ukuze wonke umphakathi, yonke i-ClickHouse ecosystem, uthole umphumela walo msebenzi.
Ake sidlulele kusizindalwazi sokuthengiselana, siye kusizindalwazi se-OLTP, esiseduze nami mathupha.
Lesi yisigaba sokuthuthukisa i-DBMS yomthombo ovulekile. Laba bafana benza umlingo wasemgwaqweni ukuthuthukisa imininingo egciniwe evuliwe yokuthengiselana.
Enye yamaphrojekthi, sisebenzisa isibonelo esingakhuluma ngaso ngokuthi senzani nokuthi senzani, i-Connection Pooler e-Postgres.
I-Postgres iyisizindalwazi senqubo. Lokhu kusho ukuthi isizindalwazi kufanele sibe noxhumo lwenethiwekhi olumbalwa ngangokunokwenzeka olusingatha ukuthengiselana.
Ngakolunye uhlangothi, endaweni yamafu, isimo esijwayelekile yilapho uxhumo oluyinkulungwane lufika eqoqweni elilodwa ngesikhathi esisodwa. Futhi umsebenzi we-pooler yoxhumano ukupakisha uxhumo oluyinkulungwane kwinani elincane lokuxhumeka kweseva.
Singasho ukuthi i-pooler yokuxhuma i-opharetha yocingo ehlela kabusha ama-byte ukuze afinyelele kahle kusizindalwazi.
Ngeshwa, alikho igama elihle lesiRashiya le-pooler yokuxhumana. Kwesinye isikhathi kuthiwa yi-multiplexer connections. Uma wazi ukuthi yini ongayibiza nge-pooler yokuxhuma, qiniseka ukuthi ungitshele, ngizojabula kakhulu ukukhuluma ulimi olufanele lobuchwepheshe lwesiRashiya.
Siphenye ama-poolers okuxhumana abefanele iqoqo lama-postgres aphethwe. Futhi i-PgBouncer bekuyisinqumo esingcono kakhulu kithi. Kodwa sihlangabezane nenani lezinkinga nge-PgBouncer. Eminyakeni eminingi edlule, uVolodya Borodin wanikeza imibiko yokuthi sisebenzisa i-PgBouncer, sithanda konke, kodwa kukhona ama-nuances, kukhona okumele kusebenze.
Futhi sasebenza. Silungise izinkinga esihlangabezane nazo, sabopha iBouncer, futhi sazama ukuphusha izicelo zokudonsa phezulu. Kodwa uchungechunge olulodwa oluyisisekelo kwakunzima ukusebenza nalo.
Bekufanele siqoqe ama-cascade kuma-Bouncers afakwe nezichibiyelo. Uma sinama-Bouncer amaningi anomucu owodwa, ukuxhunywa kwesendlalelo esiphezulu kudluliselwa kungqimba yangaphakathi yama-Bouncers. Lolu wuhlelo olungalawulwa kahle okunzima ukulakha nokukalwa liye emuva naphambili.
Sifinyelele esiphethweni sokuthi sakha eyethu i-pooler yokuxhumana, ebizwa nge-Odyssey. Sayibhala kusukela ekuqaleni.
Ngo-2019, engqungqutheleni ye-PgCon, ngethule lesi sikhwama emphakathini wonjiniyela. Manje sinezinkanyezi ezingaphansi kuka-2 ku-GitHub, okusho ukuthi iphrojekthi iyaphila, iphrojekthi idumile.
Futhi uma udala iqoqo le-Postgres ku-Yandex.Cloud, izoba yiqoqo eline-Odyssey eyakhelwe ngaphakathi, ehlelwa kabusha lapho kukala iqoqo emuva noma phambili.
Sifundeni kulo msebenzi? Ukwethula iphrojekthi eqhudelanayo kuhlale kuyisinyathelo esinolaka, kuyisinyathelo esidlulele uma sithi kunezinkinga ezingaxazululeki ngokushesha ngokwanele, ezingaxazululeki ngezikhathi ezithile ezingasifanela. Kodwa lesi isilinganiso esiphumelelayo.
I-PgBouncer yaqala ukuthuthuka ngokushesha.
Futhi manje sekuvele amanye amaphrojekthi. Isibonelo, i-pgagroal, eyakhiwe ngabathuthukisi be-Red Hat. Baphishekela imigomo efanayo futhi basebenzise imibono efanayo, kodwa, kunjalo, ngezici zabo, eziseduze nabathuthukisi be-pgagroal.
Elinye icala lokusebenza nomphakathi we-postgres libuyela endaweni ethile ngesikhathi. Lokhu ukuphinda kutholwe ngemva kokwehluleka, lokhu wukuthola kabusha kusukela kukhophi yasenqolobaneni.
Kunezipele eziningi futhi zonke zihlukile. Cishe wonke umthengisi we-Postgres unesixazululo sakhe sokulondoloza.
Uma uthatha wonke amasistimu ayisipele, dala isici se-matrix bese ubala ngokuhlekisa isihlukanisi kule matrix, kuzoba uziro. Kusho ukuthini lokhu? Kuthiwani uma uthatha ifayela elithile eliyisipele, alikwazi ukuqoqwa kusuka ezicucu zabo bonke abanye. Iyingqayizivele ekusetshenzisweni kwayo, iyingqayizivele enjongweni yayo, iyingqayizivele emibonweni efakwe kuyo. Futhi zonke ziqondile.
Ngenkathi sisasebenza kulolu daba, i-CitusData yethule iphrojekthi ye-WAL-G. Lena isistimu yokusekelayo eyenziwe ngeso lemvelo yefu. Manje i-CitusData isivele iyingxenye ye-Microsoft. Futhi ngaleso sikhathi, siyithande ngempela imibono eyayibekwe ekukhishweni kokuqala kwe-WAL-G. Futhi saqala ukufaka isandla kule phrojekthi.
Manje kunenqwaba yonjiniyela kule phrojekthi, kodwa abanikeli abaphezulu abayi-10 ku-WAL-G bahlanganisa amaYandexoids angu-6. Silethe imibono yethu eminingi lapho. Futhi, vele, sizisebenzise ngokwethu, sizivivinya ngokwethu, sazikhiphela ekukhiqizeni ngokwethu, sizisebenzisa ngokwethu, thina ngokwethu sithola ukuthi sizodlulelaphi ngokulandelayo, ngenkathi sixhumana nomphakathi omkhulu we-WAL-G.
Futhi ngokombono wethu, manje lolu hlelo lokusekelayo, okuhlanganisa ukucabangela imizamo yethu, selulungele indawo yamafu. Lezi izindleko ezinhle kakhulu zokwenza ikhophi yasenqolobaneni yama-Postgres emafini.
Kusho ukuthini? Besikhuthaza umbono omkhulu kakhulu: ukwenza ikhophi yasenqolobaneni kufanele kuvikeleke, kushibhile ukuze kusetshenziswe futhi kusheshe ngangokunokwenzeka ukubuyisela.
Kungani kufanele kube kushibhile ukusebenza? Uma kungekho okuphukile, akufanele wazi ukuthi unama-backups. Konke kusebenza kahle, uchitha i-CPU encane ngangokunokwenzeka, usebenzisa izinsiza ezincane zediski yakho ngangokunokwenzeka, futhi uthumela amabhayithi ambalwa kunethiwekhi ngangokunokwenzeka ukuze ungaphazamisi ukukhokhelwa kwezinsizakalo zakho ezibalulekile.
Futhi lapho yonke into iphuka, isibonelo, umlawuli ulahle idatha, kukhona okungahambanga kahle, futhi udinga ngokushesha ukubuyela emuva, ululama ngayo yonke imali, ngoba ufuna idatha yakho ibuye ngokushesha futhi iphelele.
Futhi sikhuthaze lo mbono olula. Futhi, kubonakala ngathi, sikwazile ukukusebenzisa.
Kodwa akugcini lapho. Besifuna enye into encane. Besifuna imininingwane eminingi ehlukene. Akuwona wonke amaklayenti ethu asebenzisa i-Postgres. Abanye abantu basebenzisa i-MySQL, i-MongoDB. Emphakathini, abanye onjiniyela baye basekela i-FoundationDB. Futhi lolu hlu lukhula njalo.
Umphakathi uyawuthanda umbono wokuthi isizindalwazi siqhutshwa endaweni ephethwe emafini. Futhi abathuthukisi bagcina imininingwane yabo egciniwe, engasekelwa ngokufana kanye ne-Postgres ngohlelo lwethu lokusekelayo.
Sifundeni endabeni le? Umkhiqizo wethu, njengesigaba sokuthuthukiswa, akuyona imigqa yekhodi, akuzona izitatimende, akuwona amafayela. Umkhiqizo wethu awuzona izicelo zokudonsa. Lena imibono esiyidlulisela emphakathini. Lona ubungcweti bezobuchwepheshe kanye nokuhamba kobuchwepheshe kubheke endaweni yamafu.
Kukhona i-database efana ne-Postgres. Ngithanda i-Postgres core kakhulu. Ngichitha isikhathi esiningi ngithuthukisa i-Postgres core nomphakathi.
Kodwa lapha kufanele kushiwo ukuthi i-Yandex.Cloud inokufakwa kwangaphakathi kolwazi oluphethwe. Futhi yaqala kudala ku-Yandex.Mail. Ubuchwepheshe manje osebuholele ekuphathweni kwe-Postgres buqoqwe lapho i-imeyili ifuna ukuthuthela e-Postgres.
Imeyili inezidingo ezifanayo kakhulu nefu. Idinga ukuthi ukwazi ukukala uye ekukhuleni okungalindelekile noma kunini kudatha yakho. Futhi i-imeyili isivele inomthwalo namakhulu ezigidi zamabhokisi eposi yenani elikhulu labasebenzisi abahlale benza izicelo eziningi.
Futhi lokhu bekuyinselelo enkulu eqenjini ebelithuthukisa i-Postgres. Ngaleso sikhathi, noma yiziphi izinkinga esihlangabezane nazo zazibikwa emphakathini. Futhi lezi zinkinga zalungiswa, futhi zalungiswa ngumphakathi kwezinye izindawo ngisho nasezingeni lokuxhaswa okukhokhelwayo kwezinye izingosi zolwazi futhi okungcono nakakhulu. Okusho ukuthi, ungathumela incwadi ku-PgSQL hacker futhi uthole impendulo kungakapheli imizuzu engama-40. Ukwesekwa okukhokhelwayo kwezinye isizindalwazi kungase kucabange ukuthi kunezinto ezibalulekile kunesiphazamisi sakho.
Manje ukufakwa kwangaphakathi kwe-Postgres amanye ama-petabytes edatha. Lezi ezinye izigidi zezicelo ngomzuzwana. Lezi izinkulungwane zamaqoqo. Inkulu kakhulu.
Kodwa kukhona i-nuance. Ayihlali kumadrayivu enethiwekhi athandwayo, kodwa kuma-hardware alula. Futhi kunendawo yokuhlola eqondene nezinto ezintsha ezithakazelisayo.
Futhi ngesikhathi esithile endaweni yokuhlola sithole umlayezo obonisa ukuthi ukungaguquki kwangaphakathi kwezinkomba zesizindalwazi kuphuliwe.
Okungaguquki uhlobo oluthile lobudlelwano esilindele ukuthi buhlale bubambe.
Isimo esibucayi kakhulu kithi. Kubonisa ukuthi enye idatha kungenzeka ilahlekile. Futhi ukulahleka kwedatha kuyinto eyinhlekelele kakhulu.
Umbono ojwayelekile esiwulandelayo kusizindalwazi esiphethwe ukuthi noma ngomzamo, kuzoba nzima ukulahlekelwa idatha. Ngisho noma uzisusa ngamabomu, usazodinga ukuziba ukungabikho kwazo isikhathi eside. Ukuphepha kwedatha kuyinkolo esiyilandela ngokuzimisela.
Futhi lapha kuvela isimo esisikisela ukuthi kungase kube nesimo okungenzeka singasilungeli. Futhi saqala ukulungiselela lesi simo.
Into yokuqala esayenza kwaba ukungcwaba izingodo kulezi zinkulungwane zamaqoqo. Sithole ukuthi imaphi amaqoqo abekwe kumadiski ane-firmware eyinkinga abelahlekelwa yizibuyekezo zekhasi ledatha. Imake yonke ikhodi yedatha ye-Postgres. Futhi simake leyo milayezo ekhombisa ukwephulwa kokushintshashintsha kwangaphakathi ngekhodi edizayinelwe ukuthola ukonakala kwedatha.
Lesi siqeshana samukelwa umphakathi ngaphandle kwezingxoxo eziningi, ngoba esimweni ngasinye kwakubonakala ukuthi kukhona okubi okwenzekile futhi okudingeka kubikwe ku-log.
Ngemuva kwalokhu, sifike lapho sinokuqapha okuskena izingodo. Futhi uma kunemiyalezo esolisayo, uvusa isikhulu esisebenza emsebenzini, futhi isikhulu somsebenzi siyasilungisa.
Kodwa! Ukuskena izingodo kuwumsebenzi oshibhile kuqoqo elilodwa futhi kubiza ngokuyingozi amaqoqo ayinkulungwane.
Sabhala isandiso esibizwa . Idala umbono we-database ongakhetha kuyo ngokushibhile nangokushesha izibalo zamaphutha adlule. Futhi uma sidinga ukuvusa isikhulu somsebenzi, khona-ke sizothola ngalokhu ngaphandle kokuskena amafayela we-gigabyte, kodwa ngokukhipha ama-byte ambalwa etafuleni le-hashi.
Lesi sandiso samukelwe, isibonelo, endaweni yokugcina ye . Uma ufuna ukuyisebenzisa, ungazifakela wena. Yebo umthombo ovulekile.
Kodwa akugcini lapho. Siqale ukusebenzisa i-Amcheck, isandiso esakhelwe umphakathi, ukuze sithole ukwephulwa okuguquguqukayo kuzinkomba.
Futhi sithole ukuthi uma usebenzisa esikalini, kukhona izimbungulu. Saqala ukuwalungisa. Ukulungisa kwethu kwamukelwe.
Sithole ukuthi lesi sandiso asikwazi ukuhlaziya izinkomba ze-GiST & GIT. Sibenze basekela. Kodwa lokhu kusekelwa kusaxoxwa ngumphakathi, ngoba lokhu kuwumsebenzi omusha futhi kunemininingwane eminingi lapho.
Futhi siphinde sathola ukuthi lapho sihlola izinkomba zokuphulwa komholi wokuphindaphinda, ku-master, konke kusebenza kahle, kodwa kuma-replicas, kumlandeli, ukucinga inkohlakalo akuphumelelanga kangako. Akukona konke okuguquguqukayo okuhloliwe. Futhi omunye ongaguquki wasihlupha kakhulu. Futhi sichithe unyaka nesigamu sixhumana nomphakathi ukuze sikwazi ukunika amandla lokhu kuhlola kumakhophi.
Sibhale ikhodi okufanele ilandele zonke ... amaphrothokholi. Sixoxe ngalesi siqeshana isikhathi eside no-Peter Gaghan ovela ku-Crunchy Data. Kwadingeka ukuthi aguqule kancane i-B-tree ekhona e-Postgres ukuze amukele lesi siqeshana. Wamukelwa. Futhi manje ukuhlola izinkomba kuma-replicas nakho sekusebenze ngokwanele ukuze kutholwe ukuphulwa esihlangabezane nakho. Okusho ukuthi, lokhu ukwephulwa okungabangelwa amaphutha ku-disk firmware, iziphazamisi ku-Postgres, izimbungulu ku-Linux kernel, nezinkinga zehadiwe. Uhlu olubanzi impela lwemithombo yezinkinga ebesizilungiselela.
Kodwa ngaphandle kwezinkomba, kunengxenye efana nenqwaba, okungukuthi indawo lapho idatha igcinwa khona. Futhi azikho izinguqulo eziningi ezingase zihlolwe.
Sinesandiso esibizwa nge-Heapcheck. Saqala ukuyithuthukisa. Futhi ngokuhambisana, kanye nathi, inkampani ye-EnterpriseDB nayo yaqala ukubhala imojula, abayibiza ngokuthi i-Heapcheck ngendlela efanayo. Kuphela esiyibize nge-PgHeapcheck, futhi bavele bayibiza ngokuthi i-Heapcheck. Banayo ngemisebenzi efanayo, isignesha ehluke kancane, kodwa enemibono efanayo. Bawasebenzise kangcono kwezinye izindawo. Futhi bayithumele kumthombo ovulekile ngaphambili.
Futhi manje sithuthukisa ukwanda kwabo, ngoba akusekho ukwanda kwabo, kodwa ukwanda komphakathi. Futhi ngokuzayo, lokhu kuyingxenye ye-kernel ezonikezwa wonke umuntu ukuze azi ngezinkinga zesikhathi esizayo kusengaphambili.
Kwezinye izindawo, size safinyelela esiphethweni sokuthi sinemibono engamanga ezinhlelweni zethu zokuqapha. Isibonelo, uhlelo lwe-1C. Uma usebenzisa i-database, i-Postgres ngezinye izikhathi ibhala idatha kuyo engakwazi ukuyifunda, kodwa i-pg_dump ayikwazi ukufunda.
Lesi simo besibukeka njengenkohlakalo ohlelweni lwethu lokuhlonza izinkinga. Iphoyisa elisebenza emsebenzini lavuswa. Umsebenzi wabheka ukuthi kwenzekani. Ngemva kwesikhathi esithile, kwafika iklayenti lathi nginezinkinga. Isikhonzi sachaza ukuthi yayiyini inkinga. Kodwa inkinga iku-Postgres core.
Ngithole ingxoxo ngalesi sici. Futhi wabhala ukuthi sihlangabezane nalesi sici futhi kwakungemnandi, umuntu wayevuka ebusuku ukuze abone ukuthi yini.
Umphakathi waphendula, “O, sidinga ngempela ukukulungisa.”
Nginesifaniso esilula. Uma uhamba ngesicathulo esinohlamvu lwesihlabathi kuso, khona-ke, ngokomthetho, ungaqhubeka - akukho nkinga. Uma uthengisa amabhuzu ezinkulungwaneni zabantu, ake senze amabhuzu angenaso isihlabathi nhlobo. Futhi uma omunye wabasebenzisi bezicathulo zakho ezogijima i-marathon, khona-ke ufuna ukwenza izicathulo ezinhle kakhulu, bese uzikala kubo bonke abasebenzisi bakho. Futhi abasebenzisi abanjalo abangalindelekile bahlala besendaweni yamafu. Kuhlala kukhona abasebenzisi abaxhaphaza iqoqo ngendlela ethile yoqobo. Kufanele uhlale ukulungele lokhu.
Sifundeni lapha? Sifunde into elula: okubalulekile wukuchazela umphakathi ukuthi kunenkinga. Uma umphakathi usuyibonile inkinga, kuphakama ukuncintisana kwemvelo ukuze kuxazululwe inkinga. Ngoba wonke umuntu ufuna ukuxazulula inkinga ebalulekile. Bonke abathengisi, bonke abaduni bayaqonda ukuthi bona ngokwabo bangakwazi ukunyathela leli reki, ngakho bafuna ukubaqeda.
Uma usebenza ngenkinga, kodwa akukhathazi muntu ngaphandle kwakho, kodwa usebenza kuyo ngokuhlelekile futhi ekugcineni kubhekwa njengenkinga, khona-ke isicelo sakho sokudonsa sizokwamukelwa nakanjani. Isiqephu sakho sizokwamukelwa, ukuthuthukiswa kwakho noma izicelo zokuthuthukiswa zizobuyekezwa umphakathi. Ekupheleni kosuku, senza isizindalwazi sibe ngcono komunye nomunye.
I-database ethokozisayo yiGreenplum. Kuyisizindalwazi esihambisana kakhulu esisekelwe ku-Postgres codebase, engiyijwayele kakhulu.
Futhi iGreenplum inomsebenzi othokozisayo - faka amatafula athuthukisiwe. Lawa amathebula ongawengeza ngokushesha. Zingaba yikholomu noma umugqa.
Kodwa bekungekho ukuhlanganisa, okusho ukuthi bekungekho ukusebenza lapho ungahlela khona idatha etholakala kuthebula ngokuvumelana nokuhleleka okukolunye lwezikhombo.
Abafana basetekisini beza kimi bathi: “Andrey, uyamazi uPostgres. Futhi lapha kucishe kufane. Shintshela kumaminithi angu-20. Kuthathe ukwenze." Ngicabange ukuthi yebo, ngiyawazi ama-Postgres, ngishintsha imizuzu engama-20 - ngidinga ukwenza lokhu.
Kodwa cha, akuphelanga imizuzu engama-20, ngayibhala ngezinyanga. Emhlanganweni wePgConf.Russia, ngaya kuHeikki Linakangas wasePivotal ngabuza: “Ingabe zikhona izinkinga ngalokhu? Kungani kungekho ukuhlanganisa okuthuthukisiwe kwetafula?” Uthi: “Uthatha idatha. Uyahlunga, uhlela kabusha. Umsebenzi nje kuphela." Mina: "Oh, yebo, udinga nje ukukuthatha futhi ukwenze." Uthi: “Yebo, sidinga izandla zamahhala ukwenza lokhu.” Ngacabanga ukuthi kufanele ngempela ngikwenze lokhu.
Futhi ezinyangeni ezimbalwa kamuva ngathumela isicelo sokudonsa esenze lokhu kusebenza. Lesi sicelo sokudonsa sibuyekezwe yi-Pivotal kanye nomphakathi. Yiqiniso, kwakukhona izimbungulu.
Kodwa okuthakazelisa kakhulu ukuthi lapho lesi sicelo sokudonsa sihlanganiswa, izimbungulu zatholakala kuGreenplum ngokwayo. Sithole ukuthi amathebula enqwaba kwesinye isikhathi aphula umsebenzi uma ehlangene. Futhi lokhu kuyinto edinga ukulungiswa. Futhi usendaweni engisanda kuyithinta. Futhi ukusabela kwami kwemvelo kwaba - kulungile, ake ngenze lokhu nami.
Ngilungise lesi siphazamisi. Ithumele isicelo sokudonsa kubalungisi. Wabulawa.
Ngemuva kwalokho kuvele ukuthi lokhu kusebenza kumele kutholwe enguqulweni yeGreenplum ye-PostgreSQL 12. Okusho ukuthi, uhambo lwemizuzu engama-20 luyaqhubeka nezigigaba ezintsha ezithokozisayo. Bekujabulisa ukuthinta intuthuko yamanje, lapho umphakathi usika izici ezintsha nezibaluleke kakhulu. Iqhwa.
Kodwa akugcinanga lapho. Ngemuva kwakho konke, kwavela ukuthi sidinga ukubhala imibhalo yakho konke lokhu.
Ngaqala ukubhala imibhalo. Ngenhlanhla, ababhali bedokhumentari abavela ku-Pivotal beza. IsiNgisi ulimi lwabo lwendabuko. Bangisiza ngemibhalo. Eqinisweni, bona ngokwabo babhala kabusha lokho engikuhlosile ngesiNgisi sangempela.
Futhi lapha, kubonakala sengathi, i-adventure iphelile. Uyazi ukuthi kwenzekani ngaleso sikhathi? Abafana basetekisini beza kimi bathi: “Kusenezigigaba ezimbili, ngayinye imizuzu eyi-10.” Futhi kufanele ngibatshele ukuthini? Ngithe manje ngizonikeza umbiko esikalini, bese sibona izigigaba zakho, ngoba lona umsebenzi othokozisayo.
Sifundeni kuleli cala? Ngoba ukusebenza ngomthombo ovulekile kuhlala kusebenza nomuntu othize, kuhlala kusebenza nomphakathi. Ngoba kuzo zonke izigaba ngasebenza nonjiniyela othile, umhloli othile, isigebengu se-inthanethi, i-documentarian ethile, umakhi othile. Angizange ngisebenze neGreenplum, ngasebenza nabantu baseGreenplum.
Kodwa! Kukhona elinye iphuzu elibalulekile - kuwumsebenzi nje. Okusho ukuthi, woza, uphuze ikhofi, ubhale ikhodi. Zonke izinhlobo zokuguquguquka ezilula ziyasebenza. Kwenze ngokujwayelekile - kuzolunga! Futhi kuwumsebenzi othokozisayo. Kunesicelo salo msebenzi esivela kumakhasimende e-Yandex.Cloud, abasebenzisi bamaqoqo ethu kokubili ngaphakathi kwe-Yandex nangaphandle. Futhi ngicabanga ukuthi inani lamaphrojekthi esibamba iqhaza kuzo lizokhula futhi ukujula kokubandakanyeka kwethu kuzokwanda.
Yilokho kuphela. Asiqhubekele emibuzweni.
Iseshini yemibuzo
Sawubona! Sinesinye isigaba semibuzo nezimpendulo. Futhi ku-studio u-Andrei Borodin. Lona umuntu osanda kukutshela mayelana nomnikelo we-Yandex.Cloud ne-Yandex ukuze uvule umthombo. Umbiko wethu manje awuphathelene ne-Cloud ngokuphelele, kodwa ngesikhathi esifanayo sisekelwe kubuchwepheshe obunjalo. Ngaphandle kwalokho okwenze ngaphakathi kwe-Yandex, bekungeke kube khona isevisi ku-Yandex.Cloud, ngakho-ke ngiyabonga kimi mathupha. Futhi umbuzo wokuqala ovela ekusakazweni: "Iyini iphrojekthi ngayinye okhulume ngayo ebhalwe kuyo?"
Isistimu yokusekelayo ku-WAL-G ibhalwe kokuthi Go. Lona omunye wamaphrojekthi amasha esisebenze kuwo. Uneminyaka emi-3 kuphela ubudala. Futhi i-database ngokuvamile imayelana nokwethembeka. Futhi lokhu kusho ukuthi i-database indala kakhulu futhi ngokuvamile ibhalwa ngo-C. Iphrojekthi ye-Postgres yaqala cishe eminyakeni engu-30 edlule. Khona-ke i-C89 bekuyisinqumo esifanele. Futhi i-Postgres ibhalwe kuyo. Imininingwane eminingi yesimanje njenge-ClickHouse ivamise ukubhalwa ku-C++. Konke ukuthuthukiswa kwesistimu kusekelwe ku-C no-C++.
Umbuzo ovela kumphathi wethu wezezimali, obhekele izindleko kwa-Cloud: "Kungani i-Cloud isebenzisa imali ekusekeleni umthombo ovulekile?"
Kunempendulo elula yomphathi wezezimali lapha. Senza lokhu ukuze senze amasevisi ethu abe ngcono. Iziphi izindlela esingenza ngazo kangcono? Singenza izinto ngempumelelo kakhudlwana, ngokushesha, futhi senze izinto zilinganiswe kakhudlwana. Kodwa kithi, le ndaba ngokuyinhloko imayelana nokwethembeka. Isibonelo, kusistimu yokwenza isipele sibuyekeza u-100% wamapeshi asebenza kuyo. Siyazi ukuthi yini ikhodi. Futhi sikhululeke kakhudlwana ukukhipha izinguqulo ezintsha ekukhiqizeni. Okusho ukuthi, okokuqala, kumayelana nokuzethemba, ukulungela intuthuko kanye nokwethembeka
Omunye umbuzo: "Ingabe izidingo zabasebenzisi bangaphandle abahlala ku-Yandex.Cloud zihlukile kubasebenzisi bangaphakathi abahlala ku-Cloud yangaphakathi?"
Iphrofayili yomthwalo, yebo, ihlukile. Kodwa ngokombono womnyango wami, wonke amacala akhethekile futhi athakazelisayo adalwe ngomthwalo ongajwayelekile. Onjiniyela abanomcabango, onjiniyela abenza okungalindelekile, kungenzeka batholakale ngaphakathi nangaphandle. Mayelana nalokhu, sonke sicishe sifane. Futhi, cishe, isici esibalulekile kuphela ngaphakathi kokusebenza kwe-Yandex yolwazi kuzoba ukuthi ngaphakathi kwe-Yandex sinemfundiso. Ngesinye isikhathi, indawo ethile yokutholakala ingena ngokuphelele emthunzini, futhi zonke izinsizakalo ze-Yandex kufanele ngandlela-thile ziqhubeke nokusebenza naphezu kwalokhu. Lona umehluko omncane. Kepha kudala ukuthuthukiswa kocwaningo okuningi kusixhumi esibonakalayo se-database kanye nesitaki senethiwekhi. Uma kungenjalo, ukufakwa kwangaphandle nokwangaphakathi kukhiqiza izicelo ezifanayo zezici nezicelo ezifanayo zokuthuthukisa ukwethembeka nokusebenza.
Umbuzo olandelayo: "Uzizwa kanjani wena mathupha mayelana nokuthi okuningi okwenzayo kusetshenziswa amanye amafu?" Ngeke sisho amagama athile, kodwa amaphrojekthi amaningi enziwe ku-Yandex.Cloud asetshenziswa emafwini abanye abantu.
Lokhu kuhle. Okokuqala, kuwuphawu lokuthi kukhona esikwenzile kahle. Futhi iklwebha i-ego. Futhi siqiniseka ngokwengeziwe ukuthi senze isinqumo esifanele. Ngakolunye uhlangothi, leli ithemba lokuthi esikhathini esizayo lokhu kuzosilethela imibono emisha, izicelo ezintsha ezivela kubasebenzisi bezinkampani zangaphandle. Izinkinga eziningi ku-GitHub zidalwe ama-sysadmins ngamanye, ama-DBA angawodwana, abaklami bezakhiwo ngabanye, onjiniyela ngabanye, kodwa ngezinye izikhathi abantu abanolwazi oluhlelekile bayafika bathi kuma-30% wamacala athile sinale nkinga futhi ake sicabange ukuthi singayixazulula kanjani. Yilokhu esikubheke ngabomvu kakhulu. Sibheke ngabomvu ukwabelana ngezipiliyoni nezinye izinkundla zamafu.
Ukhulume kakhulu ngemarathon. Ngiyazi ukuthi ugijime ibanga elide eMoscow. Ngenxa yalokho? Uthole abafana abavela ku-PostgreSQL?
Cha, u-Oleg Bartunov ugijima kakhulu. Waqeda ihora ngaphambi kwami. Sekukonke, ngijabule ngendlela engiyifinyelele. Kimina, ukuqeda nje kwaba impumelelo. Sekukonke, kuyamangaza ukuthi baningi abagijimi emphakathini we-postgres. Kimina kubonakala sengathi kukhona uhlobo oluthile lobudlelwano phakathi kwemidlalo ye-aerobic kanye nesifiso sokuhlela izinhlelo.
Ingabe uthi abekho abagijimi eClickHouse?
Ngiyazi impela ukuthi bakhona. I-ClickHouse nayo iyisizindalwazi. Phela, u-Oleg manje usengibhalela: “Sizogijima ngemva kombiko?” Lona umqondo omuhle.
Omunye umbuzo ovela ekusakazweni okuvela ku-Nikita: "Kungani uzilungisele wena isiphazamisi eGreenplum futhi ungasiniki abancane?" Yiqiniso, akucaci kahle ukuthi iyini isiphazamisi nokuthi iyiphi isevisi, kodwa mhlawumbe isho leyo okhulume ngayo.
Yebo, empeleni, yayinganikezwa othile. Bekuyikhodi engisanda kuyishintsha. Futhi kwakungokwemvelo ukuqhubeka nokwenza ngaso leso sikhathi. Empeleni, umqondo wokwabelana ngobungcweti neqembu umqondo omuhle. Nakanjani sizokwabelana ngemisebenzi yeGreenplum phakathi kwawo wonke amalungu esigaba sethu.
Njengoba sikhuluma ngabancane, nanku umbuzo. Umuntu uthathe isinqumo sokudala ukuzibophezela kokuqala ku-Postgres. Yini okufanele ayenze ukuze enze isibophezelo sokuqala?
Lona umbuzo othakazelisayo: "Ungaqala kuphi?" Ngokuvamile kunzima kakhulu ukuqala ngokuthile ku-kernel. Ku-Postgres, ngokwesibonelo, kunohlu lwezinto okufanele zenziwe. Kodwa empeleni, leli yishidi lalokho abazama ukukwenza, kodwa abazange baphumelele. Lezi yizinto eziyinkimbinkimbi. Futhi ngokuvamile ungathola izinsiza ku-ecosystem, ezinye izandiso ezingathuthukiswa, eziheha ukunakwa okuncane okuvela kubathuthukisi be-kernel. Futhi, ngokufanele, kunamaphuzu amaningi okukhula lapho. Kuhlelo Lwekhodi Lwehlobo le-Google, minyaka yonke umphakathi we-postgres ubeka phambili izihloko eziningi ezahlukene ezingase zixazululwe. Kulo nyaka, ngicabanga ukuthi sinabafundi abathathu. Omunye waze wabhala ku-WAL-G ngezihloko ezibalulekile ku-Yandex. E-Greenplum, yonke into ilula kunomphakathi wase-Postgres, ngoba abaduni be-Greenplum baphatha izicelo zokudonsa kahle kakhulu futhi baqale ukubuyekeza ngaso leso sikhathi. Ukuthumela isiqeshana ku-Postgres kuyindaba yezinyanga, kodwa iGreenplum izofika ngosuku futhi ibone okwenzile. Enye into ukuthi iGreenplum idinga ukuxazulula izinkinga zamanje. IGreenplum ayisetshenziswa kakhulu, ngakho ukuthola inkinga yakho kunzima impela. Futhi okokuqala, sidinga ukuxazulula izinkinga, kunjalo.
Source: www.habr.com