Ngiphakamisa ukuthi ufunde okulotshiwe kombiko kusukela ekuqaleni kuka-2019 ka-Andrey Borodin "Izipele nge-WAL-G. Yini ekhona ngo-2019?"
Sanibonani nonke! Igama lami ngingu-Andrey Borodin. Ngingunjiniyela kwa-Yandex. Ngibe nesithakazelo ku-PostgreSQL kusukela ngo-2016, ngemva kokuba ngikhulume nabathuthukisi futhi bathi konke kulula - uthatha ikhodi yomthombo futhi uyakha, futhi konke kuzosebenza. Futhi kusukela ngaleso sikhathi angikwazi ukuyeka - ngibhala zonke izinhlobo zezinto ezahlukene.
Enye yezinto engisebenza kuzo isistimu yokusekelayo. . Ngokuvamile, kwa-Yandex sesisebenze ezinhlelweni zokulondoloza i-PostgreSQL isikhathi eside kakhulu. Futhi ungathola ku-inthanethi uchungechunge lwemibiko eyisithupha yokuthi senza kanjani amasistimu okulondoloza ikhophi. Futhi njalo ngonyaka zishintsha kancane, zithuthuke kancane, futhi zithembeke kakhulu.
Kodwa namuhla umbiko awugcini nje ngalokho esikwenzile, kodwa futhi mayelana nokuthi kulula kangakanani nokuthi yini. Bangaki kini asebeyibukile imibiko yami mayelana ne-WAL-G? Kuhle ukuthi abantu abambalwa ababukanga, ngoba ngizoqala ngento elula kakhulu.
Uma kungazelelwe uneqoqo le-PostgreSQL, futhi ngicabanga ukuthi wonke umuntu unazo ezimbalwa, futhi kungazelelwe alukho uhlelo lokusekelayo okwamanje, udinga ukuthola noma yisiphi isitoreji se-S3 noma isitoreji esihambisanayo se-Google Cloud.
Isibonelo, ungeza endaweni yethu futhi uthathe ikhodi yokuphromotha ye-Yandex Object Storage, ehambisana ne-S3.
Bese udala ibhakede. Isitsha nje solwazi.
Dala umsebenzisi wesevisi.
Dala ukhiye wokufinyelela womsebenzisi wesevisi: aws-s3-key.
Landa ukukhishwa kwakamuva okuzinzile kwe-WAL-G.
Ingabe ukukhishwa kwethu kwangaphambilini kwehluke kanjani kokukhishiwe? Ngivame ukucelwa ukuthi ngikhulule kusenesikhathi. Futhi uma kungekho bug enguqulweni isikhathi esanele, isibonelo, inyanga, bese ngiyayikhulula. Nakhu lokhu kukhululwa kukaNovemba. Futhi lokhu kusho ukuthi nyanga zonke sithole uhlobo oluthile lwesiphazamisi, ngokuvamile ekusebenzeni okungabalulekile, kodwa asikakakhiphi ukukhishwa. Inguqulo yangaphambilini ingoNovemba kuphela. Azikho iziphazamisi ezaziwa yithi kuyo, okungukuthi iziphazamisi zengeziwe njengoba iphrojekthi iqhubeka.
Uma usuyidawunilode i-WAL-G, ungasebenzisa umyalo olula “wohlu lwesipele”, udlule ezintweni eziguquguqukayo zemvelo. Futhi izoxhuma kokuthi Isitoreji Sento futhi ikutshele ukuthi yiziphi izipele onazo. Ekuqaleni, kunjalo, akufanele ube nama-backups. Iphuzu lalesi silayidi ukukhombisa ukuthi yonke into ilula. Lona umyalo wekhonsoli owamukela okuguquguqukayo kwemvelo futhi wenze imiyalo engaphansi.
Ngemva kwalokhu, ungenza isipele sakho sokuqala. Ithi “backup-push” ku-WAL-G futhi ucacise ku-WAL-G indawo ye-pgdata yeqoqo lakho. Futhi cishe, i-PostgreSQL izokutshela, uma ungakabi nalo uhlelo lokusekelayo, udinga ukunika amandla "imodi yomlando".
Lokhu kusho ukuthi udinga ukuya kuzilungiselelo bese uvule okuthi “archive_mode = on” bese wengeza okuthi “archive_command”, okuphinde kube umyalo ongaphansi ku-WAL-G. Kodwa ngesizathu esithile abantu bavame ukusebenzisa imibhalo yebha kulesi sihloko bese beyisonga nge-WAL-G. Ngicela ungakwenzi lokhu. Sebenzisa ukusebenza okutholakala ku-WAL-G. Uma kukhona okushodayo, bhalela . I-WAL-G ithatha ngokuthi iwukuphela kohlelo olusebenza ku-archive_command.
Sisebenzisa i-WAL-G ikakhulukazi ukuze sakhe Iqoqo Lokutholakala Okuphezulu ekuphathweni Kwedathabhesi ye-Yandex.
Futhi imvamisa isetshenziswa ku-topology ye-Master eyodwa kanye nezimpendulo ezimbalwa. Ngesikhathi esifanayo, yenza ikhophi yokusekelayo ku-Yandex Object Storage.
Izimo ezivame kakhulu ukudala amakhophi eqoqo kusetshenziswa i-Point ngesikhathi sokuthola kabusha. Kodwa kulokhu, ukusebenza kohlelo lokusekelayo akubalulekile kangako kithi. Sidinga nje ukulayisha iqoqo elisha kusukela kusipele.
Ngokuvamile, sidinga ukusebenza kwesistimu yokusekelayo lapho singeza inodi entsha. Kungani ibalulekile? Ngokuvamile abantu bengeza inodi entsha kuqoqo ngoba iqoqo elikhona alikwazi ukuphatha umthwalo ofundiwe. Badinga ukungeza ikhophi entsha. Uma sengeza umthwalo osuka ku-pg_basebackup ukuya ku-Master, khona-ke i-Master ingase igoqe. Ngakho-ke, bekubaluleke kakhulu kithi ukuthi sikwazi ukulayisha ngokushesha i-node entsha kusuka kungobo yomlando, sidale umthwalo omncane ku-Master.
Nesinye isimo esifanayo. Lesi isidingo sokuqala kabusha i-Master endala ngemva kokushintsha i-Cluster Master kusukela kuSikhungo Sedatha okwalahleka ngaso ukuxhumana.
- Njengomphumela, lapho sibumba izimfuneko zesistimu yokukopisha, sabona ukuthi i-pg_basebackup ayifaneleki kithi uma isebenza emafini.
- Besifuna ukukwazi ukucindezela idatha yethu. Kodwa cishe noma yiluphi uhlelo lokusekelayo ngaphandle kwalokho oluza ebhokisini luzohlinzeka ngokucindezelwa kwedatha.
- Besifuna ukufanisa yonke into ngoba umsebenzisi osefini uthenga inani elikhulu lama-processor cores. Kodwa uma singenayo i-parallelism ekusebenzeni okuthile, inani elikhulu lama-cores liba yize.
- Sidinga ukubethela ngoba ngokuvamile idatha akuyona eyethu futhi ayikwazi ukugcinwa ngombhalo ocacile. Kodwa-ke, umnikelo wethu ku-WAL-G waqala ngokubethela. Siqede ukubethela ku-WAL-G, ngemuva kwalokho sabuzwa: "Mhlawumbe omunye wethu uzothuthukisa iphrojekthi?" Futhi kusukela lapho bengisebenza ne-WAL-G isikhathi esingaphezu konyaka.
- Siphinde sadinga ukucindezelwa kwezinsiza, ngoba ngokuhamba kwesikhathi sisebenzisa ifu, sathola ukuthi ngezinye izikhathi abantu banomthwalo obalulekile wegrosa ebusuku futhi lo mthwalo awukwazi ukuphazanyiswa. Kungakho sengeze i-resource throttling.
- Kanye nokufakwa kuhlu nokuphatha.
- Nokuqinisekisa.
Sibheke amathuluzi amaningi ahlukene. Ngenhlanhla, sinokukhetha okukhulu ku-PostgreSQL. Futhi yonke indawo sasishoda ngokuthile, umsebenzi owodwa omncane, okunye okuncane okuncane.
Futhi ngemva kokuhlola izinhlelo ezikhona, sifinyelele esiphethweni sokuthi sizothuthukisa i-WAL-G. Kwakuyiphrojekthi entsha ngaleso sikhathi. Kwakulula kakhulu ukuthonya intuthuko ekuqondeni nengqalasizinda yefu yesistimu yokusekelayo.
I-ideology eyinhloko esinamathela kuyo ukuthi i-WAL-G kufanele ibe lula njenge-balalaika.
I-WAL-G inemiyalo emi-4. Lokhu:
I-WAL-PUSH – gcina i-shaft kungobo yomlando.
I-WAL-FETCH - thola shaft.
BACKUP-PUSH – yenza ikhophi yasenqolobaneni.
BACKUP-FETCH - thola ikhophi yasenqolobaneni ohlelweni lokusekelayo.
Eqinisweni, i-WAL-G iphinde ibe nokuphatha lezi zipele, okungukuthi ukufaka kuhlu nokususa amarekhodi nezipele emlandweni ezingasadingeki okwamanje.
Omunye wemisebenzi ebalulekile kithi umsebenzi wokwenza amakhophi e-delta.
Amakhophi e-Delta asho ukuthi asakhi isipele esigcwele salo lonke iqoqo, kodwa kuphela amakhasi ashintshiwe wamafayela ashintshiwe kuqoqo. Kubukeka sengathi ngokusebenza lokhu kufana ncamashi nekhono lokululama usebenzisa i-WAL. Kodwa singakwazi ukugoqa isipele se-delta esinochungechunge olulodwa lwe-WAL ngokuhambisana. Ngokufanelekile, uma sinekhophi yasenqolobaneni eyisisekelo eyenziwe ngoMgqibelo, i-delta igcina ikhophi nsuku zonke, futhi ngoLwesine siyehluleka, lapho-ke sidinga ukugoqa izipele ze-delta ezi-4 namahora angu-10 we-WAL. Kuzothatha cishe isikhathi esifanayo ngoba izipele ze-delta zigoqa ngokuhambisana.
I-deltas esekwe ku-LSN - lokhu kusho ukuthi lapho senza ikhophi yasenqolobaneni, sizodinga ukuhlanganisa ikhasi ngalinye futhi sihlole i-LSN yalo ne-LSN yesipele sangaphambilini ukuze siqonde ukuthi ishintshile. Noma yiliphi ikhasi okungenzeka liqukethe idatha eshintshiwe kufanele libe khona kusipele se-delta.
Njengoba ngishilo, kunakwe kakhulu ukuhambisana.
Kepha i-API yengobo yomlando ku-PostgreSQL iyahambisana. I-PostgreSQL igcina kungobo yomlando ifayela elilodwa le-WAL futhi lapho ilibuyisela licela ifayela elilodwa le-WAL. Kodwa uma isizindalwazi sicela ifayela elilodwa le-WAL lisebenzisa umyalo othi "WAL-FETCH", sibiza umyalo othi "WAL-PREFETCH", olungiselela amafayela angu-8 alandelayo ukuze alande idatha esitolo sezinto ngokuhambisana.
Futhi lapho isizindalwazi sisicela ukuthi sigcine ifayela elilodwa kungobo yomlando, sibheka archive_status futhi sibone ukuthi akhona yini amanye amafayela we-WAL. Futhi sizama ukulanda i-WAL ngokuhambisana. Lokhu kunikeza inzuzo enkulu yokusebenza futhi kunciphisa ngokuphawulekayo ibanga lenani lama-WAL angagciniwe. Abaningi abathuthukisi besistimu yokusekelayo bakholelwa ukuthi lena isistimu eyingozi kakhulu ngoba sithembele olwazini lwethu lwabangaphakathi bekhodi okungeyona i-PostgreSQL API. I-PostgreSQL ayikuqinisekisi ubukhona befolda ye-archive_status yethu futhi ayiqinisekisi i-semantics, ukuba khona kwezimpawu zokulungela amafayela we-WAL lapho. Noma kunjalo, sifunda ikhodi yomthombo, siyabona ukuthi lokhu kunjalo futhi sizama ukukusebenzisa. Futhi silawula isiqondiso lapho i-PostgreSQL ithuthukiswa khona; uma kungazelelwe le mishini iphukile, sizoyeka ukuyisebenzisa.
Ngendlela yayo emsulwa, i-WAL delta esekwe ku-LSN idinga ukufunda noma yiliphi ifayela leqoqo lapho isikhathi salo ohlelweni lwefayela sishintshile kusukela ekhophini yangaphambilini. Sahlala nalokhu isikhathi eside, cishe unyaka. Futhi ekugcineni sifinyelele esiphethweni sokuthi sine-WAL deltas.
Lokhu kusho ukuthi ngaso sonke isikhathi lapho sifaka kungobo yomlando i-WAL ku-Master, asigcini nje ngokuyicindezela, siyibethele futhi siyithumele kunethiwekhi, kodwa futhi siyayifunda ngesikhathi esifanayo. Sihlaziya futhi sifunde amarekhodi kuwo. Siyaqonda ukuthi yimaphi amabhlogo ashintshile futhi siqoqa amafayela e-delta.
Ifayela le-delta lichaza ububanzi obuthile bamafayela e-WAL, lichaza ulwazi mayelana nokuthi yimaphi amabhlogo ashintshiwe kulobu bubanzi be-WAL. Bese-ke lawa mafayela e-delta nawo afakwa kungobo yomlando.
Lapha sibhekene neqiniso lokuthi safanisa yonke into ngokushesha okukhulu, kodwa asikwazi ukufunda umlando olandelanayo ngokuhambisana, ngoba engxenyeni ethile singase sihlangabezane nokuphela kwerekhodi le-WAL langaphambili, esingakabi nalutho esingaxhuma nalo, ngoba ukufunda okuhambisanayo kwaholela ekutheni siqale sihlaziye ikusasa, elingakabi nalo okwedlule.
Ngenxa yalokho, kudingeke ukuthi sibeke izingcezu ezingaqondakali kumafayela angu-_delta_partial. Ngenxa yalokho, lapho sibuyela esikhathini esedlule, sizonamathisela izingcezu zerekhodi le-WAL libe linye, ngemva kwalokho sizolihlukanisa futhi siqonde ukuthi yini eshintshile kuyo.
Uma emlandweni wokuhlukanisa i-shaft yethu kukhona okungenani iphuzu elilodwa lapho singaqondi ukuthi kwenzekani, ngakho-ke, ngokufanele, ngesikhathi sokulondoloza okulandelayo sizophoqeleka ukuthi sifunde lonke iqoqo futhi, njengoba senzile nge-LSN evamile. -based delta.
Ngenxa yalokho, konke ukuhlupheka kwethu kuholele ekutheni sivule imithombo yolwazi yomtapo wolwazi we-WAL-G. Ngokwazi kwami, akekho oyisebenzisayo okwamanje, kodwa uma kukhona ofuna ukuyibhala futhi ayisebenzise, isendaweni yomphakathi. (Isixhumanisi esibuyekeziwe )
Ngenxa yalokho, konke ukugeleza kolwazi kubukeka kuyinkimbinkimbi. I-Master yethu igcina kungobo yomlando i-shaft futhi igcina kungobo yomlando amafayela e-delta. Futhi ikhophi eyenza ikhophi eyisipele kufanele yamukele amafayela e-delta ngesikhathi esidlulile phakathi kwezipele. Kulesi simo, izingxenye zomlando zizodinga ukwengezwa ngobuningi futhi zihlukaniswe, ngoba akuwona wonke umlando ongena kumasegimenti amakhulu. Futhi ngemva kwalokhu kuphela i-replica ingagcina kungobo yomlando isipele esigcwele se-delta.
Emagrafu yonke into ibonakala ilula kakhulu. Lokhu ukulanda kusuka kwelinye lamaqoqo ethu angempela. Sine-LSN-based, eyenziwe ngosuku olulodwa. Futhi siyabona ukuthi isipele se-delta esekwe ku-LSN besiqala ngehora lesithathu ekuseni kuya kwelesihlanu ekuseni. Lona umthwalo enanini lama-processor cores. I-WAL-delta isithathe lapha cishe imizuzu engu-20. Okusho ukuthi, yashesha kakhulu, kodwa ngesikhathi esifanayo kwakukhona ukushintshana okunamandla phezu kwenethiwekhi.
Njengoba sinolwazi mayelana nokuthi yimaphi amabhulokhi ashintshile nokuthi ngasiphi isikhathi emlandweni wesizindalwazi, siye phambili futhi sanquma ukuhlanganisa ukusebenza - isandiso se-PostgreSQL esibizwa ngokuthi “pg_prefaulter”
Lokhu kusho ukuthi uma isisekelo sokuma senza umyalo wokubuyisela, sitshela i-WAL-G ukuthi ilande ifayela le-WAL elilandelayo. Siyaqonda cishe ukuthi iyiphi idatha evimba inqubo yokutholwa kwe-WAL ezofinyelela maduze futhi siqale umsebenzi wokufunda kulawa mabhulokhi. Lokhu kwenziwa ukuze kwandiswe ukusebenza kwezilawuli ze-SSD. Ngoba i-WAL roll izofinyelela ekhasini elidinga ukushintshwa. Leli khasi likudiski futhi alikho kunqolobane yekhasi. Futhi uzolinda ngokuvumelanisa ukuze leli khasi lifike. Kodwa eduze kukhona i-WAL-G, eyazi ukuthi kumakhulu ambalwa amamegabyte alandelayo we-WAL sizodinga amakhasi athile futhi ngesikhathi esifanayo siqala ukuwafudumeza. Iqala ukufinyelela kumadiski amaningi ukuze kwenziwe ngokufana. Lokhu kusebenza kahle kumadrayivu e-SSD, kodwa, ngeshwa, akusebenzi neze ku-hard drive, ngoba siphazamisa kuphela ngokutshelwa kwethu.
Yilokhu okukukhodi manje.
Kunezici esingathanda ukuzengeza.
Lesi sithombe sibonisa ukuthi i-WAL-delta ithatha isikhathi esifushane uma kuqhathaniswa. Futhi lokhu ukufunda izinguquko ezenzeka kusizindalwazi phakathi nosuku. Singenza i-WAL-delta hhayi ebusuku kuphela, ngoba ayisewona umthombo obalulekile womthwalo. Singafunda i-WAL-delta njalo ngomzuzu ngoba ishibhile. Ngomzuzu owodwa singaskena zonke izinguquko ezenzeke kuqoqo. Futhi lokhu kungabizwa ngokuthi "i-WAL-delta esheshayo".
Iphuzu liwukuthi uma sibuyisela iqoqo, sinciphisa isibalo sezindaba okufanele sizigoqe ngokulandelana. Okusho ukuthi, inani le-WAL okumele i-PostgreSQL rolls lincishiswe, ngoba kuthatha isikhathi esibalulekile.
Kodwa akugcini lapho. Uma sazi ukuthi ibhulokhi elithile lizoshintshwa libe sezingeni lokuvumelana kwekhophi yasenqolobaneni, ngeke sikwazi ukukushintsha esikhathini esidlule. Okusho ukuthi, manje sesinokulungiselelwa kwefayela ngefayela lokudluliswa kwe-WAL-delta. Lokhu kusho ukuthi uma, ngokwesibonelo, ngoLwesibili elinye ithebula lisuswe ngokuphelele noma amanye amafayela asuswa ngokuphelele etafuleni, lapho-ke i-delta igingqika ngoMsombuluko bese i-pg_basebackup yangoMgqibelo ibuyiselwa, ngeke size siyidale le datha.
Sifuna ukunweba lobu buchwepheshe bufike ezingeni lekhasi. Okusho ukuthi, uma ingxenye ethile yefayela ishintsha ngoMsombuluko, kodwa izobhalwa ngaphezulu ngoLwesithathu, khona-ke lapho sibuyisela iphuzu ngoLwesine, akudingeki sibhale izinguqulo ezimbalwa zokuqala zamakhasi kudiski.
Kodwa lona kusewumbono okuxoxwa ngawo ngenkuthalo ngaphakathi kwethu, kodwa awukakafinyeleli kukhodi.
Sifuna ukwenza esinye isici ku-WAL-G. Sifuna ukuyenza inwetshwe ngoba sidinga ukusekela izingosi zolwazi ezihlukene futhi singathanda ukwazi ukubhekana nokuphathwa kokulondoloza ngendlela efanayo. Kodwa inkinga ukuthi i-MySQL APIs ihluke kakhulu. Ku-MySQL, i-PITR ayisekelwe kulogi ye-WAL ebonakalayo, kodwa ku-binlog. Futhi asinalo uhlelo lokugcina kungobo yomlando ku-MySQL olungatshela uhlelo oluthile lwangaphandle ukuthi le binlog isiqediwe futhi idinga ukufakwa kungobo yomlando. Sidinga ukuma endaweni ethile ku-cron ne-database futhi sihlole ukuthi kukhona yini okulungile?
Futhi ngendlela efanayo, ngesikhathi sokubuyiselwa kwe-MySQL, awukho umyalo wokubuyisela ongatshela uhlelo ukuthi ngidinga amafayela anjalo nanjalo. Ngaphambi kokuthi uqale ukwakha kabusha iqoqo lakho, udinga ukwazi ukuthi yimaphi amafayela ozowadinga. Wena ngokwakho udinga ukuqagela ukuthi yimaphi amafayela ozowadinga. Kodwa lezi zinkinga zingase zikwazi ukugwenywa ngandlela-thile. (Ukucaciswa: I-MySQL isivele isekelwe)
Embikweni, bengifuna futhi ukukhuluma ngalezo zimo lapho i-WAL-G ingakufanelanga.
Uma ungenayo i-replica ehambisanayo, i-WAL-G ayiqinisekisi ukuthi ingxenye yokugcina izogcinwa. Futhi uma ukufakwa kungobo yomlando kusasele ngemuva kwezingxenye ezimbalwa zokugcina zomlando, lokho kuyingozi. Uma ingekho i-replica ehambisanayo, ngeke ngincome ukusebenzisa i-WAL-G. Noma kunjalo, yakhelwe ngokuyinhloko ukufakwa kwamafu, okusho isixazululo Sokutholakala Okuphezulu nge-synchronous replica, enesibopho sokuphepha kwamabhayithi okugcina azibophezele.
Ngivame ukubona abantu bezama ukusebenzisa i-WAL-G ne-WAL-E ngesikhathi esisodwa. Sisekela ukuhambisana okusemuva ngomqondo wokuthi i-WAL-G ingakwazi ukubuyisela ifayela ukusuka ku-WAL-E futhi ingabuyisela ikhophi yasenqolobaneni eyenziwe nge-WAL-E. Kodwa njengoba zombili lezi zinhlelo zisebenzisa i-parallel wal-push, ziqala ukuntshontshelana amafayela. Uma siyilungisa ku-WAL-G, isazohlala ku-WAL-E. Ku-WAL-E, ibheka isimo sengobo yomlando, ibona amafayela aqediwe futhi iwagcine kungobo yomlando, kuyilapho ezinye izinhlelo ngeke zazi ukuthi leli fayela le-WAL belikhona, ngoba i-PostgreSQL ngeke izame ukuligcina okwesibili.
Yini esizoyilungisa lapha ohlangothini lwe-WAL-G? Ngeke sazise i-PostgreSQL ukuthi leli fayela lidluliselwe ngokufana, futhi lapho i-PostgreSQL isicela ukuthi siyigcine kungobo yomlando, sizokwazi kakade ukuthi ifayela elinjalo elinalesi sikhathi semodi futhi ngale md5 selivele ligciniwe futhi sizomane sithi i-PostgreSQL - KULUNGILE, yonke into isimi ngomumo ngaphandle kokwenza lutho.
Kodwa le nkinga ayinakwenzeka ukuthi ilungiswe ohlangothini lwe-WAL-E, ngakho-ke okwamanje akunakwenzeka ukudala umyalo wengobo yomlando ozogcina ifayela kungobo yomlando kukho kokubili i-WAL-G ne-WAL-E.
Ngaphezu kwalokho, kunezimo lapho i-WAL-G ingakulungele manje, kodwa sizoyilungisa nakanjani.
Okokuqala, okwamanje asinakho ukuqinisekiswa kwekhophi yasenqolobaneni eyakhelwe ngaphakathi. Asinakho ukuqinisekiswa ngesikhathi sokwenza ikhophi yasenqolobaneni noma sokutholwa. Yiqiniso, lokhu kwenziwa efwini. Kodwa lokhu kwenziwa ngokuhlola kuqala, ngokumane ubuyisele iqoqo. Ngingathanda ukunikeza lokhu kusebenza kubasebenzisi. Kodwa ngokuqinisekisa, ngicabanga ukuthi ku-WAL-G kuzokwazi ukubuyisela iqoqo futhi liqale, futhi liqhube ukuhlolwa kwentuthu: pg_dumpall kuya ku-/dev/null kanye nokuqinisekiswa kwenkomba ye-amcheck.
Okwamanje ku-WAL-G ayikho indlela yokuhlehlisa isipele esisodwa esivela ku-WAL. Okusho ukuthi, sisekela iwindi elithile. Isibonelo, ukugcina izinsuku eziyisikhombisa zokugcina, ukugcina izipele eziyishumi zokugcina, ukugcina izipele ezintathu zokugcina ezigcwele. Isikhathi esiningi abantu beza bathi: “Sidinga ikhophi yasenqolobaneni yalokho okwenzeka ngoNcibijane futhi sifuna ukukugcina unomphela.” I-WAL-G ayikwazi ukwenza lokhu okwamanje. (Qaphela - Lokhu sekuvele kulungisiwe. Funda kabanzi - Inketho yokulondoloza uphawu ku )
Futhi asinakho ukuhlola kwekhasi nokuhlolwa kobuqotho kwawo wonke amasegimenti e-shaft lapho kuqinisekiswa i-PITR.
Kusuka kukho konke lokhu ngihlanganise iphrojekthi ye-Google Summer of Code. Uma wazi abafundi abahlakaniphile abangathanda ukubhala okuthile kokuthi Go futhi bathole izinkulungwane ezimbalwa zamadola enkampanini eyodwa enohlamvu “G”, bese uncoma iphrojekthi yethu kubo. Ngizosebenza njengomeluleki wale phrojekthi, bangayenza. Uma bengekho abafundi, ngizoyithatha ngizenzele ngokwami ehlobo.
Futhi sinezinye izinkinga eziningi ezincane esizilungisa kancane kancane. Futhi kwenzeka izinto ezingavamile.
Isibonelo, uma unikeza i-WAL-G isipele esingenalutho, izovele iwe. Isibonelo, uma umtshela ukuthi udinga ukwenza isipele ifolda engenalutho. Ifayela le-pg_control ngeke libe khona. Futhi uzocabanga ukuthi kukhona angakuqondi. Ngombono, kulokhu udinga ukubhala umlayezo ojwayelekile kumsebenzisi ukuze umchazele indlela yokusebenzisa ithuluzi. Kodwa lokhu akusona ngisho isici sohlelo, kodwa isici solimi oluhle, oluqondakalayo.
Asazi ukwenza isipele ungaxhunyiwe ku-inthanethi. Uma isizindalwazi siqamba amanga, asikwazi ukwenza ikhophi yasenqolobaneni. Kodwa konke kulula kakhulu lapha. Sibiza izipele nge-LSN ngenkathi iqala. I-LSN yesisekelo esiyisisekelo kufanele ifundwe efayeleni lokulawula. Futhi lokhu kuyisici esinjalo esingafezeki. Amasistimu amaningi okulondoloza angakwazi ukwenza isipele isizindalwazi esiyisisekelo. Futhi kulula.
Okwamanje asikwazi ukubhekana nokushoda kwesikhala sokulondoloza ngendlela efanele. Ngoba sivame ukusebenza ngezipele ezinkulu ekhaya. Futhi abazange baqonde ngakho. Kodwa uma othile efuna ukuhlela okuthi Hamba khona manje, engeza ukuphatha amaphutha angaphandle kwesikhala ebhakedeni. Ngizosibheka nakanjani isicelo sokudonsa.
Futhi into eyinhloko esikhathazayo ukuthi sifuna izivivinyo eziningi zokuhlanganisa i-docker ngangokunokwenzeka ezihlola izimo ezihlukahlukene. Njengamanje sihlola kuphela izimo eziyisisekelo. Kukho konke ukuzibophezela, kodwa sifuna ukuhlola ukuzibophezela ngakunye konke ukusebenza esikusekelayo. Ikakhulukazi, ngokwesibonelo, sizoba nokusekelwa okwanele kwe-PostgreSQL 9.4-9.5. Siyabasekela ngoba umphakathi uyayisekela i-PostgreSQL, kodwa asibheki ukuzibophezela ngakunye ukuze siqiniseke ukuthi yonke into ayiphukile. Futhi kimi kubonakala sengathi lokhu kuyingozi enkulu.
Sine-WAL-G esebenza kumaqoqo angaphezu kwenkulungwane ku-Yandex Database management. Futhi yenza ikhophi yasenqolobaneni engamakhulu ambalwa edatha nsuku zonke.
Sine-TODO eningi kukhodi yethu. Uma ufuna ukuhlela woza, silinde izicelo zokudonsa, silinde imibuzo.
Imibuzo yakho
Sawubona! Ngiyabonga! Ukuqagela kwami ukuthi uma usebenzisa i-WAL-delta, kungenzeka ukuthi uthembele kakhulu ekubhaleni okugcwele ikhasi. Futhi uma kunjalo, uye wenza izivivinyo? Ubonise igrafu enhle. Iba muhle kangakanani uma i-FPW icishiwe?
Ukubhala kwekhasi eligcwele kunikwe amandla kithi, asizamanga ukukukhubaza. Okusho ukuthi, mina, njengonjiniyela, angizange ngizame ukuyicisha. Abaphathi besistimu abenze ucwaningo cishe balucwaningile lolu daba. Kodwa siyayidinga i-FPW. Cishe akekho oyicishayo, ngoba uma kungenjalo akunakwenzeka ukuthatha ikhophi yasenqolobaneni.
Siyabonga ngombiko! Nginemibuzo emibili. Umbuzo wokuqala uthi kuzokwenzekani ngamatafula?
Silindele isicelo sokudonsa. Imininingo yethu egciniwe iphila kuma-SSD kanye namadiski e-NMVE futhi asisidingi ngempela lesi sici. Angikakulungeli ukuchitha isikhathi esibucayi njengamanje ekwenzeni kahle. Ngincoma ngenhliziyo yonke ukuthi sikweseke lokhu. Kunabantu abebeseseka, kodwa beseka ngendlela evumelana nabo. Benza imfologo, kodwa abazenzi izicelo zokudonsa. (Kwengezwe kunguqulo 0.2.13)
Nombuzo wesibili. Ushilo ekuqaleni ukuthi i-WAL-G icabanga ukuthi isebenza yodwa futhi akukho ukusonga okudingekayo. Ngisebenzisa ama-wrapper ngokwami. Kungani kungafanele zisetshenziswe?
Sifuna kube lula njenge-balaika. Lokhu kusho ukuthi awudingi lutho ngaphandle kwe-balalaika. Sifuna uhlelo lube lula. Uma unokusebenza okudingeka ukwenze kusikripthi, woza uzositshela - sizokwenza kokuthi Go.
Sawubona! Siyabonga ngombiko! Asikwazanga ukwenza i-WAL-G isebenze ngokususa ukubethela kwe-GPG. Ibhala ngemfihlo ngokujwayelekile, kodwa ayifuni ukususa ukubethela. Ingabe yinto engazange isisebenzele? Isimo siyadabukisa.
Dala inkinga ku-GitHub futhi masiyithole.
Okusho ukuthi, awukaze uhlangabezane nalokhu?
Kunesici sombiko wephutha lapho i-WAL-G ingaqondi ukuthi hlobo luni lwefayela, iyabuza: "Mhlawumbe ibethelwe?" Mhlawumbe inkinga ayikona nhlobo ukubethela. Ngifuna ukuthuthukisa ukugawulwa kwemithi kulesi sihloko. Kumele ayichaze. Okwamanje sisebenza ngalesi sihloko ngomqondo wokuthi asithandi ngempela ukuthi isistimu yokuthola okhiye basesidlangalaleni nabayimfihlo ihlelwa kanjani. Ngoba sibiza i-GPG yangaphandle ukuze isinike okhiye bayo. Bese sithatha lezi zihluthulelo bese sizidlulisela ku-GPG yangaphakathi, eyi-PGP evulekile, ehlanganiswe kithi ngaphakathi kwe-WAL-G, futhi lapho sikubiza ukubethela. Mayelana nalokhu, sifuna ukuthuthukisa uhlelo futhi sifuna ukusekela ukubethela kwe-Libsodium (Kungezwe kunguqulo 0.2.15). Vele, ukwenza amakhodi kufanele kusebenze, ake sikuthole - udinga uphawu oluningi kunamagama ambalwa. Ungakwazi ukubuthana ekamelweni lesikhulumi ngesinye isikhathi futhi ubheke uhlelo. (Ukubethela kwe-PGP ngaphandle kwe-GPG yangaphandle - v0.2.9)
Sawubona! Siyabonga ngombiko! Nginemibuzo emibili. Nginesifiso esiyinqaba sokwenza pg_basebackup futhi WAL ungene kubahlinzeki ababili, i.e. Ngifuna ukwenza ifu elilodwa nelinye. Ingabe ikhona indlela yokwenza lokhu?
Lokhu akukho manje, kodwa umqondo othakazelisayo.
Angimethembi umhlinzeki oyedwa, ngifuna ukuba okufanayo komunye, uma kwenzeka.
Umbono uyathakazelisa. Ngobuchwepheshe, lokhu akunzima neze ukukusebenzisa. Ukuvikela umbono ukuthi ungalahleki, ngingakucela ukuthi wenze inkinga ku-GitHub?
O impela.
Bese-ke, lapho abafundi beza ku-Google Summer of Code, sizobengeza kuphrojekthi ukuze kube nomsebenzi omningi wokuthola okwengeziwe kubo.
Nombuzo wesibili. Kunenkinga ku-GitHub. Ngicabanga ukuthi isivele ivaliwe. Kukhona ukwethuka ngesikhathi sokubuyisela. Futhi ukuze uyinqobe, wenze umhlangano ohlukile. Ilungile ezindabeni. Futhi kukhona inketho yokwenza indawo eguquguqukayo emculweni owodwa. Futhi yingakho isebenza kancane kakhulu. Futhi sihlangabezane nale nkinga, futhi ayikalungiswa.
Inkinga ukuthi ngesizathu esithile isitoreji (i-CEPH) sisetha kabusha ukuxhumeka uma sifika kuso ngokuhambisana okuphezulu. Yini engenziwa ngalokhu? I-logic yokuzama kabusha ibukeka kanje. Sizama ukulanda ifayela futhi. Ngokudlula okukodwa, sibe nenani lamafayela angalandiwe, sizokwenza elesibili labo bonke abangangenanga. Futhi inqobo nje uma okungenani ifayela elilodwa lilayishwa ngokuphindaphinda, siyaphinda futhi siphinde futhi siphinde. Sithuthukise ingqondo yokuzama kabusha - i-exponential backoff. Kepha akucaci ngokuphelele ukuthi yini okufanele yenziwe ngeqiniso lokuthi ukuxhumana kumane kwephuke ohlangothini lwesistimu yokugcina. Okusho ukuthi, uma silayisha ekusakazweni okukodwa, akunqamuki lokhu kuxhumana. Yini esingayithuthukisa lapha? Sine-throttling yenethiwekhi, singakhawulela ukuxhumana ngakunye ngenani lamabhayithi eliwathumelayo. Uma kungenjalo, angazi ukuthi ngingabhekana kanjani neqiniso lokuthi isitoreji sento asisivumeli ukuthi silande noma silande kuso ngokuhambisana.
Ayikho i-SLA? Akulotshwanga yini ngabo ukuthi bazivumela kanjani ukuba bahlushwe?
Iphuzu liwukuthi abantu abaza nalo mbuzo bavame ukuba ne-vault yabo. Okusho ukuthi, akekho ovela e-Amazon noma ku-Google Cloud noma ku-Yandex Object Storage.
Mhlawumbe umbuzo awusekho kuwe?
Umbuzo lapha kulokhu akusho lutho kubani. Uma kukhona imibono yokuthi ungabhekana kanjani nalokhu, masikwenze ku-WAL-G. Kodwa kuze kube manje anginayo imibono emihle yokuthi ngingabhekana kanjani nalokhu. Kunesinye Isitoreji Sezinto esisekela ukufakwa kuhlu okuyisipele ngokuhlukile. Ubacela ukuthi babhale izinto, bese bengeza ifolda lapho. U-WAL-G uyesaba ngalokhu - kukhona uhlobo oluthile lwento lapha okungelona ifayela, angikwazi ukulibuyisela, okusho ukuthi isipele asizange sibuyiselwe. Okusho ukuthi, empeleni, uneqoqo elibuyiselwe ngokuphelele, kodwa likubuyiselela isimo esiyiphutha ngoba Isitoreji Sento sibuyise ulwazi oluyinqaba ebesingaluqondi ngokugcwele.
Lokhu kuyinto eyenzekayo efwini le-Mail.
Uma ukwazi ukwakha kabusha...
Ikhiqizwa kabusha ngokuqhubekayo...
Uma kukhona ukukhiqiza, khona-ke ngicabanga ukuthi sizozama amasu futhi sithole ukuthi singazama kanjani futhi siqonde ukuthi ifu lidingani kithi. Mhlawumbe kuzozinza kithi ekuxhumaneni okuthathu futhi ngeke kuphule uxhumano, khona-ke sizofinyelela ngokucophelela ezintathu. Ngoba manje sehlisa ukuxhumana ngokushesha okukhulu, okungukuthi, uma sethule ukutholwa ngezintambo eziyi-16, ngemuva kokuzama kabusha kuzoba nezintambo eziyi-8, imicu emi-4, imicu emi-2 kanye neyodwa. Bese izodonsa ifayela ekusakazweni okukodwa. Uma kukhona amanani omlingo afana nezintambo ezingu-7,5 ezingcono kakhulu zokupompa, khona-ke sizohlala kuwo futhi sizame ukwenza enye imicu engu-7,5. Nawu umbono.
Siyabonga ngombiko! Ngabe kubukeka kanjani ukugeleza komsebenzi okuphelele kokusebenza ne-WAL-G? Isibonelo, esimweni esiyisiphukuphuku lapho ingekho i-delta emakhasini. Futhi sithatha futhi sisuse isipele sokuqala, bese sifaka kungobo yomlando i-shaft kuze kube yilapho siluhlaza okwesibhakabhaka ebusweni. Lapha, njengoba ngikuqonda, kukhona ukuwohloka. Ngesinye isikhathi udinga ukwenza isipele se-delta yamakhasi, okungukuthi inqubo ethile yangaphandle ishayela lokhu noma lokhu kwenzeka kanjani?
I-delta backup API ilula kakhulu. Kukhona inombolo lapho - izinyathelo ezinkulu ze-delta, yilokho okubizwa ngakho. Izenzakalela ibe nguziro. Lokhu kusho ukuthi ngaso sonke isikhathi uma wenza i-backup-push, ilanda isipele esigcwele. Uma uyishintshela kunoma iyiphi inombolo ephozithivu, isibonelo, u-3, ngokuzayo lapho wenza i-backup-push, ibheka umlando wezipele zangaphambilini. Uyabona ukuthi awudluli uchungechunge lwama-delta angu-3 futhi wenza i-delta.
Okusho ukuthi, njalo uma sethula i-WAL-G, izama ukwenza isipele esigcwele?
Cha, sisebenzisa i-WAL-G, futhi izama ukwenza i-delta uma izinqubomgomo zakho ziyivumela.
Ukukhuluma nje, uma uyisebenzisa ngoziro njalo, izoziphatha njenge-pg_basebackup?
Cha, isazogijima ngokushesha ngoba isebenzisa ukuminyanisa nokuhambisana. I-Pg_basebackup izobeka i-shaft eduze kwakho. I-WAL-G ithatha ukuthi ukulondoloza kungobo yomlando okumisiwe. Futhi izokhipha isexwayiso uma ingalungiselelwe.
I-Pg_basebackup ingaqhutshwa ngaphandle kwama-shafts.
Yebo, khona-ke bazoziphatha cishe ngendlela efanayo. Pg_basebackup amakhophi ohlelweni lwefayela. Ngendlela, sinesici esisha engikhohlwe ukusisho. Manje singakwazi ukwenza isipele ohlelweni lwefayela kusuka ku-pg_basebackup. Angazi ukuthi kungani lokhu kudingeka, kodwa kukhona.
Isibonelo, ku-CephFS. Akuwona wonke umuntu ofuna ukulungisa Isitoreji Sento.
Yebo, mhlawumbe yingakho bebuze umbuzo mayelana nalesi sici ukuze sikwenze. Futhi sikwenzile.
Siyabonga ngombiko! Kunombuzo nje mayelana nokukopisha ohlelweni lwefayela. Ngaphandle kwebhokisi, ingabe manje uyasekela ukukopishela kwisitoreji esikude, isibonelo, uma kukhona ishalofu kusikhungo sedatha noma enye into?
Kulokhu kwakhiwa, lo ngumbuzo onzima. Yebo, siyakusekela, kodwa lokhu kusebenza akukafakwa kunoma yikuphi ukukhishwa okwamanje. Okusho ukuthi, konke ukukhishwa kwangaphambilini kusekela lokhu, kodwa izinguqulo zokukhishwa azikwenzi. Lokhu kusebenza kwengezwe enguqulweni engu-0.2. Izokhishwa nakanjani maduze, lapho nje silungisa zonke iziphazamisi ezaziwayo. Kodwa okwamanje lokhu kungenziwa kuphela ekukhishweni kwangaphambili. Kunamaphutha amabili ekukhishweni kwangaphambilini. Inkinga ngokutholwa kwe-WAL-E, asikayilungisi. Futhi ekukhishweni kwangaphambilini kwakamuva iphutha elimayelana nesipele se-delta sengeziwe. Ngakho-ke, sincoma ukuthi wonke umuntu asebenzise izinguqulo zokukhishwa. Ngokushesha nje lapho zingasekho iziphazamisi ekukhishweni kwangaphambilini, singasho ukuthi sisekela i-Google Cloud, izinto ezihambisana ne-S3 nokugcinwa kwamafayela.
Sawubona, ngiyabonga ngombiko. Njengoba ngikuqonda, i-WAL-G ayilona uhlobo oluthile lwesistimu emaphakathi njenge-barmen? Ingabe uhlela ukuhamba ngale ndlela?
Inkinga wukuthi sisuke kule ndlela. I-WAL-G ihlala kumsingathi oyisisekelo, kumsingathi weqoqo, nakubo bonke abasingathi abakuqoqo. Lapho sithuthela ezinkulungwaneni ezimbalwa zamaqoqo, saba nezindawo eziningi zokufaka ama-bartender. Futhi njalo lapho into ewela phakathi kuzo, kuba yinkinga enkulu. Ngoba adinga ukulungiswa, udinga ukuqonda ukuthi yimaphi amaqoqo angenawo ama-backups. Angihleli ukuthuthukisa i-WAL-G ngibheke kwehardware ebonakalayo yezinhlelo eziyisipele. Uma umphakathi ufuna ukusebenza okuthile lapha, anginankinga nhlobo.
Sinamaqembu anesibopho sokugcina. Futhi sizizwa kahle kakhulu ukuthi akuthina, ukuthi kukhona abantu abakhethekile ababeka amafayela ethu lapho amafayela ephephile. Benza yonke inhlobo yokubhala ngobuhlakani lapho ukuze bamelane nokulahleka kwenani elithile lamafayela. Banesibopho somkhawulokudonsa wenethiwekhi. Uma une-bartender, ungase uthole ngokuzumayo ukuthi imininingwane encane enethrafikhi eningi iqoqene kuseva efanayo. Ubonakala unesikhala esiningi kuso, kodwa ngesizathu esithile yonke into ayihambisani nenethiwekhi. Kungase kuvele ngenye indlela. Kunamanethiwekhi amaningi lapho, kukhona ama-processor cores, kepha awekho amadiski lapha. Futhi sakhathala ngalesi sidingo sokuhlanganisa okuthile, futhi sathuthela eqinisweni lokuthi ukugcinwa kwedatha kuyisevisi ehlukile, lapho abantu abakhethekile abahlukene banesibopho.
PS Kukhishwe inguqulo entsha , lapho ungasebenzisa khona ifayela lokumisa elithi .walg.json, elitholakala kuhla lwemibhalo lwasekhaya lwe-postgres ngokuzenzakalelayo. Ungashiya izikripthi ze-bash. Isibonelo esithi .walg.json sikulolu daba
Ividiyo:
Source: www.habr.com