Ini ndinokurudzira kuti uverenge zvinyorwa zvemushumo kubva kutanga kwa2019 naAndrey Borodin "Backups neWAL-G. Chii chiripo muna 2019?"
Mhoroi mose! Zita rangu ndiAndrey Borodin. Ini ndiri mugadziri paYandex. Ndanga ndichifarira PostgreSQL kubvira 2016, mushure mekunge ndataura nevagadziri uye vakati zvese zviri nyore - iwe unotora iyo kodhi kodhi woivaka, uye zvese zvinobuda. Uye kubvira ipapo handigoni kumira - ndinonyora marudzi ose ezvinhu zvakasiyana.
Chimwe chezvinhu zvandiri kushanda pazviri ndeye backup system. . Kazhinji, paYandex tanga tichishanda pamagadziriro ekuchengetedza muPostgreSQL kwenguva yakareba. Uye iwe unogona kuwana paInternet nhevedzano yemishumo mitanhatu pamagadzirirwo atinoita backup masisitimu. Uye gore rega rega vanoshanduka zvishoma, vanokura zvishoma, uye vanova vakavimbika.
Asi nhasi mushumo hausi wezvatakaita chete, asiwo nezvekuti zviri nyore sei uye chii. Vangani venyu vakatoona mishumo yangu nezve WAL-G? Zvakanaka kuti vanhu vashoma havana kuona, nekuti ini ndichatanga nechinhu chakareruka.
Kana kamwe kamwe iwe uine PostgreSQL cluster, uye ndinofunga munhu wese ane akati wandei navo, uye kamwe kamwe hapana backup system parizvino, saka iwe unofanirwa kuwana chero S3 yekuchengetedza kana Google Cloud inoenderana kuchengetedza.
Semuenzaniso, unogona kuuya kumire yedu uye kutora kodhi yekusimudzira yeYandex Object Storage, iyo S3 inoenderana.
Zvadaro gadzira Bucket. Ingori mudziyo weruzivo.
Gadzira mushandisi webasa.
Gadzira kiyi yekuwana yemushandisi wesevhisi: aws-s3-kiyi.
Dhawunirodha yazvino yakagadzikana kuburitswa kweWAL-G.
Zvakatanga kuburitswa zvasiyana sei nezvakaburitswa? Kazhinji ndinokumbirwa kuti ndisunungure nekukurumidza. Uye kana pasina bug mune shanduro kwenguva yakakwana, semuenzaniso, mwedzi, ipapo ndinoisunungura. Heino kuburitswa uku kubva munaNovember. Uye izvi zvinoreva kuti mwedzi wega wega takawana imwe mhando yebug, kazhinji mune isiri-yakakosha mashandiro, asi isu hatisati taburitsa kuburitswa. Shanduro yapfuura ndeyeMbudzi chete. Iko hakuna tsikidzi dzinozivikanwa kwatiri mairi, i.e. tsikidzi dzakawedzerwa sezvo chirongwa chaifambira mberi.
Paunenge uchinge wadhawunirodha WAL-G, unogona kumhanyisa yakapusa "backup list" kuraira, ichipfuura munzvimbo dzakasiyana. Uye ichabatana neObject Storage uye inokuudza kuti ndeapi ma backups aunawo. Pakutanga, chokwadi, haufanirwe kunge uine backups. Pfungwa yeiyi slide kuratidza kuti zvese zviri nyore. Uyu murairo wekoni unogamuchira nharaunda dzakasiyana uye inoita subcommands.
Mushure meizvi, iwe unogona kuita yako yekutanga backup. Iti "backup-push" muWAL-G uye tsanangura muWAL-G iyo pgdata nzvimbo yesumbu rako. Uye zvakanyanya, PostgreSQL ichakuudza, kana iwe usati watova negadziriro yeparutivi, yaunoda kugonesa "archive-mode".
Izvi zvinoreva kuti iwe unofanirwa kuenda kune zvigadziriso wobatidza "archive_mode = on" uye wowedzera "archive_command", inova zvakare subcommand muWAL-G. Asi nekuda kwechimwe chikonzero vanhu vanowanzo shandisa mabhawa zvinyorwa pane iyi musoro uye nekuiputira neWAL-G. Ndapota usaite izvi. Shandisa mashandiro anowanikwa muWAL-G. Kana uri kurasikirwa nechimwe chinhu, nyorera . WAL-G inofungidzira kuti ndiyo yega chirongwa chinomhanya mu archive_command.
Isu tinoshandisa WAL-G kunyanya kugadzira Yakanyanya Kuwanikwa cluster muYandex Database manejimendi.
Uye inowanzo shandiswa mutopology yeimwe Master uye akati wandei kudzokorora. Panguva imwecheteyo, inoita kopi yekuchengetedza muYandex Object Storage.
Mamiriro akajairika ari kugadzira makopi esumbu uchishandisa Point munguva yekupora. Asi mune iyi kesi, kuita kwegadziriro yekuchengetedza hakuna kukosha kwatiri. Isu tinongoda kurodha sumbu nyowani kubva kune backup.
Kazhinji, tinoda backup system performance kana uchiwedzera node itsva. Nei zvichikosha? Kazhinji vanhu vanowedzera node itsva kuboka nekuti sumbu riripo harigone kubata mutoro wekuverenga. Vanofanira kuwedzera replica itsva. Kana tikawedzera mutoro kubva pg_basebackup kuna Tenzi, ipapo Tenzi anogona kudonha. Naizvozvo, zvaive zvakakosha kwatiri kuti tikwanise kukurumidza kurodha node nyowani kubva mudura, tichigadzira mudiki mutoro paTenzi.
Uye imwe mamiriro akafanana. Uku ndiko kudiwa kwekutanga Master yekare mushure mekuchinja Cluster Master kubva kuData Center iyo yakarasika yekubatanidza.
- Nekuda kweizvozvo, pakugadzira zvinodiwa zveiyo kopi system, takaona kuti pg_basebackup haina kukodzera isu kana tichishanda mugore.
- Taida kukwanisa kudzvanya data redu. Asi inenge chero backup system kunze kweinouya mubhokisi ichapa data compression.
- Isu taida kufananidza zvese nekuti mushandisi ari mugore anotenga hombe ye processor cores. Asi kana isu tisina parallelism mune kumwe kushanda, saka nhamba huru yemacores inova isina basa.
- Isu tinoda encryption nekuti kazhinji data haisi yedu uye haigone kuchengetwa mumavara akajeka. Nenzira, mupiro wedu kuWAL-G wakatanga nekunyorera. Isu takapedza encryption muWAL-G, mushure mezvo takabvunzwa kuti: "Pamwe mumwe wedu achagadzira chirongwa?" Uye kubva ipapo ndanga ndichishanda neWAL-G kweanopfuura gore.
- Isu taidawo kutenderedza zviwanikwa, nekuti nekufamba kwenguva tichishandisa gore, takaona kuti dzimwe nguva vanhu vane girosari rakakosha husiku uye mutoro uyu haugone kukanganiswa. Ndosaka takawedzera resource throttling.
- Zvakare kunyora uye manejimendi.
- Uye verification.
Takatarisa maturusi akawanda akasiyana. Neraki, isu tine yakakura sarudzo muPostgreSQL. Uye kwese kwese kwataishaya chimwe chinhu, chimwe chinhu chidiki, chimwe chinhu chidiki.
Uye taongorora masisitimu aripo, takasvika pamhedzisiro yekuti tichagadzira WAL-G. Yakanga iri chirongwa chitsva ipapo. Zvakanga zviri nyore kupesvedzera budiriro yakanangana neiyo gore zvivakwa zveiyo backup system.
Pfungwa huru yatinoomerera kwairi ndeyekuti WAL-G inofanirwa kuve yakapusa senge balalaika.
WAL-G ine 4 mirairo. Izvi:
WAL-PUSH - chengetedza iyo shaft.
WAL-FETCH - tora shaft.
BACKUP-PUSH - gadzira backup.
BACKUP-FETCH - tora backup kubva kune backup system.
Muchokwadi, WAL-G zvakare ine manejimendi emabhapu aya, kureva kunyora uye kudzima marekodhi uye mabhapu munhoroondo izvo zvisingachadiwi panguva ino.
Rimwe remabasa akakosha kwatiri ibasa rekugadzira delta makopi.
Makopi eDelta anoreva kuti isu hatisi kugadzira backup yakazara yesumbu rese, asi chete mapeji akashandurwa emafaira akachinjwa musumbu. Zvinoita sekuti nekushanda izvi zvakafanana nekugona kupora uchishandisa WAL. Asi isu tinogona kupeta WAL imwe-yakarukwa delta backup mukufanana. Saizvozvo, kana isu tine basic backup yakaitwa neMugovera, delta backups zuva nezuva, uye neChina isu takundikana, saka tinoda kupeta 4 delta backups uye gumi maawa eWAL. Izvo zvichatora nguva imwe chete nekuti iyo delta backups inotenderera mukuwirirana.
LSN-based deltas - izvi zvinoreva kuti kana tichigadzira backup, isu tichada kubatanidza peji rega rega uye kutarisa LSN yayo neLSN ye backup yapfuura kuti tinzwisise kuti yachinja. Chero peji rinogona kunge riine data rakashandurwa rinofanira kunge riripo mu delta backup.
Sezvandakataura, kutarisisa kwakawanda kwakabhadharwa kune parallelism.
Asi iyo archive API muPostgreSQL inowirirana. PostgreSQL inochengeta imwe WAL faira uye painodzosera inokumbira imwe WAL faira. Asi kana dhatabhesi richikumbira faira rimwe reWAL richishandisa murairo we "WAL-FETCH", tinodaidza "WAL-PREFETCH" murairo, unogadzirira mafaira masere anotevera kutora data kubva muchitoro chechinhu chakafanana.
Uye kana dhatabhesi richitikumbira kuchengetedza faira rimwe chete, tinotarisa archive_status toona kana paine mamwe mafaera eWAL. Uye isu tiri kuyedzawo kudhawunirodha WAL nenzira yakafanana. Izvi zvinopa kukosha kwekuita kuwana uye zvakanyanya kuderedza kureba muhuwandu hweWAL dzisina kuchengetwa. Vazhinji vanogadzira backup system vanotenda kuti iyi ine njodzi system nekuti isu tinovimba neruzivo rwedu rwemukati mekodhi isiri iyo PostgreSQL API. PostgreSQL haivimbisi kuvepo kweiyo archive_status folda kwatiri uye haivimbisi semantics, kuvapo kwemasaini ekugadzirira kweWAL mafaera ipapo. Zvakadaro, isu tiri kudzidza iyo sosi kodhi, isu tinoona kuti ndizvozvo uye tiri kuedza kuishandisa. Uye isu tinodzora gwara iro PostgreSQL iri kuvandudza; kana kamwe kamwe michina iyi ikaputswa, isu tichamira kuishandisa.
Mune chimiro chayo chakachena, LSN-yakavakirwa WAL delta inoda kuverenga chero cluster faira ine mode-nguva mufaira system yachinja kubva pakare backup. Takagara neizvi kwenguva yakareba, rinenge gore. Uye pakupedzisira takasvika pakugumisa kuti tine WAL deltas.
Izvi zvinoreva kuti pese patinochengeta WAL paTenzi, isu hatingoimanikidza chete, kuinyora uye kuitumira kunetiweki, asi isu tinoiverengawo panguva imwe chete. Isu tinoongorora uye tinoverenga zvinyorwa mairi. Isu tinonzwisisa kuti ndeapi mabhuroki achinja uye kuunganidza delta mafaera.
A delta faira inotsanangura imwe siyana yemafaira eWAL, inotsanangura ruzivo rwekuti ndeapi mabhuroko akachinjwa mumhando iyi yeWAL. Uye ipapo aya mafaera edelta anochengetwa zvakare.
Pano takatarisana nenyaya yekuti takafananidza zvese nekukasira, asi hatigone kuverenga nhoroondo inoteedzana nenzira yakafanana, nekuti mune chimwe chikamu tinogona kusangana nekuguma kweiyo WAL rekodhi, izvo zvatisina chekubatana nacho, nekuti. kuverenga kwakafanana kwakatungamira kuti isu titange taongorora ramangwana, iro risati rava nekare.
Nekuda kweizvozvo, taifanira kuisa zvidimbu zvisinganzwisisike mu _delta_partial mafaera. Nekuda kweizvozvo, patinodzokera kune yakapfuura, isu tinonamira zvidimbu zveWAL rekodhi mune imwe, mushure mezvo isu tichaiparadzanisa uye tinonzwisisa chakachinja mairi.
Kana munhoroondo yeshaft yedu parsing paine kanenge imwe nzvimbo yatisinganzwisise zvaiitika, saka, maererano, panguva yeinotevera backup isu tichamanikidzwa kuverenga iyo yese cluster zvakare, sezvatakaita neyakajairwa LSN. -based delta.
Nekuda kweizvozvo, kutambura kwedu kwese kwakatungamira kune chokwadi chekuti isu takavhura sourced iyo WAL-G yekuparadzanisa raibhurari. Sekuziva kwangu, hapana ari kuishandisa parizvino, asi kana paine anoda, nyora nekuishandisa, iri munharaunda yeruzhinji. (Yakavandudzwa link )
Nekuda kweizvozvo, kuyerera kwese kweruzivo kunoratidzika kunge kwakaoma. Tenzi wedu anochengeta shaft uye anochengeta delta mafaera. Uye iyo replica inogadzira iyo backup kopi inofanirwa kugamuchira delta mafaera panguva yapfuura pakati pe backups. Muchiitiko ichi, zvikamu zvenhoroondo zvichada kuwedzerwa muhuwandu uye kuparadzaniswa, nokuti haisi nhoroondo yose inopinda muzvikamu zvakakura. Uye chete mushure meizvi replica inogona kuchengetedza yakazara delta backup.
Pamagirafu zvinhu zvese zvinotaridzika zviri nyore. Uku kudhawunirodha kubva kune rimwe remasumbu edu chaiwo. Isu tine LSN-yakavakirwa, yakagadzirwa muzuva rimwe chete. Uye isu tinoona kuti iyo LSN-yakavakirwa delta backup yaimhanya kubva kumatatu mangwanani kusvika mashanu mangwanani. Uyu ndiwo mutoro muhuwandu hwema processor cores. WAL-delta yakatitorera maminetsi anenge 20. Ndiko kuti, yakava yakakurumidza kukurumidza, asi panguva imwecheteyo pakanga pane kuchinjana kwakanyanya pamusoro pemambure.
Sezvo isu tine ruzivo rwekuti ndeapi mabhuroki akachinja uye panguva ipi munhoroondo yedatabase, takaenda mberi uye takafunga kubatanidza kushanda - yePostgreSQL yekuwedzera inonzi "pg_prefaulter"
Izvi zvinoreva kuti kana iyo yekumira-negasi ikaita yekudzoreredza kuraira, inoudza WAL-G kuti itore inotevera WAL faira. Isu tinonzwisisa kuti ndeipi data inovharira iyo WAL kudzoreredza maitiro achawana munguva pfupi iri kutevera uye kutanga chirongwa chekuverenga pazvivharo izvi. Izvi zvakaitwa kuitira kuwedzera kushanda kweSSD controllers. Nekuti WAL roll ichasvika peji inoda kuchinjwa. Iri peji riri padhisiki uye harisi mucheshi yepeji. Uye achamirira synchronously kuti peji ino isvike. Asi padyo pane WAL-G, iyo inoziva kuti mumazana mashoma anotevera megabytes eWAL isu tichada mamwe mapeji uye panguva imwechete inotanga kudziya iwo. Inotanga madhisiki akawanda ekupinda kuitira kuti aitwe mukuwirirana. Izvi zvinoshanda zvakanaka pamadhiraivha eSSD, asi, zvinosuruvarisa, hazvishande kune hard drive, nekuti isu tinongokanganisa nazvo nekukurudzira kwedu.
Izvi ndizvo zviri mukodhi izvozvi.
Pane zvinhu zvatinoda kuwedzera.
Mufananidzo uyu unoratidza kuti WAL-delta inotora nguva pfupi. Uye izvi kuverenga shanduko dzakaitika mudhatabhesi mukati mezuva. Isu taigona kuita WAL-delta kwete chete husiku, nekuti haisisiri yakakosha sosi yemutoro. Tinogona kuverenga WAL-delta miniti yega yega nekuti yakachipa. Muminiti imwe tinogona kuongorora shanduko dzese dzakaitika kuboka. Uye izvi zvinogona kunzi "instant WAL-delta".
Chiripo ndechekuti kana tadzoreredza cluster, tinodzikisira nhamba yenyaya dzatinofanira kupeta sequentially. Ndiko kuti, huwandu hweWAL iyo PostgreSQL rolls inofanirwa kudzikiswa, nekuti inotora yakakosha nguva.
Asi handizvo zvoga. Kana isu tichiziva kuti imwe bhuroka ichashandurwa kusvika padanho rekuchengetedza, isu hatigone kuichinja munguva yakapfuura. Ndokunge, ikozvino isu tine faira-ne-faira optimization yeWAL-delta kutumira. Izvi zvinoreva kuti kana, semuenzaniso, neChipiri tafura yakadzimwa zvachose kana mamwe mafaera akabviswa zvachose patafura, ipapo kana delta ichitenderera neMuvhuro uye Mugovera pg_basebackup yakadzoserwa, isu hatitombogadziri iyi data.
Tinoda kuwedzera tekinoroji iyi kusvika padanho repeji. Ndiko kuti, kana chimwe chikamu chefaira chichichinja musi weMuvhuro, asi chichanyorwa pamusoro peChitatu, zvino kana tichidzorera kune imwe nzvimbo musi weChina, hatifaniri kunyora zvinyorwa zvekutanga zvemapeji ku disk.
Asi iyi ichiri pfungwa iri kushingaira kukurukurwa mukati medu, asi haisati yasvika pakodhi.
Tinoda kugadzira chimwe chinhu muWAL-G. Isu tinoda kuita kuti iwedzere nekuti isu tinoda kutsigira akasiyana dhatabhesi uye tinoda kukwanisa kusvika kune backup manejimendi nenzira imwechete. Asi dambudziko nderekuti MySQL APIs akasiyana zvakanyanya. MuMySQL, PITR haina kubva pane yemuviri WAL log, asi pane binlog. Uye isu hatina archiving system muMySQL yaizoudza imwe yekunze system kuti iyi binlog yapera uye inoda kuchengetwa. Tinofanira kumira pane imwe nzvimbo mucron ne database uye tarisa kana pane chimwe chinhu chakagadzirira?
Uye nenzira imwecheteyo, panguva yekudzoreredza kweMySQL, hapana kudzoreredza murairo unogona kuudza sisitimu kuti ndinoda mafaera akadai. Usati watanga kuvaka patsva cluster yako, unofanirwa kuziva mafaera aunoda. Iwe pachako unofanirwa kufungidzira kuti mafaira api aunoda. Asi matambudziko aya anogona kukwanisa kunzvengwa neimwe nzira. (Kujekesa: MySQL yakatotsigirwa)
Mumushumo, ini ndaidawo kutaura nezve idzo nyaya kana WAL-G isina kufanira iwe.
Kana iwe usina synchronous replica, WAL-G haivimbisi kuti chikamu chekupedzisira chichachengetedzwa. Uye kana kuchengetedza kwasara kumashure kwezvikamu zvishoma zvekupedzisira zvenhoroondo, iyo njodzi. Kana pasina synchronous replica, ini handingakurudzire kushandisa WAL-G. Zvakadaro, inonyanya kugadzirirwa kuisirwa kwegore, izvo zvinoreva kuti Yepamusoro Kuwanikwa mhinduro ine synchronous replica, iyo ine basa rekuchengetedza mabheti ekupedzisira akaitwa.
Ndinowanzoona vanhu vachiedza kumhanya zvese WAL-G uye WAL-E panguva imwe chete. Isu tinotsigira kuenderana kumashure mupfungwa yekuti WAL-G inogona kudzoreredza faira kubva kuWAL-E uye inogona kudzoreredza backup yakagadzirwa muWAL-E. Asi sezvo ese maviri aya masisitimu anoshandisa parallel wal-push, vanotanga kuba mafaera kubva kune mumwe nemumwe. Kana tikaigadzirisa muWAL-G, icharamba ichiri muWAL-E. MuWAL-E, inotarisa archive-status, inoona mafaera akapera uye anoachengeta, nepo mamwe masisitimu asingazozive kuti iyi WAL faira yaivepo, nekuti PostgreSQL haizoyedze kuichengeta kechipiri.
Chii chatichazogadzirisa pano kudivi reWAL-G? Hatisi kuzozivisa PostgreSQL kuti faira iri rakatamiswa zvakafanana, uye PostgreSQL painotikumbira kuti tiichengete, isu tichatoziva kuti faira rakadaro rine iyi modhi-nguva uye neiyi md5 yakatochengetwa uye tichangoti PostgreSQL - Zvakanaka, zvese zvakagadzirira pasina kuita chero chinhu.
Asi dambudziko iri haringaite kuti rigadziriswe kudivi reWAL-E, saka parizvino hazvigoneke kugadzira rairo yekuchengetera iyo inochengeta faira mune zvese WAL-G uye WAL-E.
Mukuwedzera, pane zviitiko apo WAL-G haina kukodzera iwe ikozvino, asi isu tichazvigadzirisa.
Chekutanga, isu parizvino hatina yakavakirwa-mukati backup verification. Isu hatina verification panguva yekuchengeta kana kupora. Chokwadi, izvi zvinoitwa mugore. Asi izvi zvinoitwa nekungofanoona, nekungodzoreredza cluster. Ndinoda kupa mashandiro aya kune vashandisi. Asi nekusimbisa, ndinofungidzira kuti muWAL-G zvinokwanisika kudzoreredza sumbu uye kutanga, uye kumhanya bvunzo dzeutsi: pg_dumpall ku /dev/null uye amcheck index verification.
Parizvino muWAL-G hapana nzira yekumisa imwe backup kubva kuWAL. Ndiko kuti, tinotsigira mamwe hwindo. Semuenzaniso, kuchengetedza mazuva manomwe ekupedzisira, kuchengetedza gumi ekupedzisira mabhapu, kuchengetedza matatu ekupedzisira akazara mabhakiti. Kazhinji vanhu vanouya vachiti: "Tinoda backup yezvakaitika paGore Idzva uye tinoda kuzvichengeta zvachose." WAL-G haikwanise kuita izvi parizvino. (Cherechedza - Izvi zvakatogadziriswa. Verenga zvimwe - Backup-mark sarudzo mukati )
Uye isu hatina peji cheki cheki uye cheki yekuvimbika kune ese mashaft zvikamu kana uchisimbisa PITR.
Kubva pane zvese izvi ndakaisa pamwechete chirongwa cheGoogle Zhizha reCode. Kana iwe uchiziva vadzidzi vakangwara vangada kunyora chimwe chinhu muGo uye vatore zviuru zvemadhora kubva kune imwe kambani ine vara "G", saka kurudzira chirongwa chedu kwavari. Ini ndichaita semurairidzi wechirongwa ichi, vanogona kuzviita. Kana pasina vadzidzi, ipapo ini ndichaitora uye ndozviita ini muzhizha.
Uye tine mamwe matambudziko madiki akawanda atiri kugadzirisa zvishoma nezvishoma. Uye zvimwe zvinoshamisa zvinoshamisa zvinoitika.
Semuenzaniso, kana iwe ukapa WAL-G isina chinhu backup, inongodonha. Semuyenzaniso, kana ukamuudza kuti anofanirwa kusevha folda isina chinhu. Iyo pg_control faira haizovepo. Uye achafunga kuti haanzwisisi chimwe chinhu. Mune dzidziso, mune iyi kesi iwe unofanirwa kunyora yakajairika meseji kumushandisi kuti umutsanangurire maitiro ekushandisa chishandiso. Asi ichi hachisi kunyange chimiro chehurongwa, asi chimiro chemutauro wakanaka, unonzwisisika.
Hatizive maitiro ekuita backup offline. Kana dhatabhesi iri kunyepa, hatigone kuichengeta. Asi zvinhu zvose zviri nyore kwazvo pano. Tinodaidza ma backups neLSN payakatanga. Iyo LSN yehwaro hwepasi inofanira kuverengerwa kubva pakudzora faira. Uye ichi chinhu chakadaro chisingagonekwe. Mazhinji masisitimu ekuchengetedza anogona kuchengetedza dhatabhesi yepasi. Uye zviri nyore.
Parizvino hatikwanise kubata kushaikwa kwenzvimbo yekuchengetedza nemazvo. Nekuti isu tinowanzo shanda nema backups makuru kumba. Uye ivo havana kuzvibata. Asi kana mumwe munhu achida kuronga muGo izvozvi, wedzera kubata kwekunze-kwe-nzvimbo kukanganisa kubhaketi. Ini zvirokwazvo ndichatarisa mukudhonza chikumbiro.
Uye chinhu chikuru chinotinetsa ndechekuti isu tinoda akawanda docker ekubatanidza bvunzo sezvinobvira anoongorora akasiyana mamiriro. Parizvino tiri kungoyedza ma basic scenarios. Pakuita kwese, asi isu tinoda kutarisa kuzvipira-ne-kuita zvese zvatinotsigira. Kunyanya, semuenzaniso, tichava nerutsigiro rwakakwana rwePostgreSQL 9.4-9.5. Isu tinovatsigira nekuti nharaunda inotsigira PostgreSQL, asi isu hatitarise kuzvipira-ne-kuzvipira kuona kuti zvese hazvina kuputswa. Uye zvinoratidzika kwandiri kuti iyi injodzi yakakomba.
Isu tine WAL-G inomhanya pane anopfuura chiuru masumbu muYandex Database manejimendi. Uye inotsigira mazana akati wandei ematerabytes edata zuva rega rega.
Tine zvakawanda zveTODO mukodhi yedu. Kana iwe uchida kuronga, huya, takamirira zvikumbiro zvekudhonza, takamirira mibvunzo.
Mibvunzo yako
Manheru akanaka! Ndatenda! Kufungidzira kwangu ndekwekuti kana uri kushandisa WAL-delta, unogona kunge uchivimba zvakanyanya nekuzara-peji kunyora. Uye kana zvakadaro, wakamhanya bvunzo here? Makaratidza girafu rakanaka. Inowedzera kunaka sei kana FPW yakadzimwa?
Yakazara-peji inonyora inogoneswa isu, isu hatina kuedza kuidzima. Ndiko kuti, ini, semugadziri, handina kuedza kuidzima. Vatariri vehurongwa vakatsvaga vangangotsvaga nyaya iyi. Asi tinoda FPW. Anenge hapana anoidzima, nekuti kana zvisina kudaro hazvibviri kutora backup kubva kune replica.
Ndatenda nemushumo! Ndine mibvunzo miviri. Mubvunzo wekutanga ndewekuti chii chichaitika kumatafura?
Takamirira chikumbiro chekudhonza. Madhatabhesi edu anogara paSSD uye NMVE dhisiki uye isu hatidi chaizvo chimiro ichi. I'm not ready to spend serious time izvozvi pakuzviita nemazvo. Ndinokurudzira nemoyo wese kuti titsigire izvi. Kune vanhu vakaitsigira, asi vakaitsigira nenzira inovakodzera. Vakaita forogo, asi havaite zvekudhonza. (Yakawedzerwa mushanduro 0.2.13)
Uye mubvunzo wechipiri. Iwe wakataura pakutanga kuti WAL-G inofunga kuti inoshanda yega uye hapana wrappers inodiwa. Ndinoshandisa wrappers pachangu. Sei vasingafaniri kushandiswa?
Tinoda kuti ive nyore sebalalaika. Izvi zvinoreva kuti hapana chaunoda zvachose kunze kwe balalaika. Tinoda kuti sisitimu ive nyore. Kana iwe uine mashandiro aunofanirwa kuita mune script, wouya utiudze - tichazviita muGo.
Manheru akanaka! Ndatenda nemushumo! Hatina kukwanisa kuita kuti WAL-G ishande neGPG decryption. Iyo encrypts zvakajairwa, asi haidi kudzima. Icho chinhu chisina kushanda kwatiri here? Mamiriro acho ezvinhu anoodza mwoyo.
Gadzira nyaya paGitHub uye ngatizvione.
Ndiko kuti, iwe hauna kusangana neizvi?
Pane chinhu chemushumo wekukanganisa kuti kana WAL-G isinganzwisise kuti ifaira rudzii, inobvunza kuti: "Pamwe yakavharirwa?" Zvichida dambudziko harisi encryption zvachose. Ndinoda kuvandudza kutema matanda panyaya iyi. Anofanira kuzvidudzira. Parizvino tiri kushanda pane iyi nyaya mupfungwa yekuti hatinyatsoda kuti hurongwa hwekuwana makiyi eruzhinji neakavanzika akarongwa sei. Nekuti tinodaidza yekunze GPG zvekuti inotipa makiyi ayo. Uye tobva tatora makiyi aya toaendesa kune yemukati GPG, iyo yakavhurika PGP, iyo inogadzirirwa isu mukati meWAL-G, uye ipapo tinodaidza encryption. Panyaya iyi, tinoda kuvandudza sisitimu uye tinoda kutsigira Libsodium encryption (Yakawedzerwa mushanduro 0.2.15). Ehe, kudhirodha kunofanirwa kushanda, ngatizvione - iwe unoda zvakawanda zvechiratidzo pane akati wandei mazwi. Unogona kuungana mukamuri yemukurukuri imwe nguva uye wotarisa kuhurongwa. (PGP encryption isina yekunze GPG - v0.2.9)
Mhoro! Ndatenda neshumo! Ndine mibvunzo miviri. Ndine chishuwo chinoshamisa chekuita pg_basebackup uye WAL kupinda mune vaviri vanopa, i.e. ndinoda kuita gore rimwe uye rimwe. Pane imwe nzira yekuita izvi here?
Izvi hazvipo iye zvino, asi ipfungwa inofadza.
Ini handingovimbi mumwe mupi, ini ndinoda kuva nezvakafanana mune imwe, kana zvikaitika.
Pfungwa yacho inofadza. Nehunyanzvi, izvi hazvina kuoma kuita. Kuti udzivise zano kuti risarasike, ndingakumbira here kuti uite nyaya paGitHub?
Ehe saizvozvo.
Uyezve, kana vadzidzi vauya kuGoogle Zhizha reCode, isu tichavawedzera kupurojekiti kuitira kuti pave nebasa rakawanda rekuwana zvakawanda kubva kwavari.
Uye mubvunzo wechipiri. Pane nyaya paGitHub. Ndofunga zvakatovharwa. Pane kuvhunduka panguva yekudzorera. Uye kuti ukunde, makaita ungano yakaparadzana. Zvakarurama munyaya. Uye pane sarudzo yekuita inoshanduka nharaunda mune imwe shinda. Uye ndicho chikonzero chinoshanda zvishoma nezvishoma. Uye tasangana nedambudziko iri, uye harisati ragadziriswa.
Dambudziko nderekuti nekuda kwechimwe chikonzero chengetedzo (CEPH) inogadzirisa zvakare kubatana kana tauya kwairi nepamusoro concurrency. Chii chingaitwa pamusoro peizvi? Kuedzazve kufunga kunoita seizvi. Tiri kuedza kudhaunirodha faira racho zvakare. Mukupfuura kumwe, takanga tine akati wandei mafaera asina kurodha pasi, tichaita yechipiri kune vese vasina kupinda. Uye chero bedzi faira rimwe chete rakaremerwa pane imwe iteration, tinodzokorora uye tinodzokorora nekudzokorora. Isu takavandudza pfungwa yekuedza zvakare - exponential backoff. Asi hazvina kunyatso kujeka zvekuita nenyaya yekuti kubatana kunongotyora padivi rekuchengetedza system. Ndokunge, kana isu tichirodha kune imwe rukova, haiparadze aya makubatanidza. Chii chatingavandudza pano? Tine network throttling, tinogona kudzikisira kubatana kwega kwega nenhamba yemabhaiti yainotumira. Zvikasadaro, ini handizive maitiro ekuita nenyaya yekuti kuchengetedza kwechinhu hakutibvumidze kurodha kana kurodha kubva pazviri zvakafanana.
Hapana SLA? Hazvina kunyorwa here kwavari kuti vanozvibvumira sei kutambudzwa?
Icho chiripo ndechekuti vanhu vanouya nemubvunzo uyu kazhinji vane vault yavo. Ndiko kuti, hapana anobva kuAmazon kana Google Cloud kana Yandex Object Storage.
Zvichida mubvunzo hausisiri kwauri?
Mubvunzo uri pano panyaya iyi hauna basa kunaani. Kana paine mazano ekuti toita sei neizvi, ngatizviitei muWAL-G. Asi kusvika pari zvino handina mazano akanaka ekuti ndoita sei neizvi. Kune zvimwe zvinhu zvekuchengetedza izvo zvinotsigira rondedzero backups zvakasiyana. Iwe unovabvunza kuti vanyore zvinhu, uye ivo vanowedzera folda ipapo. WAL-G anotya pane izvi - pane imwe mhando yechinhu pano isiri faira, handikwanise kuidzosera, zvinoreva kuti backup haina kudzoserwa. Iko ndiko kuti, iwe une sumbu rakadzoreredzwa zvachose, asi rinokudzosera iwe chimiro chinokanganisa nekuti Chinhu Chekuchengeta chakadzosa rumwe ruzivo rusinganzwisisike rwaasina kunyatsonzwisisa.
Ichi chinhu chinoitika muMail cloud.
Kana iwe uchigona kugadzira dhizaini ...
Inogara ichidzokororwa ...
Kana paine dhizaini, saka ndinofunga isu tichayedza nzira dzekuyedza zvakare uye kuona kuti tingaedza sei zvakare uye kunzwisisa izvo gore rinoda kwatiri. Zvichida ichave yakagadzikana kwatiri pazvibatanidza zvitatu uye haizoputsi kubatana, saka tichanyatsosvika kune zvitatu. Nekuti ikozvino tinodonhedza kubatana nekukurumidza, i.e. kana isu takatanga kudzoreredza netambo gumi nematanhatu, zvino mushure mekuyedza kwekutanga pave ne16 tambo, 8 tambo, 4 tambo uye imwe. Uye zvino ichadhonza faira murukova rumwe. Kana paine mamwe mashiripiti akakosha senge 2 shinda ndiwo akanakisa kupomba, saka isu tinomira paari toedza kugadzira imwe 7,5 shinda. Heino zano.
Ndatenda nemushumo! Ko kuyerera kwekushanda kwekushanda neWAL-G kunoratidzika sei? Semuenzaniso, mune iyo benzi kesi kana pasina delta pamapeji. Uye isu tinotora uye tinobvisa yekutanga backup, tozochengeta iyo shaft kusvika isu tave bhuruu kumeso. Pano, sezvandinozvinzwisisa, pane kuparara. Pane imwe nguva iwe unofanirwa kuita delta backup yemapeji, i.e. imwe nzira yekunze iri kutyaira izvi kana izvi zvinoitika sei?
Iyo delta backup API iri nyore. Pane nhamba ipapo - max delta nhanho, ndizvo zvinodaidzwa. Iyo in default pa zero. Izvi zvinoreva kuti pese paunoita backup-push, inorodha yakazara backup. Kana iwe ukachishandura kune chero nhamba yakanaka, semuenzaniso, 3, zvino nguva inotevera iwe yaunoita backup-push, inotarisa nhoroondo yema backups apfuura. Anoona kuti haupfuure cheni ye3 deltas uye anoita delta.
Ndokunge, pese patinotanga WAL-G, inoedza kuita yakazara backup?
Kwete, isu tinomhanya WAL-G, uye inoedza kugadzira delta kana mitemo yako ichibvumira.
Zvichireva kutaura, kana ukaimhanyisa ne zero nguva dzese, ichaita senge pg_basebackup?
Kwete, icharamba ichimhanya nekukurumidza nekuti inoshandisa compression uye parallelism. Pg_basebackup ichaisa shaft padivi pako. WAL-G inofungidzira kuti une archiving yakagadzirirwa. Uye ichapa yambiro kana isina kugadzirwa.
Pg_basebackup inogona kumhanya isina shafts.
Hongu, ipapo vachazvibata zvakada kufanana. Pg_basebackup makopi kune iyo faira system. Nenzira, tine chinhu chitsva chandakanganwa kutaura. Isu tinogona ikozvino kuchengetedza kune iyo faira system kubva pg_basebackup. Handizivi kuti sei izvi zvichidiwa, asi zviripo.
Somuenzaniso, paCephFS. Haasi munhu wese anoda kugadzirisa Object Storage.
Ehe, ndosaka vakabvunza mubvunzo nezve chinhu ichi kuti isu tigone kuzviita. Uye takazviita.
Ndatenda neshumo! Pane chete mubvunzo nezve kukopa kune faira system. Kunze kwebhokisi, iwe zvino unotsigira kukopa kune kure kuchengetedza, semuenzaniso, kana paine imwe sherufu munzvimbo yedata kana chimwe chinhu?
Mukugadzirisa uku, uyu mubvunzo wakaoma. Hongu, isu tinotsigira, asi basa iri harisati rabatanidzwa mune chero kuburitswa. Ndiko kuti, zvese zvakafanoburitswa zvinotsigira izvi, asi mavhezheni ekuburitswa haadaro. Kushanda uku kwakawedzerwa mushanduro 0.2. Ichanyatso kusunungurwa munguva pfupi, pakarepo patinogadzirisa zvose zvinozivikanwa zviputi. Asi izvozvi izvi zvinogona kuitwa chete mupre-kuburitswa. Pane maviri mabhugi mune pre-kuburitswa. Dambudziko neWAL-E kupora, hatina kurigadzirisa. Uye mune ichangoburwa pre-kuburitsa bug nezve delta-backup yakawedzerwa. Naizvozvo, isu tinokurudzira munhu wese kushandisa shanduro dzekuburitsa. Pakange pasisina tsikidzi mune pre-kuburitswa, tinogona kutaura kuti tinotsigira Google Cloud, S3-inoenderana zvinhu uye faira kuchengetedza.
Mhoro, ndatenda neshumo. Sezvandinozvinzwisisa, WAL-G haisi imwe mhando yepakati system senge barmen? Unoronga kufamba nenzira iyi here?
Dambudziko nderekuti tabva kure negwara iri. WAL-G anogara pane iyo base host, pane iyo cluster host, uye pane ese mauto ari musumbu. Apo takatamira kune zviuru zvinoverengeka zvisumbu, takava nezvivako zvakawanda zvebhawa. Uye nguva imwe neimwe chimwe chinhu chinoputsika mavari, idambudziko guru. Nekuti dzinoda kugadziriswa, iwe unofanirwa kunzwisisa kuti ndeapi masumbu asina ma backups. Ini handironge kuvandudza WAL-G munzira yemidziyo yemuviri ye backup masisitimu. Kana nharaunda ichida kushanda pano, ini handina basa nazvo.
Tine zvikwata zvine basa rekuchengetedza. Uye isu tinonzwa zvakanaka kwazvo kuti hatisi isu, kuti kune vanhu vakakosha vanoisa mafaera edu uko mafaira akachengeteka. Vanoita marudzi ese ehungwaru kukodha ipapo kuti vamire kurasikirwa kweimwe nhamba yemafaira. Ivo vane mhosva kune network bandwidth. Paunenge uine bartender, iwe unogona kamwe kamwe kuona kuti diki dhatabhesi ine yakawanda traffic yakaungana pane imwechete server. Iwe unoratidzika kunge une nzvimbo yakawanda pairi, asi nokuda kwechimwe chikonzero zvinhu zvose hazvikwanisi kuburikidza netiweki. Inogona kubuda neimwe nzira. Kune akawanda network ipapo, kune processor cores, asi hapana madhisiki pano. Uye isu taneta nekudiwa uku kwejuggle chimwe chinhu, uye takatamira kune chokwadi chekuti kuchengetedza data ibasa rakasiyana, iro rakapatsanurwa vanhu vakasiyana vane basa.
PS Shanduro itsva yakaburitswa , iyo yaunogona kushandisa iyo .walg.json configuration file, iyo iri mu postgres home directory by default. Unogona kusiya bash zvinyorwa. Muenzaniso .walg.json iri munyaya ino
Vhidhiyo:
Source: www.habr.com