Ini ndinokurudzira kuti uverenge zvinyorwa zvekutanga 2020 mushumo naGeorgy Rylov "WAL-G: mikana mitsva uye kuwedzera kwenharaunda"
Vagadziri veOpen-source vanosangana nematambudziko mazhinji sezvavanokura. Maitiro ekunyora akawanda uye akawanda anodiwa maficha, kugadzirisa zvakanyanya uye zvakanyanya nyaya uye kugona kuona zvakanyanya uye zvakanyanya kudhonza zvikumbiro? Ndichishandisa WAL-G (backup-tool yePostgreSQL) semuenzaniso, ini ndichakuudza kuti takagadzirisa sei matambudziko aya nekutanga kosi yeOpen-source development kuyunivhesiti, zvatakawana uye kwatichazoenda mberi.
Mhoroi zvakare mose! Ini ndiri mugadziri weYandex anobva kuYekaterinburg. Uye nhasi ndichataura nezve WAL-G.
Musoro weshumo hauna kutaura kuti chaive chimwe chinhu nezve backups. Pane anoziva here kuti WAL-G chii? Kana kuti munhu wese anoziva? Simudza ruoko rwako kana usingazivi. Tsvina shit, wakauya kuzotaura uye hauzive kuti ndezvei.
Rega ndikuudze zvichaitika nhasi. Zvinoitika kuti timu yedu yanga ichiita backups kwenguva yakati rebei. Uye ichi chimwe chirevo munhevedzano yatinotaura nezve machengetero atinoita data zvakachengeteka, zvakachengeteka, zviri nyore uye nemazvo.
Muchikamu chakapfuura pakanga paine mishumo yakawanda yaAndrei Borodin naVladimir Leskov. Taive takawanda. Uye tanga tichitaura nezve WAL-G kwemakore mazhinji.
clck.ru/F8ioz -
clck.ru/Ln8Qw -
Uyu mushumo uchave wakasiyana zvishoma kubva kune vamwe pakuti yaive yakawanda nezvechikamu chehunyanzvi, asi pano ini ndichataura nezvekuti takasangana sei nematambudziko ane chekuita nekukura kwenharaunda. Uye kuti takauya sei nekamwe kapfungwa kanotibatsira kubata neizvi.
Makore mashoma apfuura, WAL-G yaive purojekiti diki yatakawana kubva kuCitus Data. Uye takangozvitora. Uye yakagadzirwa nemunhu mumwe chete.
Uye chete WAL-G akanga asina:
- Chengetedza kubva kune replica.
- Pakanga pasina ma backups ekuwedzera.
- Pakanga pasina WAL-Delta backups.
- Uye pakanga pachine zvakawanda zvisipo.
Kwemakore mashoma aya, WAL-G yakakura zvakanyanya.
Uye panosvika 2020, zvese zviri pamusoro zvakatoonekwa. Uye kune izvi zvakawedzerwa zvatinazvo zvino:
- Zvinopfuura chiuru chenyeredzi paGitHub.
- 150 forogo.
- Inenge 15 yakavhurika PRs.
- Uye vamwe vazhinji vanopa.
- Uye vhura nyaya nguva dzese. Uye izvi zvisinei nekuti isu tinoenda ikoko mazuva ese uye tinoita chimwe chinhu nezvazvo.
Uye isu takasvika pakugumisa kuti chirongwa ichi chinoda zvakanyanya kutarisisa kwedu, kunyangwe isu pachedu hatidi kuita chero chinhu kune yedu Managed Database service muYandex.
Uye pane imwe nzvimbo mukudonha kwe2018, imwe pfungwa yakauya mupfungwa dzedu. Kazhinji timu ine nzira dzinoverengeka dzekugadzira mamwe maficha kana kugadzirisa tsikidzi kana iwe usina maoko akakwana. Semuenzaniso, unogona kuhaya mumwe mugadziri uye kumubhadhara mari. Kana kuti iwe unogona kutora intern kwechinguva uye zvakare kumubhadhara imwe muhoro. Asi kuchine boka guru revanhu, vamwe vacho vanotoziva kunyora kodhi. Iwe hausi kugara uchiziva kuti kodhi yemhando ipi.
Takafunga nezvazvo uye takasarudza kuedza kukwezva vadzidzi. Asi vadzidzi havasi kuzotora chikamu mune zvese nesu. Vachaita chimwe chikamu chebasa chete. Uye ivo, semuenzaniso, vanonyora bvunzo, kugadzirisa tsikidzi, kushandisa maficha asingakanganisi basa guru. Chinhu chikuru ndechekugadzira mabhapu uye kudzoreredza backups. Kana tikaita chikanganiso mukugadzira backup, tinozosangana nekurasikirwa nedata. Uye hapana anoda izvi, hongu. Munhu wose anoda kuti zvinhu zvose zvive zvakachengeteka zvikuru. Naizvozvo, hongu, hatidi kurega kodhi yatinovimba nayo zvishoma pane yedu. Kureva, chero isiri-yakakosha kodhi ndiyo yatinoda kugamuchira kubva kune vamwe vashandi vedu.
Mudzidzi PR inogamuchirwa pasi pemamiriro api?
- Vanosungirwa kuvhara kodhi yavo nemiedzo. Zvese zvinofanirwa kuitika muCI.
- Uye isu zvakare tinoenda kuburikidza ne2 wongororo. Imwe naAndrey Borodin uye imwe neni.
- Uye nekuwedzera, kutarisa kuti izvi hazviputse chero chinhu musevhisi yedu, ndinoisa parutivi gungano nekuzvipira uku. Uye isu tinotarisa mukupedzisira-kusvika-kumagumo bvunzo kuti hapana chinokundikana.
Special course on Open Source
Zvishoma nezve nei izvi zvichidikanwa uye nei izvi, zvichiita kwandiri, ipfungwa yakanaka.
Kwatiri, purofiti iri pachena:
- Tinowana mamwe maoko.
- Uye isu tiri kutsvaga vamiriri vechikwata pakati pevadzidzi vakangwara vanonyora smart kodhi.
Chii chinobatsira vadzidzi?
Vanogona kunge vasina kujeka, nekuti vadzidzi, zvishoma, havagamuchire mari yekodhi yavanonyora, asi vanongogamuchira mamakisi ezvinyorwa zvevadzidzi vavo.
Ndakavabvunza nezvenyaya iyi. Uye mumashoko avo:
- Mupiro ruzivo mu Open Source.
- Tora mutsara muCV yako.
- Zviratidze iwe uye upfuure kubvunzurudza muYandex.
- Iva nhengo yeGSoC.
- +1 yakakosha kosi kune avo vanoda kunyora kodhi.
Handichataura nezvekuti kosi yacho yakarongwa sei. Ndichangoti WAL-G ndiyo yaive purojekiti huru. Isu takasanganisira mapurojekiti akadai seOdyssey, PostgreSQL uye ClickHouse mune ino kosi.
Uye vakapa matambudziko kwete chete mune ino kosi, asiwo vakapa madhipuroma uye kosi.
Zvakadini nerubatsiro rwevashandisi?
Zvino ngatienderere mberi kune chikamu chinokufarira zvakanyanya. Izvi zvinokubatsirei? Chiripo ndechekuti vadzidzi vakagadzirisa tsikidzi dzakawanda. Uye isu takaita maficha ekukumbira awakatikumbira kuti tiite.
Uye regai ndikuudze pamusoro pezvinhu zvawagara uchida uye izvo zvakazadzikiswa.
Tablespaces rutsigiro. Nzvimbo dzematafura muWAL-G dzanga dzichitarisirwa kubva pakuburitswa kweWAL-G, nekuti WAL-G ndiye anotsiva chimwe chishandiso chekuchengetedza WAL-E, uko mabhapu edhatabhesi ane nzvimbo dzetafura akatsigirwa.
Rega ndikuyeuchidze muchidimbu kuti chii uye nei zvese zvichidikanwa. Kazhinji, data rako rese rePostgres rinotora dhairekitori rimwe pane faira system, inonzi base. Uye iyi dhairekitori yatove nemafaira ese uye subdirectories inodiwa nePostgres.
Tablespaces madhairekitori ane Postgres data, asi iwo haana kuwanikwa kunze kweiyo base dhairekitori. Iyo slide inoratidza kuti matafura anowanikwa kunze kwedhairekitori.
Izvi zvinotaridzika sei kune Postgres pachayo? Pane yakaparadzana subdirectory pg_tblspc mune base dhairekitori. Uye ine symlinks kune madhairekitori ane chaizvo Postgres data kunze kweiyo base dhairekitori.
Paunoshandisa zvese izvi, saka kwauri mirairo iyi inogona kutaridzika seizvi. Ndokunge, iwe unogadzira tafura mune imwe yakatsanangurwa tablespace uye ona pairi iko zvino. Iyi ndiyo mitsetse miviri yekupedzisira, mirairo miviri yekupedzisira inonzi. Uye ipapo zviri pachena kuti pane imwe nzira. Asi muchokwadi, iyi haisi iyo nzira chaiyo. Iyi ndiyo prefixed nzira kubva pachigadziko dhairekitori kuenda kune tablespace. Uye kubva ipapo inofananidzwa ne symlink inotungamira kune yako chaiyo data.
Isu hatishandise zvese izvi muchikwata chedu, asi zvakashandiswa nevamwe vazhinji vashandisi veWAL-E vakatinyorera kuti vaida kuenda kuWAL-G, asi izvi zvaive zvavamisa. Izvi zvino zvinotsigirwa.
Chimwe chinhu chatakaunzirwa nekosi yedu yakakosha ndeyekubata. Vanhu vangangoshanda zvakanyanya neOracle pane nePostgres vanoziva nezve catchup.
Muchidimbu kuti chii. Iyo cluster topology mubasa redu inogona kutaridzika seizvi. Tine tenzi. Pane replica inoyerera kunyora-mberi logi kubva mairi. Uye replica inoudza tenzi kuti LSN iri pairi pari. Uye pane imwe nzvimbo inoenderana neizvi, iyo log inogona kuchengetwa. Uye pamusoro pekuchengeta irogi, ma backups anotumirwa zvakare kune gore. Uye delta backups inotumirwa.
Chii chingava chinetso? Paunenge uine dhatabhesi rakakura, zvinogona kuitika kuti replica yako inotanga kusarira kumashure kwatenzi. Uye anosarira kumashure zvekuti haambokwanisa kumubata. Dambudziko iri rinowanzoda kugadziriswa neimwe nzira.
Uye nzira iri nyore ndeyekubvisa replica uye kuiisa zvakare, nokuti haizombobata, uye dambudziko rinoda kugadziriswa. Asi iyi inguva yakati rebei, nekuti kudzoreredza iyo yese gumi TB dhatabhesi backup inguva yakarebesa. Uye isu tinoda kuita zvese izvi nekukurumidza sezvinobvira kana matambudziko akadai amuka. Uye ndizvo chaizvo izvo catchup ndeye.
Catchup inokubvumira kushandisa delta backups, iyo inochengetwa mugore nenzira iyi. Iwe unotaura kuti ndeipi LSN iyo yakarembera replica iripo uye itsanangure mukuraira kwekubata kuitira kuti ugadzire delta backup pakati peiyo LSN neLSN iyo cluster yako iripo. Uye mushure mezvo unodzoreredza iyi backup kune replica yaive yasara kumashure.
Mamwe mabhesi
Vadzidzi vakatiunzirawo zvinhu zvakawanda panguva imwe chete. Sezvo paYandex isu tinobika kwete Postgres chete, isu zvakare tine MySQL, MongoDB, Redis, ClickHouse, pane imwe nguva taida kukwanisa kuita mabhapu nepoint-in-time recovery yeMySQL, uye kuitira kuti pave nemukana wekuisa. vaende mugore.
Uye isu taida kuzviita neimwe nzira yakafanana kune zvinoitwa neWAL-G. Uye takasarudza kuyedza uye kuona kuti zvese zvaizotaridzika sei.
Uye pakutanga, pasina kugovera pfungwa iyi neimwe nzira, vakanyora kodhi muforogo. Vakaona kuti tine imwe mhando yemhando yekushanda uye inogona kubhururuka. Ipapo takafunga kuti nharaunda yedu huru ndeye postgresists, vanoshandisa WAL-G. Uye saka tinofanira kuparadzanisa zvikamu izvi. Ndokunge, kana isu tichigadzirisa kodhi yePostgres, isu hatityore MySQL; kana isu tichigadzirisa MySQL, isu hatiputse Postgres.
Pfungwa yekutanga nezvekuparadzanisa iyi yaive pfungwa yekushandisa nzira imwechete iyo inoshandiswa muPostgreSQL yekuwedzera. Uye, kutaura zvazviri, kugadzira MySQL backup waifanira kuisa imwe mhando yeraibhurari ine simba.
Asi pano iyo asymmetry yeiyi nzira inokurumidza kuoneka. Paunochengetedza Postgres, unoisa yakajairwa backup yePostgres pairi uye zvese zvakanaka. Uye yeMySQL inoitika kuti iwe unoisa backup yePostgres uye zvakare nekuisa ine simba raibhurari yeMySQL yayo. Zvinonzwika sechienzi. Isu takafungawo saizvozvo uye takasarudza kuti iyi yakanga isiri mhinduro yataida.
Yakasiyana-siyana inovaka yePostgres, MySQL, MongoDB, Redis
Asi izvi zvakatibvumira, zvinoratidzika kwatiri, kuti tiuye kune chisarudzo chakanaka - kugovera magungano akasiyana kune zvigaro zvakasiyana. Izvi zvakaita kuti zvikwanise kupatsanura iyo logic yakasungirirwa kune backups yeakasiyana dhatabhesi iyo inowana iyo yakajairwa API iyo WAL-G inoshandisa.
Ichi ndicho chikamu chatakanyora isu pachedu - tisati tapa vadzidzi matambudziko. Ndiko kuti, ichi ndicho chaicho chikamu chavanogona kuita chimwe chinhu chakaipa, saka takasarudza kuti zviri nani tiite chimwe chinhu chakadai uye zvese zvichanaka.
Pashure paizvozvo takapa dambudziko. Vakabva vabviswa. Vadzidzi vaifanirwa kutsigira mabhesi matatu.
Iyi iMySQL, yatave tichitsigira tichishandisa WAL-G nenzira iyi kweanopfuura gore.
Uye ikozvino MongoDB yave kusvika kugadzira, kwavari kuipedzisa nefaira. Muchokwadi, isu takanyora chimiro cheizvi zvese. Ipapo vadzidzi vakanyora zvimwe zvinhu zvinoshanda. Uye zvino tinovaunza kune imwe nyika yatinogona kugamuchira mukugadzirwa.
Matambudziko aya aisaita sekuti vadzidzi vaifanirwa kunyora maturusi ekuchengetedza akazara kune yega yega dhatabhesi. Takanga tisina dambudziko rakadaro. Dambudziko redu nderekuti taida point-in-time recovery uye taida kuchengetedza gore. Uye vakakumbira vadzidzi kuti vanyore imwe kodhi yaizogadzirisa izvi. Vadzidzi vakashandisa maturusi ekuchengetedza akange aripo, ayo neimwe nzira anotora ma backups, ndokuzoabatanidza pamwe chete neWAL-G, iyo yakaendesa ese kune gore. Uye ivo vakawedzera point-in-time kudzoreredza kune izvi.
Chii chimwe chakauya nevadzidzi? Vakaunza Libsodium encryption rutsigiro kuWAL-G.
Tine zvakare mitemo yekuchengetedza yekuchengetedza. Iye zvino ma backups anogona kumakwa seanogara. Uye neimwe nzira zviri nyore kuti sevhisi yako iite otomatiki maitiro ekuachengeta.
Chii chaive mugumisiro wekuedza uku?
Vanhu vanodarika zana vakatanga kunyoresa kuti vapinde kosi iyi. Pakutanga handina kutaura kuti yunivhesiti muYekaterinburg ndiyo Ural Federal University. Takazivisa zvese ipapo. 100 vanhu vakanyoresa. Muchokwadi, vanhu vashoma vakatanga kuita chimwe chinhu, vanenge vanhu makumi matatu.
Kunyangwe vanhu vashoma vakapedza kosi, nekuti zvaive zvakafanira kunyora bvunzo dzekodhi dzatovepo. Uye zvakare gadzirisa imwe bug kana kuita chimwe chimiro. Uye vamwe vadzidzi vachiri kuvhara kosi.
Parizvino, panguva yekosi iyi, vadzidzi vakagadzirisa zvingangosvika gumi neina uye vakaita gumi maficha ehukuru hwakasiyana. Uye, zvinoratidzika kwandiri, uku kutsiva kwakazara kweimwe kana vaviri vanogadzira.
Pakati pezvimwe zvinhu, takapa madhipuroma uye kosi. Uye 12 vakagamuchira zvitupa. 6 vavo vakatozvidzivirira pa "5". Vakasara vanga vasati vadzivirirwa, asi ndinofunga kuti zvese zvichavaitirawo zvakanaka.
Zvirongwa zvemangwana
Zvii zvatine zvirongwa zvenguva yemberi?
Zvirinani izvo zvikumbiro zvemaficha zvatakatonzwa kubva kune vashandisi uye tinoda kuita. Izvi:
- Kutarisisa iko kurongeka kwekutevera nguva muHA cluster backup archive. Unogona kuita izvi neWAL-G. Uye ndinofunga tichava nevadzidzi vachatora nyaya iyi.
- Isu tatova nemunhu ane mutoro wekuendesa backups uye WAL pakati pemakore.
- Uye isu nguva pfupi yadarika takaburitsa zano rekuti isu tinokwanisa kukurumidzira WAL-G zvakatonyanya nekuburitsa ma backups ekuwedzera pasina kunyorazve mapeji uye nekugadzirisa zvakachengetwa zvatinotumira ikoko.
Unogona kuagovera pano
Chirevo ichi chaive chei? Uyezve, ikozvino, kuwedzera kune vanhu 4 vanotsigira chirongwa ichi, tine mamwe maoko, ayo ane akawanda kwazvo. Kunyanya kana iwe ukavanyorera mumeseji yega. Uye kana iwe ukachengetedza data rako uye uchizviita uchishandisa WAL-G kana uchida kutamira kuWAL-G, saka tinogona kugashira zvishuwo zvako zviri nyore.
Iyi iQR kodhi uye chinongedzo. Unogona kuenda kuburikidza navo uye kunyora zvese zvaunoda. Semuenzaniso, isu hatisi kugadzirisa imwe bug. Kana kuti iwe unonyatsoda chimwe chimiro, asi nekuda kwechimwe chikonzero haisati yave mune chero backup, kusanganisira yedu. Iva nechokwadi chekunyora pamusoro peizvi.
Mibvunzo yako
Mhoro! Ndatenda neshumo! Mubvunzo nezve WAL-G, asi kwete nezvePostgres. WAL-G inotsigira MySQL uye inodaidza imwe backup. Kana tikatora kuiswa kwemazuva ano paCentOS uye kana ukaita yum install MySQL, MariDB ichaiswa. Kubva mushanduro 10.3 yekuwedzera backup haitsigirwe, MariDB backup inotsigirwa. Uri kuita sei nazvo?
Parizvino hatina kuyedza kuchengetedza MariDB. Takava nezvikumbiro zvekutsigira FoundationDB, asi kazhinji, kana pane chikumbiro chakadaro, saka tinogona kuwana vanhu vanozviita. Haina kureba kana kuoma sezvandinofunga.
Masikati akanaka Ndatenda nemushumo! Mubvunzo pamusoro pezvitsva zvingangoitika. Wagadzirira here kuita kuti WAL-G ishande nematepi kuitira kuti ugone kuchengetedza matepi?
Negadziriro yeparutivi tepi kuchengetedza sezviri pachena zvinoreva?
Hongu.
Pane Andrei Borodin, uyo anogona kupindura mubvunzo uyu zviri nani kupfuura ini.
(Andrey) Hongu, ndatenda nemubvunzo! Taive nechikumbiro chekuendesa backup kune tepi kubva kuchengetedzo yegore. Uye nokuda kweizvi kutamisana pakati pemakore. Nekuti gore-kune-gore kutamisa ishanduro yakajairwa yekufambisa tepi. Pamusoro pezvo, isu tine yakawedzera dhizaini maererano neStorages. Nenzira, dzakawanda Storoges dzakanyorwa nevadzidzi. Uye kana iwe ukanyora Kuchengetera tepi, zvino ichave, zvechokwadi, inotsigirwa. Takagadzirira kufunga zvikumbiro zvekudhonza. Ikoko unoda kunyora faira, verenga faira. Kana iwe ukaita zvinhu izvi muGo, iwe kazhinji unoguma uine makumi mashanu mitsara yekodhi. Uye ipapo tepi inotsigirwa muWAL-G.
Ndatenda neshumo! Inonakidza nzira yekuvandudza. Backup chinhu chakakomba chekushanda chinofanirwa kufukidzwa nemayedzo. Pawakaita basa remadatabase matsva, vadzidzi vakanyorawo bvunzo here, kana kuti wakanyora bvunzo iwe wozopa kuitiswa kuvadzidzi?
Vadzidzi vakanyorawo bvunzo. Asi vadzidzi vakanyora zvimwe zvezvinhu zvakaita sedhatabhesi idzva. Vakanyora bvunzo dzekubatanidza. Uye vakanyora mayunitsi bvunzo. Kana iyo yekubatanidza ikapfuura, ndiko kuti, panguva ino, iyi script yaunoshandisa nemaoko kana iwe une cron uchizviita, semuenzaniso. Ndiko kuti, script ipapo zvakajeka.
Vadzidzi havana ruzivo rwakawanda. Kuongorora kunotora nguva yakawanda here?
Hongu, kuongorora kunotora nguva yakawanda. Kureva kuti, kazhinji, kana vamiriri vakati wandei vauya kamwechete ndokuti ndakaita izvi, ndakaita izvo, saka iwe unofanirwa kufunga uye kuisa parutivi hafu yezuva kuti uone zvavakanyora ipapo. Nokuti code inofanira kuverengwa nokungwarira. Vakanga vasina interview. Hativazivi zvakanyanya, saka zvinotora nguva yakati wandei.
Ndatenda nemushumo! Pakutanga, Andrey Borodin akataura kuti archive_command muWAL-G inofanirwa kudaidzwa zvakananga. Asi kana iri imwe mhando ye cluster cartridge, tinoda imwe pfungwa yekuona node kubva pakutumira shafts. Unogadzirisa sei dambudziko iri iwe pachako?
Dambudziko rako nderei apa? Ngatiti iwe une synchronous replica yauri kugadzira backup? Kana kuti chii?
(Andrey) Chokwadi ndechekuti zvechokwadi WAL-G inotarisirwa kushandiswa pasina magwaro egoko. Kana chimwe chinhu chisipo, saka ngatiwedzerei pfungwa inofanirwa kunge iri mukati meWAL-G. Kana zviri izvo zvinofanirwa kubva kuchengetedza zvinyorwa, isu tinotenda kuti kuchengetedza kunofanirwa kubva kuna tenzi aripo musumbu. Kuchengeta kubva kune replica ipfungwa yakaipa. Pane zvakasiyana-siyana zvinogoneka nezvinetso. Kunyanya, matambudziko nearchive timelines uye chero ruzivo rwekuwedzera. Ndatenda nemubvunzo!
(Kujekesa: Takabvisa zvinyorwa zvemagoko )
Manheru akanaka! Ndatenda neshumo! Ndiri kufarira nyaya yekubata yawataura nezvayo. Takanga takatarisana nemamiriro ezvinhu apo mufananidzo wakanga uri shure uye haugone kubata. Uye ini handina kuwana tsananguro yechinhu ichi muWAL-G zvinyorwa.
Catchup yakaonekwa chaizvo musi wa20 Ndira 2020. Zvinyorwa zvingada rimwe basa. Isu tinozvinyora isu uye hatizvinyore zvakanyanya. Uye zvichida tinofanira kutanga kuda kuti vadzidzi vanyore.
Yakatosunungurwa here?
The pull request yatofa, i.e. ndakaitarisa. Ndakaedza izvi pane test cluster. Parizvino hatisati tave nemamiriro ezvinhu atinogona kuyedza izvi mumuenzaniso wekurwa.
Kutarisira rini?
Handizive. Mirira mwedzi, tichatarisa zvechokwadi.
Source: www.habr.com