Ndicebisa ukuba ufunde umbhalo wengxelo ukususela ekuqaleni kwe-2019 ngu-Andrey Borodin "Ii-backups nge-WAL-G. Yintoni ekhoyo kwi-2019?"
Molweni nonke! Igama lam nguAndrey Borodin. Ndingumphuhlisi eYandex. Ndinomdla kwi-PostgreSQL ukususela ngo-2016, emva kokuba ndithethe nabaphuhlisi kwaye bathi yonke into ilula - uthatha ikhowudi yomthombo kwaye uyakha, kwaye yonke into iya kusebenza. Kwaye ukususela ngoko andinakuyeka - ndibhala zonke iintlobo zezinto ezahlukeneyo.
Enye yezinto endisebenza kuyo yinkqubo yokugcina. . Ngokubanzi, kwiYandex sisebenze kwiinkqubo zokugcina kwi-PostgreSQL ixesha elide kakhulu. Kwaye ungafumana kwi-Intanethi uthotho lweengxelo ezintandathu malunga nendlela esenza ngayo iinkqubo zokugcina. Kwaye minyaka le ziguquka kancinci, ziphuhlise kancinci, kwaye zithembeke ngakumbi.
Kodwa namhlanje ingxelo ayikho kuphela malunga nento esiyenzileyo, kodwa kunye nokuba ilula kangakanani kwaye yintoni. Bangaphi kuni esele bebukele iingxelo zam malunga ne-WAL-G? Kulungile ukuba abantu abambalwa khange babukele, kuba ndiza kuqala ngeyona nto ilula.
Ukuba ngequbuliso uneqela lePostgreSQL, kwaye ndicinga ukuba wonke umntu unazo isibini, kwaye ngequbuliso akukho nkqubo yogcino okwangoku, ke kuya kufuneka ufumane ugcino lwe-S3 okanye ugcino oluhambelana neLifu likaGoogle.
Ngokomzekelo, unokuza kwisigxina sethu kwaye uthathe ikhowudi yokuphromotha ye-Yandex Object Storage, ehambelana ne-S3.
Emva koko yenza iBhakethi. Sisikhongozeli nje solwazi.
Yenza umsebenzisi wenkonzo.
Yenza iqhosha lofikelelo lomsebenzisi wenkonzo: aws-s3-key.
Khuphela ukhupho lwamva nje oluzinzileyo lwe-WAL-G.
Ingaba ukhupho lwethu lwangaphambili lwahluke njani kukhupho? Ndihlala ndicelwa ukuba ndikhulule kwangoko. Kwaye ukuba akukho bug kwinguqulo ixesha elaneleyo, umzekelo, inyanga, ngoko ndiyayikhulula. Nalu olu khupholo lukaNovemba. Kwaye oku kuthetha ukuba inyanga nenyanga sifumene uhlobo oluthile lwe-bug, ngokuqhelekileyo ekusebenzeni okungabalulekanga, kodwa asikakhuphi ukukhululwa. Inguqulelo yangaphambili ingoNovemba kuphela. Akukho bugs ezaziwayo kuthi kuyo, o.k.t. iibugs zongezwa njengoko iprojekthi iqhubela phambili.
Nje ukuba ukhuphele i-WAL-G, unokusebenzisa umyalelo olula βwoluhlu logcinoβ, udlula kwizinto eziguquguqukayo zokusingqongileyo. Kwaye iya kuqhagamshelwa kwi-Object Storage kwaye ikuxelele ukuba zeziphi ii-backups onazo. Ekuqaleni, ngokuqinisekileyo, akufanele ube nee-backups. Inqaku lesi silayidi kukubonisa ukuba yonke into ilula kakhulu. Lo ngumyalelo weconsole owamkela iimeko-bume ezahlukeneyo kwaye wenze imiyalelo engaphantsi.
Emva koko, unokwenza i-backup yakho yokuqala. Yithi "backup-push" kwi-WAL-G kwaye ucacise kwi-WAL-G indawo yepgdata yeqela lakho. Kwaye kunokwenzeka ukuba, iPostgreSQL iya kukuxelela, ukuba awunayo inkqubo yokugcina, kufuneka wenze "imowudi yogcino".
Oku kuthetha ukuba kufuneka uye kwisethingi kwaye uvule "archive_mode = on" kwaye wongeze "archive_command", ekwangumyalelo ongaphantsi kwi-WAL-G. Kodwa ngesizathu esithile abantu bahlala besebenzisa izikripthi zebar kwesi sihloko kwaye bayisongele nge-WAL-G. Nceda ungayenzi le nto. Sebenzisa umsebenzi ofunyenwe kwi-WAL-G. Ukuba kukho into oyiphosayo, bhalela . I-WAL-G ithatha ukuba kuphela kwenkqubo eqhutywa kwi-archive_command.
Sisebenzisa i-WAL-G ngokuyinhloko ukwenza i-High Availability cluster in Yandex Database management.
Kwaye ihlala isetyenziswa kwi-topology ye-Master enye kunye neempendulo ezininzi. Ngexesha elifanayo, yenza ikopi yokugcina kwi-Yandex Object Storage.
Ezona meko zixhaphakileyo kukudala iikopi zeqela kusetyenziswa iPoint ngexesha lokubuyisela. Kodwa kule meko, ukusebenza kwenkqubo yokugcina akubalulekanga kangako kuthi. Sidinga nje ukulayisha iqela elitsha kwi-backup.
Ngokuqhelekileyo, sifuna ukusebenza kwenkqubo yogcino xa songeza i-node entsha. Kutheni ibalulekile nje? Ngokuqhelekileyo abantu bongeza i-node entsha kwiqela kuba iqela elikhoyo alikwazi ukuphatha umthwalo ofundiweyo. Kufuneka bongeze ikopi entsha. Ukuba songeza umthwalo kwi-pg_basebackup ukuya kwi-Master, ngoko i-Master ingawa. Ke ngoko, bekubaluleke kakhulu kuthi ukuba silayishe ngokukhawuleza i-node entsha kwindawo yogcino, senze umthwalo omncinci kwi-Master.
Kwaye enye imeko efanayo. Esi sisidingo sokuphinda uqalise i-Master endala emva kokutshintsha i-Cluster Master kwiZiko leDatha ekuphulukene nalo uqhagamshelwano.
- Ngenxa yoko, xa siqulunqa iimfuno zenkqubo yokukopisha, saqaphela ukuba i-pg_basebackup ayifanelekanga kuthi xa isebenza kwifu.
- Besifuna ukukwazi ukucinezela idatha yethu. Kodwa phantse nayiphi na inkqubo yogcino ngaphandle koko kuza kwibhokisi iya kubonelela ngoxinzelelo lwedatha.
- Besifuna ukuthelekisa yonke into kuba umsebenzisi osefini uthenga inani elikhulu leprosesa. Kodwa ukuba asinakuhambelana kwintsebenzo ethile, ke inani elikhulu lee-cores liba lilize.
- Sifuna uguqulelo oluntsonkothileyo kuba amaxesha amaninzi idatha ayiyoyethu kwaye ayinakugcinwa kumbhalo ocacileyo. Ngendlela, igalelo lethu kwi-WAL-G yaqala ngokufihlwa. Sigqibe ukubethelwa kwi-WAL-G, emva koko sabuzwa: "Mhlawumbi omnye wethu uza kuphuhlisa iprojekthi?" Kwaye ukusukela ngoko bendisebenza ne-WAL-G ngaphezulu konyaka.
- Sikwafuna ukuxinana kwezixhobo, kuba ekuhambeni kwexesha sisebenzisa ilifu, safumanisa ukuba ngamanye amaxesha abantu banomthwalo obalulekileyo wegrosari ebusuku kwaye lo mthwalo awunakuphazanyiswa. Yiyo loo nto songeze i-resource throttling.
- Kunye noluhlu kunye nolawulo.
- Nokuqinisekisa.
Sijonge izixhobo ezininzi ezahlukeneyo. Ngethamsanqa, sinokhetho olukhulu kwiPostgreSQL. Kwaye kuyo yonke indawo besiphosa into ethile, enye into encinci, enye into encinci.
Kwaye emva kokuvavanya iinkqubo ezikhoyo, safikelela kwisigqibo sokuba siya kuphuhlisa i-WAL-G. Yayiyiprojekthi entsha ngoko. Kwakulula kakhulu ukuphembelela uphuhliso olubhekiselele kwisiseko selifu senkqubo yogcino.
Iingcamango eziphambili esizithobelayo kukuba i-WAL-G kufuneka ibe lula njenge-balalaika.
I-WAL-G inemiyalelo emi-4. Oku:
WAL-PUSH β gcina i-shaft.
I-WAL-FETCH - fumana i-shaft.
BACKUP-PUSH β yenza i-backup.
BACKUP-FETCH β fumana i-backup kwisistim yogcino.
Enyanisweni, i-WAL-G ikwanayo nolawulo lwezi zigcino, oko kukuthi ukudwelisa kunye nokucima iirekhodi kunye nokugcinwa kwe-backups kwimbali engasadingekiyo okwangoku.
Omnye wemisebenzi ebalulekileyo kuthi ngumsebenzi wokwenza iikopi ze-delta.
Iikopi ze-Delta zithetha ukuba asenzi i-backup epheleleyo yeqela lonke, kodwa kuphela amaphepha atshintshiweyo eefayile ezitshintshileyo kwiqela. Kubonakala ngathi ngokusebenzayo oku kufana kakhulu nokukwazi ukubuyisela kwakhona usebenzisa i-WAL. Kodwa sinokusonga i-delta enomsonto omnye we-WAL ngokuhambelanayo. Ngokufanelekileyo, xa sine-backup esisiseko eyenziwe ngoMgqibelo, i-delta backups imihla ngemihla, kwaye ngoLwesine siyasilela, ngoko kufuneka sikhuphe i-4 delta backups kunye ne-10 iiyure ze-WAL. Kuya kuthatha malunga nexesha elifanayo kuba i-delta backups iqengqeleka ngokuhambelana.
I-deltas esekelwe kwi-LSN - oku kuthetha ukuba xa udala i-backup, kuya kufuneka sidibanise iphepha ngalinye kwaye sijonge i-LSN yayo kunye ne-LSN ye-backup yangaphambili ukuze siqonde ukuba itshintshile. Naliphi na iphepha elinokuthi liqulathe idatha etshintshileyo kufuneka libekho kugcino lwe-delta.
Njengoko benditshilo, ingqwalasela eninzi ihlawulwe kwi-parallelism.
Kodwa i-API yogcino kwiPostgreSQL iyahambelana. I-PostgreSQL igcina ifayile enye ye-WAL kwaye xa ibuyisela icela ifayile enye ye-WAL. Kodwa xa i-database icela ifayile ye-WAL enye isebenzisa umyalelo othi "WAL-FETCH", sibiza umyalelo othi "WAL-PREFETCH", olungiselela iifayile ezi-8 ezilandelayo zokulanda idatha kwivenkile yezinto ngokuhambelanayo.
Kwaye xa i-database isicela ukuba sigcine ifayile enye, sijonga i-archive_status kwaye sibone ukuba kukho ezinye iifayile ze-WAL. Kwaye sizama ukukhuphela i-WAL ngokuhambelanayo. Oku kubonelela ngenzuzo ebalulekileyo yokusebenza kwaye kunciphisa kakhulu umgama kwinani le-WAL ezingagcinwanga. Uninzi lwabaphuhlisi benkqubo yogcino bakholelwa ukuba le yinkqubo enobungozi kuba sithembele kulwazi lwethu lwangaphakathi lwekhowudi engeyiyo iPostgreSQL API. I-PostgreSQL ayiqinisekisi ubukho be-archive_status ifolda kuthi kwaye ayiqinisekisi i-semantics, ubukho beempawu zokulungela iifayile ze-WAL apho. Nangona kunjalo, sifunda ikhowudi yomthombo, siyabona ukuba oku kunjalo kwaye sizama ukuyisebenzisa. Kwaye silawula isalathiso apho i-PostgreSQL iphuhlisa; ukuba ngequbuliso le ndlela yophukile, siya kuyeka ukuyisebenzisa.
Kwimo ecocekileyo, i-LSN-based WAL delta ifuna ukufunda nayiphi na ifayile yeqela elinemowudi yexesha kwinkqubo yefayile itshintshile ukususela kwi-backup yangaphambili. Sahlala nale nto ixesha elide, phantse unyaka. Kwaye ekugqibeleni safikelela kwisigqibo sokuba sine-WAL deltas.
Oku kuthetha ukuba ngalo lonke ixesha sigcina i-WAL kwi-Master, asiyicinezeli nje kuphela, siyifihla kwaye siyithumele kwinethiwekhi, kodwa siyayifunda kwangaxeshanye. Sihlalutya kwaye sifunde iirekhodi ezikuyo. Siyaqonda ukuba zeziphi iibhloko ezitshintshileyo kwaye ziqokelele iifayile ze-delta.
Ifayile ye-delta ichaza uluhlu oluthile lweefayile ze-WAL, ichaza ulwazi malunga nokuba zeziphi iibhloko ezitshintshiweyo kolu luhlu lwe-WAL. Kwaye ke ezi fayile ze-delta nazo zigcinwe.
Apha sijongene nesibakala sokuba sangqamanisa yonke into ngokukhawuleza, kodwa asikwazi ukufunda imbali elandelelanayo ngokunxuseneyo, kuba kwicandelo elithile sinokudibana nesiphelo serekhodi ye-WAL yangaphambili, esingenanto yokudibanisa nayo okwangoku, ngenxa yokuba. ukufunda ngokuhambelanayo kwakhokelela ekubeni siqale sihlalutye ikamva, elingekabinalo ixesha elidlulileyo.
Ngenxa yoko, kwafuneka sibeke iziqwenga ezingaqondakaliyo kwiifayile ze_delta_partial. Ngenxa yoko, xa sibuyela kwixesha elidlulileyo, siya kunamathisela iziqwenga ze-WAL kwirekhodi enye, emva koko siya kuyicalula kwaye siqonde ukuba yintoni etshintshileyo kuyo.
Ukuba kwimbali yokwahlulahlula i-shaft kukho ubuncinci indawo enye apho singaqondi ukuba kwenzeka ntoni, ngoko, ngokufanelekileyo, ngexesha lokugcinwa kwe-backup elandelayo siya kunyanzeliswa ukuba sifunde lonke iqela kwakhona, njengoko senzile nge-LSN eqhelekileyo. -based delta.
Ngenxa yoko, konke ukubandezeleka kwethu kukhokelele ekubeni sivule ithala leencwadi le-WAL-G lokwahlulahlula. Ngokwazi kwam, akukho mntu uyisebenzisayo okwangoku, kodwa ukuba nabani na ufuna ukuyibhala kwaye ayisebenzise, ββikwindawo yoluntu. (Ikhonkco elihlaziyiweyo )
Ngenxa yoko, lonke ulwazi lokuhamba lubonakala luntsonkothile. Inkosi yethu igcina i-shaft kwaye igcina iifayile zedelta. Kwaye i-replica eyenza ikopi yokugcina kufuneka ifumane iifayile ze-delta ngexesha elidlulile phakathi kwee-backups. Kule meko, iinxalenye zembali ziya kufuneka zongezwe ngobuninzi kwaye zihlulwe, kuba akuyiyo yonke imbali ehambelana namacandelo amakhulu. Kwaye kuphela emva koku i-replica inokugcina i-delta epheleleyo yokugcina.
Kwiigrafu yonke into ibonakala ilula kakhulu. Oku kukhutshelwa kwelinye lamaqela ethu okwenene. Sine-LSN-based, eyenziwe ngosuku olunye. Kwaye siyabona ukuba i-LSN-based delta backup yayiqhuba ukusuka kweyesithathu ekuseni ukuya kweyesihlanu ekuseni. Lo ngumthwalo kwinani le processor cores. I-WAL-delta yasithatha apha malunga nemizuzu engama-20. Oko kukuthi, iye yakhawuleza kakhulu, kodwa kwangaxeshanye kwakukho ukutshintshiselana okukhulu ngakumbi kwinethiwekhi.
Kuba sinolwazi malunga nokuba zeziphi iibhloko ezitshintshileyo kwaye ngeliphi ixesha kwimbali yesiseko sedatha, siye saya phambili kwaye sagqiba kwelokuba sidibanise ukusebenza - ulwandiso lwePostgreSQL olubizwa ngokuba "pg_prefaulter"
Oku kuthetha ukuba xa isiseko sokuma senza umyalelo wokubuyisela, ixelela i-WAL-G ukuba ilande ifayile ye-WAL elandelayo. Siyaqonda malunga nokuba yeyiphi idatha evimba inkqubo yokubuyisela i-WAL eya kufikelela kwikamva elikufutshane kwaye iqalise umsebenzi wokufunda kwezi bhloko. Oku kwenziwa ukwenzela ukwandisa ukusebenza kwabalawuli be-SSD. Ngenxa yokuba i-WAL roll iya kufikelela kwiphepha elifuna ukutshintshwa. Eli phepha likwidiski kwaye alikho kwindawo yokugcina yephepha. Kwaye uya kulinda ngaxeshanye ukuba eli phepha lifike. Kodwa kufuphi ne-WAL-G, eyazi ukuba kumakhulu ambalwa alandelayo e-megabytes ye-WAL siya kufuna amaphepha athile kwaye kwangaxeshanye siqala ukuwafudumeza. Iqalisa ufikelelo lwediski ezininzi ukuze ziphunyezwe ngaxeshanye. Oku kusebenza kakuhle kwi-SSD drives, kodwa, ngelishwa, ayisebenzi ngokupheleleyo kwi-hard drive, kuba siphazamisana nayo kuphela ngeengcebiso zethu.
Yile nto ikwikhowudi ngoku.
Kukho iimpawu esingathanda ukuzongeza.
Lo mfanekiso ubonisa ukuba i-WAL-delta ithatha ixesha elifutshane. Kwaye oku kufunda utshintsho olwenzeke kwisiseko sedatha emini. Sinokwenza i-WAL-delta kungekuphela nje ebusuku, kuba ayisenguye umthombo obalulekileyo womthwalo. Singakwazi ukufunda i-WAL-delta yonke imzuzu kuba iyabiza. Ngomzuzu omnye sinokuskena zonke iinguqu ezenzeke kwiqela. Kwaye oku kunokubizwa ngokuba yi "instant WAL-delta".
Ingongoma kukuba xa sibuyisela i-cluster, sinciphisa inani lamabali ekufuneka siwasonge ngokulandelelana. Oko kukuthi, inani le-WAL ukuba i-PostgreSQL rolls kufuneka incitshiswe, kuba kuthatha ixesha elibalulekileyo.
Kodwa akuphelelanga apho. Ukuba siyazi ukuba ibhloko ethile iya kutshintshwa ukuya kwinqanaba lokuhambelana kwe-backup, asinakukwazi ukuyitshintsha kwixesha elidlulileyo. Oko kukuthi, ngoku sinefayile-nge-fayile yolungiselelo logqithiso lwe-WAL-delta. Oku kuthetha ukuba, umzekelo, ngoLwesibini itafile yacinywa ngokupheleleyo okanye ezinye iifayile zicinywe ngokupheleleyo etafileni, ngoko xa i-delta iqengqeleka ngoMvulo nangoMgqibelo pg_basebackup ibuyiselwe, asiyi nokuyidala le datha.
Sifuna ukwandisa le teknoloji ukuya kwinqanaba lephepha. Oko kukuthi, ukuba inxalenye yefayile iyatshintsha ngoMvulo, kodwa iya kubhalwa ngaphezulu ngoLwesithathu, ngoko xa ubuyisela kwinqanaba ngoLwesine, akufuneki ukuba sibhale iinguqulelo zokuqala zamaphepha kwidiski.
Kodwa oku kusengumbono oxoxwa ngenkuthalo ngaphakathi kwethu, kodwa awukafiki kwikhowudi.
Sifuna ukwenza into enye kwi-WAL-G. Sifuna ukuyenza yandiswe kuba kufuneka sixhase oovimba beenkcukacha ezahlukeneyo kwaye singathanda ukukwazi ukujongana nolawulo logcino ngendlela efanayo. Kodwa ingxaki kukuba i-MySQL APIs yahluke kakhulu. Kwi-MySQL, i-PITR ayisekelwanga kwilog ye-WAL ebonakalayo, kodwa kwi-binlog. Kwaye asinayo inkqubo yokugcina kwi-MySQL eya kuxelela inkqubo yangaphandle ukuba le bhinlog igqityiwe kwaye kufuneka igcinwe. Kufuneka sime kwindawo ethile kwi-cron kunye ne-database kwaye sikhangele ukuba kukho into ekulungeleyo?
Kwaye ngendlela efanayo, ngexesha lokubuyisela i-MySQL, akukho myalelo wokubuyisela onokuxelela inkqubo ukuba ndifuna iifayile ezinjalo kunye nezinje. Ngaphambi kokuba uqale ukwakha kwakhona iqela lakho, kufuneka wazi ukuba zeziphi iifayile oza kuzifuna. Wena ngokwakho kufuneka uqikelele ukuba zeziphi iifayile oza kuzifuna. Kodwa ezi ngxaki zisenokuthintelwa ngandlelβ ithile. (Ingcaciso: I-MySQL sele ixhasiwe)
Kwingxelo, ndandifuna ukuthetha ngaloo matyala xa i-WAL-G ingafanelekanga kuwe.
Ukuba awunayo i-synchronous replica, i-WAL-G ayiqinisekisi ukuba icandelo lokugqibela liyakugcinwa. Kwaye ukuba ugcino lwazi lusalele ngasemva kumacandelo ambalwa okugqibela embali, oko kuyingozi. Ukuba akukho replica ehambelanayo, andizukucebisa ukusebenzisa i-WAL-G. Nangona kunjalo, yenzelwe ukufakwa kwamafu, oku kuthetha isisombululo sokuFumana okuPhezulu kunye ne-synchronous replica, ejongene nokhuseleko lwee-byte zokugqibela ezizinikezele.
Ndihlala ndibona abantu bezama ukuqhuba zombini i-WAL-G kunye ne-WAL-E ngexesha elinye. Sixhasa ukuhambelana ngasemva ngengqiqo yokuba i-WAL-G inokubuyisela ifayile kwi-WAL-E kwaye inokubuyisela i-backup eyenziwe kwi-WAL-E. Kodwa ekubeni zombini ezi nkqubo zisebenzisa i-parallel wal-push, ziqala ukweba iifayile omnye komnye. Ukuba siyilungisa kwi-WAL-G, iya kuhlala ikwi-WAL-E. Kwi-WAL-E, ijonga kwi-archive-status, ibona iifayile ezigqityiweyo kwaye zigcinwe, ngelixa ezinye iinkqubo aziyi kwazi ukuba le fayile ye-WAL ikhona, kuba i-PostgreSQL ayiyi kuzama ukuyigcina okwesibini.
Siza kulungisa ntoni apha kwicala le-WAL-G? Asiyi kwazisa i-PostgreSQL ukuba le fayile idluliselwe ngokufana, kwaye xa i-PostgreSQL isicela ukuba siyigcine, siya kuba siyazi ukuba ifayile elolu hlobo kunye nale md5 sele igcinwe kwaye siya kuthi nje i-PostgreSQL - Kulungile, yonke into ilungile ngaphandle kokwenza nantoni na.
Kodwa le ngxaki ayinakwenzeka ukuba ilungiswe kwicala le-WAL-E, ngoko ke akunakwenzeka ngoku ukwenza umyalelo wogcino oluzakugcina ifayile kuzo zombini i-WAL-G kunye ne-WAL-E.
Ukongeza, kukho iimeko apho i-WAL-G ayikufanelanga ngoku, kodwa siya kuyilungisa ngokuqinisekileyo.
Okokuqala, okwangoku asinayo isiqinisekiso sogcino esakhelwe ngaphakathi. Asinaso isiqinisekiso nokuba ngexesha lokugcinwa okanye ukubuyisela. Ewe, oku kuphunyezwa kwilifu. Kodwa oku kuphunyezwa ngokujonga kwangaphambili, ngokubuyisela iqela. Ndingathanda ukunika lo msebenzi kubasebenzisi. Kodwa ngokuqinisekiswa, ndicinga ukuba kwi-WAL-G kuya kwenzeka ukubuyisela iqela kwaye liqalise, kwaye liqhube iimvavanyo zomsi: pg_dumpall ukuya /dev/null kunye nokuqinisekiswa kwesalathisi se-amcheck.
Okwangoku kwi-WAL-G akukho ndlela yokuhlehlisa i-backup enye kwi-WAL. Oko kukuthi, sixhasa ifestile ethile. Umzekelo, ukugcina iintsuku ezisixhenxe zokugqibela, ukugcina ii-backups ezilishumi zokugqibela, ukugcina ii-backups ezipheleleyo ezintathu zokugqibela. Ngokuqhelekileyo abantu beza bathi: "Sifuna i-backup yento eyenzekayo ngoNyaka oMtsha kwaye sifuna ukuyigcina ngonaphakade." U-WAL-G akakayazi indlela yokwenza oku. (Qaphela - Oku sele kulungisiwe. Funda ngakumbi - I-Backup-mark ukhetho kwi )
Kwaye asinazo iitshekhi zephepha kunye nokuhlolwa kwemfezeko kuwo onke amacandelo e-shaft xa kuqinisekiswa i-PITR.
Ukusuka kuyo yonke le nto ndidibanise iprojekthi yeHlobo likaGoogle leKhowudi. Ukuba uyabazi abafundi abakrelekrele abangathanda ukubhala into ethi Go kwaye bafumane amawaka eedola aliqela kwinkampani enye enobumba βGβ, ke ucebise iprojekthi yethu kubo. Ndiza kusebenza njengomcebisi wale projekthi, bangayenza. Ukuba akukho bafundi, ngoko ndiya kuyithatha kwaye ndiyenze ngokwam ehlotyeni.
Kwaye sinezinye iingxaki ezininzi ezincinci esizilungisa ngokuthe ngcembe. Kwaye kwenzeka izinto ezingaqhelekanga.
Umzekelo, ukuba unika i-WAL-G i-backup engenanto, iya kuwa ngokulula. Umzekelo, ukuba umxelela ukuba kufuneka agcine ifolda engenanto. Ifayile ye pg_control ayizubakho. Kwaye uya kucinga ukuba akayiqondi into ethile. Kwithiyori, kule meko kufuneka ubhale umyalezo oqhelekileyo kumsebenzisi ukuba amchazele indlela yokusebenzisa isixhobo. Kodwa oku akulophawu lwenkqubo, kodwa luphawu lolwimi oluhle, oluqondakalayo.
Asazi ukuba singenza njani i-backup ngaphandle kweintanethi. Ukuba i-database ixoka, asinako ukuyigcina. Kodwa yonke into ilula kakhulu apha. Sibiza ii-backups nge-LSN xa iqalile. I-LSN yesiseko esisezantsi mayifundwe kwifayili yolawulo. Kwaye olu luphawu olungafezekanga. Iinkqubo ezininzi zogcino zinokugcina isiseko sedatha. Kwaye ifanelekile.
Okwangoku asikwazi ukumelana nokunqongophala kwendawo yokugcina ngokufanelekileyo. Kuba sihlala sisebenza ngee-backups ezinkulu ekhaya. Kwaye abazange bafikelele kuyo. Kodwa ukuba umntu ufuna ukwenza inkqubo kwi-Go ngoku, yongeza ukuphatha iimpazamo ezingaphandle kwendawo kwibhakethi. Ngokuqinisekileyo ndiya kujonga kwisicelo sokutsala.
Kwaye eyona nto isikhathazayo kukuba sifuna iimvavanyo ezininzi zokudityaniswa kwe-docker kangangoko sijonga iimeko ezahlukeneyo. Okwangoku sivavanya kuphela iimeko ezisisiseko. Kuzo zonke izibophelelo, kodwa sifuna ukujonga ukuzibophelela ngokuzibophelela kuyo yonke imisebenzi esiyixhasayo. Ngokukodwa, umzekelo, siya kuba nenkxaso eyaneleyo ye-PostgreSQL 9.4-9.5. Siyabaxhasa kuba uluntu luyayixhasa iPostgreSQL, kodwa asijongi ukuzibophelela ngokuqinisekisa ukuba yonke into ayophukanga. Kwaye kubonakala kum ukuba oku kuyingozi enkulu.
Sine-WAL-G eqhuba kumaqela angaphezu kwewaka kwi-Yandex Database management. Kwaye ixhasa amakhulu aliqela eeterabytes zedatha yonke imihla.
Sine-TODO eninzi kwikhowudi yethu. Ukuba ufuna ukwenza inkqubo, yiza, silindele izicelo zokutsalwa, silindele imibuzo.
Imibuzo yakho
Mholo ngokuhlwa nje! Enkosi! Uqikelelo lwam kukuba ukuba usebenzisa i-WAL-delta, mhlawumbi uthembele kakhulu kumaphepha apheleleyo abhalayo. Kwaye ukuba kunjalo, ulwenzile uvavanyo? Ubonise igrafu entle. Iba ntle kangakanani ukuba i-FPW icinyiwe?
Iphepha elipheleleyo libhala livulelwe thina, asizange sizame ukuyicima. Oko kukuthi, mna, njengomphuhlisi, andizange ndizame ukuyicima. Abalawuli benkqubo abaye baphanda mhlawumbi baye baphanda ngalo mba. Kodwa sifuna iFPW. Phantse akukho mntu uyivalayo, kuba kungenjalo akunakwenzeka ukuthatha i-backup kwi-replica.
Enkosi ngengxelo! Ndinemibuzo emibini. Umbuzo wokuqala ngowokuba kuya kwenzeka ntoni kwiindawo zetafile?
Silindele isicelo sokutsalwa. Oovimba bethu bogcino-lwazi bahlala kwi-SSD kunye neediski ze-NMVE kwaye asiyifuni ngokwenene eli nqaku. andikakulungeli ukuchitha ixesha eli serious ngoku ndiyenze kakuhle. Ndiyithethelela ngentliziyo yam yonke ukuba siyixhase le nto. Kukho abantu ababeyixhasa, kodwa bayixhasa ngendlela ebafaneleyo. Benza ifolokhwe, kodwa abenzi zicelo zokutsalwa. (Yongezwe kwinguqulo 0.2.13)
Kwaye umbuzo wesibini. Utshilo kwasekuqaleni ukuba i-WAL-G ithatha ukuba isebenza yodwa kwaye akukho zisongelo ezifunekayo. Ndisebenzisa izisongelo ngokwam. Kutheni zingasetyenziswa?
Sifuna ukuba kube lula njenge-balalaika. Oku kuthetha ukuba awudingi nantoni na ngaphandle kwe-balalaika. Sifuna inkqubo ibe lula. Ukuba unomsebenzi ekufuneka uyenzile kwiskripthi, yiza ke usixelele- siza kuyenza kuGo.
Mholo ngokuhlwa nje! Enkosi ngengxelo! Asikwazanga ukufumana i-WAL-G isebenze nge-GPG decryption. Iguqulela ngokuntsonkothileyo ngokwesiqhelo, kodwa ayifuni kukucima ukuntsonkotha. Ngaba yinto engazange isebenze kuthi? Le meko iyadandathekisa.
Yenza umba kwi-GitHub kwaye masiyibone.
Oko kukuthi, awuzange udibane nale nto?
Kukho inqaku lengxelo yempazamo xa i-WAL-G ingaqondi ukuba luhlobo luni lwefayile, ibuza: "Mhlawumbi ifihliwe?" Mhlawumbi ingxaki ayilulo uguqulelo oluntsonkothileyo konke konke. Ndifuna ukuphucula ukugawulwa kwemithi kwesi sihloko. Umele ayicacise. Ngoku sisebenza kwesi sihloko ngengqiqo yokuba asiyithandi ncam indlela inkqubo yokufumana izitshixo zikawonke-wonke nezabucala ezicwangciswe ngayo. Kuba siyibiza iGPG yangaphandle ukuze isinike izitshixo zayo. Kwaye ke sithatha ezi zitshixo kwaye sizigqithisele kwi-GPG yangaphakathi, evulekileyo ye-PGP, ehlanganiswe kuthi ngaphakathi kwe-WAL-G, kwaye apho siyibiza i-encryption. Kule nkalo, sifuna ukuphucula inkqubo kwaye sifuna ukuxhasa i-encryption ye-Libsodium (Eyongeziweyo kwi-version 0.2.15). Ewe kunjalo, ukwenza iikhowudi kufuneka kusebenze, masiyiqonde - udinga uphawu oluninzi kunamagama ambalwa. Unokuhlanganisana kwigumbi lesithethi ngelinye ixesha kwaye ujonge inkqubo. (Uguqulelo oluntsonkothileyo lwePGP ngaphandle kweGPG yangaphandle - v0.2.9)
Mholo! Enkosi ngengxelo! Ndinemibuzo emibini. Ndinomnqweno ongaqhelekanga wokwenza i-pg_basebackup kunye ne-WAL ungene kubaboneleli ababini, oko kukuthi, ndifuna ukwenza ilifu elinye kunye nelinye. Ngaba ikho indlela yokwenza oku?
Oku akukho ngoku, kodwa ngumbono onomdla.
Andimthembi umnikezeli omnye, ndifuna ukuba okufanayo kwenye, ukuba kunokwenzeka.
Ingcamango inomdla. Ngokobuchwephesha, oku akunzima kwaphela ukukuphumeza. Ukuthintela umbono ukuba ungalahleki, ndingakucela ukuba wenze umcimbi kwiGitHub?
Ewe kunjalo.
Kwaye ke, xa abafundi beza kwiHlobo likaGoogle leKhowudi, siya kubongeza kwiprojekthi ukuze kubekho umsebenzi omninzi wokufumana okungakumbi kubo.
Kwaye umbuzo wesibini. Kukho ingxaki kwi-GitHub. Ndicinga ukuba sele ivaliwe. Kukho ukuxhalaba ngexesha lokubuyisela. Kwaye ukuyoyisa, wenze indibano eyahlukileyo. Ilungile kwimiba. Kwaye kukho ukhetho ukwenza imeko-bume eguquguqukayo kumsonto omnye. Yiyo loo nto isebenza ngokucothayo. Kwaye sidibene nale ngxaki, kwaye ayikalungiswa okwangoku.
Ingxaki kukuba ngenxa yesizathu esithile ugcino (CEPH) luseta kwakhona uqhagamshelo xa sifika kuyo ngeconcurrency ephezulu. Yintoni enokwenziwa ngale nto? Ukuzama kwakhona ukucinga kujongeka ngolu hlobo. Sizama ukukhuphela ifayile kwakhona. Kwipasi elinye, sineefayile ezininzi ezingakhutshelwanga, siya kwenza eyesibini kubo bonke abo bangangenanga. Kwaye okoko nje ifayile enye ilayishiwe ngokuphindaphindiweyo, siyaphinda kwaye siphinde siphinde. Siyiphucule ingqiqo yokuzama kwakhona - umva we-exponential backoff. Kodwa ayicacanga ngokupheleleyo into emayiyenze ngenyani yokuba uqhagamshelo luqhekeka nje kwicala lenkqubo yokugcina. Oko kukuthi, xa silayisha kumlambo omnye, ayikophuli olu nxibelelwano. Yintoni esinokuyiphucula apha? Sine-network throttling, sinokukhawulela uxhulumaniso ngalunye ngenani leebhayithi eziyithumelayo. Ngaphandle koko, andiyazi indlela yokujongana nenyaniso yokuba ukugcinwa kwento akusivumeli ukuba sikhuphe okanye sikhuphe kuyo ngokuhambelanayo.
Akukho SLA? Ngaba ayibhalelwanga bona indlela abazivumela ngayo ukuba bathuthunjiswe?
Inqaku lelokuba abantu abeza nalo mbuzo badla ngokuba nesabo ivault. Oko kukuthi, akukho mntu uvela kwi-Amazon okanye kwi-Google Cloud okanye kwi-Yandex Object Storage.
Mhlawumbi umbuzo awusekho kuwe?
Umbuzo apha kule meko awukhathali kubani. Ukuba kukho naziphi na izimvo malunga nendlela yokujongana nale nto, masiyenze kwi-WAL-G. Kodwa ukuza kuthi ga ngoku andinazimvo zilungileyo malunga nendlela yokujongana nale nto. Kukho enye into yoGcino oluxhasa uluhlu lwee-backups ngokwahlukileyo. Ubacela ukuba badwelise izinto, kwaye bongeza ulawulo apho. U-WAL-G uyoyika oku - kukho into ethile apha engeyiyo ifayile, andinako ukuyibuyisela, okuthetha ukuba ugcino aluzange lubuyiselwe. Oko kukuthi, eneneni, uneqela elibuyiselwe ngokupheleleyo, kodwa likubuyisela ubume obunempazamo kuba uGcino Lwento lubuyise ulwazi olungaqhelekanga olungazange luluqonde ngokupheleleyo.
Le yinto eyenzekayo kwiMeyile ilifu.
Ukuba unokwakha ukuvelisa kwakhona...
Iveliswa rhoqo...
Ukuba kukho ukuveliswa kwakhona, ngoko ke ndicinga ukuba siza kuzama amaqhinga kwakhona kwaye sifumanise ukuba singazama njani kwakhona kwaye siqonde ukuba ilifu lifuna ntoni kuthi. Mhlawumbi iya kuzinza kuthi kwiidibaniso ezintathu kwaye ayiyi kuphula uxhulumaniso, ngoko siya kufikelela ngononophelo ezintathu. Ngenxa yokuba ngoku silahla uxhulumaniso ngokukhawuleza, oko kukuthi, ukuba siqalise ukubuyiswa kunye neentambo ezili-16, emva kokuzama kwakhona kuya kubakho iintambo ezi-8, iintambo ezi-4, iintambo ezi-2 kunye nomnye. Kwaye ke iyakutsala ifayile ibe ngumsinga omnye. Ukuba kukho amaxabiso omlingo afana ne-7,5 imisonto yeyona ilungileyo yokumpompa, siya kuhlala kuzo kwaye sizame ukwenza enye imisonto eyi-7,5. Nantsi imbono.
Enkosi ngengxelo! Ingaba ukuhamba komsebenzi okupheleleyo kokusebenza nge-WAL-G kujongeka njani? Umzekelo, kwimeko yobudenge xa ββkungekho delta ngapha kwamaphepha. Kwaye sithatha kwaye sisuse i-backup yokuqala, emva koko sigcine i-shaft de sibe luhlaza okwesibhakabhaka ebusweni. Apha, njengoko ndiyiqonda, kukho ukuphazamiseka. Ngexesha elithile kufuneka wenze i-delta backup yamaphepha, oko kukuthi inkqubo yangaphandle iqhuba oku okanye kwenzeka njani oku?
I-API yogcino lwe-delta ilula kakhulu. Kukho inani apho - amanyathelo aphezulu e-delta, yiloo nto ebizwa ngayo. Ihlala ibe nguziro. Oku kuthetha ukuba ngalo lonke ixesha usenza i-backup-push, ikhuphela i-backup epheleleyo. Ukuba uyitshintshela kulo naliphi na inani eliqinisekileyo, umzekelo, 3, ngoko kwixesha elizayo usenza i-backup-push, ijonga kwimbali yee-backups zangaphambili. Uyabona ukuba awugqithisi ikhonkco le-3 deltas kwaye wenze i-delta.
Oko kukuthi, ngalo lonke ixesha sisungula i-WAL-G, izama ukwenza ugcino olupheleleyo?
Hayi, siqhuba i-WAL-G, kwaye izama ukwenza i-delta ukuba imigaqo-nkqubo yakho iyayivumela.
Ukuthetha nje, ukuba uyiqhuba ngo-zero ngalo lonke ixesha, iyakuziphatha njenge pg_basebackup?
Hayi, iya kuqhubeka ibaleka ngokukhawuleza kuba isebenzisa ucinezelo kunye nongqamaniso. Pg_basebackup iya kubeka ishaft ecaleni kwakho. I-WAL-G ithatha ukuba ugcino olugciniweyo luqwalaselwe. Kwaye iya kukhupha isilumkiso ukuba ayilungiswanga.
Pg_basebackup inokuqhutywa ngaphandle kweeshafts.
Ewe, ngoko baya kuziphatha phantse ngokufanayo. Pg_basebackup iikopi kwisixokelelwano sefayile. Ngendlela, sinento entsha endilibele ukuyikhankanya. Ngoku sinokukhuphela kwisixokelelwano sefayile kwi-pg_basebackup. Andazi ukuba kutheni le nto ifunwa, kodwa ikhona.
Umzekelo, kwiCephFS. Ayinguye wonke umntu ofuna ukuqwalasela uGcino lweNto.
Ewe, yiloo nto kanye bebuza umbuzo malunga neli nqaku ukuze sikwazi ukulenza. Kwaye siyenzile.
Enkosi ngengxelo! Kukho nje umbuzo malunga nokukhuphela kwisixokelelwano sefayile. Ngaphandle kwebhokisi, ngaba ngoku uyakuxhasa ukukhuphela kwindawo yokugcina kude, umzekelo, ukuba kukho ishelufu kwiziko ledatha okanye enye into?
Kolu qulunqo, lo ngumbuzo onzima. Ewe, siyaxhasa, kodwa oku kusebenza akukabandakanywa nakuphi na ukukhutshwa okwangoku. Oko kukuthi, konke ukukhutshwa kwangaphambili kuyakuxhasa oku, kodwa iinguqulelo zokukhupha azikwenzi. Lo msebenzi wongezwa kwinguqulo 0.2. Ngokuqinisekileyo iya kukhutshwa kungekudala, ngokukhawuleza xa silungisa zonke iibhugi ezaziwayo. Kodwa ngoku oku kunokwenziwa kuphela ekukhutshweni kwangaphambili. Kukho iibhugi ezimbini kukhupho lwangaphambili. Ingxaki nge-WAL-E yokubuyisela, asikayilungisi. Kwaye kukhululo lwangaphambili lwamva nje ibug malunga ne-delta-backup yongezwa. Ke ngoko, sicebisa wonke umntu ukuba asebenzise iinguqulelo zokukhupha. Ngokukhawuleza ukuba akukho zibhugi kwi-pre-release, sinokuthi sixhasa i-Google Cloud, izinto ezihambelana ne-S3 kunye nokugcinwa kwefayile.
Molo, enkosi ngengxelo. Njengoko ndiyiqonda, i-WAL-G ayilohlobo oluthile lwenkqubo esembindini efana ne-barman? Ngaba uceba ukuhamba kweli cala?
Ingxaki kukuba siye sasuka kwelicala. U-WAL-G uhlala kwi-base host host, kwi-cluster host, nakuzo zonke iinginginya kwiqela. Xa sasifudukela kumawaka aliqela amaqela, saba neendawo ezininzi zokufakelwa kwebartender. Kwaye ngalo lonke ixesha into isiwa phakathi kwabo, yingxaki enkulu. Ngenxa yokuba kufuneka zilungiswe, kufuneka uqonde ukuba zeziphi na amaqela angenazo ii-backups. Andicwangcisi ukuphuhlisa i-WAL-G kwicala lezinto eziphathekayo kwiinkqubo zokulondoloza. Ukuba uluntu lufuna ukusebenza apha, andinangxaki kwaphela.
Sinamaqela anoxanduva lokugcina. Kwaye sivakalelwa kukuba akuthina, ukuba kukho abantu abakhethekileyo ababeka iifayile zethu apho iifayile zikhuselekile. Benza zonke iintlobo zeekhowudi zobuqili apho ukuze zimelane nokulahlekelwa kwenani elithile leefayile. Banoxanduva lwe-bandwidth yenethiwekhi. Xa une-bartender, unokufumanisa ngokukhawuleza ukuba i-database encinci ene-traffic eninzi iqokelele kwi-server efanayo. Ubonakala unendawo eninzi kuyo, kodwa ngesizathu esithile yonke into ayihambelani nenethiwekhi. Isenokujika ngenye indlela. Zininzi iinethiwekhi apho, kukho ii-processor cores, kodwa akukho diski apha. Kwaye siye sadinwa yile mfuneko yokudibanisa into ethile, kwaye safudukela kwinto yokuba ukugcinwa kwedatha yinkonzo eyahlukileyo, apho abantu abakhethekileyo abahlukeneyo banoxanduva.
P.S Kukhutshwe inguqulelo entsha , apho ungasebenzisa i .walg.json ifayile yoqwalaselo, ebekwe kulawulo lwasekhaya lwe-postgres ngokungagqibekanga. Uyakwazi ukulahla izikripthi ze-bash. Umzekelo .walg.json ukulo mba
Ividiyo:
umthombo: www.habr.com