Ndikupangira kuti muwerenge zolembedwa za lipoti kuyambira koyambirira kwa 2019 ndi Andrey Borodin "Zosunga zobwezeretsera ndi WAL-G. Kodi mu 2019 ndi chiyani?"
Moni nonse! Dzina langa ndine Andrey Borodin. Ndine wopanga pa Yandex. Ndakhala ndi chidwi ndi PostgreSQL kuyambira 2016, nditatha kulankhula ndi okonza mapulogalamuwo ndipo adanena kuti chirichonse chiri chophweka - mutenge kachidindo kachitsime ndikuchimanga, ndipo zonse zidzayenda bwino. Ndipo kuyambira pamenepo sindingathe kuyima - ndimalemba mitundu yonse ya zinthu zosiyanasiyana.
Chimodzi mwazinthu zomwe ndikugwira ntchito ndikusunga zosunga zobwezeretsera. . Nthawi zambiri, ku Yandex takhala tikugwira ntchito zosunga zobwezeretsera mu PostgreSQL kwa nthawi yayitali kwambiri. Ndipo mutha kupeza pa intaneti mndandanda wa malipoti asanu ndi limodzi amomwe timapangira zosunga zobwezeretsera. Ndipo chaka chilichonse amasintha pang'ono, amakula pang'ono, ndipo amakhala odalirika.
Koma lero lipotilo silimangonena zomwe tachita, komanso momwe zilili zosavuta komanso zomwe zili. Ndi angati mwa inu omwe mwawonera kale malipoti anga okhudza WAL-G? Ndibwino kuti anthu ochepa sanawonere, chifukwa ndiyamba ndi chinthu chophweka.
Ngati mwadzidzidzi muli ndi gulu la PostgreSQL, ndipo ndikuganiza kuti aliyense ali nawo angapo, ndipo mwadzidzidzi palibe dongosolo losunga zobwezeretsera, ndiye kuti muyenera kupeza yosungirako S3 kapena Google Cloud yosungirako.
Mwachitsanzo, mutha kubwera kudzayimilira ndikutenga kachidindo kotsatsira Yandex Object Storage, yomwe ndi yogwirizana ndi S3.
Kenako pangani Chidebe. Ndi chidebe chodziwitsa zambiri.
Pangani wogwiritsa ntchito.
Pangani kiyi yofikira kwa wogwiritsa ntchito: aws-s3-key.
Tsitsani kutulutsidwa kwaposachedwa kwa WAL-G.
Kodi zomwe tatulutsa kale zikusiyana bwanji ndi zotulutsidwa? Nthawi zambiri ndimafunsidwa kuti nditulutse msanga. Ndipo ngati palibe cholakwika mu Baibulo kwa nthawi yokwanira, mwachitsanzo, mwezi, ndiye ndimamasula. Nayi kutulutsidwa kwa Novembala. Ndipo izi zikutanthauza kuti mwezi uliwonse tidapeza mtundu wina wa cholakwika, nthawi zambiri muzochita zosafunikira, koma sitinatulutsebe kumasulidwa. Baibulo lapitalo ndi November yekha. Palibe nsikidzi zomwe timazidziwa mmenemo, mwachitsanzo, nsikidzi zinawonjezeredwa pamene polojekiti ikupita.
Mukatsitsa WAL-G, mutha kuyendetsa lamulo losavuta la "zosunga zobwezeretsera", ndikudutsa pazosintha zachilengedwe. Ndipo ilumikizana ndi Object Storage ndikuwuzani zosunga zobwezeretsera zomwe muli nazo. Poyamba, ndithudi, simuyenera kukhala ndi zosunga zobwezeretsera. Mfundo ya slide iyi ndikuwonetsa kuti zonse ndi zophweka. Ili ndi lamulo la console lomwe limavomereza kusintha kwa chilengedwe ndikuchita ma subcommands.
Pambuyo pake, mutha kupanga zosunga zobwezeretsera zanu zoyamba. Nenani "zosunga zobwezeretsera" mu WAL-G ndipo tchulani mu WAL-G malo a pgdata a gulu lanu. Ndipo mwina, PostgreSQL idzakuwuzani, ngati mulibe makina osunga zobwezeretsera, muyenera kuyambitsa "archive-mode".
Izi zikutanthauza kuti muyenera kupita ku zoikamo ndi kuyatsa "archive_mode = on" ndikuwonjezera "archive_command", yomwe ilinso subcommand mu WAL-G. Koma pazifukwa zina anthu nthawi zambiri amagwiritsa ntchito zilembo za bar pamutuwu ndikuzikulunga mozungulira WAL-G. Chonde musachite izi. Gwiritsani ntchito zomwe zikupezeka mu WAL-G. Ngati mukusowa chinachake, lembani kwa . WAL-G amaganiza kuti ndi pulogalamu yokhayo yomwe imayenda mu archive_command.
Timagwiritsa ntchito WAL-G makamaka kupanga gulu la High Availability mu Yandex Database management.
Ndipo nthawi zambiri amagwiritsidwa ntchito mu topology ya Master imodzi ndi kubwereza zingapo. Nthawi yomweyo, imapanga zosunga zobwezeretsera mu Yandex Object Storage.
Zochitika zofala kwambiri ndikupanga makope a gulu pogwiritsa ntchito Point pakuchira nthawi. Koma mu nkhani iyi, ntchito ya dongosolo zosunga zobwezeretsera si zofunika kwambiri kwa ife. Timangofunika kukweza gulu latsopano kuchokera ku zosunga zobwezeretsera.
Nthawi zambiri, timafunikira machitidwe osunga zobwezeretsera powonjezera node yatsopano. Nβchifukwa chiyani kuli kofunikira? Nthawi zambiri anthu amawonjezera node yatsopano kumagulu chifukwa gulu lomwe lilipo silingathe kuwerengera. Ayenera kuwonjezera chofananira chatsopano. Ngati tiwonjezera katundu kuchokera pg_basebackup kupita kwa Master, ndiye Master akhoza kugwa. Chifukwa chake, zinali zofunika kwambiri kwa ife kuti titha kukweza mwachangu node yatsopano kuchokera kumalo osungira, kupanga katundu wocheperako pa Master.
Ndipo mkhalidwe wina wofananawo. Uku ndiye kufunikira koyambitsanso Master wakale mutasintha Cluster Master kuchokera ku Data Center yomwe kulumikizana kudatayika.
- Zotsatira zake, popanga zofunikira pamakina okopera, tidazindikira kuti pg_basebackup siyoyenera kwa ife tikamagwira ntchito mumtambo.
- Tinkafuna kuti tithe kufinya deta yathu. Koma pafupifupi dongosolo lililonse losunga zobwezeretsera kupatula zomwe zimabwera m'bokosilo limapereka kupsinjika kwa data.
- Tinkafuna kufananiza chilichonse chifukwa wogwiritsa ntchito pamtambo amagula ma processor cores ambiri. Koma ngati tilibe kufanana mu ntchito ina, ndiye kuti ma cores ambiri amakhala opanda pake.
- Timafunika kubisa chifukwa nthawi zambiri deta si yathu ndipo singasungidwe m'mawu omveka bwino. Mwa njira, chothandizira chathu ku WAL-G chinayamba ndi kubisa. Tidamaliza kubisa mu WAL-G, pambuyo pake tidafunsidwa kuti: "Mwina mwa ife apanga ntchitoyi?" Ndipo kuyambira pamenepo ndakhala ndikugwira ntchito ndi WAL-G kwa nthawi yopitilira chaka.
- Tinkafunikanso kuwongolera zinthu, chifukwa pakapita nthawi timagwiritsa ntchito mtambo, tidazindikira kuti nthawi zina anthu amakhala ndi katundu wofunikira usiku ndipo katunduyu sangasokonezedwe. Ichi ndichifukwa chake tawonjezera gwero lazinthu.
- Komanso ndandanda ndi kasamalidwe.
- Ndipo kutsimikizira.
Tinayang'ana zida zosiyanasiyana. Mwamwayi, tili ndi zosankha zazikulu mu PostgreSQL. Ndipo kulikonse tinali kuphonya chinachake, chinachake chaching'ono, china chaching'ono.
Ndipo titayang'ana machitidwe omwe alipo, tidazindikira kuti tipanga WAL-G. Inali ntchito yatsopano panthawiyo. Zinali zophweka kukopa chitukuko cha makina osungira mitambo.
Lingaliro lalikulu lomwe timatsatira ndikuti WAL-G iyenera kukhala yophweka ngati balalaika.
WAL-G ili ndi malamulo 4. Izi:
WAL-PUSH - sungani shaft.
WAL-FETCH - pezani shaft.
BACKUP-PUSH - sungani zosunga zobwezeretsera.
BACKUP-FETCH - pezani zosunga zobwezeretsera kuchokera ku zosunga zobwezeretsera.
M'malo mwake, WAL-G ilinso ndi kasamalidwe ka zosunga zobwezeretsera izi, mwachitsanzo, kulembetsa ndikuchotsa zolemba ndi zosunga zobwezeretsera m'mbiri zomwe sizikufunikanso pakadali pano.
Imodzi mwa ntchito zofunika kwa ife ndi ntchito yopanga delta makope.
Makope a Delta amatanthauza kuti sitipanga zosunga zobwezeretsera zonse za gulu lonse, koma masamba osinthidwa a mafayilo osinthidwa mgululi. Zikuwoneka kuti mwachidwi izi ndizofanana kwambiri ndi kuthekera kochira pogwiritsa ntchito WAL. Koma titha kuyika zosunga zobwezeretsera za WAL zamtundu umodzi mofananira. Chifukwa chake, tikakhala ndi zosunga zobwezeretsera zomwe zidapangidwa Loweruka, zosunga zobwezeretsera za delta tsiku lililonse, ndipo Lachinayi tilephera, ndiye kuti tifunika kukweza ma backups 4 a delta ndi maola 10 a WAL. Zidzatenga nthawi yomweyo chifukwa zosunga zobwezeretsera za delta zimayendera limodzi.
LSN-based deltas - izi zikutanthauza kuti popanga zosunga zobwezeretsera, tidzafunika kuphatikiza tsamba lililonse ndikuyang'ana LSN yake ndi LSN ya zosunga zobwezeretsera zam'mbuyomu kuti timvetsetse kuti zasintha. Tsamba lililonse lomwe lingakhale ndi data yosinthidwa liyenera kupezeka muzosunga zobwezeretsera za delta.
Monga ndidanenera, chidwi chochuluka chidaperekedwa ku kufanana.
Koma API yosungidwa ku PostgreSQL ndiyofanana. PostgreSQL imasunga fayilo imodzi ya WAL ndipo ikabwezeretsa imapempha fayilo imodzi ya WAL. Koma pamene nkhokwe ikupempha fayilo imodzi ya WAL pogwiritsa ntchito lamulo la "WAL-FETCH", timatcha lamulo la "WAL-PREFETCH", lomwe limakonzekera mafayilo 8 otsatirawa kuti atenge deta kuchokera ku sitolo ya chinthu mofanana.
Ndipo pamene nkhokwe imatifunsa kuti tisunge fayilo imodzi, timayang'ana archive_status ndikuwona ngati pali mafayilo ena a WAL. Ndipo tikuyeseranso kutsitsa WAL mofananira. Izi zimapereka phindu lalikulu la ntchito ndipo zimachepetsa kwambiri mtunda wa ma WAL osasungidwa. Opanga makina ambiri osunga zobwezeretsera amakhulupirira kuti iyi ndi njira yowopsa chifukwa timadalira chidziwitso chathu cha omwe ali mkati mwa code omwe si PostgreSQL API. PostgreSQL sizikutsimikizira kupezeka kwa archive_status foda kwa ife ndipo sizikutsimikizira semantics, kukhalapo kwa zizindikiro zokonzekera mafayilo a WAL kumeneko. Komabe, tikuphunzira ma code source, tikuwona kuti izi ndi choncho ndipo tikuyesera kuzigwiritsa ntchito. Ndipo timawongolera komwe PostgreSQL ikupanga; ngati mwadzidzidzi makinawa athyoka, tidzasiya kugwiritsa ntchito.
Mu mawonekedwe ake oyera, LSN-based WAL delta imafuna kuwerenga fayilo iliyonse yamagulu yomwe nthawi yake mumafayilo yasintha kuyambira zosunga zakale. Tinakhala ndi izi kwa nthawi yaitali, pafupifupi chaka. Ndipo pamapeto tidafika pomaliza kuti tili ndi ma WAL deltas.
Izi zikutanthauza kuti nthawi iliyonse yomwe timasunga WAL pa Master, sitimangoyikakamiza, kuilemba ndikuitumiza ku netiweki, koma timawerenganso nthawi yomweyo. Timasanthula ndi kuwerenga zolemba zomwe zili mmenemo. Timamvetsetsa ndi ma block omwe asintha ndikusonkhanitsa mafayilo a delta.
Fayilo ya delta imafotokoza zamitundu ina ya mafayilo a WAL, imafotokoza zambiri zamabuloko omwe adasinthidwa mu WAL iyi. Ndiyeno mafayilo a delta awa amasungidwanso.
Apa tikukumana ndi mfundo yakuti tinafananiza chirichonse mofulumira, koma sitingathe kuwerenga mbiri yotsatizana mofanana, chifukwa mu gawo lina tikhoza kukumana ndi mapeto a mbiri ya WAL yapitayi, yomwe tilibe kanthu kugwirizana nayo, chifukwa. Kuwerenga kofanana kunapangitsa kuti tiyambe kusanthula zam'tsogolo, zomwe sizikhala ndi zakale.
Chotsatira chake, tinayenera kuyika zidutswa zosamvetsetseka mu _delta_partial owona. Zotsatira zake, tikabwerera ku zakale, tidzamata zidutswa za mbiri ya WAL kukhala imodzi, pambuyo pake tidzayigawa ndikumvetsetsa zomwe zidasintha mmenemo.
Ngati m'mbiri ya shaft yathu parsing pali mfundo imodzi yomwe sitikumvetsetsa zomwe zikuchitika, ndiye, potsatira zosunga zobwezeretsera tidzakakamizika kuwerenganso gulu lonse, monga momwe timachitira ndi LSN wamba. -kuchokera ku delta.
Zotsatira zake, kuvutika kwathu konse kudapangitsa kuti titsegule laibulale ya WAL-G yogawa. Momwe ndikudziwira, palibe amene akuigwiritsa ntchito panobe, koma ngati wina akufuna, kulemba ndikuigwiritsa ntchito, ili pagulu. (Ulalo wosinthidwa )
Chifukwa chake, mayendedwe a chidziwitso chonse amawoneka ovuta kwambiri. Master athu amasunga ma shaft ndikusunga mafayilo a delta. Ndipo chofanizira chomwe chimapanga zosunga zobwezeretsera chiyenera kulandira mafayilo a delta panthawi yomwe yadutsa pakati pa zosunga zobwezeretsera. Pamenepa, mbali za mbiriyakale zidzafunika kuwonjezeredwa mochuluka ndi kugawa, chifukwa si mbiri yonse yomwe imalowa m'magawo akuluakulu. Ndipo zitangochitika izi chofananacho chimatha kusungitsa zosunga zobwezeretsera zonse za delta.
Pazithunzi zonse zimawoneka zosavuta kwambiri. Uku ndikutsitsa kuchokera kugulu lathu lenileni. Tili ndi LSN-based, yopangidwa tsiku limodzi. Ndipo tikuwona kuti zosunga zobwezeretsera za LSN zochokera ku delta zinali kuyambira 20 koloko m'mawa mpaka XNUMX koloko m'mawa. Uku ndiye kuchuluka kwa ma processor cores. WAL-delta inatitengera ife pafupi maminiti a XNUMX. Ndiko kuti, idakhala mofulumira kwambiri, koma panthawi imodzimodziyo panali kusinthanitsa kwakukulu pa intaneti.
Popeza tili ndi chidziwitso chomwe midadada idasinthidwa komanso nthawi yanji m'mbiri ya nkhokwe, tidapitilira ndikusankha kuphatikiza magwiridwe antchito - kukulitsa kwa PostgreSQL kotchedwa "pg_prefaulter"
Izi zikutanthauza kuti choyimira-choyimira chikapereka lamulo lobwezeretsa, chimauza WAL-G kuti atenge fayilo yotsatira ya WAL. Timamvetsetsa kuti ndi data iti yomwe imalepheretsa njira yobwezeretsa ya WAL posachedwa ndikuyamba ntchito yowerengera pama block awa. Izi zidachitika kuti muwonjezere magwiridwe antchito a owongolera a SSD. Chifukwa WAL roll idzafika patsamba lomwe likufunika kusinthidwa. Tsambali lili pa disk ndipo mulibe posungira masamba. Ndipo adikira synchronously kuti tsamba ili lifike. Koma pafupi ndi WAL-G, yomwe ikudziwa kuti m'ma megabytes mazana angapo otsatirawa a WAL tidzafunika masamba ena ndipo nthawi yomweyo amayamba kutenthetsa. Imayambitsa ma disk angapo kuti azitha kuchitidwa mofanana. Izi zimagwira ntchito bwino pama drive a SSD, koma, mwatsoka, sizigwira ntchito pa hard drive, chifukwa timangosokoneza ndi zomwe tikufuna.
Izi ndi zomwe zili mu code tsopano.
Pali zinthu zomwe tikufuna kuwonjezera.
Chithunzichi chikuwonetsa kuti WAL-delta imatenga nthawi yochepa. Ndipo uku ndikuwerenga zosintha zomwe zidachitika mu database masana. Titha kuchita WAL-delta osati usiku wokha, chifukwa sichikhalanso gwero lalikulu la katundu. Titha kuwerenga WAL-delta mphindi iliyonse chifukwa ndiyotsika mtengo. Mu miniti imodzi tikhoza kuyang'ana zosintha zonse zomwe zachitika pamagulu. Ndipo izi zitha kutchedwa "instant WAL-delta".
Mfundo ndi yakuti pamene tibwezeretsa masango, timachepetsa chiwerengero cha nkhani zomwe tiyenera kuzilemba motsatizana. Ndiko kuti, kuchuluka kwa WAL komwe PostgreSQL imagudubuza iyenera kuchepetsedwa, chifukwa zimatenga nthawi yofunikira.
Koma si zokhazo. Ngati tidziwa kuti chipika china chidzasinthidwa mpaka kusasinthika kwa zosunga zobwezeretsera, sitingasinthe m'mbuyomu. Ndiye kuti, tsopano tili ndi kukhathamiritsa kwa fayilo ndi fayilo ya WAL-delta kutumiza. Izi zikutanthauza kuti ngati, mwachitsanzo, Lachiwiri tebulo lidachotsedwa kwathunthu kapena mafayilo ena adachotsedwa patebulo, ndiye kuti delta ikadutsa Lolemba ndi Loweruka pg_basebackup ibwezeretsedwa, sitipanganso izi.
Tikufuna kukulitsa lusoli mpaka patsamba. Ndiko kuti, ngati gawo lina la fayilo likusintha Lolemba, koma lidzalembedwa Lachitatu, ndiye pobwezeretsanso Lachinayi, sitiyenera kulemba masamba oyambirira a masamba ku disk.
Koma ili ndi lingaliro lomwe likukambidwa mwachangu mkati mwathu, koma silinafike pa code.
Tikufuna kupanga gawo lina mu WAL-G. Tikufuna kuti izi ziwonjezeke chifukwa tikufunika kuthandizira ma database osiyanasiyana ndipo tikufuna kuti titha kulumikizana ndi oyang'anira zosunga zobwezeretsera mwanjira yomweyo. Koma vuto ndiloti MySQL APIs ndi yosiyana kwambiri. Mu MySQL, PITR sikutengera zolemba za WAL, koma pa binlog. Ndipo tilibe makina osungira mu MySQL omwe angauze machitidwe ena akunja kuti binlog iyi yatha ndipo ikufunika kusungidwa. Tiyenera kuima penapake mu cron ndi database ndikuyang'ana ngati pali chinachake chokonzeka?
Ndipo momwemonso, pakubwezeretsa kwa MySQL, palibe lamulo lobwezeretsa lomwe lingauze dongosolo kuti ndikufunika mafayilo otere. Musanayambe kumanganso gulu lanu, muyenera kudziwa mafayilo omwe mukufuna. Inu nokha muyenera kulingalira mafayilo omwe mungafune. Koma mavutowa akhoza kuthetsedwa mwanjira ina. (Kufotokozera: MySQL yathandizidwa kale)
Mu lipotilo, ndidafunanso kulankhula za milanduyi pomwe WAL-G siyiyenera kwa inu.
Ngati mulibe chofananira chofananira, WAL-G sichitsimikizira kuti gawo lomaliza lidzasungidwa. Ndipo ngati kusungitsa zakale kutsalira kumbuyo kwa magawo angapo omaliza a mbiri yakale, ndiye ngozi. Ngati palibe chofananira chofananira, sindingalimbikitse kugwiritsa ntchito WAL-G. Komabe, zimapangidwira kuyika kwamtambo, zomwe zikutanthauza yankho la Kupezeka Kwapamwamba ndi chofanana ndi synchronous replica, chomwe chimayang'anira chitetezo cha ma byte omaliza.
Nthawi zambiri ndimawona anthu akuyesera kuyendetsa WAL-G ndi WAL-E nthawi imodzi. Timathandizira kuyanjana chakumbuyo chifukwa WAL-G imatha kubwezeretsa fayilo kuchokera ku WAL-E ndikubwezeretsanso zosunga zobwezeretsera zopangidwa mu WAL-E. Koma popeza machitidwe onsewa amagwiritsa ntchito parallel wal-push, amayamba kuba mafayilo kuchokera kwa wina ndi mnzake. Tikayikonza mu WAL-G, ikhalabe mu WAL-E. Mu WAL-E, imayang'ana pa archive-status, imawona mafayilo omalizidwa ndikuwasunga, pamene machitidwe ena sangadziwe kuti fayilo ya WAL inalipo, chifukwa PostgreSQL sidzayesa kuisunga kachiwiri.
Tikonza chiyani pano kumbali ya WAL-G? Sitidziwitsa PostgreSQL kuti fayiloyi idasamutsidwa mofanana, ndipo PostgreSQL ikatipempha kuti tiyisunge, tidzadziwa kale kuti fayilo yotereyi yomwe ili ndi nthawiyi ndipo md5 iyi yasungidwa kale ndipo tidzangonena kuti PostgreSQL - Chabwino, zonse zakonzeka popanda kuchita chilichonse.
Koma vutoli silingathe kukhazikitsidwa kumbali ya WAL-E, kotero sikutheka kupanga lamulo losunga zakale lomwe lidzasunga fayilo mu WAL-G ndi WAL-E.
Kuphatikiza apo, pali zochitika zomwe WAL-G sizoyenera kwa inu tsopano, koma tidzazikonza.
Choyamba, pakadali pano tilibe zotsimikizira zosunga zobwezeretsera. Sitikhala ndi chitsimikizo panthawi yosunga zobwezeretsera kapena kuchira. Inde, izi zimayendetsedwa mumtambo. Koma izi zimakhazikitsidwa ndikungoyang'anatu, ndikungobwezeretsa masango. Ndikufuna kupereka izi kwa ogwiritsa ntchito. Koma potsimikizira, ndikuganiza kuti mu WAL-G kudzakhala kotheka kubwezeretsa tsango ndikuyambitsa, ndikuyesa mayeso a utsi: pg_dumpall to /dev/null ndi amcheck index verification.
Pakali pano mu WAL-G palibe njira yochepetsera zosunga zobwezeretsera kuchokera ku WAL. Ndiye kuti, timathandizira zenera lina. Mwachitsanzo, kusunga masiku asanu ndi awiri apitawo, kusunga zosunga zobwezeretsera khumi zomaliza, kusunga zosunga zitatu zomaliza. Nthawi zambiri anthu amabwera kudzanena kuti: "Tikufuna zosunga zobwezeretsera zomwe zidachitika pa Chaka Chatsopano ndipo tikufuna kuzisunga kosatha." WAL-G sakudziwa momwe angachitire izi. (Dziwani - Izi zakhazikitsidwa kale. Werengani zambiri - Chosunga-chizindikiro mu )
Ndipo tilibe macheke amasamba ndi kukhulupirika kwa magawo onse a shaft potsimikizira PITR.
Kuchokera pa izi zonse ndapanga pulojekiti ya Google Summer of Code. Ngati mumadziwa ophunzira anzeru omwe angafune kulemba china chake mu Go ndikupeza madola masauzande angapo kuchokera ku kampani imodzi yokhala ndi chilembo "G", ndiye alimbikitseni ntchito yathu kwa iwo. Ndikhala ngati mlangizi wa polojekitiyi, atha kuchita. Ngati palibe ophunzira, ndiye kuti nditenga ndikuzichita ndekha m'chilimwe.
Ndipo tili ndi mavuto enanso angβonoangβono ambiri amene pangβonopangβono tikuyesetsa kuwathetsa. Ndipo zinthu zina zodabwitsa zimachitika.
Mwachitsanzo, ngati mupatsa WAL-G zosunga zobwezeretsera zopanda kanthu, zimangogwa. Mwachitsanzo, mukamuuza kuti akufunika kusunga chikwatu chopanda kanthu. Fayilo ya pg_control sidzakhalapo. Ndipo adzaganiza kuti sakumvetsa kanthu. Mwachidziwitso, munkhaniyi muyenera kulemba uthenga wabwinobwino kwa wogwiritsa ntchito kuti amufotokozere momwe angagwiritsire ntchito chidacho. Koma ichi sichiri ngakhale mawonekedwe a mapulogalamu, koma mawonekedwe a chinenero chabwino, chomveka.
Sitikudziwa momwe tingasungire zosunga pa intaneti. Ngati database ikunama, sitingathe kuisunga. Koma zonse ndi zophweka apa. Timatcha ma backups ndi LSN pomwe idayamba. LSN ya maziko oyambira iyenera kuwerengedwa kuchokera pa fayilo yolamulira. Ndipo ichi ndi mawonekedwe osakwaniritsidwa. Makina ambiri osunga zobwezeretsera amatha kusunga nkhokwe yachinsinsi. Ndipo ndi yabwino.
Panopa sitingathe kuthana ndi kusowa kwa malo osungira bwino. Chifukwa nthawi zambiri timagwira ntchito ndi zosunga zobwezeretsera zazikulu kunyumba. Ndipo iwo sanabwere kwa izo. Koma ngati wina akufuna kupanga pulogalamu mu Go pompano, onjezani kuwongolera zolakwika zomwe zachitika mu chidebe. Ndidzayang'ana pempho lachikoka.
Ndipo chachikulu chomwe chimatidetsa nkhawa ndichakuti tikufuna mayeso ophatikiza ma docker ambiri momwe tingathere omwe amayang'ana zochitika zosiyanasiyana. Pakali pano tikungoyesa zochitika zoyambira. Pazochita zilizonse, koma tikufuna kuyang'ana kudzipereka ndikuchita zonse zomwe timathandizira. Makamaka, mwachitsanzo, tidzakhala ndi chithandizo chokwanira cha PostgreSQL 9.4-9.5. Timawathandiza chifukwa anthu ammudzi amathandizira PostgreSQL, koma sitiyang'ana kudzipereka kuti tiwonetsetse kuti zonse sizikusweka. Ndipo zikuwoneka kwa ine kuti ichi ndi chiwopsezo chachikulu.
Tili ndi WAL-G yomwe ikuyenda pamagulu opitilira chikwi mu kasamalidwe ka Yandex Database. Ndipo imathandizira ma terabytes mazana angapo tsiku lililonse.
Tili ndi zambiri za TODO mu code yathu. Ngati mukufuna kupanga pulogalamu, bwerani, tikuyembekezera zopempha, tikuyembekezera mafunso.
Mafunso anu
Madzulo abwino! Zikomo! Lingaliro langa ndikuti ngati mukugwiritsa ntchito WAL-delta, mwina mukudalira kwambiri zolemba zonse patsamba. Ndipo ngati ndi choncho, mudayesa mayeso? Mwawonetsa chithunzi chokongola. Kodi zimakhala zokongola kwambiri bwanji ngati FPW yazimitsidwa?
Zolemba zamasamba zonse zatithandizira, sitinayese kuziletsa. Ndiye kuti, ine, monga wopanga, sindinayesere kuzimitsa. Oyang'anira machitidwe omwe adafufuza mwina adafufuzapo nkhaniyi. Koma tikufuna FPW. Pafupifupi palibe amene amachiletsa, chifukwa mwina ndizosatheka kutenga zosunga zobwezeretsera kuchokera pachifaniziro.
Zikomo chifukwa cha lipoti! Ndili ndi mafunso awiri. Funso loyamba ndilakuti chidzachitika ndi ma tablespaces?
Tikuyembekezera kupempha kukoka. Ma database athu amakhala pa SSD ndi ma disks a NMVE ndipo sitifunikira izi. Sindinakonzekere kuwononga nthawi yofunikira pochita bwino. Ndikukhulupirira ndi mtima wonse kuti tikuchirikiza izi. Pali anthu omwe adachithandizira, koma adachithandizira m'njira yowakomera. Iwo anapanga mphanda, koma iwo samakoka zopempha. (Yowonjezedwa mu mtundu 0.2.13)
Ndipo funso lachiwiri. Mudanena koyambirira kuti WAL-G imaganiza kuti imagwira ntchito yokha ndipo palibe zokutira zomwe zimafunikira. Ndimagwiritsa ntchito zokutira ndekha. Chifukwa chiyani sayenera kugwiritsidwa ntchito?
Tikufuna kuti ikhale yosavuta ngati balalaika. Izi zikutanthauza kuti simukusowa chilichonse kupatula balalaika. Tikufuna kuti dongosolo likhale losavuta. Ngati muli ndi magwiridwe antchito omwe muyenera kuchita muzolemba, bwerani mudzatiuze - tizichita mu Go.
Madzulo abwino! Zikomo chifukwa cha lipoti! Sitinathe kuti WAL-G igwire ntchito ndi GPG decryption. Imabisa nthawi zonse, koma safuna kubisa. Kodi ndi chinthu chomwe sichinagwire ntchito kwa ife? Mkhalidwewu ndi wokhumudwitsa.
Pangani vuto pa GitHub ndipo tiyeni tiwone.
Ndiko kuti, simunakumanepo ndi izi?
Pali mbali ya lipoti lolakwika lomwe WAL-G ikapanda kumvetsetsa kuti ndi fayilo yamtundu wanji, imafunsa kuti: "Mwina idabisidwa?" Mwina vuto si kubisa konse. Ndikufuna kukonza zodula mitengo pamutuwu. Iye ayenera kuzimasulira izo. Pakali pano tikugwira ntchito pamutuwu m'lingaliro lakuti sitikukonda momwe dongosolo lopezera makiyi agulu ndi achinsinsi amapangidwira. Chifukwa timatcha GPG yakunja kotero kuti imatipatsa makiyi ake. Ndiyeno timatenga makiyi awa ndikuwasamutsira ku GPG yamkati, yomwe ili yotseguka PGP, yomwe imapangidwira ife mkati mwa WAL-G, ndipo kumeneko timatcha encryption. Pachifukwa ichi, tikufuna kukonza dongosolo ndipo tikufuna kuthandizira Libsodium encryption (Yowonjezera mu 0.2.15). Zachidziwikire, kujambula kuyenera kugwira ntchito, tiyeni tiwone - muyenera chizindikiro chochulukirapo kuposa mawu angapo. Mutha kusonkhana m'chipinda cha wokamba nkhani nthawi ina ndikuyang'ana dongosolo. (Kubisa kwa PGP popanda GPG yakunja - v0.2.9)
Moni! Zikomo chifukwa cha lipoti! Ndili ndi mafunso awiri. Ndili ndi chikhumbo chachilendo chochita pg_basebackup ndi WAL kulowa mu opereka awiri, mwachitsanzo, ndikufuna kuchita mtambo umodzi ndi wina. Kodi pali njira ina yochitira izi?
Izi palibe pano, koma ndi lingaliro losangalatsa.
Sindimangodalira wothandizira m'modzi, ndikufuna kukhala ndi zomwezo mwa wina, ngati zingachitike.
Lingaliroli ndi losangalatsa. Mwaukadaulo, izi sizili zovuta kukhazikitsa. Kuletsa lingalirolo kuti lisatayike, ndingakufunseni kuti mupange vuto pa GitHub?
Inde ndithudi.
Ndiyeno, ophunzira akabwera ku Google Summer of Code, tidzawawonjezera pulojekitiyi kuti pakhale ntchito yochulukirapo kuti apindule nawo.
Ndipo funso lachiwiri. Pali vuto pa GitHub. Ndikuganiza kuti yatsekedwa kale. Pali mantha panthawi yobwezeretsa. Ndipo kuti muligonjetse, munapanga msonkhano wosiyana. Ndi bwino mu nkhani. Ndipo pali mwayi wochita malo osinthika mu ulusi umodzi. Ndi chifukwa chake zimagwira ntchito pang'onopang'ono. Ndipo tinakumana ndi vuto ili, ndipo silinakonzedwebe.
Vuto ndiloti pazifukwa zina zosungirako (CEPH) zimakhazikitsanso kugwirizanako tikabwera ndi ndalama zambiri. Kodi chingachitike nβchiyani pa zimenezi? Njira yoyeseranso ikuwoneka motere. Tikuyesera kutsitsanso fayilo. Pakupita kumodzi, tinali ndi mafayilo angapo omwe sanatsitsidwe, tipanga yachiwiri kwa onse omwe sanalowemo. Ndipo bola ngati fayilo imodzi imakwezedwa pakubwereza, timabwereza ndikubwereza ndikubwereza. Tinawongolera malingaliro oyeseranso - exponential backoff. Koma sizodziwikiratu chochita ndi chakuti kulumikizana kumangosweka kumbali yosungirako. Ndiye kuti, tikayika pamtsinje umodzi, sizimaphwanya maulalo awa. Kodi tingawongolere chiyani apa? Tili ndi network throttling, titha kuchepetsa kulumikizana kulikonse ndi kuchuluka kwa ma byte omwe amatumiza. Kupanda kutero, sindikudziwa momwe ndingathanirane ndi mfundo yakuti kusungirako zinthu sikumatilola kutsitsa kapena kukopera kuchokera pamenepo mofanana.
Palibe SLA? Kodi sikunalembedwe kwa iwo momwe amalolera kuzunzidwa?
Mfundo ndi yakuti anthu omwe amabwera ndi funso ili nthawi zambiri amakhala ndi chipinda chawo. Ndiye kuti, palibe amene amachokera ku Amazon kapena Google Cloud kapena Yandex Object Storage.
Mwina funso silikhalanso kwa inu?
Funso pano pankhaniyi ilibe kanthu kwa ndani. Ngati pali malingaliro amomwe mungachitire ndi izi, tiyeni tichite mu WAL-G. Koma mpaka pano ndilibe malingaliro abwino a momwe ndingathanirane ndi izi. Pali Zina Zosungira Zomwe Zimathandizira Kusunga Zosunga Mosiyana. Muwafunsa kuti alembe zinthu, ndipo amawonjezera chikwatu pamenepo. WAL-G akuchita mantha ndi izi - pali mtundu wina wa chinthu pano chomwe si fayilo, sindingathe kuyibwezeretsa, zomwe zikutanthauza kuti zosunga zobwezeretsera sizinabwezeretsedwe. Izi zikutanthauza kuti, muli ndi gulu lobwezeretsedwa, koma limakubwezerani malo olakwika chifukwa Object Storage idabweza chidziwitso chachilendo chomwe sichinamvetsetse bwino.
Izi ndi zomwe zimachitika mumtambo wa Mail.
Ngati mutha kupanga chojambula ...
Amapangidwanso nthawi zonse...
Ngati pali kuberekana, ndiye ndikuganiza kuti tidzayesa njira zoyeseranso ndikuwona momwe tingayeserenso ndikumvetsetsa zomwe mtambo umafuna kwa ife. Mwinamwake izo zidzakhala zokhazikika kwa ife pazolumikizidwe zitatu ndipo sizidzaphwanya kugwirizana, ndiye ife tidzafika mosamala atatu. Chifukwa tsopano timagwetsa kulumikizana mwachangu kwambiri, i.e. ngati tidayambitsa kuchira ndi ulusi 16, ndiye kuti mutatha kuyesanso koyamba padzakhala ulusi 8, ulusi 4, ulusi 2 ndi imodzi. Ndiyeno izo kukoka wapamwamba mu mtsinje umodzi. Ngati pali zinthu zamatsenga monga ulusi 7,5 ndi zabwino kwambiri kupopera, ndiye tidzakhala pa iwo ndi kuyesa kupanga ulusi wina 7,5. Nali lingaliro.
Zikomo chifukwa cha lipoti! Kodi mayendedwe athunthu ogwirira ntchito ndi WAL-G amawoneka bwanji? Mwachitsanzo, mu nkhani yopusa pamene palibe delta kudutsa masamba. Ndipo timatenga ndikuchotsa zosunga zobwezeretsera zoyambira, ndikusunga kutsinde mpaka titakhala buluu kumaso. Apa, monga ndikumvetsetsa, pali kusokonekera. Panthawi ina muyenera kupanga zosunga zobwezeretsera zamasamba, mwachitsanzo, njira ina yakunja ikuyendetsa izi kapena izi zimachitika bwanji?
The delta backup API ndiyosavuta. Pali nambala pamenepo - max delta masitepe, ndizomwe zimatchedwa. Imasinthidwa kukhala zero. Izi zikutanthauza kuti nthawi iliyonse mukamasunga zosunga zobwezeretsera, imatsitsa zosunga zobwezeretsera zonse. Ngati musintha kukhala nambala iliyonse yabwino, mwachitsanzo, 3, ndiye nthawi ina mukadzapanga zosunga zobwezeretsera, imayang'ana mbiri yakale yosunga zobwezeretsera. Amawona kuti simudutsa unyolo wa 3 deltas ndikupanga delta.
Ndiye kuti, nthawi iliyonse tikayambitsa WAL-G, imayesa kupanga zosunga zobwezeretsera zonse?
Ayi, timayendetsa WAL-G, ndipo imayesa kupanga delta ngati ndondomeko zanu zilola.
Mwachidule, ngati mumayendetsa ndi zero nthawi iliyonse, kodi izikhala ngati pg_basebackup?
Ayi, idzathamangabe mwachangu chifukwa imagwiritsa ntchito kukanikiza ndi kufanana. Pg_basebackup idzayika shaft pafupi ndi inu. WAL-G akuganiza kuti mwakonza zosungira. Ndipo idzapereka chenjezo ngati silinakonzedwe.
Pg_basebackup imatha kuyendetsedwa popanda shafts.
Inde, ndiye adzachita pafupifupi mofanana. Pg_basebackups ku fayilo yamafayilo. Mwa njira, tili ndi chinthu chatsopano chomwe ndinayiwala kutchula. Tsopano titha kusungira ku fayilo kuchokera ku pg_basebackup. Sindikudziwa chifukwa chake izi zikufunika, koma zilipo.
Mwachitsanzo, pa CephFS. Sikuti aliyense akufuna kukhazikitsa Object Storage.
Inde, mwina nβchifukwa chake anafunsa funso lokhudza mbali imeneyi kuti tichite. Ndipo ife tinachita izo.
Zikomo chifukwa cha lipoti! Pali funso chabe lokhudza kukopera ku fayilo yamafayilo. Kuchokera m'bokosilo, kodi tsopano mukuthandizira kukopera kumalo osungirako akutali, mwachitsanzo, ngati pali alumali mu data center kapena china?
Pakukonza uku, ili ndi funso lovuta. Inde, timathandizira, koma izi sizinaphatikizidwebe pakumasulidwa kulikonse. Ndiye kuti, zotulutsidwa zonse zisanachitike zimathandizira izi, koma zomasulira sizimatero. Ntchitoyi idawonjezedwa mu mtundu 0.2. Idzatulutsidwa posachedwa, tikangokonza nsikidzi zonse zodziwika. Koma pakali pano izi zitha kuchitika potulutsidwa kale. Pali nsikidzi ziwiri mu pre-kumasulidwa. Vuto ndi kuchira kwa WAL-E, sitinakonze. Ndipo pakutulutsidwa kwaposachedwa kwambiri cholakwika chokhudza delta-backup chidawonjezedwa. Chifukwa chake, timalimbikitsa aliyense kuti agwiritse ntchito mitundu yotulutsidwa. Mwamsanga pamene palibenso nsikidzi pakumasulidwa kusanachitike, tinganene kuti timathandizira Google Cloud, zinthu zogwirizana ndi S3 ndi kusunga mafayilo.
Moni, zikomo chifukwa cha lipoti. Monga ndikumvetsetsa, WAL-G simtundu wamtundu wapakati ngati barmen? Kodi mukukonzekera kusunthira mbali iyi?
Vuto ndilakuti tachokapo. WAL-G amakhala pagulu loyambira, pagulu lamagulu, komanso pamagulu onse amgululi. Pamene tinasamukira kumagulu masauzande angapo, tinali ndi makhazikitsidwe ambiri a bartender. Ndipo nthawi iliyonse pamene chinachake chikugwa mwa iwo, ndi vuto lalikulu. Chifukwa amafunika kukonzedwa, muyenera kumvetsetsa kuti ndi magulu ati omwe alibe zosunga zobwezeretsera. Sindikukonzekera kupanga WAL-G molunjika kwa zida zakuthupi zamakina osunga zobwezeretsera. Ngati anthu ammudzi akufuna magwiridwe antchito pano, sindisamala konse.
Tili ndi magulu omwe ali ndi udindo wosunga. Ndipo timamva bwino kwambiri kuti si ife, kuti pali anthu apadera omwe amaika mafayilo athu kumene mafayilo ali otetezeka. Iwo amachita mitundu yonse yochenjera yokhotakhota kumeneko kuti athe kupirira kutayika kwa chiwerengero cha mafayilo. Iwo ali ndi udindo pa network bandwidth. Mukakhala ndi bartender, mutha kupeza mwadzidzidzi kuti ma database ang'onoang'ono okhala ndi magalimoto ambiri asonkhana pa seva yomweyo. Mukuwoneka kuti muli ndi malo ambiri, koma pazifukwa zina zonse sizikugwirizana ndi intaneti. Zitha kuchitika mwanjira ina mozungulira. Pali maukonde ambiri kumeneko, pali ma processor cores, koma palibe ma disks pano. Ndipo tidatopa ndi kufunikira kosintha zina, ndipo tinasamukira ku mfundo yakuti kusungirako deta ndi ntchito yosiyana, yomwe anthu apadera ali ndi udindo.
PS Mtundu watsopano watulutsidwa , momwe mungagwiritsire ntchito fayilo yokonzekera .walg.json, yomwe ili m'ndandanda wa kunyumba ya postgres mwachisawawa. Mutha kusiya zolemba za bash. Chitsanzo .walg.json ili m'nkhaniyi
Video:
Source: www.habr.com