KuCitymobil tinoshandisa dhatabhesi yeMySQL seyedu huru inoshingirira kuchengetedza data. Isu tine akati wandei database masumbu emasevhisi akasiyana uye zvinangwa.

Kuwanikwa nguva dzose kwetenzi chiratidzo chakakosha chekushanda kwehurongwa hwose uye zvikamu zvayo. Otomatiki cluster kudzoreredza muchiitiko chekutadza kwatenzi kunoderedza zvakanyanya nguva yekupindura chiitiko uye system downtime. Muchikamu chino, ini ndichatarisa kuwanikwa kwepamusoro (HA) dhizaini yeMySQL cluster yakavakirwa MySQL Orchestrator uye virtual IP kero (VIP).

HA mhinduro yakavakirwa paVIP

Kutanga, ini ndichakutaurira muchidimbu kuti yedu yekuchengetedza data chii.

Isu tinoshandisa kirasi yekudzokorora chirongwa neinyore-inopindika tenzi uye akawanda ekuverenga-chete replicas. Cluster inogona kunge ine tenzi wepakati - node iri zvese replica uye tenzi kune vamwe. Vatengi vanowana replicas kuburikidza neHAProxy, iyo inobvumira kugovera kunyange mutoro uye kuyera nyore. Kushandiswa kweHAProxy kunokonzerwa nezvikonzero zvekare, uye isu tiri mubishi rekutamira kuProxySQL.

Kudzokorora kunoitwa mune semi-synchronous mode yakavakirwa pa GTID. Izvi zvinoreva kuti kana imwe replica inofanira kunyora kutengeserana isati yaonekwa seyabudirira. Iyi yekudzokorodza modhi inopa yakaringana chiyero pakati pekuita uye kuchengetedza data muchiitiko chekutadza kwe master node. Chaizvoizvo shanduko dzese dzinotamiswa kubva kuna tenzi kuenda kune replicas uchishandisa Row Based Replication (RBR), asi mamwe mapfundo anogona kuva nawo mixed binlog format.

Iyo orchestrator nguva nenguva inogadzirisa mamiriro eiyo cluster topology, inoongorora ruzivo rwakagamuchirwa, uye kana matambudziko amuka, inogona kuvhura otomatiki kudzoreredza maitiro. Mugadziri ane basa rekuita iyo pachayo, nekuti inogona kuitwa nenzira dzakasiyana: zvichibva paVIP, DNS, uchishandisa masevhisi ekutsvaga masevhisi kana nzira dzekuzvinyora.

Imwe nzira iri nyore yekudzoreredza tenzi kana ikatadza ndeye kushandisa anoyangarara maVIP kero.

Zvaunoda kuziva nezvemhinduro iyi usati waenda kumberi:

• A VIP ikero yeIP iyo isingabatanidzwe neiyo chaiyo yemuviri network interface. Kana node ikatadza kana panguva yakarongwa kugadzirisa, tinogona kushandura VIP kune imwe sosi ine shoma yekudzikira.
• Kusunungura uye kupa chaiyo IP kero ibasa rakachipa uye rinokurumidza.
• Kuti ushande neVIP, unoda kuwana sevha kuburikidza neSSH, kana kushandiswa kwezvinhu zvakakosha, semuenzaniso, keepalived.

Ngatitarisei matambudziko anogona kuitika ne wizard yedu uye fungidzira kuti otomatiki yekudzoreredza michina inofanira kushanda sei.

Network yekubatanidza kuna tenzi yanyangarika, kana dambudziko ramuka padanho rehardware, uye sevha haisi kuwanikwa.

1. Iyo orchestrator inogadziridza cluster topology, imwe neimwe replica inoshuma kuti tenzi haasi kuwanikwa. Mutevedzeri anotanga maitiro ekusarudza mufananidzo wakakodzera basa reishe mutsva uye anotanga kupora.
2. Tiri kuedza kubvisa VIP kubva kuna tenzi wekare - pasina kubudirira.
3. Replica inochinja kuita basa ratenzi. Iyo topology iri kuvakwa patsva.
4. Kuwedzera itsva network interface neVIP. Sezvo zvaisaita kubvisa VIP, tinotanga nguva nenguva kutumira chikumbiro kumashure mahara ARP. Iyi mhando yekukumbira / mhinduro inobvumidza iwe kugadzirisa iyo IP uye MAC kero yemepu tafura pane yakabatana switch, nekudaro kukuzivisa iwe kuti VIP yedu yafamba. Izvi zvinoderedza mukana split brain pakudzoka tenzi mukuru.
5. Zvese zvinongedzo zvitsva zvinodzoserwa pakarepo kuna tenzi mutsva. Kubatana kwekare kunotadza uye kudzokororwa kufona kune dhatabhesi kunoitwa padanho rekushandisa.

Sevha iri kushanda nenzira yakajairika, kukundikana kwakaitika padanho reDBMS

Iyo algorithm yakafanana neyekare kesi: kuvandudza iyo topology uye kutanga maitiro ekudzoreredza. Sezvo sevha iripo, isu takabudirira kusunungura VIP pane yekare tenzi, toiendesa kune itsva, uye kutumira akati wandei ARP zvikumbiro. Iyo inogona kudzoka yatenzi wekare haifanire kukanganisa sumbu rakavakwazve uye kushanda kwechikumbiro.

Mamwe matambudziko

Kutadza kwe replicas kana masters epakati haitungamiri kuita otomatiki zviito uye zvinoda kupindira kwemaoko.

A virtual network interface inogara ichiwedzerwa kwechinguva, ndiko kuti, mushure mesevha reboot, VIP haina kupihwa otomatiki. Imwe neimwe dhatabhesi chiitiko chinotanga mukuverenga-chete modhi nekukasira, orchestrator inochinja otomatiki tenzi mutsva kunyora uye kuyedza kuisa. read only pana tenzi mukuru. Zviito izvi zvine chinangwa chekudzikisa mukana split brain.

Matambudziko anogona kumuka panguva yekudzoreredza maitiro, ayo anofanirwa kuziviswa kuburikidza ne orchestrator UI mukuwedzera kune akajairwa ekutarisa maturusi. Isu takawedzera iyo REST API nekuwedzera chinhu ichi (PR parizvino iri kuongororwa).

Iyo dhayagiramu yakawanda yeHA mhinduro inoratidzwa pazasi.

Kusarudza tenzi mutsva

The orchestrator akachenjera zvakakwana uye anoedza kusarudza mufananidzo wakakodzera kwazvo satenzi mutsva zvichienderana nemaitiro anotevera:

• mufananidzo unosara shure kwatenzi;
• MySQL vhezheni ye master uye replica;
• replication type (RBR, SBR kana yakasanganiswa);
• nzvimbo munzvimbo imwechete kana dzakasiyana data data;
• kuwanikwa errant GTID - kutengeserana kwakaitwa pane replica uye kwete pane tenzi;
• mitemo yekusarudza tsika inotorwawo.

Haisi yega cue inokodzera mumiriri we master. Semuenzaniso, replica inogona kushandiswa kuchengetedza data, kana sevha ine isina kusimba magadzirirwo ehardware. Orchestrator zvitsigiso Mitemo yemanyorero iyo iwe yaunogona kugadzirisa yako yemunhu sarudzo yesarudzo kubva kune vakawanda vanofarirwa kusateererwa.

Mhinduro uye nguva yekudzorera

Muchiitiko chechiitiko, zvakakosha kudzikisa kuderera kwehurongwa, saka ngatitarisei iyo MySQL maparamita anobata kusikwa uye kuvandudzwa kweiyo cluster topology neaimbi:

• slave_net_timeout - nhamba yemasekonzi panguva iyo replica inomirira data nyowani kana chiratidzo chekurova kwemoyo kuti chisvike kubva kuna tenzi kusati kwabatana kunozivikanwa sekunge kurasika uye kubatanidzwazve. Iyo yakaderera kukosha, nekukurumidza iyo replica inogona kuona kuti kutaurirana natenzi kwakaputsika. Isu tinoisa kukosha uku kumasekonzi mashanu.
• MASTER_CONNECT_RETRY - nhamba yemasekonzi pakati pekuedzazve kubatana. Muchiitiko chematambudziko etiweki, kukosha kwakaderera kweiyi parameter kunobvumira kukurumidza kubatanidza uye kudzivirira iyo cluster kudzoreredza maitiro kubva pakutanga. Nhamba inokurudzirwa i1 sekondi.
• MASTER_RETRY_COUNT - nhamba yakawanda yekuedza kubatanidza zvakare.
• MASTER_HEARTBEAT_PERIOD - nguva mumasekondi mushure mokunge tenzi atumira chiratidzo chemoyo. Defaults kusvika pahafu yeukoshi slave_net_timeout.

Orchestrator parameters:

• DelayMasterPromotionIfSQLThreadNotUpToDate - kana zvakaenzana true, ipapo iro basa ratenzi harizoshandiswe pane replica yemumiriri kudzamara replica's SQL thread yapedza zvese zvisina kushandiswa kutengeserana kubva kuRelay Log. Isu tinoshandisa iyi sarudzo kudzivirira kurasikirwa nekutengeserana kana vese vamiriri vevamiriri vasara kumashure.
• InstancePollSeconds - kuwanda kwekuvaka uye kugadzirisa iyo topology.
• RecoveryPollSeconds - kuwanda kwekuongorora kwetopology. Kana dambudziko rikaonekwa, topology recovery inotangwa. Izvi chinogara, yakaenzana nesekondi 1.

Imwe neimwe cluster node inovhoterwa ne orchestrator kamwe chete InstancePollSeconds seconds Kana dambudziko rikaonekwa, iyo cluster state inomanikidzwa updated, uye ipapo chisarudzo chekupedzisira chinoitwa kuita kupora. Nekuyedza neakasiyana dhatabhesi uye orchestrator paramita, isu takakwanisa kudzikisa mhinduro uye kudzoreredza nguva kusvika 30 masekonzi.

test bhenji

Takatanga kuyedza chirongwa cheHA nekuvandudzwa kwenzvimbo test bhenji uye kuenderera mberi kwekuita munzvimbo dzekuyedza nekugadzira. Iyo yenzvimbo inomira yakanyatso otomatiki yakavakirwa paDocker uye inokutendera kuti uedze nekumisikidzwa kweiyo orchestrator uye network, kuyera sumbu kubva 2-3 maseva kusvika gumi nemaviri, uye kuita maekisesaizi munzvimbo yakachengeteka.

Munguva yezviitwa, isu tinosarudza imwe yematambudziko emulation nzira: ipapo pfura tenzi uchishandisa kill -9, zvinyoro kumisa maitiro uye kumisa sevha (docker-compose stop), tevedzera matambudziko etiweki uchishandisa iptables -j REJECT kana iptables -j DROP. Tinotarisira zvinotevera zvinotevera:

• orchestrator ichaona matambudziko natenzi uye inovandudza topology pasina kupfuura 10 seconds;
• iyo nzira yekudzoreredza ichatanga otomatiki: iyo network yekumisikidza ichachinja, basa ratenzi richapfuura kune replica, iyo topology ichavakwazve;
• tenzi mutsva anozonyorwa, mhenyu replicas haizorasikirwe panguva yekuvakazve;
• data ichatanga kunyorwa kune tenzi mutsva uye kudzokororwa;
• Nguva yekudzosa yese haizopfuuri masekonzi makumi matatu.

Sezvaunoziva, iyo sisitimu inogona kuita zvakasiyana muyedzo uye yekugadzira nharaunda nekuda kweakasiyana Hardware uye network masisitimu, misiyano yekugadzira uye chaiyo mutoro, nezvimwe. Naizvozvo, isu nguva nenguva tinoita maekisesaizi mumamiriro ezvinhu chaiwo, tichitarisa kuti hurongwa hunoita sei kana network yekubatanidza yarasika kana kuti zvikamu zvayo zvega zvadzikiswa. Mune ramangwana, isu tinoda kuvaka chivakwa chakafanana chese nharaunda uye otomatiki kuyedzwa kwayo.

zvakawanikwa

Hutano hweiyo huru yekuchengetedza system node nderimwe remabasa makuru eSRE uye timu yekushanda. Kuitwa kweiyo orchestrator uye HA mhinduro yakavakirwa paVIP yakatibvumira kuwana zvinotevera mhedzisiro:

• kuona kwakavimbika kwezvinetso netopology yedatabase cluster;
• otomatiki uye nekukurumidza kupindura kune zviitiko zvine chekuita ne master, kuderedza system downtime.

Nekudaro, iyo mhinduro ine painogumira uye nekuipa kwayo:

• kuyera iyo HA chirongwa kune akati wandei data data kunoda imwechete L2 network pakati pavo;
• Tisati tapa VIP pane tenzi mutsva, tinoda kuisunungura pane yekare. Iyo nzira inoteedzana, iyo inowedzera kupora nguva;
• kuburitsa iyo VIP kunoda SSH kuwana kune sevha, kana chero imwe nzira yekudaidza kure kure maitiro. Sezvo sevha kana dhatabhesi iri kusangana nematambudziko akakonzeresa maitiro ekudzoreredza, hatigone kuve nechokwadi chekuti kubviswa kweVIP kuchapedza zvinobudirira. Uye izvi zvinogona kutungamira pakuonekwa kwemaseva maviri ane yakafanana virtual IP kero uye dambudziko split brain.

Kunzvenga split brain, unogona kushandisa nzira STONITH ("Pfura Iyo Imwe Node Mumusoro"), iyo inoparadzanisa zvachose kana kudzima dambudziko node. Kune dzimwe nzira dzekushandisa cluster yakakwirira kuwanikwa: musanganiswa weVIP neDNS, kuwanikwa kwesevhisi uye masevhisi eproxy, kudzokorora kupindirana uye dzimwe nzira dzine zvadzinokanganisa uye zvakanakira.

Ndakataura nezve maitiro edu ekugadzira MySQL failover cluster. Zviri nyore kushandisa uye inopa chiyero chinogamuchirwa chekuvimbika pasi pemamiriro ezvinhu aripo. Sezvo hurongwa hwese huzhinji uye zvivakwa zvakanyanya zvichikura, nzira iyi pasina mubvunzo ichashanduka.

Source: www.habr.com