E-Citymobil sisebenzisa isizindalwazi se-MySQL njengokugcinwa kwethu kwedatha okuyinhloko eziphikelelayo. Sinamaqoqo amaningana esizindalwazi ngezinkonzo nezinjongo ezahlukene.

Ukutholakala njalo kwenkosi kuyinkomba ebalulekile yokusebenza kwalo lonke uhlelo kanye nezingxenye zalo ngazinye. Ukuthola kabusha kweqoqo okuzenzakalelayo uma kwenzeka ukwehluleka okuyinhloko kunciphisa kakhulu isikhathi sokuphendula kwesigameko kanye nokwehla kwesistimu. Kulesi sihloko, ngizobheka ukutholakala okuphezulu (HA) kwe-Cluster ye-MySQL esekelwe I-MySQL Orchestrator kanye namakheli e-IP (VIP).

Isixazululo se-HA esisekelwe ku-VIP

Okokuqala, ngizokutshela kafushane ukuthi iyini isistimu yethu yokugcina idatha.

Sisebenzisa isikimu sokuphindaphinda sakudala esinenkosi eyodwa efinyeleleka kalula kanye nezifaniso eziningi zokufunda kuphela. Iqoqo lingaqukatha okuyinhloko okumaphakathi - inodi eyikhophi kanye nenkosi kwabanye. Amaklayenti afinyelela izifaniso nge-HAProxy, evumela ukusatshalaliswa komthwalo ngisho nokukala kalula. Ukusetshenziswa kwe-HAProxy kungenxa yezizathu zomlando, futhi okwamanje siphezu kwenqubo yokuthuthela ku-ProxySQL.

Ukuphindaphinda kwenziwa ngemodi ye-semi-synchronous ngokusekelwe ku GTID. Lokhu kusho ukuthi okungenani isifaniso esisodwa kufanele sibhale umsebenzi ngaphambi kokuthi kuthathwe njengokuyimpumelelo. Le modi yokuphindaphinda inikeza ibhalansi elungile phakathi kokusebenza nokuphepha kwedatha uma kwenzeka ukwehluleka kwenodi eyinhloko. Ngokuyisisekelo zonke izinguquko zidluliswa zisuka ku-master ziye kuzifaniso kusetshenziswa Row Based Replication (RBR), kodwa amanye ama-node angase abe nawo mixed binlog format.

I-orchestrator ngezikhathi ezithile ibuyekeza isimo se-cluster topology, ihlaziya ulwazi olutholiwe, futhi uma izinkinga ziphakama, ingaqalisa inqubo yokutakula ngokuzenzakalelayo. Umthuthukisi unesibopho senqubo ngokwayo, njengoba ingasetshenziswa ngezindlela ezihlukene: ngokusekelwe ku-VIP, i-DNS, isebenzisa izinsizakalo zokuthola isevisi noma izindlela zokuzibhalela.

Enye indlela elula yokubuyisela i-master uma ihluleka ukusebenzisa amakheli e-VIP antantayo.

Odinga ukukwazi ngalesi sixazululo ngaphambi kokuqhubekela phambili:

• I-VIP ikheli le-IP elingahlobene nesixhumi esibonakalayo senethiwekhi ethile. Uma i-node ihluleka noma phakathi nokulungiswa okuhleliwe, singashintshela i-VIP kwesinye isisetshenziswa esinesikhathi esincane sokuphumula.
• Ukukhulula kanye nokukhipha ikheli le-IP elibonakalayo kuwumsebenzi oshibhile futhi osheshayo.
• Ukuze usebenze ne-VIP, udinga ukufinyelela kuseva nge-SSH, noma ukusetshenziswa kwezinsiza ezikhethekile, isibonelo, keepalived.

Ake sibheke izinkinga ezingaba khona ngewizadi yethu futhi sicabange ukuthi indlela yokutakula okuzenzakalelayo kufanele isebenze kanjani.

Uxhumano lwenethiwekhi ku-master lunyamalele, noma kuvele inkinga ezingeni le-hardware, futhi iseva ayitholakali.

1. I-orchestrator ibuyekeza i-cluster topology, isifaniso ngasinye sibika ukuthi okuyinhloko akatholakali. I-orchestrator iqala inqubo yokukhetha i-replica efanele indima yenkosi entsha futhi iqala ukululama.
2. Sizama ukususa i-VIP kunkosi endala - ngaphandle kwempumelelo.
3. Isifaniso sishintshela endimeni yokuba umpetha. I-topology iyakhiwa kabusha.
4. Yengeza inethiwekhi entsha yokuxhumana nge-VIP. Njengoba kungenzeki ukususa i-VIP, siqala ukuthumela isicelo ngemuva I-ARP yamahhala. Lolu hlobo lwesicelo/impendulo ikuvumela ukuthi ubuyekeze ithebula lemephu yekheli le-IP kanye ne-MAC kumaswishi axhunyiwe, ngaleyo ndlela ikwazise ukuthi i-VIP yethu isihambile. Lokhu kunciphisa amathuba split brain lapho ebuya inkosi endala.
5. Konke ukuxhumeka okusha kuqondiswa kabusha ngokushesha kumphathi omusha. Ukuxhumana okudala kwehluleka futhi izingcingo eziphindaphindiwe kusizindalwazi zenziwa ezingeni lesicelo.

Iseva isebenza ngemodi evamile, ukwehluleka kwenzeke ezingeni le-DBMS

I-algorithm ifana necala langaphambilini: ukuvuselela i-topology nokuqala inqubo yokutakula. Njengoba iseva itholakala, sikhulula ngempumelelo i-VIP kunkosi endala, siyidlulisele kwentsha, bese sithumela izicelo ezimbalwa ze-ARP. Ukubuyiselwa okungenzeka kwenkosi endala akumele kuthinte iqoqo elakhiwe kabusha kanye nokusebenza kohlelo lokusebenza.

Ezinye izinkinga

Ukuhluleka kwezifaniso noma izingcweti ezimaphakathi ayiholeli ukwenza izenzo ezizenzakalelayo futhi kudinga ukungenelela okwenziwa ngesandla.

I-interface yenethiwekhi ebonakalayo ihlale yengezwa okwesikhashana, okungukuthi, ngemva kokuqaliswa kabusha kweseva, i-VIP ayinikezwa ngokuzenzakalelayo. Isenzakalo ngasinye sesizindalwazi siqala ngemodi yokufunda kuphela ngokuzenzakalelayo, i-orchestrator ishintsha ngokuzenzakalelayo umphathi omusha ukuze abhale futhi azame ukuyifaka. read only kumphathi omdala. Lezi zenzo kuhloswe ngazo ukunciphisa amathuba split brain.

Izinkinga zingase ziphakame phakathi nenqubo yokutakula, okufanele futhi yaziswe nge-orchestrator UI ngaphezu kwamathuluzi okuqapha ajwayelekile. Sinwebe i-REST API ngokwengeza lesi sici (PR okwamanje esabuyekezwa).

Umdwebo ojwayelekile wesixazululo se-HA uvezwa ngezansi.

Ukukhetha inkosi entsha

I-orchestrator ihlakaniphe ngokwanele futhi izama ukukhetha ikhophi efaneleke kakhulu njengomphathi omusha ngokwemibandela elandelayo:

• i-replica isala ngemuva kwenkosi;
• Inguqulo ye-MySQL ye-master and replica;
• uhlobo lokuphindaphinda (RBR, SBR noma okuxubile);
• indawo endaweni efanayo noma izikhungo zedatha ezahlukene;
• ukutholakala errant GTID - ukuthengiselana okwenziwe kumfanekiso futhi akukho ku-master;
• kucatshangelwa nemithetho yokukhetha ngokwezifiso.

Akuwona wonke ama-cue ayikhandidethi elifanele le-master. Isibonelo, i-replica ingasetshenziswa ukwenza ikhophi yasenqolobaneni idatha, noma iseva inokucushwa okubuthakathaka kwehadiwe. I-Orchestrator isekela imithetho ongayenza ngayo ukuze wenze ngendlela oyifisayo ukukhetha kwakho kwekhandidethi kusuka kokuningi okuncanyelwayo kunokushaywa indiva.

Isikhathi sokuphendula nesokuthola kabusha

Esimeni sesigameko, kubalulekile ukunciphisa isikhathi sokuphumula sesistimu, ngakho-ke ake sicabangele amapharamitha we-MySQL athinta ukudalwa nokubuyekezwa kwe-topology yeqoqo yi-orchestrator:

• slave_net_timeout — inani lamasekhondi lapho i-replica ilinda idatha entsha noma isignali yokushaya kwenhliziyo ukuthi ifike ivela kumphathi ngaphambi kokuthi uxhumo lubonakale njengolulahlekile futhi luxhunywe kabusha. Lapho inani liphansi, i-replica inganquma ngokushesha ukuthi ukuxhumana nomphathi kuphukile. Simisa leli nani kumasekhondi angu-5.
• MASTER_CONNECT_RETRY - inani lamasekhondi phakathi kwemizamo yokuxhuma kabusha. Esimeni sezinkinga zenethiwekhi, inani eliphansi lale parameter lizovumela ukuxhuma kabusha okusheshayo futhi kuvimbele inqubo yokutakula iqoqo kusukela ekuqaleni. Inani elinconyiwe isekhondi elingu-1.
• MASTER_RETRY_COUNT - inombolo enkulu yemizamo yokuxhuma kabusha.
• MASTER_HEARTBEAT_PERIOD — isikhawu ngemizuzwana emva kwalokho inkosi ithumela uphawu lokushaya kwenhliziyo. Okuzenzakalelayo kuya kuhhafu yenani slave_net_timeout.

Amapharamitha we-Okhestra:

• DelayMasterPromotionIfSQLThreadNotUpToDate - uma ilingana true, lapho-ke indima eyinhloko ngeke isetshenziswe kusifaniso sekhandidethi kuze kube yilapho uchungechunge lwe-SQL lwe-replica seluqede konke ukuthengiselana okungasetshenzisiwe okuvela Kulogi Yokudlulisa. Sisebenzisa le nketho ukugwema ukulahlekelwa okwenziwayo uma zonke izifaniso zekhandidethi zisalela ngemuva.
• InstancePollSeconds - imvamisa yokwakha kanye nokuvuselela i-topology.
• RecoveryPollSeconds - imvamisa yokuhlaziywa kwe-topology. Uma inkinga itholwa, ukutholwa kwe-topology kuyaqalwa. Lokhu njalo, ilingana nesekhondi elingu-1.

I-cluster node ngayinye iphendulwa i-orchestrator kanye njalo InstancePollSeconds imizuzwana Uma inkinga itholwa, isimo se-cluster siyaphoqelelwa kuvuselelwe, bese kwenziwa isinqumo sokugcina sokwenza alulame. Ngokuhlola idatha egciniwe ehlukene namapharamitha we-orchestrator, sikwazile ukunciphisa impendulo nesikhathi sokutakula sibe amasekhondi angu-30.

Isitendi sokuhlola

Saqala ukuhlola uhlelo lwe-HA ngokuthuthukiswa kwendawo ibhentshi lokuhlola kanye nokuqaliswa okuqhubekayo ezindaweni zokuhlola nezokukhiqiza. Isitendi sasendaweni sizenzakalela ngokugcwele ngokusekelwe ku-Docker futhi sikuvumela ukuthi uzame ukucushwa kwe-orchestrator nenethiwekhi, ukale iqoqo lisuka kumaseva angu-2-3 liye kuma-dazeni ambalwa, futhi wenze izivivinyo endaweni ephephile.

Ngesikhathi sokuzivocavoca, sikhetha enye yezindlela zokulingisa inkinga: dubula inkosi ngokushesha usebenzisa kill -9, nqamula inqubo kancane bese umisa iseva (docker-compose stop), belingisa izinkinga zenethiwekhi usebenzisa iptables -j REJECT noma iptables -j DROP. Silindele imiphumela elandelayo:

• i-orchestrator izothola izinkinga nge-master futhi ibuyekeze i-topology ngemizuzwana engaphansi kwe-10;
• inqubo yokutakula izoqala ngokuzenzakalelayo: ukucushwa kwenethiwekhi kuzoshintsha, indima yenkosi izodlulela ku-replica, i-topology izokwakhiwa kabusha;
• inkosi entsha izobhaleka, amakhophi abukhoma ngeke alahleke ngesikhathi senqubo yokwakha kabusha;
• idatha izoqala ukubhalelwa umphathi omusha futhi iphindwe;
• Isikhathi sokutakula esiphelele ngeke sibe ngaphezu kwamasekhondi angu-30.

Njengoba wazi, uhlelo lungase luziphathe ngendlela ehlukile ezindaweni zokuhlola nezokukhiqiza ngenxa yezingxenyekazi zekhompuyutha ezihlukene kanye nokulungiselelwa kwenethiwekhi, umehluko kumthwalo wokwenziwa nowangempela, njll. Ngakho-ke, ngezikhathi ezithile senza izivivinyo ezimweni zangempela, sihlola ukuthi uhlelo luziphatha kanjani lapho ukuxhumeka kwenethiwekhi kulahleka noma izingxenye zayo ngazinye zonakaliswa. Ngokuzayo, sifuna ukwakha ingqalasizinda efana ngokuphelele yazo zombili izindawo futhi sizenzele ukuhlola kwayo.

okutholakele

Impilo yenodi yesistimu yokugcina ingenye yezinjongo eziyinhloko ze-SRE kanye nethimba lokusebenza. Ukuqaliswa kwe-orchestrator nesisombululo se-HA esisekelwe ku-VIP kusivumele ukuthi sizuze imiphumela elandelayo:

• ukutholwa okuthembekile kwezinkinga nge-topology yeqoqo lesizindalwazi;
• impendulo ezenzakalelayo futhi esheshayo ezehlakalweni ezihlobene nobuciko, ukunciphisa isikhathi sokuphumula kwesistimu.

Kodwa-ke, isixazululo sinokulinganiselwa kanye nokubi:

• ukukala uhlelo lwe-HA ezikhungweni zedatha eziningana kuzodinga inethiwekhi eyodwa ye-L2 phakathi kwabo;
• Ngaphambi kokwabela i-VIP kunkosi entsha, sidinga ukuyikhulula kwendala. Inqubo ilandelana, okwandisa isikhathi sokutakula;
• ukukhulula i-VIP kudinga ukufinyelela kwe-SSH kuseva, nanoma iyiphi enye indlela yokubiza izinqubo ezikude. Njengoba iseva noma isizindalwazi sihlangabezana nezinkinga ezibangele inqubo yokuthola kabusha, asikwazi ukuqiniseka ukuthi ukususwa kwe-VIP kuzoqeda ngempumelelo. Futhi lokhu kungaholela ekubukeni kwamaseva amabili anekheli elifanayo le-IP elibonakalayo kanye nenkinga split brain.

Ukuze ugweme split brain, ungasebenzisa indlela USTONITH (“Dubula Enye I-Node Ekhanda”), ehlukanisa ngokuphelele noma ekhubaza inodi yenkinga. Kunezinye izindlela zokuqalisa ukutholakala okuphezulu kweqoqo: inhlanganisela ye-VIP ne-DNS, ukutholwa kwesevisi kanye nezinsizakalo ze-proxy, ukuphindaphinda okuvumelanayo nezinye izindlela ezinobubi bazo kanye nezinzuzo zazo.

Ngikhulume ngendlela yethu yokwenza iqoqo le-MySQL le-failiver. Kulula ukuyisebenzisa futhi inikeza izinga elamukelekayo lokwethembeka ngaphansi kwezimo zamanje. Njengoba lonke uhlelo jikelele kanye nengqalasizinda ikakhulukazi ithuthuka, le ndlela ngokungangabazeki izothuthuka.

Source: www.habr.com