Ku Citymobil timagwiritsa ntchito database ya MySQL ngati kusungirako kwathu kwakukulu kosalekeza. Tili ndi magulu angapo osungiramo ntchito zosiyanasiyana komanso zolinga.

Kupezeka kosalekeza kwa mbuye ndi chizindikiro chofunikira cha machitidwe a dongosolo lonse ndi zigawo zake. Kubwezeretsa masango modzidzimutsa pakalephera bwino kumachepetsa kwambiri nthawi yoyankha komanso kutsika kwadongosolo. M'nkhaniyi, ndiwona mapangidwe apamwamba (HA) a gulu la MySQL kutengera MySQL Orchestrator ndi ma adilesi enieni a IP (VIP).

HA yankho lochokera pa VIP

Choyamba, ndikuwuzani mwachidule zomwe dongosolo lathu losungiramo deta liri.

Timagwiritsa ntchito sikimu yachikale yobwereza yomwe ili ndi mbuye wina wosavuta kulemba komanso zolemba zingapo zowerengera zokha. Gulu litha kukhala ndi master wapakatikati - mfundo yomwe ili yofananira komanso master kwa ena. Makasitomala amapeza zofananira kudzera mu HAProxy, yomwe imalola kugawa ngakhale katundu komanso kukulitsa kosavuta. Kugwiritsiridwa ntchito kwa HAProxy ndi chifukwa cha zifukwa za mbiri yakale, ndipo panopa tikupita ku ProxySQL.

Kubwereza kumachitika mu semi-synchronous mode kutengera GTID. Izi zikutanthauza kuti chifaniziro chimodzi chiyenera kuyika malonda asanaonedwe kuti ndi opambana. Njira yobwereza iyi imapereka kulinganiza koyenera pakati pa magwiridwe antchito ndi chitetezo cha data pakagwa kulephera kwa node. Kwenikweni zosintha zonse zimasamutsidwa kuchokera ku master kupita ku replicas pogwiritsa ntchito Row Based Replication (RBR), koma mfundo zina zingakhale nazo mixed binlog format.

Woyimba nyimbo nthawi ndi nthawi amasintha mawonekedwe a cluster topology, amasanthula zomwe alandila, ndipo ngati pabuka mavuto, amatha kuyambitsa njira yobwezeretsa yokha. Wopanga mapulogalamuwa ali ndi udindo pa ndondomekoyi, chifukwa ikhoza kukhazikitsidwa m'njira zosiyanasiyana: kutengera VIP, DNS, pogwiritsa ntchito ntchito zodziwira ntchito kapena njira zolembera.

Njira imodzi yosavuta yobwezeretsanso mbuye ngati ikulephera ndikugwiritsa ntchito ma adilesi akuyandama a VIP.

Zomwe muyenera kudziwa za yankho ili musanapite patsogolo:

• VIP ndi adilesi ya IP yomwe simalumikizidwa ndi mawonekedwe enaake amtaneti. Ngati node ikulephera kapena panthawi yokonzekera, tikhoza kusintha VIP kuzinthu zina ndi nthawi yochepa.
• Kumasula ndi kupereka adilesi ya IP ndi ntchito yotsika mtengo komanso yachangu.
• Kuti mugwire ntchito ndi VIP, muyenera kupeza seva kudzera pa SSH, kapena kugwiritsa ntchito zida zapadera, mwachitsanzo, keepalived.

Tiyeni tiwone zovuta zomwe zingatheke ndi wizard yathu ndikulingalira momwe makina odziwira okha ayenera kugwirira ntchito.

Kulumikizana kwa netiweki kwa mbuye kwatha, kapena vuto labuka pamlingo wa hardware, ndipo seva sichikupezeka.

1. Oyimba amakonzanso cluster topology, chofananira chilichonse chimanena kuti mbuyeyo kulibe. Woyimba akuyamba njira yosankha choyimira choyenera pa gawo la mbuye watsopano ndikuyamba kuchira.
2. Tikuyesera kuchotsa VIP kwa mbuye wakale - popanda kupambana.
3. Chofananacho chimasintha kukhala master. Topology ikumangidwanso.
4. Kuwonjezera mawonekedwe atsopano a netiweki ndi VIP. Popeza sikunali kotheka kuchotsa VIP, timayamba nthawi ndi nthawi kutumiza pempho kumbuyo ARP kwaulere. Pempho/mayankhidwe amtunduwu amakulolani kuti musinthe tebulo la mapu a IP ndi adilesi ya MAC pama switch olumikizidwa, ndikukudziwitsani kuti VIP yathu yasuntha. Izi zimachepetsa mwayi split brain pobwerera mbuye wakale.
5. Malumikizidwe onse atsopano amatumizidwa nthawi yomweyo kwa mbuye watsopano. Malumikizidwe akale amalephera ndipo kuyimba mobwerezabwereza ku database kumapangidwa pamlingo wofunsira.

Seva ikugwira ntchito mwachizolowezi, kulephera kunachitika pamlingo wa DBMS

Algorithm ndi yofanana ndi yapitayi: kukonzanso topology ndikuyamba kuchira. Popeza seva ilipo, timamasula bwino VIP pa mbuye wakale, kutumiza kwa watsopano, ndikutumiza zopempha zingapo za ARP. Kubwerera kotheka kwa mbuye wakale sikuyenera kukhudza gulu lomangidwanso ndikugwiritsa ntchito ntchitoyo.

Mavuto ena

Kulephera kwa replicas kapena masters apakatikati samatsogolera kuchita zodziwikiratu ndipo kumafuna kuchitapo kanthu pamanja.

Mawonekedwe a netiweki nthawi zonse amawonjezedwa kwakanthawi, ndiye kuti, seva ikayambiranso, VIP siyimaperekedwa. Chiwonetsero chilichonse cha database chimayamba ndikungowerenga kokha mwachisawawa, woyimba nyimbo amangosintha mbuye watsopanoyo kuti alembe ndikuyesa kukhazikitsa. read only pa bwana wakale. Zochita izi ndi cholinga chochepetsa mwayi split brain.

Mavuto angabwere panthawi yobwezeretsa, yomwe iyeneranso kudziwitsidwa kudzera mu orchestrator UI kuwonjezera pa zida zowunikira. Takulitsa REST API powonjezera izi (PR pano tikuwunikidwa).

Chithunzi chachikulu cha yankho la HA chikuwonetsedwa pansipa.

Kusankha mbuye watsopano

Woimbayo ndi wanzeru mokwanira ndipo amayesa kusankha chofanizira choyenera kwambiri monga mbuye watsopano malinga ndi izi:

• chofananacho chimatsalira kumbuyo kwa mbuye;
• Mtundu wa MySQL wa master ndi replica;
• mtundu wobwereza (RBR, SBR kapena wosakanikirana);
• malo omwe ali m'malo amodzi kapena osiyana;
• kupezeka errant GTID - zochitika zomwe zidachitika pachifanizirocho ndipo sizili pa mbuye;
• malamulo osankhidwa mwachizolowezi amaganiziridwanso.

Osati cue iliyonse yomwe ili yoyenera kwa master. Mwachitsanzo, chofananira chingagwiritsidwe ntchito posunga zosunga zobwezeretsera, kapena seva ili ndi kasinthidwe ka Hardware. Oyimba zogwiriziza malamulo apamanja omwe mungasinthire makonda anu osankhidwa kuchokera kwa omwe amakonda kunyalanyazidwa.

Yankho ndi kuchira nthawi

Pakachitika chochitika, ndikofunikira kuchepetsa nthawi yochepetsera dongosolo, ndiye tiyeni tiganizire magawo a MySQL omwe amakhudza kupanga ndi kukonzanso kwa cluster topology ndi oimba:

• slave_net_timeout - chiwerengero cha masekondi pamene chofananacho chimadikirira deta yatsopano kapena chizindikiro cha kugunda kwa mtima kuti chifike kuchokera kwa mbuye chisanayambe kugwirizana kudziwika ngati kutayika ndi kulumikizidwanso. Kutsika kwamtengo wapatali, mofulumira chofananacho chingadziwe kuti kulankhulana ndi mbuyeyo kwasweka. Timayika mtengowu kukhala masekondi asanu.
• MASTER_CONNECT_RETRY - chiwerengero cha masekondi pakati pa kuyesa kulumikizanso. Pakachitika mavuto a netiweki, mtengo wotsika wa parameter iyi udzalola kulumikizanso mwachangu ndikuletsa njira yobwezeretsa masango kuti isayambike. Mtengo wovomerezeka ndi sekondi imodzi.
• MASTER_RETRY_COUNT - kuchuluka kwa kuchuluka kwa kuyesa kulumikizanso.
• MASTER_HEARTBEAT_PERIOD - nthawi mumasekondi pambuyo pake mbuyeyo amatumiza chizindikiro cha kugunda kwa mtima. Zosasintha mpaka theka la mtengowo slave_net_timeout.

Zoyimira za Orchestrator:

• DelayMasterPromotionIfSQLThreadNotUpToDate - ngati ofanana true, ndiye kuti gawo lalikulu silidzagwiritsidwa ntchito pa choyimira mpaka ulusi wa SQL wa replica utamaliza zonse zomwe sizinagwiritsidwe ntchito kuchokera pa Relay Log. Timagwiritsa ntchito njirayi kuti tipewe kutayika ngati zofananira zonse zatsalira m'mbuyo.
• InstancePollSeconds - pafupipafupi pomanga ndi kukonzanso topology.
• RecoveryPollSeconds - pafupipafupi kusanthula kwa topology. Ngati vuto lipezeka, kuchira kwa topology kumayambika. Izi mosalekeza, zofanana ndi sekondi imodzi.

Gulu lililonse lamagulu limasankhidwa ndi oimba kamodzi InstancePollSeconds masekondi Vuto likapezeka, gulu lamagulu limakakamizika zasinthidwa, ndiyeno chigamulo chomaliza chimapangidwa kuti achire. Poyesa magawo osiyanasiyana a database ndi orchestrator, tinatha kuchepetsa nthawi yoyankha ndi kuchira mpaka masekondi a 30.

benchi yoyesera

Tinayamba kuyesa ndondomeko ya HA ndi chitukuko cha m'deralo benchi yoyesera ndi kukhazikitsanso kwina m'malo oyesera ndi kupanga. Maimidwe akomweko amakhala okhazikika pa Docker ndipo amakulolani kuyesa kasinthidwe ka oimba ndi netiweki, sinthani gululo kuchokera pa maseva 2-3 mpaka khumi ndi awiri, ndikuchita masewera olimbitsa thupi pamalo otetezeka.

Pazochita zolimbitsa thupi, timasankha imodzi mwazovuta njira zotsatsira: nthawi yomweyo kuwombera mbuyeyo pogwiritsa ntchito kill -9, chetsani pang'onopang'ono ntchitoyi ndikuyimitsa seva (docker-compose stop), yerekezerani mavuto a netiweki pogwiritsa ntchito iptables -j REJECT kapena iptables -j DROP. Tikuyembekeza zotsatirazi:

• woyimba adzazindikira mavuto ndi mbuye ndikusintha topology pasanathe masekondi 10;
• njira yobwezeretsa idzangoyambira: kasinthidwe ka netiweki kadzasintha, gawo la mbuye lidzapita ku choyimira, topology idzamangidwanso;
• mbuye watsopanoyo azitha kulembedwa, zolemba zamoyo sizidzatayika panthawi yomanganso;
• deta idzayamba kulembedwa kwa mbuye watsopano ndikubwereza;
• Nthawi yonse yochira idzakhala yosaposa masekondi 30.

Monga mukudziwira, dongosololi likhoza kuchita mosiyana m'malo oyesera ndi kupanga chifukwa cha hardware zosiyanasiyana ndi masanjidwe a maukonde, kusiyana kwa katundu ndi katundu weniweni, etc. Chifukwa chake, nthawi ndi nthawi timachita masewera olimbitsa thupi munthawi yeniyeni, kuyang'ana momwe dongosolo limakhalira pomwe kulumikizidwa kwa netiweki kutayika kapena magawo ake akuwonongeka. M'tsogolomu, tikufuna kupanga maziko ofanana m'malo onse awiri ndikuyesa kuyesa kwake.

anapezazo

Thanzi la node yayikulu yosungiramo zinthu ndi imodzi mwantchito zazikulu za SRE ndi gulu lantchito. Kukhazikitsidwa kwa oimba ndi yankho la HA kutengera VIP kunatilola kukwaniritsa zotsatirazi:

• kuzindikira kodalirika kwa zovuta ndi topology ya gulu la database;
• kuyankha mwachangu komanso mwachangu pazochitika zokhudzana ndi mbuye, kuchepetsa nthawi yopumira.

Komabe, yankholi lili ndi malire ake ndi zovuta zake:

• kukulitsa chiwembu cha HA kumalo angapo a data kudzafuna netiweki imodzi ya L2 pakati pawo;
• Tisanagawire VIP kwa mbuye watsopano, tiyenera kumasula pa yakaleyo. Njirayi ndi yotsatizana, yomwe imawonjezera nthawi yobwezeretsa;
• kutulutsa VIP kumafuna mwayi wa SSH ku seva, kapena njira ina iliyonse yoyimbira njira zakutali. Popeza seva kapena database ikukumana ndi mavuto omwe adayambitsa kuchira, sitingakhale otsimikiza kuti kuchotsa VIP kudzatha bwino. Ndipo izi zitha kuwonetsa ma seva awiri omwe ali ndi adilesi yofanana ya IP komanso vuto split brain.

Kupewa split brain, mungagwiritse ntchito njira STONITH ("Shoot The Other Node in the Head"), yomwe imalekanitsa kapena kulepheretsa node yamavuto. Palinso njira zina zoyendetsera kupezeka kwakukulu kwamagulu: kuphatikiza kwa VIP ndi DNS, kupeza ntchito ndi ma proxy services, kubwereza kofananira ndi njira zina zomwe zili ndi zovuta ndi zabwino zake.

Ndinayankhula za njira yathu yopangira gulu la MySQL failover. Ndizosavuta kukhazikitsa ndikupereka mlingo wovomerezeka wa kudalirika pansi pa zomwe zikuchitika. Pamene dongosolo lonse ndi zomangamanga makamaka zikukula, njira iyi mosakayika idzasintha.

Source: www.habr.com