Mhoro, Habr! Ini ndiri Artem Karamyshev, mukuru wetimu manejimendi timu . Takava nezvakawanda zvitsva zvigadzirwa mugore rapfuura. Isu taida kuona kuti masevhisi eAPI ari nyore scalable, kukanganisa-kushivirira, uye akagadzirira kukurumidza kukura mumutoro wemushandisi. Yedu puratifomu inoshandiswa paOpenStack, uye ini ndinoda kukuudza kuti ndeapi chikamu chematambudziko ekushivirira ataifanira kugadzirisa kuti tiwane yekutadza-kushivirira system. Ini ndinofunga izvi zvichave zvinonakidza kune avo vanogadzirawo zvigadzirwa paOpenStack.
Iyo yakazara kukanganisa kushivirira kwepuratifomu kunosanganisira kusimba kwezvikamu zvayo. Saka isu tichaenda zvishoma nezvishoma nematanho ese atakaona njodzi nekudzivhara.
Vhidhiyo vhezheni yeiyi nyaya, yekutanga sosi yaive shumo paUptime day 4 musangano, wakarongwa na , unogona kuona .
Resiliency yechimiro chekuvaka
Nzvimbo yeruzhinji yegore reMCS ikozvino yakavakirwa munzvimbo mbiri dzeTier III data, pakati pavo pane yayo yakasviba fiber, yakachengetwa pamwero wenyama nenzira dzakasiyana, ine 200 Gbit/s. Tier III inopa iyo inodiwa mwero wekutadza kushivirira kune yemuviri masisitimu.
Fiber yakasviba inochengeterwa pane zvese zvemuviri uye zvine musoro mazinga. Iyo nzira yekuchengeta chiteshi yaive yekudzokorora, matambudziko akamuka, uye isu tiri kugara tichivandudza kutaurirana pakati penzvimbo dzedata.
Semuenzaniso, nguva pfupi yapfuura, pakushanda mutsime pedyo neimwe yedata data, mucheri akapwanya pombi, uye mukati meiyi pombi maive netambo huru uye yekuchengetedza optical cable. Yedu yekukanganisa-inoshivirira nzira yekutaurirana neye data data yakave isina njodzi pane imwe nguva, mutsime. Naizvozvo, takarasikirwa nechikamu chezvivakwa. Takatora mhedziso uye takaita akati wandei, kusanganisira kuisa mamwe optics mutsime riri padyo.
Munzvimbo dzedata pane mapoinzi ekuvapo kwevanopa zvekutaurirana vatinotepfenyura maprefixes edu kuburikidza neBGP. Kune yega yega yetiweki nzira, yakanakisa metric inosarudzwa, iyo inobvumira vatengi vakasiyana kupihwa yakanakisa yekubatanidza mhando. Kana kutaurirana kuburikidza nemupi mumwe kukadzikira, tinovakazve nzira yedu kuburikidza nevanopa varipo.
Kana mupi akatadza, isu tinochinja kune inotevera. Muchiitiko chekukundikana kweimwe yedata data, isu tine girazi kopi yemasevhisi edu mune yechipiri data center, iyo inotora mutoro wese.
Resiliency of physical infrastructure
Zvatinoshandisa kune application-level kukanganisa kushivirira
Sevhisi yedu yakavakirwa pane akati wandei maopensource components.
ExaBGP ibasa rinoshandisa akati wandei mabasa uchishandisa iyo BGP-based dynamic routing protocol. Isu tinoshingaira kuishandisa kushambadzira yedu yakacheneswa IP kero kuburikidza iyo vashandisi vanowana iyo API.
HAProxy ndeye yakakwira-mutoro inoremedza iyo inokutendera iwe kuti ugadzirise inochinjika kwazvo traffic balancing mitemo pamatanho akasiyana eiyo OSI modhi. Isu tinoishandisa kuenzanisa pamberi pese masevhisi: dhatabhesi, meseji broker, API masevhisi, webhu masevhisi, yedu yemukati mapurojekiti - zvese zviri kuseri kweHAProxy.
API application - web application yakanyorwa mu python, iyo mushandisi anotarisira hupfumi hwake nebasa rake.
Chikumbiro chevashandi (pano mubatsiri chete) - muOpenStack masevhisi, iyi idhimoni rezvivakwa rinokutendera kuti utepfenyure mirairo yeAPI kune zvivakwa. Semuenzaniso, dhisiki kusikwa kunoitika mumushandi, uye chikumbiro chekugadzira chinoitika muApp API.
Standard OpenStack Application Architecture
Mazhinji masevhisi akagadzirirwa OpenStack anoedza kutevedzera imwe paradigm. Sevhisi inowanzove nezvikamu zviviri: API nevashandi (backend executors). Semutemo, API ndeye WSGI application mupython, iyo inotangwa ingave yakazvimirira maitiro (daemon), kana kushandisa yakagadzirira-yakagadzirwa Nginx kana Apache web server. Iyo API inobata chikumbiro chemushandisi uye inopfuudza mimwe mirairo kune yevashandi application yekuurayiwa. Kuchinjisa kunoitika uchishandisa meseji broker, kazhinji RabbitMQ, mamwe acho haana kutsigirwa zvakanaka. Kana mameseji asvika kune broker, anogadziriswa nevashandi uye, kana zvichidikanwa, anodzosera mhinduro.
Iyi paradigm inosanganisira ega akajairwa mapoinzi ekutadza: RabbitMQ uye dhatabhesi. Asi RabbitMQ yakaparadzaniswa mukati mesevhisi imwe uye, muchirevo, inogona kuve yega yega sevhisi. Saka kuMCS tinopatsanura masevhisi aya zvakanyanya sezvinobvira; kune yega yega purojekiti tinogadzira yakaparadzana dhatabhesi, yakaparadzana RabbitMQ. Iyi nzira yakanaka nekuti kana pakaitika tsaona kune dzimwe nzvimbo dzisina njodzi, haisi iyo yese sevhisi inoputsika, asi chikamu chayo chete.
Huwandu hwemashandisirwo evashandi hauna muganho, saka API inogona kuyera nyore yakachinjika kuseri kwevarengi kuitira kuti iwedzere kuita uye kukanganisa kushivirira.
Mamwe masevhisi anoda kurongeka mukati mesevhisi kana zvakaoma sequential mashandiro anoitika pakati peAPI nevashandi. Muchiitiko ichi, imwe nzvimbo yekubatanidza inoshandiswa, hurongwa hwemasumbu hwakadai seRedis, Memcache, etcd, iyo inobvumira mumwe mushandi kuti audze mumwe kuti basa iri rakapiwa kwaari ("ndapota usatora"). Isu tinoshandisa etcd. Sezvo mutemo, vashandi vanoshingairira kutaurirana ne database, kunyora uye kuverenga ruzivo kubva ipapo. Isu tinoshandisa mariadb sedhatabhesi, iyo iri mune multimaster cluster.
Iyi yemhando yepamusoro sevhisi imwe yakarongeka nenzira inogamuchirwa kazhinji yeOpenStack. Inogona kutorwa seyakavharwa sisitimu, iyo iyo nzira dzekuyera uye kushivirira kukanganisa dziri pachena. Semuenzaniso, kune API kukanganisa kukanganisa, zvakakwana kuisa balancer pamberi pavo. Kuwedzera vashandi kunowanikwa nekuwedzera nhamba yavo.
Nzvimbo isina simba muchirongwa chose ndeye RabbitMQ uye MariaDB. Mavakirwo avo anokodzera chinyorwa chakasiyana.Muchinyorwa chino ndinoda kutarisa nezve API kukanganisa kushivirira.
Openstack Application Architecture. Kuenzanisa uye kukanganisa kushivirira kwepuratifomu yegore
Kuita iyo HAProxy balancer kukanganisa-kushivirira uchishandisa ExaBGP
Kuita kuti ma API edu awedzere, anokurumidza uye anokanganisa-kutadza, tinoisa chiyero chemutoro pamberi pavo. Isu takasarudza HAProxy. Mukuona kwangu, ine zvese zvinodikanwa zvebasa redu: kuenzana pamazinga akati wandei eOSI, manejimendi interface, kuchinjika uye scalability, nhamba huru yekuenzanisa nzira, tsigiro yematafura esesheni.
Dambudziko rekutanga raida kugadziriswa raive kukanganisa kushivirira kweiyo balancer pachayo. Kungoisa balancer zvakare kunogadzira poindi yekutadza: iyo balancer inotyora uye sevhisi inopunzika. Kuti izvi zvisaitike, takashandisa HAProxy takabatana neExaBGP.
ExaBGP inokutendera kuti ushandise nzira yekutarisa mamiriro esevhisi. Takashandisa nzira iyi kutarisa kushanda kweHAProxy uye, kana pane matambudziko, bvisa basa reHAProxy kubva kuBGP.
ExaBGP + HAProxy chirongwa
- Isu tinoisa software inodiwa, ExaBGP uye HAProxy, pamaseva matatu.
- Isu tinogadzira loopback interface pane yega yega server.
- Pane ese matatu maseva tinopa imwechete chena IP kero kune ino interface.
- A chena IP kero inoshambadzirwa kuInternet kuburikidza neExaBGP.
Kukanganisa kushivirira kunowanikwa nekushambadza iyo yakafanana IP kero kubva kune ese matatu maseva. Kubva pane network yekuona, iyo kero imwe chete inowanikwa kubva kune matatu akasiyana anotevera hops. Iyo router inoona nzira nhatu dzakafanana, inosarudza iyo inonyanya kukosha yadzo zvichibva pane yayo metric (iyi kazhinji ndiyo imwechete sarudzo), uye traffic inoenda chete kune imwe yemaseva.
Muchiitiko chezvinetso nekushanda kweHAProxy kana kukundikana kwevhavha, ExaBGP inomira kuzivisa nzira, uye motokari inoshandura zvakanaka kune imwe sevha.
Nekudaro, isu takawana kukanganisa kushivirira kweiyo balancer.
Kukanganisa kushivirira kweHAProxy balancers
Chirongwa chakazove chisina kukwana: takadzidza kuchengetedza HAProxy, asi hatina kudzidza kugovera mutoro mukati mesevhisi. Naizvozvo, isu takawedzera chirongwa ichi zvishoma: isu takaenderera mberi nekuenzanisa pakati akati wandei machena IP kero.
Kuenzanisa kwakavakirwa paDNS pamwe neBGP
Nyaya yekuremerwa kwekutakura kune yedu HAProxy inoramba isina kugadziriswa. Nekudaro, inogona kugadziriswa zviri nyore, sezvatakaita pano.
Kuti uenzanise maseva matatu iwe uchada 3 chena IP kero uye yakanaka yekare DNS. Imwe neimwe yemakero aya inotemerwa pane loopback interface yeHAProxy yega yega uye inoshambadzirwa kuInternet.
MuOpenStack, kubata zviwanikwa, dhairekitori rebasa rinoshandiswa, iro rinotsanangura iyo endpoint API yeimwe sevhisi. Mune ino dhairekitori tinonyoresa zita rezita - public.infra.mail.ru, iro rinogadziriswa kuburikidza neDNS nematatu akasiyana IP kero. Nekuda kweizvozvo, tinowana kugoverwa kwemutoro pakati pemakero matatu kuburikidza neDNS.
Asi kubva pakuzivisa chena IP kero isu hatidzore zvekutanga sevhavha sarudzo, izvi hazvisati zvakaenzana. Kazhinji, sevha imwe chete ndiyo ichasarudzwa zvichienderana nehukuru hwe IP kero, uye mamwe maviri achange asina chaanoita nekuti hapana mametric anotsanangurwa muBGP.
Takatanga kutumira nzira kuburikidza neExaBGP nemametric akasiyana. Imwe neimwe balancer inoshambadza ese matatu machena IP kero, asi imwe yacho, iyo huru yeiyi balancer, inoshambadzirwa neiyo shoma metric. Saka apo ese ari matatu mabharanzi ari kushanda, nhare kukero yekutanga yeIP inoenda kune yekutanga balancer, inodana kune yechipiri kune yechipiri, uye inodana kune yechitatu kune yechitatu.
Chii chinoitika kana imwe yezviyereso ikadonha? Kana chero balancer ikatadza, kero yayo huru ichiri kushambadzwa kubva kune mamwe maviri, uye traffic inogovaniswa pakati pavo. Nekudaro, isu tinopa mushandisi akati wandei IP kero kamwechete kuburikidza neDNS. Nekuenzanisa neDNS uye akasiyana metrics, tinowana kugovaniswa kwemutoro pane ese matatu mabharari. Uye panguva imwe chete isu hatirasikirwi nemhosva kushivirira.
Kuenzanisa HAProxy yakavakirwa paDNS + BGP
Kudyidzana pakati peExaBGP neHAProxy
Saka, isu takashandisa kukanganisa kushivirira kana sevha ichibva, zvichibva pakumisa kuziviswa kwenzira. Asi HAProxy inogona kuvhara nekuda kwezvimwe zvikonzero kunze kwekutadza kweserver: zvikanganiso zvekutonga, kutadza mukati mesevhisi. Tinoda kubvisa yakatyoka balancer kubva pasi pemutoro mune idzi kesi zvakare, uye isu tinoda imwe nzira.
Naizvozvo, kuwedzera chirongwa chakapfuura, takaita kurova kwemoyo pakati peExaBGP neHAProxy. Uku ndiko kusevenzesa kwesoftware yekudyidzana pakati peExaBGP neHAProxy, apo ExaBGP inoshandisa zvinyorwa zvetsika kutarisa mamiriro ekushandisa.
Kuti uite izvi, unofanirwa kugadzirisa muchengeti wehutano muExaBGP config, iyo inogona kutarisa mamiriro eHAProxy. Mune yedu kesi, isu takagadzirisa hutano backend muHAProxy, uye kubva kudivi reExaBGP tinotarisa nechikumbiro chiri nyore cheGET. Kana chiziviso chikamira kuitika, saka HAProxy ingangove isingashande uye hapana chikonzero chekuchishambadza.
HAProxy Health Check
HAProxy Peers: kuwiriranisa kwechikamu
Chinhu chinotevera chekuita kwaive kuwiriranisa zvirongwa. Paunenge uchishanda kuburikidza neakagoverwa mabharani, zvakaoma kuronga kuchengetwa kweruzivo nezvesesheni dzevatengi. Asi HAProxy ndeimwe yevashoma mabharani anogona kuita izvi nekuda kweKushanda kwePeers - kugona kuendesa matafura esesheni pakati peHAProxy maitiro akasiyana.
Kune nzira dzakasiyana dzekuenzanisa: dzakareruka dzakadai se , uye yakawedzerwa, kana chikamu chemutengi chichiyeukwa, uye nguva imwe neimwe yaanoguma ari pane imwechete sevha sepakutanga. Taida kushandisa sarudzo yechipiri.
HAProxy inoshandisa zvimiti-matafura kuchengetedza zvikamu zvevatengi zvemuchina uyu. Ivo vanochengetedza kero yemutengi yepakutanga, kero yakasarudzwa yakanangana (backend) uye rumwe ruzivo rwesevhisi. Kazhinji, matafura anonamira anoshandiswa kuchengetedza sosi-IP + yekuenda-IP pair, iyo inonyanya kubatsira kune zvikumbiro zvisingakwanisi kuendesa mushandisi wesesheni mamiriro kana uchichinja kune imwe balancer, semuenzaniso, muRoundRobin balancing mode.
Kana tafura yetsvimbo ichidzidziswa kufamba pakati pemaitiro akasiyana eHAProxy (pakati peiyo kuenzana kunoitika), mabharani edu achakwanisa kushanda nedziva rimwe rematafura etsvimbo. Izvi zvinoita kuti zvikwanise kushandura network yemutengi zvisina mutsindo kana imwe yemabharanzi ikatadza; shanda nemakasitoma emutengi anoenderera mberi pane mamwe mabackends akasarudzwa kare.
Nekushanda kwakakodzera, dambudziko reiyo IP kero yeiyo balancer kubva iyo musangano wakatangwa inofanira kugadziriswa. Kwatiri, iyi ikero ine simba pane loopback interface.
Basa rakarurama revezera rinowanikwa chete mune mamwe mamiriro ezvinhu. Kureva kuti, TCP timeouts inofanira kunge yakakura zvakakwana kana kuti kuchinja kunofanira kukurumidza zvakakwana kuitira kuti sangano reTCP risave nenguva yekugumisa. Zvisinei, inobvumira kuchinja kusina musono.
MuIaaS tine sevhisi yakavakwa uchishandisa tekinoroji imwechete. Izvi , iyo inonzi Octavia. Inobva pane maviri HAProxy maitiro uye pakutanga inosanganisira kutsigirwa kwevezera. Vakaratidza kuti vakanakisisa mubasa iri.
Mufananidzo wacho unoratidza kufamba kwematafura evezera pakati pezviitiko zvitatu zveHAProxy, gadziriso inorongwa pamusoro pekuti izvi zvinogona kugadzirwa sei:
HAProxy Peers (sesion synchronization)
Kana iwe ukashandisa chirongwa chimwechete, kushanda kwayo kunofanira kunyatsoongororwa. Haisi chokwadi chekuti ichashanda nenzira imwechete 100% yenguva. Asi zvirinani hauzorasikirwe nematafura etsvimbo kana iwe uchifanira kurangarira mutengi sosi IP.
Kudzikamisa nhamba yezvikumbiro zvenguva imwe chete kubva kumutengi mumwe chete
Chero masevhisi anowanikwa pachena, kusanganisira maAPI edu, anogona kuve pasi pezvikumbiro zvakawanda. Zvikonzero zvavo zvinogona kunge zvakasiyana zvachose, kubva pazvikanganiso zvevashandisi kuenda kune zvakanangwa kurwisa. Isu tiri nguva nenguva DDoSed ne IP kero. Vatengi vanowanzoita zvikanganiso muzvinyorwa zvavo uye vanotipa mini-DDoSs.
Imwe nzira kana imwe, kuchengetedzwa kwekuwedzera kunofanirwa kupihwa. Mhinduro iri pachena ndeyekudzikamisa nhamba yezvikumbiro zveAPI uye kwete kutambisa CPU nguva kugadzirisa zvikumbiro zvakashata.
Kuti tishandise zvirambidzo zvakadaro, tinoshandisa miganhu yechiyero, yakarongeka pahwaro hweHAProxy, tichishandisa matafura akafanana. Kumisa miganhu kuri nyore uye kunobvumidza iwe kudzikamisa mushandisi nehuwandu hwezvikumbiro kuAPI. Iyo algorithm inorangarira kunobva IP kubva kunoitwa zvikumbiro uye inodzikamisa nhamba yezvikumbiro panguva imwe chete kubva kune mumwe mushandisi. Ehe, isu takaverenga avhareji API mutoro profil yesevhisi yega yega uye isa muganhu we ≈ 10 nguva kukosha uku. Tinoramba tichinyatsocherechedza mamiriro acho ezvinhu uye chengetedza munwe wedu pamhepo.
Izvi zvinoratidzika sei mukuita? Tine vatengi vanoshandisa yedu autoscaling APIs nguva dzese. Ivo vanogadzira anosvika mazana maviri kusvika matatu emagetsi chaiwo muchina mangwanani uye vanodzima manheru. YeOpenStack, kugadzira muchina chaiwo, zvakare nePaaS masevhisi, inoda angangoita zana zvikumbiro zveAPI, sezvo kudyidzana pakati pemasevhisi kunoitikawo kuburikidza neAPI.
Kuendeswa kwakadaro kwemabasa kunokonzeresa mutoro wakakura. Isu takaongorora mutoro uyu, takaunganidza nhongonya dzemazuva ese, tikawedzera kakapetwa kagumi, uye izvi zvakava chiyero chedu. Isu tinochengeta munwe wedu pane pulse. Isu tinowanzoona bots uye scanner vari kuedza kutitarisa kuti vaone kana tine chero CGA zvinyorwa zvinogona kumhanyiswa, isu tiri kushingaira kuacheka.
Maitiro ekugadzirisa codebase yako pasina vashandisi kucherechedza
Isu tinoshandisawo kukanganisa kushivirira pamwero wekodhi deployment process. Panogona kunge paine glitches panguva yekuburitswa, asi kukanganisa kwavo pakuwanikwa kwesevhisi kunogona kudzikiswa.
Isu tinogara tichivandudza masevhisi edu uye tinofanira kuona kuti iyo codebase inovandudzwa pasina kukanganisa vashandisi. Isu takakwanisa kugadzirisa dambudziko iri tichishandisa kugona kwekutungamira kweHAProxy uye kuita kweGraceful Shutdown mumasevhisi edu.
Kuti ugadzirise dambudziko iri, zvaive zvakafanira kuve nechokwadi chekudzora kweiyo balancer uye "chaiyo" kuvharika kwemasevhisi:
- Panyaya yeHAProxy, kutonga kunoitwa kuburikidza neiyo stats faira, iyo inonyanya socket uye inotsanangurwa muHAProxy config. Unogona kutumira mirairo kwairi kuburikidza ne stdio. Asi yedu huru yekumisikidza dhizaini inonzwisisika, saka ine yakavakirwa-mukati module yekutarisira HAProxy. Yatinoshandisa nesimba.
- Mazhinji edu API uye Injini masevhisi anotsigira ane nyasha kuvharika matekinoroji: kana ichivhara, ivo vanomirira kuti basa razvino ripere, ingave iri http chikumbiro kana rimwe basa rebasa. Chinhu chimwe chetecho chinoitika nemushandi. Inoziva mabasa ose airi kuita uye inopera kana ichinge yapedza zvese.
Kutenda kune aya mapoinzi maviri, iyo yakachengeteka algorithm yekutumirwa kwedu inotaridzika seizvi.
- Iyo inovandudza inounganidza pasuru nyowani yekodhi (kwedu iyi iRPM), inoiedza munzvimbo yedev, inoiedza padanho, uye inoisiya iri mudanho repository.
- Mugadziri anogadzirisa basa rekutumirwa netsanangudzo yakadzama ye "artifacts": iyo vhezheni yepakeji nyowani, tsananguro yekushanda kutsva uye zvimwe zvinhu nezve kutumirwa kana zvichidikanwa.
- Iyo system administrator inotanga iyo update. Inotangisa Ansible playbook, iyo inozoita zvinotevera:
- Inotora pasuru kubva pachikuva repository uye inoishandisa kugadzirisa vhezheni yepakeji mune yechigadzirwa repository.
- Inonyora runyoro rwekumashure kweiyo yakagadziridzwa sevhisi.
- Inovhara sevhisi yekutanga kuti igadziriswe muHAProxy uye inomirira kuti maitiro ayo apedze kumhanya. Nekuda kwekuvharika kwakanaka, tine chivimbo chekuti zvese zvazvino zvikumbiro zvevatengi zvinopedza zvinobudirira.
- Mushure mokunge API nevashandi vakamira zvachose, uye HAProxy yakadzimwa, code inovandudzwa.
- Ansible anomhanyisa masevhisi.
- Pasevhisi yega yega, mamwe "mabato" anodhonzwa, ayo anoita bvunzo dzeyuniti pane akati wandei akatsanangurwa makiyi bvunzo. Cheki yekutanga yekodhi nyowani inoitika.
- Kana pasina zvikanganiso zvakawanikwa munhanho yapfuura, iyo backend inogadziriswa.
- Ngatiendei kune inotevera backend.
- Mushure mekunge mabackend ese agadziridzwa, bvunzo dzinoshanda dzinotangwa. Kana ivo vasipo, saka mugadziri anotarisa chero hutsva hutsva hwaakagadzira.
Izvi zvinopedzisa kutumirwa.
Service update cycle
Chirongwa ichi hachingashande dai tisina mutemo mumwe chete. Isu tinotsigira ese ekare uye matsva shanduro muhondo. Pamberi payo, padanho rekuvandudza software, zvakaiswa pasi kuti kunyangwe paine shanduko mudhatabhesi rebasa, havazotyora kodhi yapfuura. Somugumisiro, nheyo yekodhi inovandudzwa zvishoma nezvishoma.
mhedziso
Kugovera pfungwa dzangu pamusoro pekukanganisa-kushivirira WEB dhizaini, ndinoda zvakare kucherechedza ayo akakosha mapoinzi:
- kushivirira kukanganisa kwemuviri;
- network kukanganisa kushivirira (valancers, BGP);
- kukanganisa kushivirira kwesoftware yakashandiswa uye yakagadziridzwa.
Yakagadzikana uptime munhu wese!
Source: www.habr.com
