Mhoro, ini ndiri Sergey Elantsev, ndinokudziridza muYandex.Cloud. Pakutanga, ini ndakatungamira kukudziridzwa kweL7 balancer yeYandex portal - vamwe vaaishanda navo vanoseka kuti zvisinei nezvandinoita, zvinova muyedzo. Ini ndichaudza vaverengi veHabr maitiro ekugadzirisa mutoro mupuratifomu yegore, izvo zvatinoona sechishandiso chakakodzera chekuzadzisa chinangwa ichi, uye mafambiro atiri kuenda pakuvaka chishandiso ichi.
Kutanga, ngatisume mamwe mazwi:
- VIP (Virtual IP) - balancer IP kero
- Server, backend, semuenzaniso - muchina chaiwo unomhanyisa application
- RIP (Real IP) - server IP kero
- Healthcheck - yekutarisa server kugadzirira
- Availability Zone, AZ - yakasarudzika zvivakwa munzvimbo yedata
- Nharaunda - mubatanidzwa weAZ akasiyana
Mitoro inoyera inogadzirisa mabasa makuru matatu: ivo vanoita kuyera pachayo, vanovandudza kukanganisa kushivirira kwesevhisi, uye kurerutsa kuyera kwayo. Kukanganisa kushivirira kunochengetedzwa kuburikidza neatomatiki traffic manejimendi: iyo balancer inotarisisa mamiriro ekushandisa uye isingabatanidzi zviitiko kubva pakuyera izvo zvisingapfuure cheki yehupenyu. Kuyera kunovimbiswa nekugovera zvakaenzana mutoro muzviitiko zvese, pamwe nekugadzirisa rondedzero yezviitiko panhunzi. Kana iyo kuenzanisa isina yunifomu yakakwana, mamwe ezviitiko anogashira mutoro unodarika chiyero chavo chekugona, uye sevhisi ichave isingavimbike.
Mutoro wemutoro unowanzo kurongedzerwa neprotocol layer kubva kuOSI modhi painomhanya. Iyo Cloud Balancer inoshanda padanho reTCP, rinoenderana neyechina layer, L4.
Ngatienderere mberi kune pamusoro peiyo Cloud balancer architecture. Tichawedzera zvishoma nezvishoma huwandu hwemashoko. Isu tinoparadzanisa zvikamu zvebalancer mumakirasi matatu. Iyo config plane class ine basa rekudyidzana kwevashandisi uye inochengeta iyo inotarirwa mamiriro ehurongwa. Iyo ndege inodzora inochengetedza iyo yazvino mamiriro eiyo system uye inogadzirisa masisitimu kubva kune data ndege kirasi, ayo ane mutoro wakananga kuendesa traffic kubva kune vatengi kune ako mamiriro.
Data ndege
Iyo traffic inopera pamidziyo inodhura inonzi mabhodha routers. Kuwedzera kukanganisa kushivirira, akati wandei maturusi anoshanda panguva imwe chete mune imwe data data. Tevere, iyo traffic inoenda kune vanoenzanisa, iyo inozivisa chero cast IP kero kune ese maAZs kuburikidza neBGP kune vatengi.
Traffic inofambiswa pamusoro peECMP - iyi inzira yekufambisa zvichienderana nekuti panogona kunge paine nzira dzakati wandei dzakaenzana kune chinangwa (kwedu, tarisiro ichave yekuenda IP kero) uye mapaketi anogona kutumirwa pamwe nechero ipi zvayo. Isu tinotsigirawo basa munzvimbo dzinoverengeka dziripo zvichienderana nechirongwa chinotevera: isu tinoshambadza kero munharaunda yega yega, traffic inoenda kune iri pedyo uye haipfuuri miganhu yayo. Gare gare mune positi isu tichatarisa zvakadzama pane zvinoitika kune traffic.
Gadzirisa ndege
Chinhu chakakosha cheiyo config plane ndiyo API, kuburikidza neayo maitiro ekutanga ane mabharari anoitwa: kugadzira, kudzima, kushandura kuumbwa kwezviitiko, kuwana hutano cheki, nezvimwe. Kune rumwe rutivi, iyi REST API, uye pane zvimwe, isu tiri muGore tinowanzo shandisa chimiro chegRPC, saka "tinoshandura" REST kuenda ku gRPC uye toshandisa gRPC chete. Chero chikumbiro chinotungamira mukusikwa kweakatevedzana easynchronous idempotent mabasa anoitwa pane yakajairika dziva reYandex.Cloud vashandi. Mabasa anonyorwa nenzira yekuti anogona kumiswa chero nguva obva atangazve. Izvi zvinovimbisa scalability, kudzokorora uye kutema matanda ekushanda.
Nekuda kweizvozvo, basa kubva kuAPI richaita chikumbiro kune iyo balancer service controller, iyo yakanyorwa muGo. Inogona kuwedzera uye kubvisa zvienzaniso, shandura kuumbwa kwemashure uye zvigadziriso.
Iyo sevhisi inochengeta mamiriro ayo muYandex Database, yakagoverwa yakachengetedzwa dhatabhesi iyo iwe uchakurumidza kukwanisa kushandisa. MuYandex.Cloud, sezvatatova , pfungwa yekudya kwembwa inoshanda: kana isu pachedu tikashandisa masevhisi edu, saka vatengi vedu vachafarawo kuvashandisa. Yandex Database ndiyo muenzaniso wekushandiswa kwemafungiro akadaro. Isu tinochengeta data redu rese muYDB, uye isu hatifanirwe kufunga nezve kuchengetedza uye kuyera dhatabhesi: matambudziko aya anogadziriswa isu, isu tinoshandisa dhatabhesi sesevhisi.
Ngatidzokei kune balancer controller. Basa rayo ndere kuchengetedza ruzivo nezve balancer uye kutumira basa rekutarisa kugadzirira kweiyo chaiyo muchina kune healthcheck controller.
Healthcheck controller
Iyo inogamuchira zvikumbiro zvekuchinja cheki mitemo, inovachengeta muYDB, inogovera mabasa pakati peheatcheck node uye inounganidza mhedzisiro, iyo inozochengetwa kune dhatabhesi uye inotumirwa kune loadbalancer controller. Iyo, zvakare, inotumira chikumbiro chekushandura kuumbwa kwesumbu mundege yedata kune loadbalancer-node, yandichakurukura pazasi.
Ngatikurukurei zvakawanda nezveutano. Vanogona kukamurwa kuva makirasi akawanda. Ongororo dzine maitiro akasiyana ekubudirira. TCP cheki inoda kubudirira kumisikidza kubatana mukati menguva yakatarwa. Macheki eHTTP anoda zvese zviri zviviri kubatana kwakabudirira uye mhinduro ine 200 mamiriro kodhi.
Zvakare, cheki dzinosiyana mukirasi yechiito - inoshanda uye inongoita. Passive cheki inongotarisa zviri kuitika netraffic pasina kutora chero chakakosha chiito. Izvi hazvishande zvakanyanya paL4 nekuti zvinoenderana nemafungiro eiyo yepamusoro-level protocol: paL4 hapana ruzivo rwekuti kuvhiyiwa kwakatora nguva yakareba sei kana kuti kupedzwa kwekubatanidza kwaive kwakanaka kana kwakaipa. Cheki cheki chinoda kuti mubhadhari atumire zvikumbiro kune yega yega sevha.
Mazhinji mabharani anotakura zvinhu anoongorora hupenyu pachawo. PaCloud, takasarudza kuparadzanisa zvikamu zvehurongwa kuti uwedzere scalability. Iyi nzira ichatibvumira kuwedzera nhamba yevalancers tichichengetedza nhamba yehutano zvikumbiro kune sevhisi. Macheki anoitwa neakasiyana healthcheck node, pane iyo cheki tarisiro inogovaniswa uye inodzokororwa. Iwe haugone kuita cheki kubva kune mumwe mugamuchiri, sezvo angatadza. Ipapo isu hatizowana mamiriro ezviitiko zvaakatarisa. Isu tinoita macheki pane chero ezviitiko kubva angangoita matatu healthcheck node. Isu tinogovanisa zvinangwa zvecheki pakati penodhi tichishandisa anowirirana hashing algorithms.
Kuparadzanisa kuenzanisa uye kutarisa kwehutano kunogona kutungamirira kumatambudziko. Kana iyo healthcheck node ichiita zvikumbiro kumuenzaniso, ichipfuura iyo balancer (iyo isiri kushandira traffic parizvino), ipapo mamiriro anoshamisa anomuka: iyo sosi inoita kunge mhenyu, asi traffic haisvike pairi. Isu tinogadzirisa dambudziko iri nenzira iyi: isu takavimbiswa kutanga hutano hwekutarisa traffic kuburikidza nemabalancers. Mune mamwe mazwi, chirongwa chekufambisa mapaketi ane traffic kubva kune vatengi uye kubva kune hutano cheki inosiyana zvishoma: muzviitiko zvese izvi, mapaketi achasvika kune vanoenzanisa, izvo zvinovaendesa kune zvakanangwa zviwanikwa.
Musiyano ndewekuti vatengi vanoita zvikumbiro kuVIP, nepo hutano hunoita zvikumbiro kune yega yega RIP. Dambudziko rinonakidza rinomuka pano: isu tinopa vashandisi vedu mukana wekugadzira zviwanikwa mune grey IP network. Ngatifungei kuti kune vaviri vakasiyana varidzi vemakore vakavanza masevhisi avo kuseri kwemabharanzi. Mumwe nemumwe wavo ane zviwanikwa mu10.0.0.1/24 subnet, ine kero dzakafanana. Iwe unofanirwa kukwanisa kuvasiyanisa neimwe nzira, uye pano iwe unofanirwa kunyura muchimiro cheYandex.Cloud chaiyo network. Zviri nani kuti uwane rumwe ruzivo mukati , zvakakosha kwatiri ikozvino kuti network ine multi-layered uye ine tunnels inogona kusiyaniswa ne subnet id.
Healthcheck nodes inobata mabharanzi uchishandisa anonzi quasi-IPv6 kero. A quasi-kero i IPv6 kero ine IPv4 kero uye mushandisi subnet id yakaiswa mukati mayo. Iyo traffic inosvika kune balancer, iyo inobvisa iyo IPv4 resource kero kubva mairi, inotsiva IPv6 neIPv4 uye inotumira pakiti kune network yemushandisi.
Iyo reverse traffic inoenda nenzira imwecheteyo: muyedzo anoona kuti kwainoenda igrey network kubva kune hutano, uye inoshandura IPv4 kuita IPv6.
VPP - mwoyo we data ndege
Iyo balancer inoshandiswa uchishandisa Vector Packet Processing (VPP) tekinoroji, dhizaini kubva kuCisco ye batch processing ye network traffic. Kwatiri, chimiro chinoshanda pamusoro pemushandisi-nzvimbo network network manejimendi raibhurari - Data Plane Development Kit (DPDK). Izvi zvinogonesa kuita kwepamusoro kwepakeji kugadzirisa: kwakawanda kunokanganisa kunoitika mu kernel, uye hapana shanduko yemamiriro ezvinhu pakati pe kernel nzvimbo nenzvimbo yemushandisi.
VPP inoenda zvakatoenda mberi uye inosvina kutowedzera kuita kunze kweiyo sisitimu nekubatanidza mapakeji kuita mabhechi. Kuwana kwekuita kunobva mukushandiswa kwehasha kwecache pane ma processors emazuva ano. Zvose zvinyorwa zvinyorwa zvinoshandiswa (mapakiti anogadziriswa mu "vectors", data iri pedyo kune mumwe nemumwe) uye caches yekuraira: muVPP, kushandiswa kwepakeji kunotevera girafu, node dzine mabasa anoita basa rimwe chete.
Semuenzaniso, kushandiswa kwe IP packets muVPP kunoitika nenzira inotevera: kutanga, misoro yepakiti inoparadzaniswa mu node yekuparadzanisa, uye inotumirwa kune node, iyo inoendesa mberi mapepa mberi maererano nematafura ekufambisa.
A little hardcore. Vanyori veVPP havashiviriri kukanganisa mukushandiswa kwema processor caches, saka yakajairika kodhi yekugadzira vector yemapakiti ine manual vectorization: pane yekugadzira loop umo mamiriro akaita se "tine mapaketi mana mumutsara" inogadziriswa, zvino zvakafanana kune vaviri, ipapo - kune imwe. Prefetch mirairo inowanzo shandiswa kurodha data mumacache kuti ikurumidze kupinda kwairi mune dzinotevera iterations.
n_left_from = frame->n_vectors;
while (n_left_from > 0)
{
vlib_get_next_frame (vm, node, next_index, to_next, n_left_to_next);
// ...
while (n_left_from >= 4 && n_left_to_next >= 2)
{
// processing multiple packets at once
u32 next0 = SAMPLE_NEXT_INTERFACE_OUTPUT;
u32 next1 = SAMPLE_NEXT_INTERFACE_OUTPUT;
// ...
/* Prefetch next iteration. */
{
vlib_buffer_t *p2, *p3;
p2 = vlib_get_buffer (vm, from[2]);
p3 = vlib_get_buffer (vm, from[3]);
vlib_prefetch_buffer_header (p2, LOAD);
vlib_prefetch_buffer_header (p3, LOAD);
CLIB_PREFETCH (p2->data, CLIB_CACHE_LINE_BYTES, STORE);
CLIB_PREFETCH (p3->data, CLIB_CACHE_LINE_BYTES, STORE);
}
// actually process data
/* verify speculative enqueues, maybe switch current next frame */
vlib_validate_buffer_enqueue_x2 (vm, node, next_index,
to_next, n_left_to_next,
bi0, bi1, next0, next1);
}
while (n_left_from > 0 && n_left_to_next > 0)
{
// processing packets by one
}
// processed batch
vlib_put_next_frame (vm, node, next_index, n_left_to_next);
}Saka, Healthchecks inotaura pamusoro peIPv6 kuVPP, inovashandura kuita IPv4. Izvi zvinoitwa node mugirafu, yatinodaidza kuti algorithmic NAT. Kune reverse traffic (uye kushandura kubva IPv6 kuenda IPv4) kune yakafanana algorithmic NAT node.
Yakananga traffic kubva kune vanoenzanisa vatengi inoenda nepakati pemagirafu node, iyo inoita iyo yekuyera pachayo.
Yokutanga node inonamira zvikamu. Inochengeta hashi ye kumisangano yakasimbiswa. 5-tuple inosanganisira kero uye chiteshi chemutengi kubva kunofambiswa ruzivo, kero uye zviteshi zvezviwanikwa zviripo zvekugamuchira traffic, pamwe netiweki protocol.
Iyo 5-tuple hashi inotibatsira kuita shoma computation mune inotevera inopindirana hashing node, pamwe nekubata zvirinani resource list shanduko kuseri kwebalancer. Kana pakiti isina chikamu inosvika pakuenzanisa, inotumirwa kune inopindirana hashing node. Apa ndipo panoitika kuenzanisa uchishandisa hashing inowirirana: isu tinosarudza sosi kubva pane rondedzero inowanikwa "mhenyu" zviwanikwa. Tevere, mapaketi anotumirwa kuNAT node, iyo inotsiva kero yekuenda uye inoverengerazve macheki. Sezvauri kuona, isu tinotevera mirairo yeVPP - sekuda, kurongedza maverengero akafanana kuti uwedzere kushanda kwe processor cache.
Consistent hashing
Sei takasarudza iyo uye chii icho kunyange? Kutanga, ngatitarisei basa rakapfuura - kusarudza sosi kubva pane rondedzero.
Nehashi isingaenderane, iyo hashi yepakiti iri kuuya inoverengerwa, uye sosi inosarudzwa kubva pane inorongwa neinosara yekukamura iyi hashi nehuwandu hwezviwanikwa. Chero bedzi iyo rondedzero inoramba isina kuchinjika, chirongwa ichi chinoshanda nemazvo: isu tinogara tichitumira mapaketi ane akafanana 5-tuple kune imwecheteyo muenzaniso. Kana, semuenzaniso, imwe sosi yakamira kupindura kune hutano, saka kune yakakosha chikamu chehashi sarudzo ichachinja. Kubatana kweTCP yemutengi kuchatyorwa: pakiti yakambosvika muenzaniso A inogona kutanga kusvika pamuenzaniso B, isingazive nezvesesheni yepakiti iyi.
Consistent hashing inogadzirisa dambudziko rakatsanangurwa. Nzira iri nyore yekutsanangura iyi pfungwa ndeiyi: fungidzira kuti une mhete yaunogovera zviwanikwa ne hashi (semuenzaniso, ne IP: port). Kusarudza sosi kushandura vhiri nekona, iyo inotarirwa nehashi yepakiti.
Izvi zvinoderedza kugovaniswa kwemotokari kana kuumbwa kwezviwanikwa kuchichinja. Kudzima sosi kunongokanganisa chikamu cheiyo inopindirana hashing ring iyo sosi yaivemo. Kuwedzera sosi zvakare kunoshandura kugovera, asi isu tine inonamira zvikamu node, izvo zvinotitendera kuti tisachinjire zvakatotangwa zvikamu kune zvitsva zviwanikwa.
Takatarisa zvinoitika kutungamira traffic pakati peiyo balancer uye zviwanikwa. Zvino ngatitarisei traffic yekudzoka. Inoteedzera patani yakafanana neyekutarisa traffic - kuburikidza nealgorithmic NAT, kureva, kuburikidza nereverse NAT 44 yevatengi traffic uye kuburikidza neNAT 46 yehutanochecks traffic. Isu tinonamatira kune yedu chirongwa: isu tinobatanidza hutano cheki traffic uye chaiyo mushandisi traffic.
Loadbalancer-node uye yakaungana zvikamu
Kuumbwa kwezviyereso uye zviwanikwa muVPP zvinoshumwa nebasa renzvimbo - loadbalancer-node. Iyo inonyorera kune kuyerera kwezviitiko kubva kuloadbalancer-controller uye inokwanisa kuronga mutsauko pakati peiyo ikozvino VPP mamiriro uye inotarirwa nyika inogamuchirwa kubva kumutongi. Isu tinowana yakavharwa system: zviitiko kubva kuAPI zvinouya kune iyo balancer controller, iyo inopa mabasa kune healthcheck controller kutarisa "liveness" yezviwanikwa. Izvo, zvakare, zvinopa mabasa kune healthcheck-node uye inounganidza mhedzisiro, mushure mezvo inovadzosera kune iyo balancer controller. Loadbalancer-node inonyorera kune zviitiko kubva kumutongi uye inoshandura mamiriro eVPP. Muhurongwa hwakadaro, sevhisi yega yega inoziva chete izvo zvinodiwa nezve masevhisi evavakidzani. Nhamba yekubatanidza ishoma uye isu tine kugona kushanda uye kuyera zvikamu zvakasiyana takazvimirira.
Inyaya dzipi dzaidziviswa?
Ese masevhisi edu mundege yekudzora akanyorwa muGo uye ane kuyera kwakanaka uye kuvimbika hunhu. Go ine akawanda akavhurika sosi maraibhurari ekuvaka akagoverwa masisitimu. Isu tinoshingairira kushandisa GRPC, zvese zvikamu zvine yakavhurika sosi yekumisikidzwa kwesevhisi kuwanikwa - masevhisi edu anotarisisa mashandiro emumwe nemumwe, anogona kushandura kuumbwa kwawo zvine simba, uye isu takabatanidza izvi neGRPC kuenzanisa. Kune metrics, isu zvakare tinoshandisa yakavhurika sosi mhinduro. Mundege yedata, takawana kuita kwakanaka uye hombe yezviwanikwa: zvakazonetsa kuunganidza kumira kwataigona kuvimba nekuita kweVPP, pane simbi network kadhi.
Matambudziko Nekugadzirisa
Chii chisina kushanda zvakanaka? Go ine otomatiki ndangariro manejimendi, asi ndangariro kuvuza kuchiri kuitika. Nzira iri nyore yekubata navo ndeyekumhanyisa goroutines uye rangarira kuvamisa. Takeaway: Tarisa yako Go zvirongwa 'memory mashandisiro. Kazhinji chiratidzo chakanaka nhamba yegoroutines. Pane kuwedzera mune iyi nyaya: muGo zviri nyore kuwana runtime data - ndangariro mashandisiro, huwandu hwekumhanya goroutines, uye mamwe akawanda ma paramita.
Zvakare, Go inogona kunge isiri iyo yakanakisa sarudzo yekushanda bvunzo. Iwo ane verbose, uye nzira yakajairwa ye "kumhanyisa zvese muCI mubatch" haina kunyatsovakodzera. Icho chokwadi ndechekuti bvunzo dzekushanda dzinoda zviwanikwa uye dzinokonzeresa nguva chaiyo yekubuda. Nekuda kweizvi, bvunzo dzinogona kukundikana nekuti iyo CPU yakabatikana neyuniti bvunzo. Mhedziso: Kana zvichiita, ita bvunzo βzvinoremaβ wakaparadzana neyuniti bvunzo.
Microservice chiitiko chekuvaka chakanyanya kuomarara kupfuura monolith: kuunganidza matanda pamakumi emakumi emakina akasiyana hazvina kunyatsoita. Mhedziso: kana iwe ukaita microservices, pakarepo funga nezvekutsvaga.
Zvirongwa zvedu
Tichatangisa balancer yemukati, IPv6 balancer, towedzera tsigiro yeKubernetes zvinyorwa, rambai muchigovana masevhisi edu (parizvino chete healthcheck-node uye healthcheck-ctrl zvakakamurwa), wedzera hutano hutsva, uye zvakare shandisa smart aggregation yemacheki. Tiri kufunga nezve mukana wekuita kuti masevhisi edu awedzere kuzvimiririra - kuti ataurirane kwete zvakanangana, asi vachishandisa mutsara wemeseji. Iyo SQS-inoenderana sevhisi ichangobva kuoneka muCloud .
Munguva pfupi yapfuura, kuburitswa pachena kweYandex Load Balancer kwakaitika. Explore Sevhisi, maneja mabharani nenzira yakakunakira iwe uye wedzera kukanganisa kushivirira kwemapurojekiti ako!
Source: www.habr.com