Sawubona, ngingu-Sergey Elantsev, ngiyathuthuka ku-Yandex.Cloud. Ngaphambilini, ngihole ukuthuthukiswa kwe-balancer ye-L7 ye-portal ye-Yandex - ozakwethu benza ihlaya ukuthi kungakhathaliseki ukuthi ngenzani, kuvela ukuthi ibhalansi. Ngizotshela abafundi bakaHabr ukuthi bawuphathe kanjani umthwalo endaweni yesikhulumi samafu, lokho esikubona njengethuluzi elikahle lokufeza lo mgomo, nokuthi siqhubekela phambili kanjani ekwakheni leli thuluzi.
Okokuqala, ake sethule amagama athile:
- I-VIP (Virtual IP) - ibhalansi ikheli le-IP
- Iseva, i-backend, isibonelo - umshini obonakalayo osebenzisa uhlelo lokusebenza
- I-RIP (I-IP Yangempela) - ikheli le-IP leseva
- I-Healthcheck - ihlola ukulungela kweseva
- I-Availability Zone, AZ - ingqalasizinda ehlukanisiwe esikhungweni sedatha
- Isifunda - inyunyana yama-AZ ahlukene
Izikali zomthwalo zixazulula imisebenzi emithathu eyinhloko: zizenzela ukulinganisa, zithuthukisa ukubekezelela amaphutha kwesevisi, futhi zenze ukukalwa kwayo kube lula. Ukubekezelelwa kwamaphutha kuqinisekiswa ngokulawulwa kwethrafikhi okuzenzakalelayo: obhalansi uqapha isimo sohlelo futhi akubandakanyi izimo zokulinganisa ezingaphumeleli isheke lokuphila. Ukukala kuqinisekiswa ngokusabalalisa umthwalo ngokulinganayo kuzo zonke izimo, kanye nokubuyekeza uhlu lwezenzakalo ngokuhamba kwesikhathi. Uma ukulinganisa kungalingani ngokwanele, ezinye zezimo zizothola umthwalo oweqa umkhawulo wazo womthamo, futhi isevisi ngeke ithembeke kancane.
Isilinganisi somthwalo ngokuvamile sihlukaniswa ngelendlalelo yephrothokholi kusukela kumodeli ye-OSI esebenza kuyo. I-Cloud Balancer isebenza ezingeni le-TCP, elihambisana nesendlalelo sesine, i-L4.
Ake siqhubekele ekubukezeni konke kwe-Cloud balancer architecture. Sizokwandisa kancane kancane izinga lemininingwane. Sihlukanisa izingxenye ze-balancer zibe amakilasi amathathu. Isigaba sendiza yokulungisa sinesibopho sokusebenzelana komsebenzisi futhi sigcina isimo esiqondiwe sohlelo. Indiza elawulayo igcina isimo samanje sesistimu futhi iphatha amasistimu asuka esigabeni sendiza yedatha, anesibopho esiqondile sokuletha ithrafikhi kusuka kumakhasimende kuya ezimweni zakho.
Indiza yedatha
I-traffic iphelela kumadivayisi abizayo abizwa ngokuthi ama-border routers. Ukwandisa ukubekezelela amaphutha, amadivaysi anjalo amaningana asebenza ngesikhathi esisodwa esikhungweni sedatha esisodwa. Okulandelayo, ithrafikhi iya kubalinganisi, abamemezela noma yimaphi amakheli e-IP kuwo wonke ama-AZ nge-BGP kumakhasimende.
Ithrafikhi idluliswa nge-ECMP - leli yisu lomzila lapho kungaba khona imizila eminingana emihle ngokulinganayo eya lapho okuqondiwe khona (kithi, okuqondiwe kuzoba yikheli le-IP okuyiwa kulo) futhi amaphakethe angathunyelwa kunoma iyiphi yawo. Siphinde futhi sisekele umsebenzi ezindaweni ezimbalwa ezitholakalayo ngokuya ngohlelo olulandelayo: sikhangisa ikheli endaweni ngayinye, ithrafikhi iya kweseduze futhi ayidluli imingcele yayo. Kamuva kokuthunyelwe sizobheka kabanzi ukuthi kwenzekani ngethrafikhi.
Lungiselela indiza
Ingxenye eyinhloko yendiza yokuhlela i-API, lapho kwenziwa imisebenzi eyisisekelo namabhalansi: ukudala, ukususa, ukushintsha ukwakheka kwezimo, ukuthola imiphumela yokuhlolwa kwezempilo, njll. Ngakolunye uhlangothi, lena i-REST API, futhi okunye, thina emafini sivame ukusebenzisa uhlaka lwe-gRPC, ngakho "sihumusha" i-REST kuya ku-gRPC bese sisebenzisa kuphela i-gRPC. Noma yisiphi isicelo siholela ekwakhiweni kochungechunge lwemisebenzi engabonakali evumelanayo eyenziwa echibini elivamile labasebenzi be-Yandex.Cloud. Imisebenzi ibhalwa ngendlela yokuthi ingamiswa noma nini bese iqalwa kabusha. Lokhu kuqinisekisa ukuqina, ukuphindaphinda nokungena kwemisebenzi.
Njengomphumela, umsebenzi ovela ku-API uzokwenza isicelo kusilawuli sesevisi yebhalansi, esibhalwe kokuthi Go. Ingakwazi ukwengeza futhi isuse abalinganisi, iguqule ukwakheka kwama-backends nezilungiselelo.
Isevisi igcina isimo sayo ku-Yandex Database, isizindalwazi esiphethwe ozokwazi ukusisebenzisa maduzane. Ku-Yandex.Cloud, njengoba sesivele vele , kuyasebenza umqondo wokudla kwezinja: uma thina ngokwethu sisebenzisa izinsiza zethu, amaklayenti ethu nawo azokujabulela ukuwasebenzisa. I-Yandex Database iyisibonelo sokuqaliswa komqondo onjalo. Sigcina yonke idatha yethu ku-YDB, futhi akudingekile ukuthi sicabange ngokugcina kanye nokukala isizindalwazi: lezi zinkinga zixazululiwe ngenxa yethu, sisebenzisa isizindalwazi njengesevisi.
Masibuyele kusilawuli sebhalansi. Umsebenzi wawo ukulondoloza ulwazi mayelana ne-balancer nokuthumela umsebenzi ukuze uhlole ukulungela komshini obonakalayo kusilawuli se-healthcheck.
Isilawuli se-Healthcheck
Ithola izicelo zokushintsha imithetho yesheke, izigcine ku-YDB, isabalalise imisebenzi phakathi kwamanodi e-healtcheck futhi ihlanganise imiphumela, bese igcinwa kusizindalwazi bese ithunyelwa kusilawuli se-loadbalancer. Futhi, ithumela isicelo sokushintsha ukwakheka kweqoqo endizeni yedatha ku-loadbalancer-node, engizoxoxa ngayo ngezansi.
Ake sikhulume kabanzi ngokuhlolwa kwezempilo. Zingahlukaniswa zibe amakilasi amaningana. Ukucwaningwa kwamabhuku kunemibandela yempumelelo eyahlukene. Ukuhlola kwe-TCP kudinga ukusungula uxhumano ngempumelelo phakathi nesikhathi esinqunyiwe. Ukuhlola i-HTTP kudinga kokubili uxhumano oluyimpumelelo kanye nempendulo enekhodi yesimo engu-200.
Futhi, amasheke ayahluka ekilasini lesenzo - ayasebenza futhi awasebenzi. Ukuhlola okwenziwayo kumane kuqaphe ukuthi kwenzekani ngethrafikhi ngaphandle kokuthatha isenzo esikhethekile. Lokhu akusebenzi kahle kakhulu ku-L4 ngoba kuncike ekucabangeni kwezivumelwano zezinga eliphezulu: ku-L4 alukho ulwazi mayelana nokuthi ukuhlinzwa kuthathe isikhathi esingakanani noma ukuthi ukuqedwa koxhumano bekukuhle noma kubi. Ukuhlola okusebenzayo kudinga ukuthi obhalansi athumele izicelo esimweni ngasinye seseva.
Iningi labalinganisi bomthwalo lizihlola ngokwazo ukuthi liyaphila. Kwamafu, sinqume ukuhlukanisa lezi zingxenye zesistimu ukuze sandise ukukala. Le ndlela izosivumela ukuthi sinyuse isibalo sababhalansi ngesikhathi sigcina inani lezicelo zokuhlolwa kwezempilo kusevisi. Ukuhlola kwenziwa ngamanodi okuhlola impilo ahlukene, lapho okuhlosiwe kokuhlola kuhlukaniswa khona futhi kuphindaphindwe. Awukwazi ukuhlola umsingathi oyedwa, njengoba kungase kuhluleke. Khona-ke ngeke sisithole isimo samacala awahlolile. Senza amasheke kunoma yiziphi izimo okungenani ama-healthcheck node amathathu. Sihlukanisa izinjongo zokuhlola phakathi kwamanodi sisebenzisa ama-algorithms we-hashing angashintshi.
Ukwehlukanisa ukulinganisa kanye nokuhlolwa kwezempilo kungaholela ezinkingeni. Uma i-healthcheck node yenza izicelo kusibonelo, ukudlula i-balancer (okwamanje engakhonzi ithrafikhi), khona-ke kuvela isimo esiyinqaba: insiza ibonakala iyaphila, kodwa ithrafikhi ngeke ifinyelele kuyo. Sixazulula le nkinga ngale ndlela: siqinisekisiwe ukuqalisa ithrafikhi yokuhlola impilo ngokusebenzisa ababhalansi. Ngamanye amazwi, uhlelo lokuhambisa amaphakethe ngethrafikhi evela kumakhasimende futhi kusuka kumasheke wezempilo ahluka kancane: kuzo zombili izimo, amaphakethe azofinyelela kubalinganisi, okuzobaletha ezinsizeni ezihlosiwe.
Umehluko ukuthi amaklayenti enza izicelo ku-VIP, kuyilapho abahloli bezempilo benza izicelo ku-RIP ngayinye. Kuvela inkinga ethokozisayo lapha: sinikeza abasebenzisi bethu ithuba lokudala izinsiza kumanethiwekhi we-IP ampunga. Ake sicabange ukuthi kukhona abanikazi bamafu ababili abahlukene abafihle izinsiza zabo ngemuva kwabalinganisi. Ngamunye wabo unezinsiza ku-subnet engu-10.0.0.1/24, enamakheli afanayo. Udinga ukwazi ukuhlukanisa ngandlela thize, futhi lapha udinga ukungena esakhiweni senethiwekhi ebonakalayo ye-Yandex.Cloud. Kungcono ukuthola imininingwane eyengeziwe ku , kubalulekile kithi manje ukuthi inethiwekhi inezendlalelo eziningi futhi inemigudu engahlukaniswa nge-id ye-subnet.
I-Healthcheck nodes ixhumana nabalinganisi besebenzisa lokho okubizwa ngamakheli e-quasi-IPv6. I-quasi-address ikheli le-IPv6 elinekheli le-IPv4 kanye ne-id subnet yomsebenzisi eshumekwe ngaphakathi kwalo. Ithrafikhi ifinyelela kusilinganisi, esikhipha ikheli lensiza ye-IPv4 kuyo, imisele i-IPv6 nge-IPv4 futhi ithumele iphakethe kunethiwekhi yomsebenzisi.
Ithrafikhi ehlehlayo ihamba ngendlela efanayo: ibhalansi ibona ukuthi indawo iyinethiwekhi empunga esuka kubahloli bezempilo, futhi iguqula i-IPv4 iye ku-IPv6.
I-VPP - inhliziyo yendiza yedatha
Ibhalansi isetshenziswa kusetshenziswa ubuchwepheshe be-Vector Packet Processing (VPP), uhlaka oluvela kwa-Cisco lokucubungula inqwaba yethrafikhi yenethiwekhi. Esimweni sethu, uhlaka lusebenza phezu kwelabhulali yokuphatha idivayisi yenethiwekhi yendawo yomsebenzisi - I-Data Plane Development Kit (DPDK). Lokhu kuqinisekisa ukusebenza okuphezulu kokucubungula iphakethe: iziphazamiso ezimbalwa kakhulu ezenzeka ku-kernel, futhi azikho izinguquko zomongo phakathi kwesikhala se-kernel nesikhala somsebenzisi.
I-VPP iqhubekela phambili futhi incipha ukusebenza okwengeziwe ngaphandle kwesistimu ngokuhlanganisa amaphakheji abe amaqoqo. Izinzuzo zokusebenza zivela ekusebenziseni okunamandla kwama-cache kumaphrosesa esimanje. Zombili izinqolobane zedatha zisetshenziswa (amaphakethe acutshungulwa "kuma-vectors", idatha isondele komunye nomunye) kanye nezinqolobane zomyalelo: ku-VPP, ukucubungula iphakethe kulandela igrafu, ama-node aqukethe imisebenzi eyenza umsebenzi ofanayo.
Isibonelo, ukucutshungulwa kwamaphakethe we-IP ku-VPP kwenzeka ngokulandelana okulandelayo: okokuqala, izihloko zephakethe zihlukaniswa endaweni yokuhlaziya, bese zithunyelwa ku-node, edlulisela phambili amaphakethe ngokuya ngamatafula omzila.
I-hardcore encane. Ababhali be-VPP abakubekezeleli ukuyekethisa ekusetshenzisweni kwe-caches processor, ngakho-ke ikhodi evamile yokucubungula i-vector yamaphakethe iqukethe i-vectorization ye-manual: kukhona i-loop yokucubungula lapho isimo esifana nokuthi "sinamaphakethe amane kulayini" sicutshungulwa, bese kuba okufanayo kokubili, bese - ngeyodwa. Imiyalo yokulanda kuqala ivamise ukusetshenziselwa ukulayisha idatha kunqolobane ukuze kusheshiswe ukufinyelela kuyo ngokuphindaphinda okulandelayo.
n_left_from = frame->n_vectors;
while (n_left_from > 0)
{
vlib_get_next_frame (vm, node, next_index, to_next, n_left_to_next);
// ...
while (n_left_from >= 4 && n_left_to_next >= 2)
{
// processing multiple packets at once
u32 next0 = SAMPLE_NEXT_INTERFACE_OUTPUT;
u32 next1 = SAMPLE_NEXT_INTERFACE_OUTPUT;
// ...
/* Prefetch next iteration. */
{
vlib_buffer_t *p2, *p3;
p2 = vlib_get_buffer (vm, from[2]);
p3 = vlib_get_buffer (vm, from[3]);
vlib_prefetch_buffer_header (p2, LOAD);
vlib_prefetch_buffer_header (p3, LOAD);
CLIB_PREFETCH (p2->data, CLIB_CACHE_LINE_BYTES, STORE);
CLIB_PREFETCH (p3->data, CLIB_CACHE_LINE_BYTES, STORE);
}
// actually process data
/* verify speculative enqueues, maybe switch current next frame */
vlib_validate_buffer_enqueue_x2 (vm, node, next_index,
to_next, n_left_to_next,
bi0, bi1, next0, next1);
}
while (n_left_from > 0 && n_left_to_next > 0)
{
// processing packets by one
}
// processed batch
vlib_put_next_frame (vm, node, next_index, n_left_to_next);
}Ngakho-ke, ama-Healthchecks akhuluma nge-IPv6 ku-VPP, ewaguqula abe yi-IPv4. Lokhu kwenziwa nge-node kugrafu, esiyibiza nge-algorithmic NAT. Kuthrafikhi ehlehlayo (kanye nokuguqulwa kusuka ku-IPv6 kuya ku-IPv4) kunenodi ye-algorithmic ye-NAT efanayo.
Ithrafikhi eqondile evela kumakhasimende ebhalansi idlula kuma-graph node, enza ukulinganisa ngokwawo.
I-node yokuqala amaseshini anamathelayo. Igcina i-hash ye kumaseshini amisiwe. I-5-tuple ihlanganisa ikheli nechweba leklayenti lapho ulwazi ludluliselwa khona, ikheli nezimbobo zezinsiza ezitholakalayo zokwamukela ithrafikhi, kanye nephrothokholi yenethiwekhi.
I-hashi ye-5-tuple isisiza ukuba senze ukubala okuncane ku-hashing node elandelayo, kanye nokubamba kangcono izinguquko zohlu lwensiza ngemuva kwesilinganisi. Uma iphakethe elingekho iseshini yalo lifika kusilinganisi, lithunyelwa ku-hashing node engaguquki. Kulapho ukulinganisa kwenzeka khona kusetshenziswa i-hashing engaguquki: sikhetha insiza ohlwini lwezinsiza ezitholakalayo "ezibukhoma". Okulandelayo, amaphakethe athunyelwa endaweni ye-NAT, ethatha indawo yekheli okuyiwa kuyo futhi ibala kabusha amasheke. Njengoba ubona, silandela imithetho ye-VPP - ukuthanda ukuthanda, ukuqoqa izibalo ezifanayo ukuze kwandiswe ukusebenza kahle kwama-cache okucubungula.
I-hashing engaguquki
Kungani siyikhethile futhi siyini? Okokuqala, ake sicabangele umsebenzi odlule - ukukhetha insiza ohlwini.
Nge-hashing engahambisani, i-hashi yephakethe engenayo iyabalwa, futhi insiza ikhethwa ohlwini ngensalela yokuhlukanisa le hashi ngenani lezisetshenziswa. Uma nje uhlu luhlala lungashintshiwe, lolu hlelo lusebenza kahle: sihlala sithumela amaphakethe ane-5-tuple efanayo esihlokweni esifanayo. Uma, ngokwesibonelo, insiza ethile iyeka ukuphendula ekuhlolweni kwezempilo, lapho-ke ingxenye ebalulekile yamaheshi ukukhetha kuzoshintsha. Izixhumanisi ze-TCP zeklayenti zizophulwa: iphakethe elifinyelele kusibonelo A lingase liqale ukufika kusibonelo B, elingajwayelene neseshini yaleli phakethe.
I-hashing engaguquki ixazulula inkinga echaziwe. Indlela elula yokuchaza lo mqondo yilena: cabanga ukuthi unendandatho owasabalalisa kuyo izinsiza nge-hashi (isibonelo, nge-IP:port). Ukukhetha insiza kungukuguqula isondo nge-engeli, enqunywa i-hashi yephakethe.
Lokhu kunciphisa ukusatshalaliswa kabusha kwethrafikhi lapho ukwakheka kwezisetshenziswa kushintsha. Ukususa insiza kuzothinta kuphela ingxenye ye-hashing ring engaguquki lapho isisetshenziswa besitholakala khona. Ukwengeza insiza nakho kushintsha ukusatshalaliswa, kodwa sinenodi yeseshini enamathelayo, esivumela ukuthi singashintshi amaseshini asevele amisiwe aye kuzinsiza ezintsha.
Sibheke ukuthi kwenzekani ukuqondisa ithrafikhi phakathi kwe-balancer nezinsiza. Manje ake sibheke ithrafikhi ebuyayo. Ilandela iphethini efanayo nethrafikhi yokuhlola - nge-algorithmic NAT, okungukuthi, ngokuhlehla i-NAT 44 yethrafikhi yeklayenti kanye ne-NAT 46 yethrafikhi yokuhlolwa kwezempilo. Sinamathela esimisweni sethu: sihlanganisa ithrafikhi yokuhlola impilo kanye nethrafikhi yangempela yabasebenzisi.
I-Loadbalancer-node kanye nezingxenye ezihlanganisiwe
Ukwakhiwa kwabalinganisi nezinsiza ku-VPP kubikwa isevisi yendawo - i-loadbalancer-node. Ibhalisela ukusakazwa kwemicimbi kusuka kusilawuli se-loadbalancer futhi iyakwazi ukuhlela umehluko phakathi kwesimo samanje se-VPP nesimo esiqondiwe esitholwe kusilawuli. Sithola uhlelo oluvaliwe: imicimbi evela ku-API iza kusilawuli sebhalansi, esinikeza isilawuli se-healthcheck imisebenzi ukuze sihlole βukuba bukhomaβ kwezinsiza. Lokho, nakho, kwabela imisebenzi ku-healthcheck-node futhi kuhlanganisa imiphumela, ngemva kwalokho iyibuyisele kusilawuli sebhalansi. I-Loadbalancer-node ibhalisela imicimbi evela kusilawuli futhi ishintsha isimo se-VPP. Kulolu hlelo lokusebenza, isevisi ngayinye yazi kuphela okudingekayo mayelana nezinsizakalo ezingomakhelwane. Inombolo yokuxhumana inomkhawulo futhi sinamandla okusebenza nokukala amasegimenti ahlukene ngokuzimela.
Yiziphi izinkinga ezagwenywa?
Wonke amasevisi ethu endizeni yokulawula abhalwe kokuthi Go futhi anezici ezinhle zokukala nokuthembeka. I-Go inemitapo yolwazi eminingi evulekile yokwakha amasistimu asabalalisiwe. Sisebenzisa ngokugcwele i-GRPC, zonke izingxenye ziqukethe ukusetshenziswa komthombo ovulekile wokutholwa kwesevisi - amasevisi ethu aqapha ukusebenza komunye nomunye, angashintsha ukwakheka kwawo ngokushintshashintshayo, futhi sikuxhume lokhu nokulinganisa kwe-GRPC. Kumamethrikhi, sisebenzisa nesixazululo somthombo ovulekile. Endizeni yedatha, sithole ukusebenza okuhloniphekile kanye nendawo enkulu yokugcina imithombo: kube nzima kakhulu ukuhlanganisa isitendi esingathembela kuso ekusebenzeni kwe-VPP, kunekhadi lenethiwekhi yensimbi.
Izinkinga Nezixazululo
Yini engasebenzi kahle? I-Go inokuphathwa kwememori okuzenzakalelayo, kodwa ukuvuza kwememori kusenzeka. Indlela elula yokubhekana nazo ukusebenzisa ama-goroutines futhi ukhumbule ukuwaqeda. I-Takeaway: Buka ukusetshenziswa kwememori yezinhlelo zakho ze-Go. Ngokuvamile inkomba enhle yinani lama-goroutines. Kukhona okuhlanganisayo kule ndaba: ku-Go kulula ukuthola idatha yesikhathi sokusebenza - ukusetshenziswa kwememori, inani lama-goroutine asebenzayo, namanye amapharamitha amaningi.
Futhi, i-Go ingase ingabi inketho engcono kakhulu yokuhlolwa kokusebenza. Ziyi-verbose impela, futhi indlela ejwayelekile "yokusebenzisa yonke into ku-CI ngenqwaba" ayibafanele kakhulu. Iqiniso liwukuthi ukuhlola okusebenzayo kufuna kakhulu izinsiza futhi kubangela ukuphela kwesikhathi kwangempela. Ngenxa yalokhu, ukuhlola kungase kuhluleke ngoba i-CPU imatasa nokuhlolwa kweyunithi. Isiphetho: Uma kungenzeka, yenza izivivinyo βezisindayoβ ngokwehlukana nezivivinyo zamayunithi.
Ukwakhiwa komcimbi we-Microservice kuyinkimbinkimbi kakhulu kune-monolith: ukuqoqa izingodo emishinini eminingi ehlukene akulula kakhulu. Isiphetho: uma wenza ama-microservices, cabanga ngokushesha ngokulandelela.
Izinhlelo zethu
Sizokwethula ibhalansi yangaphakathi, ibhalansi ye-IPv6, sengeze usekelo lwemibhalo ye-Kubernetes, siqhubeke nokwaba izinsizakalo zethu (okwamanje yi-healthcheck-node kuphela ne-healthcheck-ctrl ehlukanisiwe), sengeze ukuhlolwa kwezempilo okusha, futhi sisebenzise ukuhlanganisa amasheke okuhlakaniphile. Sicabanga ukuthi kungenzeka senze amasevisi ethu azimele nakakhulu - ukuze angaxhumani ngokuqondile, kodwa asebenzise umugqa wemilayezo. Isevisi ehambisana ne-SQS isanda kuvela ku-Cloud .
Muva nje, ukukhululwa komphakathi kwe-Yandex Load Balancer kwenzeka. Hlola enkonzweni, phatha abalinganisi ngendlela ekulungele futhi wandise ukubekezelelana kwamaphutha amaphrojekthi akho!
Source: www.habr.com