Molo, ndinguSergey Elantsev, ndiyaphuhlisa kwiYandex.Cloud. Ngaphambili, ndikhokele ukuphuhliswa kwe-balancer ye-L7 kwi-portal ye-Yandex - oogxa bahlekisa ukuba kungakhathaliseki ukuba ndenza ntoni, kuya kuba yi-balancer. Ndiza kuxelela abafundi bakaHabr indlela yokulawula umthwalo kwiqonga lelifu, into esiyibonayo njengesixhobo esifanelekileyo sokufezekisa le njongo, kunye nendlela esihamba ngayo ekwakheni esi sixhobo.
Okokuqala, makhe sazise amagama athile:
- I-VIP (i-IP ebonakalayo) - idilesi ye-IP yokulinganisa
- Umncedisi, umva, umzekelo - umatshini wenyani osebenzisa usetyenziso
- I-RIP (I-IP yokwenene) - idilesi ye-IP yomncedisi
- Healthcheck - ukujonga ukulungela umncedisi
- Indawo yokufumaneka, i-AZ - iziseko ezizimeleyo kwiziko ledatha
- Ummandla - umanyano lwee-AZs ezahlukeneyo
Abalinganisi bomthwalo basombulula imisebenzi emithathu ephambili: benza ukulinganisa ngokwayo, baphucula ukunyamezela kwesiphoso senkonzo, kunye nokwenza lula ukukala kwayo. Ukunyamezela iziphene kuqinisekiswa ngolawulo oluzenzekelayo lwetrafikhi: umlinganisi ubeka iliso kwimeko yesicelo kwaye ungabandakanyi iimeko ukusuka ekulinganiseni okungaphumeleli ukukhangela ukuphila. Ukukala kuqinisekiswa ngokusasazwa komthwalo ngokulinganayo kuzo zonke iimeko, kunye nokuhlaziya uluhlu lweziganeko ngokubhabha. Ukuba ulungelelwaniso alufani ngokwaneleyo, ezinye iimeko ziya kufumana umthwalo odlula umda wabo wamandla, kwaye inkonzo iya kuba yinto ethembekileyo.
Isilinganisi somthwalo sihlala sihlelwa ngumaleko weprotocol ukusuka kwimodeli yeOSI esebenza kuyo. I-Cloud Balancer isebenza kwinqanaba le-TCP, elihambelana noluhlu lwesine, i-L4.
Masiqhubele phambili kwisishwankathelo se-Cloud balancer Architecture. Siza kwandisa ngokuthe ngcembe inqanaba leenkcukacha. Sahlula amacandelo e-balancer kwiiklasi ezintathu. Iklasi ye-config plane inoxanduva lokusebenzisana komsebenzisi kwaye igcina imeko ekujoliswe kuyo yenkqubo. Inqwelomoya yokulawula igcina imeko yangoku yenkqubo kwaye ilawula iinkqubo ezivela kwiklasi yendiza yedatha, ejongene ngqo nokuhambisa i-traffic kubathengi kwiimeko zakho.
Inqwelomoya yedatha
I-traffic iphelela kwizixhobo ezibizayo ezibizwa ngokuba yi-border routers. Ukwandisa ukunyamezela kweempazamo, ezininzi izixhobo ezinjalo zisebenza ngaxeshanye kwiziko ledatha enye. Okulandelayo, i-traffic iya kubalinganisi, ababhengeza nayiphi na idilesi ye-IP kuzo zonke ii-AZs nge-BGP kubathengi.
I-Traffic isasazwa ngaphezulu kwe-ECMP - esi sisicwangciso-qhinga esinokuthi sibe neendlela ezininzi ezifanelekileyo ngokulinganayo kwithagethi (kwimeko yethu, ekujoliswe kuyo kuya kuba yidilesi ye-IP yendawo ekusingwa kuyo) kwaye iipakethi zingathunyelwa kunye nayo nayiphi na kuzo. Sikwaxhasa umsebenzi kwiindawo ezininzi zokufumaneka ngokwesikimu esilandelayo: sithengisa idilesi kwindawo nganye, i-traffic iya kwindawo ekufutshane kwaye ayihambi ngaphaya kwemida yayo. Kamva kwisithuba siza kujonga ngokubanzi ukuba kwenzekani kwi-traffic.
Qwalasela inqwelomoya
Inxalenye ephambili ye-config plane yi-API, apho imisebenzi esisiseko kunye nabalinganisi benziwa: ukudala, ukucima, ukutshintsha ukubunjwa kweemeko, ukufumana iziphumo zezempilo, njl. Kwelinye icala, le yi-REST API, kwaye enye, thina kwiLifu sihlala sisebenzisa isakhelo se-gRPC, ke "siguqulela" REST kwi-gRPC kwaye sisebenzise kuphela i-gRPC. Nasiphi na isicelo sikhokelela ekudalweni koluhlu lwemisebenzi engabonakaliyo ye-asynchronous eyenziwa kwi-pool eqhelekileyo yabasebenzi baseYandex.Cloud. Imisebenzi ibhalwa ngendlela yokuba inokunqunyanyiswa nangaliphi na ixesha ize iqaliswe kwakhona. Oku kuqinisekisa ukuqina, ukuphinda-phinda kunye nokugawulwa kwemisebenzi.
Ngenxa yoko, umsebenzi ovela kwi-API uya kwenza isicelo kumlawuli wenkonzo ye-balancer, ebhalwe kwi-Go. Inokongeza kwaye isuse abalinganisi, itshintshe ukubunjwa kwee-backends kunye nezicwangciso.
Inkonzo igcina imeko yayo kwiYandex Database, i-database elawulwayo esasazwayo oya kukwazi ukuyisebenzisa ngokukhawuleza. KwiYandex.Cloud, njengoko sele sikhona , ingcamango yokutya kwenja iyasebenza: ukuba thina ngokwethu sisebenzisa iinkonzo zethu, ngoko abathengi bethu baya kukuvuyela ukuzisebenzisa. Yandex Database ngumzekelo wokuphunyezwa kwengcamango enjalo. Sigcina yonke idatha yethu kwi-YDB, kwaye akufuneki sicinge malunga nokugcina kunye nokulinganisa i-database: ezi ngxaki zixazululwa kuthi, sisebenzisa i-database njengenkonzo.
Masibuyele kwi-balancer controller. Umsebenzi wayo kukugcina ulwazi malunga ne-balancer kunye nokuthumela umsebenzi ukujonga ukulungela umatshini wenyani kumlawuli wezempilo.
Healthcheck isilawuli
Ifumana izicelo zokutshintsha imigaqo yokutshekisha, igcine kwi-YDB, isasaze imisebenzi phakathi kweenodi ze-healtcheck kwaye idibanise iziphumo, eziye zigcinwe kwisiseko sedatha kwaye zithunyelwe kumlawuli we-loadbalancer. Yona, ithumela isicelo sokutshintsha ukubunjwa kweqela kwi-plane yedatha kwi-loadbalancer-node, endiya kuxoxa ngayo ngezantsi.
Makhe sithethe ngakumbi malunga nokuhlolwa kwezempilo. Zinokwahlulwa zibe ziiklasi ezininzi. Uphicotho luneekhrayitheriya ezahlukeneyo zempumelelo. Iitshekhi ze-TCP zifuna ukuseka ngempumelelo unxibelelwano phakathi kwexesha elimiselweyo. Ukuhlolwa kwe-HTTP kufuna zombini uxhulumaniso oluyimpumelelo kunye nempendulo ngekhowudi yesimo se-200.
Kwakhona, iitshekhi ziyahluka kudidi lwesenzo - ziyasebenza kwaye ziyasebenza. Iitshekhi zijonga nje ukuba kwenzeka ntoni ngetrafikhi ngaphandle kokuthatha naliphi na inyathelo elikhethekileyo. Oku akusebenzi kakuhle kakhulu kwi-L4 kuba kuxhomekeke kwingqiqo yeprothokholi ezikwinqanaba eliphezulu: kwi-L4 akukho lwazi malunga nokuba uqhagamshelo luthathe ixesha elingakanani okanye ukuba ukugqityezelwa koqhagamshelwano bekukuhle okanye kubi. Iitshekhi ezisebenzayo zifuna ukuba umlinganisi athumele izicelo kumzekelo ngamnye weseva.
Uninzi lwezilinganisi zomthwalo ziyazihlola ngokwazo. Kwi-Cloud, sagqiba ekubeni sahlule ezi nxalenye zenkqubo ukunyusa ukulinganisa. Le ndlela iza kusivumela ukuba sandise inani labalinganisi ngelixa sigcina inani lezicelo zokuhlolwa kwezempilo kwinkonzo. Iitshekhi zenziwa ngoogqirha bezempilo abahlukeneyo, apho iithagethi zetshekhi zihlulwe kwaye ziphindaphindwe. Awunako ukwenza iitshekhi kwinginginya enye, njengoko inokusilela. Emva koko asiyi kufumana imeko yeemeko awayezijongile. Senza uhlolo kuzo naziphi na iimeko ukusuka ubuncinane kwiindawo ezintathu zokuhlola impilo. Sabelana ngeenjongo zokuhlola phakathi kweendawo zisebenzisa i-algorithms ye-hashing engaguqukiyo.
Ukwahlula ukulinganisa kunye nokuhlolwa kwezempilo kunokukhokelela kwiingxaki. Ukuba i-node ye-healthcheck yenza izicelo kumzekelo, ukudlula i-balancer (engasetyenziswanga ngoku i-traffic), ngoko kuvela imeko engaqhelekanga: ubutyebi bubonakala buphila, kodwa i-traffic ayiyi kufikelela kuyo. Le ngxaki siyisombulula ngolu hlobo: siqinisekisiwe ukuba siza kuqalisa ukuxilonga impilo ngokusebenzisa abalinganisi. Ngamanye amazwi, iskimu sokuhambisa iipakethi kunye ne-traffic evela kubaxhasi kunye nokusuka kwii-healthchecks zihluke kancinci: kuzo zombini iimeko, iipakethi ziya kufikelela kubalinganisi, eziza kuzinikezela kwizibonelelo ezijoliswe kuyo.
Umahluko kukuba abathengi benza izicelo kwi-VIP, ngelixa i-healthchecks yenza izicelo kwi-RIP nganye. Ingxaki enomdla ivela apha: sinika abasebenzisi bethu ithuba lokudala izixhobo kwiinethiwekhi ze-IP ezingwevu. Makhe sicinge ukuba kukho abanini bamafu ababini abahlukeneyo abafihle iinkonzo zabo ngasemva kwabalinganisi. Ngamnye wabo unezibonelelo kwi-10.0.0.1/24 subnet, kunye needilesi ezifanayo. Kufuneka ukwazi ngandlela ithile ukubahlula, kwaye apha kufuneka ungene kwisakhiwo seYandex.Cloud yenethiwekhi yenyani. Kungcono ukufumana iinkcukacha ezingaphezulu kwi , kubalulekile kuthi ngoku ukuba inethiwekhi ine-multi-layered kwaye ineetonela ezinokuhlukaniswa nge-subnet id.
I-Healthcheck node zoqhagamshelwano zabalinganisi zisebenzisa into ebizwa ngokuba ziidilesi ze-quasi-IPv6. Idilesi eyi-quasi yidilesi ye-IPv6 enedilesi ye-IPv4 kunye ne-id subnet yomsebenzisi efakwe ngaphakathi kuyo. I-traffic ifikelela kwi-balancer, ekhupha idilesi ye-IPv4 yomthombo kuyo, ithatha indawo ye-IPv6 nge-IPv4 kwaye ithumela ipakethi kwinethiwekhi yomsebenzisi.
I-traffic ebuyayo ihamba ngendlela efanayo: i-balancer ibona ukuba indawo ekuyiwa kuyo yinethiwekhi engwevu ukusuka kwi-healthcheckers, kwaye iguqula i-IPv4 ukuya kwi-IPv6.
I-VPP - intliziyo yendiza yedatha
I-balancer iphunyezwa ngokusebenzisa i-teknoloji ye-Vector Packet Processing (VPP), isakhelo esivela kwi-Cisco ye-batch processing ye-traffic network. Kwimeko yethu, isakhelo sisebenza phezulu kwilayibrari yolawulo lwesixhobo senethiwekhi yomsebenzisi-isithuba seDatha yoPhuhliso lweDatha (DPDK). Oku kuqinisekisa ukusebenza kwepakethi ephezulu yokusetyenzwa: kuncinci ukuphazamiseka okwenzekayo kwi-kernel, kwaye akukho kutshintsha komxholo phakathi kwendawo ye-kernel kunye nesithuba somsebenzisi.
I-VPP iqhubela phambili kwaye icinezele ukusebenza ngakumbi ngaphandle kwenkqubo ngokudibanisa iipakethe kwiibhetshi. Iinzuzo zokusebenza zivela kusetyenziso olungqongqo lwee-cache kwiiprosesa zanamhlanje. Zomibini ii-cache zedatha zisetyenzisiweyo (iipakethi zicutshungulwa "kwi-vectors", idatha isondele komnye nomnye) kunye nee-caches zomyalelo: kwi-VPP, ukusetyenzwa kwepakethe kulandela igrafu, ii-nodes eziqulethe imisebenzi eyenza umsebenzi ofanayo.
Ngokomzekelo, ukucutshungulwa kweepakethi ze-IP kwi-VPP kwenzeka ngolu hlobo lulandelayo: okokuqala, iintloko zeepakethi zihlanjululwe kwi-node yokuhlaziya, kwaye emva koko zithunyelwa kwi-node, ehambisa iipakethi phambili ngokuhambelana neetafile zomzila.
I-hardcore encinci. Ababhali be-VPP abanyamezeli ukuxhatshazwa ekusebenziseni i-cache yeprosesa, ngoko ke ikhowudi eqhelekileyo yokucubungula i-vector yeepakethi iqulethe i-vectorization ye-manual: kukho i-loop yokucubungula apho imeko efana "sineepakethi ezine emgceni" iqhutywe, kwaze kwafana kwababini, emva koko - kwenye. Imiyalelo yokulanda isoloko isetyenziselwa ukulayisha idatha kwiicache ukukhawulezisa ufikelelo kuzo kuphindaphindo olulandelayo.
n_left_from = frame->n_vectors;
while (n_left_from > 0)
{
vlib_get_next_frame (vm, node, next_index, to_next, n_left_to_next);
// ...
while (n_left_from >= 4 && n_left_to_next >= 2)
{
// processing multiple packets at once
u32 next0 = SAMPLE_NEXT_INTERFACE_OUTPUT;
u32 next1 = SAMPLE_NEXT_INTERFACE_OUTPUT;
// ...
/* Prefetch next iteration. */
{
vlib_buffer_t *p2, *p3;
p2 = vlib_get_buffer (vm, from[2]);
p3 = vlib_get_buffer (vm, from[3]);
vlib_prefetch_buffer_header (p2, LOAD);
vlib_prefetch_buffer_header (p3, LOAD);
CLIB_PREFETCH (p2->data, CLIB_CACHE_LINE_BYTES, STORE);
CLIB_PREFETCH (p3->data, CLIB_CACHE_LINE_BYTES, STORE);
}
// actually process data
/* verify speculative enqueues, maybe switch current next frame */
vlib_validate_buffer_enqueue_x2 (vm, node, next_index,
to_next, n_left_to_next,
bi0, bi1, next0, next1);
}
while (n_left_from > 0 && n_left_to_next > 0)
{
// processing packets by one
}
// processed batch
vlib_put_next_frame (vm, node, next_index, n_left_to_next);
}Ke, iiHealthchecks zithetha nge-IPv6 ukuya kwi-VPP, ezijika zibe yi-IPv4. Oku kwenziwa nge-node kwigrafu, esiyibiza ngokuba yi-algorithmic NAT. Ukubuyisela umva itrafikhi (kunye nokuguqulwa ukusuka kwi-IPv6 ukuya kwi-IPv4) kukho i-algorithmic node ye-NAT efanayo.
I-traffic ngqo evela kubaxhasi be-balancer ihamba ngeegrafu zegrafu, ezenza ukulinganisa ngokwayo.
Indawo yokuqala ziiseshoni ezincangathi. Igcina ihashi ye kwiiseshoni ezimiselweyo. I-5-tuple ibandakanya idilesi kunye nechweba lomxhasi apho ulwazi ludluliselwa khona, idilesi kunye namachweba ezixhobo ezikhoyo zokufumana i-traffic, kunye neprotocol yenethiwekhi.
I-5-tuple hash isinceda senze ubalo oluncinci kwi-hashing node elandelayo, kunye noluhlu olungcono lokuphatha utshintsho emva kwesikali. Xa ipakethi ekungekho seshoni ifika kwi-balancer, ithunyelwa kwi-hashing node ehambelanayo. Kulapho ulungelelwaniso lwenzeka kusetyenziswa i-hashing engaguqukiyo: sikhetha uvimba kuluhlu lwemithombo ekhoyo “ephilayo”. Emva koko, iipakethi zithunyelwa kwindawo ye-NAT, eneneni ithatha indawo yedilesi yendawo kwaye iphinda ibale iitshekhi. Njengoko ubona, silandela imithetho ye-VPP-ukuthanda ukuthanda, ukwahlula izibalo ezifanayo ukwandisa ukusebenza kakuhle kwee-cache zeprosesa.
I-hashing ehambelanayo
Kutheni siyikhethile kwaye yintoni na? Okokuqala, makhe siqwalasele umsebenzi wangaphambili - ukukhetha uvimba kuluhlu.
Nge-hashing engahambelaniyo, i-hash yepakethi engenayo ibalwa, kwaye isibonelelo sikhethwa kuluhlu ngentsalela yokwahlula le hashi ngenani lezibonelelo. Ngethuba nje uluhlu luhlala lungatshintshi, esi sikimu sisebenza kakuhle: sihlala sithumela iipakethi ezine-5-tuple efanayo kumzekelo ofanayo. Ukuba, umzekelo, ezinye izixhobo ziye zayeka ukuphendula kwi-healthchecks, ngoko inxalenye ebalulekileyo ye-hashes ukhetho luya kutshintsha. Uqhagamshelo lwe-TCP yomxhasi luya kwaphulwa: ipakethi efikelelweyo ngaphambili umzekelo A inokuqala ukufikelela kumzekelo B, ongaqhelananga neseshoni yale pakethi.
I-hashing ehambelanayo isombulula ingxaki echazwe. Eyona ndlela ilula yokuchaza le ngcamango yile: cinga ukuba unendandatho osasaza kuyo izixhobo nge-hash (umzekelo, nge-IP: port). Ukukhetha ubutyebi kukuguqula ivili nge-angle, echongwa yi-hash yepakethi.
Oku kunciphisa ukwabiwa kwakhona kwezithuthi xa ukubunjwa kwezibonelelo kutshintsha. Ukucima isibonelelo kuya kuchaphazela kuphela inxalenye yeringi ye-hashing engaguqukiyo apho uvimba wawubekwe khona. Ukongeza iresource kukwatshintsha ukuhanjiswa, kodwa sineeseshoni ezincangathi, ezisivumela ukuba singatshintshi iiseshoni esele zisekiwe kwizixhobo ezintsha.
Sijonge ukuba kwenzekani kwi-traffic ngqo phakathi kwe-balancer kunye nezibonelelo. Ngoku makhe sijonge ukubuya kwetrafikhi. Ilandela ipateni efanayo njengetshekhi yetrafikhi - nge-algorithmic NAT, oko kukuthi, nge-reverse NAT 44 yetrafikhi yabaxhasi kunye ne-NAT 46 yetrafikhi yezempilo. Sithobela isikimu sethu: sidibanisa itrafikhi yezempilo kunye nokugcwala kwabasebenzisi bokwenyani.
I-Loadbalancer-node kunye namacandelo adibeneyo
Ukubunjwa kwabalinganisi kunye nezibonelelo kwi-VPP kuchazwe yinkonzo yendawo - i-loadbalancer-node. Ibhalisa kwimijelo yeziganeko ukusuka kwi-loadbalancer-controller kwaye iyakwazi ukucwangcisa umehluko phakathi kwe-VPP yangoku kunye nesimo esijoliswe kuyo esifunyenwe kumlawuli. Sifumana inkqubo evaliweyo: iziganeko ezivela kwi-API ziza kumlawuli we-balancer, onikezela imisebenzi kumlawuli wezempilo ukujonga "ukuphila" kwezibonelelo. Oko ke, kwabela imisebenzi kwi-healthcheck-node kwaye idibanise iziphumo, emva koko izibuyisele kumlawuli we-balancer. I-Loadbalancer-node ibhalisela iziganeko ezivela kumlawuli kwaye itshintshe imeko yeVPP. Kwinkqubo enjalo, inkonzo nganye iyazi kuphela into efunekayo malunga neenkonzo zommelwane. Inani loqhagamshelwano lilinganiselwe kwaye sinamandla okusebenza kunye nokulinganisa amacandelo ahlukeneyo ngokuzimeleyo.
Yiyiphi imiba eyayiphetshwa?
Zonke iinkonzo zethu kwinqwelomoya yokulawula zibhalwe kwi-Go kwaye zineempawu ezintle zokulinganisa kunye nokuthembeka. I-Go inamathala eencwadi amaninzi avulelekileyo okwakha iinkqubo ezisasazwayo. Siyisebenzisa ngenkuthalo i-GRPC, onke amacandelo aqulathe uphumezo lomthombo ovulelekileyo wokufunyanwa kwenkonzo - iinkonzo zethu zijongana nokusebenza kwazo, zinokutshintsha ukuqamba kwazo ngokutshintshatshintshayo, kwaye siye sanxulumanisa oku nokulungelelaniswa kwe-GRPC. Kwiimetrics, sikwasebenzisa isisombululo somthombo ovulekileyo. Kwinqwelo-moya yedatha, sifumene ukusebenza okuhloniphekileyo kunye novimba omkhulu wemithombo: kwaba nzima kakhulu ukudibanisa ukuma apho sinokuthembela ekusebenzeni kweVPP, kunekhadi lenethiwekhi yentsimbi.
Iingxaki kunye nezisombululo
Yintoni engazange isebenze kakuhle? I-Go inolawulo lwememori oluzenzekelayo, kodwa ukuvuza kwememori kusenzeka. Eyona ndlela ilula yokujongana nazo kukuqhuba ii-goroutines kwaye ukhumbule ukuziphelisa. Thatha: Jonga ukusetyenziswa kwememori kweenkqubo zakho. Amaxesha amaninzi isalathisi esilungileyo linani lee-goroutines. Kukho ukudibanisa kweli bali: kwi-Go kulula ukufumana idatha yexesha lokusebenza - ukusetyenziswa kwememori, inani lee-goroutines ezisebenzayo, kunye nezinye iiparameters ezininzi.
Kwakhona, i-Go isenokungabi lolona khetho lulungileyo lovavanyo olusebenzayo. Zizi-verbose, kwaye indlela esemgangathweni "yokuqhuba yonke into kwi-CI kwibhetshi" ayibafanelanga kakhulu. Inyani yeyokuba iimvavanyo ezisebenzayo zifuna kakhulu izixhobo kwaye zibangela ukuphelelwa lixesha kwangempela. Ngenxa yoku, iimvavanyo zinokusilela kuba iCPU ixakeke ziimvavanyo zeyunithi. Isiphelo: Ukuba kuyenzeka, yenza uvavanyo “olunzima” ngokwahlukileyo kuvavanyo lweyunithi.
Uyilo lweminyhadala yeMicroservice luntsonkothile ngakumbi kune-monolith: ukuqokelela izingodo kuninzi koomatshini abahlukeneyo akulula kakhulu. Isiphelo: ukuba wenza i-microservices, cinga ngokukhawuleza malunga nokulandela umkhondo.
Izicwangciso zethu
Siza kuphehlelela i-balancer yangaphakathi, i-balancer ye-IPv6, songeze inkxaso yeskripthi se-Kubernetes, siqhubeke nokwabelana ngeenkonzo zethu (okwangoku kuphela i-healthcheck-node kunye ne-healthcheck-ctrl ezahluliwe), yongeza i-healthchecks entsha, kwaye siphumeze ukudityaniswa kweetshekhi. Sicinga ukuba kunokwenzeka ukwenza iinkonzo zethu zizimele ngakumbi-ukuze zinxibelelane ngokuthe ngqo omnye nomnye, kodwa zisebenzisa umgca womyalezo. Inkonzo ehambelana ne-SQS isanda kuvela kwiLifu .
Kungekudala, ukukhululwa koluntu kweYandex Load Balancer kwenzeka. Phonononga kwinkonzo, lawula abalinganisi ngendlela ekulungeleyo kwaye wandise ukunyamezela iimpazamo kwiiprojekthi zakho!
umthombo: www.habr.com