Moni, ndine Sergey Elantsev, ndimakula mu Yandex.Cloud. M'mbuyomu, ndidatsogolera chitukuko cha L7 balancer ya Yandex portal - anzanga nthabwala kuti ziribe kanthu zomwe ndingachite, zimakhala zowerengera. Ndiuza owerenga a Habr momwe angayendetsere katundu pamtambo wamtambo, zomwe tikuwona ngati chida choyenera chokwaniritsira cholinga ichi, ndi momwe tikuyendera pomanga chida ichi.
Choyamba, tiyeni titchule mawu ena:
- VIP (Virtual IP) - adilesi ya IP ya balancer
- Seva, backend, chitsanzo - makina enieni omwe ali ndi ntchito yothamanga
- RIP (Real IP) - adilesi ya IP ya seva
- Healthcheck - kuyang'ana kukonzekera kwa seva
- Availability Zone, AZ - maziko akutali mu data center
- Dera - mgwirizano wa ma AZ osiyanasiyana
Ma balancers amathetsa ntchito zazikulu zitatu: amadzilinganiza okha, amawongolera kulolerana kwantchito, ndikuchepetsa kukulitsa kwake. Kulekerera zolakwika kumatsimikiziridwa kudzera mu kasamalidwe ka magalimoto okha: wolinganiza amayang'anira momwe ntchito ikugwiritsidwira ntchito ndikuchotsa zochitika zomwe sizikudutsa cheke. Kuchulukitsa kumatsimikizidwa pogawa katunduyo mofananamo nthawi zonse, komanso kukonzanso mndandanda wa zochitika pa ntchentche. Ngati kusanja sikuli kofanana mokwanira, zochitika zina zidzalandira katundu wopitirira malire awo, ndipo ntchitoyo idzakhala yodalirika.
Cholemetsa cholemetsa nthawi zambiri chimasankhidwa ndi gawo la protocol kuchokera ku mtundu wa OSI womwe umayendera. Cloud Balancer imagwira ntchito pamlingo wa TCP, womwe umagwirizana ndi gawo lachinayi, L4.
Tiyeni tipitirire ku chithunzithunzi cha kamangidwe ka Cloud balancer. Tidzawonjezera pang'onopang'ono mlingo wa tsatanetsatane. Timagawaniza zigawo za balancer m'magulu atatu. Gulu la config plane limayang'anira kuyanjana kwa ogwiritsa ntchito ndikusunga momwe dongosololi likufunira. Ndege yowongolera imasunga momwe dongosololi lilili pano ndikuwongolera machitidwe kuchokera kugulu la ndege za data, omwe ali ndi udindo wopereka magalimoto kuchokera kwa makasitomala kupita kuzochitika zanu.
Ndege ya data
Magalimoto amathera pazida zodula zotchedwa ma border routers. Kuti muwonjezere kulolerana kwa zolakwika, zida zingapo zotere zimagwira ntchito nthawi imodzi pamalo amodzi a data. Kenako, magalimoto amapita kwa owerengera, omwe amalengeza ma adilesi a IP aliwonse ku ma AZ onse kudzera pa BGP kwa makasitomala.
Magalimoto amafalitsidwa kudzera pa ECMP - iyi ndi njira yoyendetsera malinga ndi momwe pangakhale njira zingapo zabwino zolowera komwe mukufuna (kwa ife, chandamale ndi adilesi ya IP) ndipo mapaketi amatha kutumizidwa pamtundu uliwonse. Timathandiziranso ntchito m'magawo angapo opezeka molingana ndi dongosolo ili: timalengeza adilesi m'dera lililonse, magalimoto amapita kufupi ndipo sadutsa malire ake. Pambuyo pake mu positi tiwona mwatsatanetsatane zomwe zimachitika pamagalimoto.
Konzani ndege
Gawo lofunika kwambiri pa ndege yokonzekera ndi API, yomwe ntchito zoyambira ndi owerengera zimachitidwa: kupanga, kuchotsa, kusintha mawonekedwe a zochitika, kupeza zotsatira za thanzi, ndi zina zotero. Mbali imodzi, iyi ndi REST API, ndi pa kwina, ife mumtambo nthawi zambiri timagwiritsa ntchito chimango cha gRPC, kotero "timamasulira" REST ku gRPC kenako kugwiritsa ntchito gRPC yokha. Pempho lililonse limapangitsa kuti pakhale ntchito zingapo zopanda pake zomwe zimachitidwa padziwe wamba la antchito a Yandex.Cloud. Ntchito zimalembedwa m'njira yoti zitha kuyimitsidwa nthawi iliyonse ndikuyambiranso. Izi zimatsimikizira scalability, kubwereza komanso kudulidwa kwa ntchito.
Chotsatira chake, ntchito yochokera ku API idzapempha kwa woyang'anira utumiki wa balancer, zomwe zalembedwa mu Go. Ikhoza kuwonjezera ndi kuchotsa zowerengera, kusintha mapangidwe a backends ndi zoikamo.
Utumikiwu umasunga dziko lake mu Yandex Database, malo osungirako omwe amagawidwa omwe mudzatha kugwiritsa ntchito posachedwa. Mu Yandex.Cloud, monga ife kale , lingaliro la chakudya cha agalu likugwiritsidwa ntchito: ngati ife eni timagwiritsa ntchito mautumiki athu, ndiye kuti makasitomala athu adzakhala okondwa kuzigwiritsa ntchito. Yandex Database ndi chitsanzo cha kukhazikitsidwa kwa lingaliro lotere. Timasunga zidziwitso zathu zonse mu YDB, ndipo sitiyenera kuganiza za kusunga ndi kukulitsa nkhokwe: mavutowa amathetsedwa kwa ife, timagwiritsa ntchito nkhokwe ngati ntchito.
Tiyeni tibwerere ku controller. Ntchito yake ndikusunga zambiri za balancer ndikutumiza ntchito yoyang'ana kukonzekera kwa makinawo kwa woyang'anira zaumoyo.
Healthcheck controller
Imalandira zopempha kuti zisinthe malamulo a cheke, zimawapulumutsa ku YDB, zimagawira ntchito pakati pa ma node za healtcheck ndikuphatikiza zotsatira, zomwe zimasungidwa ku database ndikutumizidwa kwa wolamulira wa loadbalancer. Iwo, nawonso, amatumiza pempho kusintha zikuchokera tsango mu ndege deta kuti loadbalancer-node, amene ine kukambirana pansipa.
Tiyeni tikambirane zambiri zokhudza thanzi. Iwo akhoza kugawidwa m'magulu angapo. Maudindo ali ndi njira zosiyanasiyana zopambana. Macheke a TCP amafunikira kukhazikitsa kulumikizana bwino mkati mwa nthawi yokhazikika. Macheke a HTTP amafunikira kulumikizana kopambana komanso kuyankha ndi ma code 200.
Komanso, macheke amasiyana m'magulu a zochita - amakhala achangu komanso osachita chilichonse. Macheke a Passive amangoyang'ana zomwe zikuchitika ndi kuchuluka kwa magalimoto popanda kuchitapo kanthu mwapadera. Izi sizikugwira ntchito bwino pa L4 chifukwa zimadalira malingaliro a ma protocol apamwamba: pa L4 palibe chidziwitso chokhudza nthawi yomwe opaleshoniyo inatenga kapena ngati kutsirizidwa kwa kugwirizana kunali kwabwino kapena koipa. Macheke okhazikika amafuna kuti wowerengera atumize zopempha ku seva iliyonse.
Ambiri onyamula katundu amayesa moyo wawo. Ku Cloud, tidaganiza zolekanitsa magawo awa adongosolo kuti tiwonjezere scalability. Njirayi itithandiza kuti tiwonjezere chiwerengero cha owerengera pamene tikusunga chiwerengero cha zopempha zaumoyo ku utumiki. Macheke amachitidwa ndi ma node osiyana aumoyo, pomwe macheke amagawidwa ndikusinthidwanso. Simungathe kuchita cheke kuchokera kwa wolandira m'modzi, chifukwa zitha kulephera. Ndiye sitipeza momwe zochitika zomwe adazifufuza. Timafufuza pazochitika zilizonse kuchokera ku ma nodi atatu aumoyo. Timagawana zolinga zamacheke pakati pa node pogwiritsa ntchito ma hashing algorithms.
Kulekanitsa kusanja ndi kuyang'anira thanzi kungayambitse mavuto. Ngati node yaumoyo ikupempha mwachitsanzo, kudutsa balancer (yomwe sikugwira ntchito pakali pano), ndiye kuti pali vuto lachilendo: gwero likuwoneka ngati lamoyo, koma magalimoto sangafikire. Timathetsa vutoli motere: ndife otsimikizika kuti tiyambitsa magalimoto oyendera zaumoyo kudzera m'mabalancers. Mwa kuyankhula kwina, ndondomeko yosuntha mapaketi ndi magalimoto kuchokera kwa makasitomala komanso kuchokera ku thanzi labwino amasiyana pang'ono: muzochitika zonsezi, mapaketiwo adzafika kwa owerengera, omwe adzawapereka kuzinthu zomwe akufuna.
Kusiyana kwake ndikuti makasitomala amapanga zopempha ku VIP, pomwe oyang'anira zaumoyo amapanga zopempha kwa aliyense RIP. Vuto losangalatsa limabwera apa: timapatsa ogwiritsa ntchito mwayi wopanga zinthu mu ma network a imvi a IP. Tiyerekeze kuti pali eni ake amtambo awiri osiyana omwe abisa ntchito zawo kumbuyo kwa ma balancers. Aliyense wa iwo ali ndi zothandizira mu subnet ya 10.0.0.1/24, yokhala ndi ma adilesi omwewo. Muyenera kutha kuwasiyanitsa, ndipo apa muyenera kulowa mumpangidwe wa Yandex.Cloud pafupifupi network. Ndi bwino kuti mudziwe zambiri mu , ndizofunika kwa ife tsopano kuti maukonde ali ndi magawo ambiri ndipo ali ndi tunnel omwe amatha kusiyanitsa ndi subnet id.
Healthcheck node imalumikizana ndi owerengera pogwiritsa ntchito ma adilesi otchedwa quasi-IPv6. Adilesi ya quasi ndi adilesi ya IPv6 yokhala ndi adilesi ya IPv4 ndi id yolumikizidwa mkati mwake. Magalimoto amafika pa balancer, yomwe imatulutsa adilesi ya IPv4 kuchokera pamenepo, imalowetsa IPv6 ndi IPv4 ndikutumiza paketi ku netiweki ya wogwiritsa ntchito.
Magalimoto obwereranso amapita chimodzimodzi: wowerengera amawona kuti komwe akupita ndi netiweki yotuwa kuchokera kwa akatswiri azaumoyo, ndikusintha IPv4 kukhala IPv6.
VPP - mtima wa ndege ya data
Balancer ikugwiritsidwa ntchito pogwiritsa ntchito ukadaulo wa Vector Packet Processing (VPP), chimango chochokera ku Cisco pakukonza batch ya traffic network. Kwa ife, chimangochi chimagwira ntchito pamwamba pa laibulale yoyang'anira chipangizo cha ogwiritsa ntchito - Data Plane Development Kit (DPDK). Izi zimawonetsetsa kuti mapaketi akugwira ntchito kwambiri: zosokoneza zocheperako zimachitika mu kernel, ndipo palibe masinthidwe apakati pakati pa malo a kernel ndi malo ogwiritsa ntchito.
VPP imapitanso patsogolo ndikufinya magwiridwe antchito ambiri mwa kuphatikiza mapaketi kukhala magulu. Kupindula kwa magwiridwe antchito kumabwera chifukwa chogwiritsa ntchito mwaukali cache pa mapurosesa amakono. Zosungira zonse ziwiri zimagwiritsidwa ntchito (mapaketi amapangidwa mu "vectors", deta ili pafupi ndi wina ndi mzake) ndi ma cache a malangizo: mu VPP, kukonza paketi kumatsatira graph, node zomwe zimakhala ndi ntchito zomwe zimagwira ntchito yomweyo.
Mwachitsanzo, kukonza mapaketi a IP mu VPP kumachitika motere: choyamba, mitu ya paketi imagawidwa mu node yodutsa, kenako imatumizidwa ku node, yomwe imatumiza mapaketiwo mopitilira malinga ndi matebulo olowera.
Wolimba pang'ono. Olemba a VPP samalekerera kusagwirizana pakugwiritsa ntchito ma processor cache, kotero kuti code yosinthira vekitala ya mapaketi imakhala ndi ma vectorization amanja: pali njira yosinthira momwe zinthu ngati "tili ndi mapaketi anayi pamzere" amakonzedwa, ndiye chimodzimodzi kwa awiri, ndiye - kwa mmodzi. Malangizo otengeratu nthawi zambiri amagwiritsidwa ntchito kuyika deta mu cache kuti mufulumizitse kuwafikira pakubwereza kotsatira.
n_left_from = frame->n_vectors;
while (n_left_from > 0)
{
vlib_get_next_frame (vm, node, next_index, to_next, n_left_to_next);
// ...
while (n_left_from >= 4 && n_left_to_next >= 2)
{
// processing multiple packets at once
u32 next0 = SAMPLE_NEXT_INTERFACE_OUTPUT;
u32 next1 = SAMPLE_NEXT_INTERFACE_OUTPUT;
// ...
/* Prefetch next iteration. */
{
vlib_buffer_t *p2, *p3;
p2 = vlib_get_buffer (vm, from[2]);
p3 = vlib_get_buffer (vm, from[3]);
vlib_prefetch_buffer_header (p2, LOAD);
vlib_prefetch_buffer_header (p3, LOAD);
CLIB_PREFETCH (p2->data, CLIB_CACHE_LINE_BYTES, STORE);
CLIB_PREFETCH (p3->data, CLIB_CACHE_LINE_BYTES, STORE);
}
// actually process data
/* verify speculative enqueues, maybe switch current next frame */
vlib_validate_buffer_enqueue_x2 (vm, node, next_index,
to_next, n_left_to_next,
bi0, bi1, next0, next1);
}
while (n_left_from > 0 && n_left_to_next > 0)
{
// processing packets by one
}
// processed batch
vlib_put_next_frame (vm, node, next_index, n_left_to_next);
}Chifukwa chake, Healthchecks imalankhula za IPv6 kupita ku VPP, zomwe zimawasandutsa IPv4. Izi zimachitika ndi mfundo mu graph, yomwe timatcha algorithmic NAT. Pamagalimoto obwerera m'mbuyo (ndikusintha kuchokera ku IPv6 kupita ku IPv4) pali njira yofananira ya NAT.
Magalimoto olunjika kuchokera kwa makasitomala olinganiza amadutsa ma graph node, omwe amadzipangira okha.
Node yoyamba ndi magawo omata. Imasunga hashi ya kwa magawo okhazikika. 5-tuple imaphatikizapo adilesi ndi doko la kasitomala komwe chidziwitso chimatumizidwa, ma adilesi ndi madoko azinthu zomwe zilipo kuti mulandire magalimoto, komanso ma protocol a network.
The 5-tuple hashi imatithandiza kuwerengera pang'ono muzotsatira zotsatizana za hashing, komanso kusintha kwa mndandanda wazinthu zomwe zikusintha kumbuyo kwa balancer. Pamene paketi yomwe palibe gawolo ifika pa balancer, imatumizidwa ku node yokhazikika ya hashing. Apa ndipamene kusanja kumachitika pogwiritsa ntchito hashing yosasinthika: timasankha chida kuchokera pamndandanda wazinthu zomwe zilipo. Kenako, mapaketiwo amatumizidwa ku node ya NAT, yomwe imalowetsanso adilesi yopita ndikuwerengeranso macheke. Monga mukuonera, timatsatira malamulo a VPP - monga kukonda, kugawa mawerengedwe ofanana kuti awonjezere mphamvu ya cache purosesa.
Hashing yokhazikika
Chifukwa chiyani tasankha ndipo ndi chiyani? Choyamba, tiyeni tiganizire ntchito yapitayi - kusankha gwero kuchokera pamndandanda.
Ndi hashi yosagwirizana, hashi ya paketi yomwe ikubwera imawerengedwa, ndipo gwero limasankhidwa pamndandanda ndi gawo lotsala logawa hashi iyi ndi kuchuluka kwazinthu. Malingana ngati mndandandawo sunasinthidwe, ndondomekoyi imagwira ntchito bwino: nthawi zonse timatumiza mapaketi okhala ndi 5-tuple yofanana pazochitika zomwezo. Ngati, mwachitsanzo, gwero lina lasiya kuyankha pazaumoyo, ndiye kuti gawo lalikulu la ma hashes chisankho chidzasintha. Malumikizidwe a TCP a kasitomala aphwanyidwa: paketi yomwe idafikirapo kale A ikhoza kuyamba kufika pa chitsanzo B, chomwe sichidziwika bwino ndi gawo la paketi iyi.
Hashing yokhazikika imathetsa vuto lomwe lafotokozedwa. Njira yosavuta yofotokozera mfundoyi ndi iyi: taganizirani kuti muli ndi mphete yomwe mumagawira zothandizira ndi hashi (mwachitsanzo, ndi IP: doko). Kusankha gwero ndikutembenuza gudumu ndi ngodya, yomwe imatsimikiziridwa ndi hashi ya paketi.
Izi zimachepetsa kugawikana kwa magalimoto pamene zinthu zikusintha. Kuchotsa gwero kumangokhudza mbali ya hashing ring yosasinthika yomwe chidacho chinali. Kuwonjezera gwero kumasinthanso kugawa, koma tili ndi mfundo zomata, zomwe zimatilola kuti tisasinthe magawo omwe adakhazikitsidwa kale kuzinthu zatsopano.
Tinayang'ana zomwe zimachitika kuwongolera magalimoto pakati pa balancer ndi zothandizira. Tsopano tiyeni tiwone zamayendedwe obwerera. Imatsata njira yofananira ndi kuchuluka kwa magalimoto - kudzera mu algorithmic NAT, ndiye kuti, kudzera m'mbuyo NAT 44 ya kuchuluka kwamakasitomala komanso kudzera mu NAT 46 pamagalimoto achitetezo. Timatsatira dongosolo lathu: timagwirizanitsa kuchuluka kwa magalimoto ndi kuchuluka kwa ogwiritsa ntchito.
Loadbalancer-node ndi zida zosonkhanitsidwa
Kapangidwe ka balancers ndi zothandizira mu VPP zimanenedwa ndi ntchito yakomweko - loadbalancer-node. Imalembetsa ku mtsinje wa zochitika kuchokera kwa loadbalancer-controller ndipo imatha kukonza kusiyana pakati pa dziko la VPP lomwe lilipo panopa ndi dziko lachindunji lomwe linalandira kuchokera kwa wolamulira. Timapeza dongosolo lotsekedwa: zochitika kuchokera ku API zimabwera kwa wolamulira wa balancer, yemwe amapereka ntchito kwa woyang'anira zaumoyo kuti ayang'ane "moyo" wazinthu. Izi, nazonso, zimapatsa ntchito ku healthcheck-node ndikuphatikiza zotsatira, pambuyo pake zimawatumizanso kwa wowongolera. Loadbalancer-node imalembetsa zochitika kuchokera kwa wolamulira ndikusintha dziko la VPP. M'dongosolo loterolo, ntchito iliyonse imadziwa zomwe zili zofunika zokhudzana ndi mautumiki oyandikana nawo. Chiwerengero cha maulumikizidwe ndi chochepa ndipo timatha kugwira ntchito ndikukulitsa magawo osiyanasiyana paokha.
Ndi nkhani ziti zimene anapewa?
Ntchito zathu zonse mu ndege yowongolera zimalembedwa mu Go ndipo zimakhala ndi mawonekedwe abwino komanso odalirika. Go ili ndi malaibulale ambiri otseguka omangira machitidwe ogawidwa. Timagwiritsa ntchito GRPC mwachangu, zigawo zonse zili ndi kukhazikitsidwa kwa gwero lotseguka la ntchito - ntchito zathu zimawunika momwe zimagwirira ntchito, zimatha kusintha mawonekedwe ake, ndipo tidalumikiza izi ndi kusanja kwa GRPC. Kwa ma metric, timagwiritsanso ntchito njira yotsegulira gwero. Mu ndege ya data, tinali ndi magwiridwe antchito abwino komanso nkhokwe yayikulu: zidakhala zovuta kusonkhanitsa choyimira chomwe titha kudalira momwe VPP ikuyendera, m'malo mwa chitsulo chachitsulo.
Mavuto ndi Mayankho
Ndi chiyani chomwe sichinagwire bwino kwambiri? Go imakhala ndi kasamalidwe ka kukumbukira, koma kutayikira kukumbukira kumachitikabe. Njira yosavuta yothana nawo ndikuyendetsa ma goroutines ndikukumbukira kuwathetsa. Takeaway: Onani kugwiritsa ntchito kukumbukira kwa mapulogalamu anu a Go. Nthawi zambiri chizindikiro chabwino ndi chiwerengero cha goroutines. Pali chowonjezera m'nkhaniyi: mu Go ndikosavuta kupeza nthawi yogwiritsira ntchito - kugwiritsa ntchito kukumbukira, kuchuluka kwa ma goroutines, ndi zina zambiri.
Komanso, Go sangakhale chisankho chabwino kwambiri pamayeso ogwira ntchito. Iwo ndi omveka ndithu, ndipo njira yokhazikika ya "kuyendetsa chirichonse mu CI mu batch" si yoyenera kwa iwo. Chowonadi ndi chakuti kuyesa kogwira ntchito kumakhala kofunikira kwambiri ndipo kumayambitsa nthawi yeniyeni. Chifukwa cha izi, mayeso amatha kulephera chifukwa CPU ili yotanganidwa ndi mayeso a unit. Kutsiliza: Ngati nkotheka, yesani mayeso βolemeraβ mosiyana ndi mayunitsi.
Kapangidwe ka zochitika za Microservice ndizovuta kwambiri kuposa monolith: kusonkhanitsa zipika pamakina ambiri osiyanasiyana sikothandiza kwambiri. Kutsiliza: ngati mupanga ma microservices, nthawi yomweyo ganizirani za kutsatira.
Zolinga zathu
Tikhazikitsa chowerengera chamkati, chowerengera cha IPv6, kuwonjezera kuthandizira zolemba za Kubernetes, kupitiliza kugawana ntchito zathu (pakali pano ndi healthcheck-node ndi healthcheck-ctrl zomwe zimagawidwa), kuwonjezera macheke atsopano azaumoyo, ndikukhazikitsanso macheke anzeru. Tikulingalira za kuthekera kopanga mautumiki athu kukhala odziyimira pawokha - kuti azilumikizana osati mwachindunji, koma kugwiritsa ntchito mzere wa mauthenga. Ntchito yogwirizana ndi SQS yawoneka posachedwa mu Cloud .
Posachedwapa, kutulutsidwa kwapagulu kwa Yandex Load Balancer kunachitika. Onani ku ntchito, wongolerani zowongolera m'njira yabwino kwa inu ndikuwonjezera kulolerana kwamapulojekiti anu!
Source: www.habr.com