🥇 Kusintha kwa Tinder kupita ku Kubernetes

Zindikirani. transl.: Ogwira ntchito pagulu lodziwika bwino la Tinder posachedwapa adagawana zambiri zaukadaulo zakusamuka ku Kubernetes. Ntchitoyi inatenga pafupifupi zaka ziwiri ndipo inachititsa kuti pakhale nsanja yaikulu kwambiri pa K8s, yokhala ndi mautumiki 200 omwe amasungidwa pa makontena 48. Kodi ndizovuta zotani zomwe mainjiniya a Tinder anakumana nazo ndipo zotsatira zake zidakhala zotani? Werengani kumasuliraku.

Chifukwa chiyani?

Pafupifupi zaka ziwiri zapitazo, Tinder adaganiza zosamukira ku Kubernetes. Kubernetes angalole kuti gulu la Tinder likhazikike ndikusunthira kukupanga molimbika pang'ono kudzera pakutumiza kosasinthika. (kutumiza kosasinthika). Pachifukwa ichi, kusonkhana kwa mapulogalamu, kutumizidwa kwawo, ndi zomangamanga zokhazokha zidzatanthauzidwa mwapadera ndi code.

Tinkafunanso njira yothetsera vuto la scalability ndi kukhazikika. Pamene makulitsidwe anali ovuta, nthawi zambiri tinkadikirira mphindi zingapo kuti zochitika zatsopano za EC2 zisinthe. Lingaliro lokhazikitsa zotengera ndikuyamba kutumiza magalimoto mumasekondi m'malo mwa mphindi lidakhala lokongola kwa ife.

Ntchitoyi inakhala yovuta. Pakusamuka kwathu koyambirira kwa 2019, gulu la Kubernetes lidafika povuta kwambiri ndipo tidayamba kukumana ndi mavuto osiyanasiyana chifukwa cha kuchuluka kwa magalimoto, kukula kwamagulu, ndi DNS. Panjira, tidathetsa zovuta zambiri zosangalatsa zokhudzana ndi kusamuka kwa mautumiki 200 ndikusunga gulu la Kubernetes lomwe lili ndi ma node 1000, ma pod 15000 ndi zotengera 48000 zothamanga.

Motani?

Kuyambira Januware 2018, tadutsa magawo osiyanasiyana osamuka. Tinayamba ndikuyika ntchito zathu zonse ndikuzitumiza ku Kubernetes test mtambo. Kuyambira mu Okutobala, tidayamba kusamutsa ntchito zonse zomwe zidalipo kale ku Kubernetes. Pofika mwezi wa Marichi chaka chotsatira, tidamaliza kusamuka ndipo tsopano nsanja ya Tinder imangoyendera Kubernetes.

Kupanga zithunzi za Kubernetes

Tili ndi zopitilira 30 zosungiramo ma microservices omwe akuyenda pagulu la Kubernetes. Khodi yomwe ili m'mabuku awa imalembedwa m'zilankhulo zosiyanasiyana (mwachitsanzo, Node.js, Java, Scala, Go) yokhala ndi malo angapo ogwiritsira ntchito chinenero chimodzi.

Dongosolo lomanga limapangidwa kuti lipereke "zomangamanga" zomwe mungasinthire makonda pa microservice iliyonse. Nthawi zambiri imakhala ndi Dockerfile ndi mndandanda wamalamulo a zipolopolo. Zomwe zili m'mawu ake ndizosintha mwamakonda, ndipo nthawi yomweyo, zonse zomwe zimamangidwa zimalembedwa molingana ndi mtundu wokhazikika. Kukhazikika kwazomwe zimapangidwira kumalola dongosolo limodzi lomanga kuti ligwiritse ntchito ma microservices onse.

Chithunzi 1-1. Njira yokhazikika yomanga kudzera pa chidebe cha Builder

Kuti mukwaniritse kusasinthasintha kwakukulu pakati pa nthawi zothamanga (malo othamanga) njira yomanga yomweyi imagwiritsidwa ntchito panthawi ya chitukuko ndi kuyesa. Tinakumana ndi vuto lochititsa chidwi kwambiri: tinayenera kupanga njira yowonetsetsa kuti malo omangawo agwirizane papulatifomu yonse. Kuti izi zitheke, njira zonse zochitira msonkhano zimachitikira mkati mwa chidebe chapadera. womanga.

Kukhazikitsa kwake kotengera kumafuna njira zapamwamba za Docker. Omanga amatenga cholowa cha wosuta wamba komanso zinsinsi (monga kiyi ya SSH, zidziwitso za AWS, ndi zina zambiri.) zomwe zimafunikira kuti mupeze nkhokwe zachinsinsi za Tinder. Imayika maulalo am'deralo omwe ali ndi kochokera kuti asungidwe mwachilengedwe. Njira iyi imathandizira magwiridwe antchito chifukwa imachotsa kufunika kokopera zomanga pakati pa chotengera cha Builder ndi wolandila. Zinthu zakale zomwe zasungidwa zitha kugwiritsidwanso ntchito popanda zina zowonjezera.

Kwa mautumiki ena, tidayenera kupanga chidebe china kuti tipange mapu a malo ophatikizana ndi nthawi yoyendetsera ntchito (mwachitsanzo, laibulale ya Node.js bcrypt imapanga zida zabinare za pulatifomu panthawi yoyika). Pakuphatikiza, zofunikira zimatha kusiyana pakati pa mautumiki, ndipo Dockerfile yomaliza imapangidwa powuluka.

Zomangamanga zamagulu a Kubernetes ndi kusamuka

Kuwongolera kukula kwamagulu

Tinaganiza zogwiritsa ntchito kukhala-aw pakuyika magulu odzichitira okha pazochitika za Amazon EC2. Pachiyambi choyamba, zonse zinkagwira ntchito mu dziwe limodzi la node. Tinazindikira mwamsanga kufunika kolekanitsa ntchito ndi kukula ndi mtundu wa zitsanzo kuti tigwiritse ntchito bwino zinthu. Lingaliro lake linali loti kuyendetsa ma podo angapo okhala ndi ulusi wambiri kunakhala kodziwikiratu potengera momwe amagwirira ntchito kuposa kukhalirana kwawo ndi ma pod ambiri okhala ndi ulusi umodzi.

Pomaliza tinakhala pa:

m5.4 kukula - pakuwunika (Prometheus);
c5.4 kukula - pa ntchito ya Node.js (ntchito yamtundu umodzi);
c5.2 kukula - kwa Java ndi Go (ntchito zambirimbiri);
c5.4 kukula - kwa gulu lowongolera (node 3).

Kusamuka

Chimodzi mwazinthu zokonzekera kusamuka kuchoka kumalo akale kupita ku Kubernetes chinali kuwongolera kulumikizana kwachindunji komwe kulipo pakati pa mautumiki kupita kwa onyamula katundu watsopano (Elastic Load Balancers (ELB). Adapangidwa pamtundu wina wamtambo wachinsinsi (VPC). Subnet iyi idalumikizidwa ndi Kubernetes VPC. Izi zinatilola kusamuka ma modules pang'onopang'ono, osaganizira dongosolo lenileni la kudalira kwa ntchito.

Mapeto awa adapangidwa pogwiritsa ntchito zolemba zolemetsa za DNS zomwe zinali ndi ma CNAME akulozera ku ELB iliyonse yatsopano. Kuti tisinthe, tinawonjezera cholowa chatsopano cholozera ku ntchito yatsopano ya Kubernetes ELB yokhala ndi kulemera kwa 0. Kenaka tinakhazikitsa Nthawi Yokhala ndi Moyo (TTL) yolowera ku 0. Pambuyo pa izi, zolemera zakale ndi zatsopano zinasinthidwa pang'onopang'ono. , ndipo pamapeto pake 100% ya katunduyo idatumizidwa ku seva yatsopano. Kusinthako kutatha, mtengo wa TTL unabwereranso pamlingo wokwanira.

Ma module a Java omwe tinali nawo amatha kuthana ndi TTL DNS yochepa, koma mapulogalamu a Node sanathe. Mmodzi mwa mainjiniyawo adalembanso gawo la dziwe lolumikizira ndikulikulunga ndi manejala yemwe amasinthira maiwewo masekondi 60 aliwonse. Njira yosankhidwa idagwira ntchito bwino kwambiri komanso popanda kuwonongeka kwa magwiridwe antchito.

Maphunziro ake

Malire a Network Fabric

M'mawa kwambiri pa Januware 8, 2019, nsanja ya Tinder idagwa mwadzidzidzi. Poyankha kuwonjezereka kosagwirizana kwa pulatifomu latency m'mawa womwewo, kuchuluka kwa ma pods ndi node mu masango kunakula. Izi zidapangitsa kuti cache ya ARP itope pama node athu onse.

Pali njira zitatu za Linux zokhudzana ndi cache ya ARP:

(gwero)

gc_thresh3 - ichi ndi malire ovuta. Maonekedwe a "oyandikana nawo tebulo akusefukira" zolemba mu chipika zikutanthauza kuti ngakhale pambuyo synchronous zinyalala zosonkhanitsira (GC), panalibe malo okwanira mu cache ARP kusunga lolowera oyandikana. Pamenepa, kernel idangotaya paketi kwathunthu.

Timagwiritsa ntchito Flannel ngati nsalu ya network ku Kubernetes. Mapaketi amafalitsidwa kudzera pa VXLAN. VXLAN ndi njira ya L2 yomwe idakwezedwa pamwamba pa netiweki ya L3. Ukadaulowu umagwiritsa ntchito encapsulation ya MAC-in-UDP (MAC Address-in-User Datagram Protocol) ndipo imalola kufalikira kwa magawo a Layer 2 network. Protocol yoyendera pa network data center network ndi IP kuphatikiza UDP.

Chithunzi 2-1. Chithunzi cha Flannel (gwero)

Chithunzi 2-2. VXLAN phukusi (gwero)

Node iliyonse ya ogwira ntchito a Kubernetes imagawa malo adilesi ndi / 24 chigoba kuchokera ku block / 9 block. Kwa node iliyonse izi ndi amatanthauza kulowa kumodzi patebulo lolowera, kulowa kumodzi patebulo la ARP (pa mawonekedwe a flannel.1), ndi kulowa kumodzi patebulo losinthira (FDB). Amawonjezedwa nthawi yoyamba pomwe nodi ya antchito ikayambika kapena nthawi iliyonse pomwe nodi yatsopano ipezeka.

Kuphatikiza apo, kulumikizana kwa node-pod (kapena pod-pod) kumadutsa mawonekedwe eth0 (monga momwe tawonetsera pa chithunzi cha Flannel pamwambapa). Izi zimapangitsa kuti pakhale chowonjezera patebulo la ARP pa gwero lililonse lofananira ndi komwe mukupita.

M’malo athu, kulankhulana kotereku n’kofala kwambiri. Pazinthu zothandizira ku Kubernetes, ELB imapangidwa ndipo Kubernetes amalembetsa node iliyonse ndi ELB. ELB sadziwa kanthu za pods ndipo node yosankhidwa ikhoza kukhala komaliza kwa paketi. Mfundo ndi yakuti pamene node ilandira paketi kuchokera ku ELB, imawona kuti ikuganizira malamulo iptables pa ntchito inayake ndikusankha mwachisawawa poto pa mfundo ina.

Pa nthawi ya kulephera, panali ma node 605 mgululi. Pazifukwa zomwe tazitchula pamwambapa, izi zinali zokwanira kuthana ndi tanthauzo gc_thresh3, chomwe chiri chokhazikika. Izi zikachitika, osati mapaketi okha omwe amayamba kugwetsedwa, koma malo onse a adiresi a Flannel okhala ndi / 24 chigoba amasowa patebulo la ARP. Kuyankhulana kwa node-pod ndi mafunso a DNS amasokonezedwa (DNS imasungidwa m'magulu; werengani pambuyo pake m'nkhaniyi kuti mudziwe zambiri).

Kuti muthane ndi vutoli, muyenera kuwonjezera zikhalidwe gc_thresh1, gc_thresh2 и gc_thresh3 ndikuyambitsanso Flannel kuti mulembetsenso maukonde omwe akusowa.

Makulitsidwe a DNS mosayembekezereka

Panthawi yosamuka, tidagwiritsa ntchito mwachangu DNS kuyang'anira magalimoto ndikusamutsa ntchito pang'onopang'ono kuchokera kuzinthu zakale kupita ku Kubernetes. Timayika ma TTL otsika kwambiri a RecordSets ogwirizana mu Route53. Pamene zomangamanga zakale zinkagwira ntchito pazochitika za EC2, kasinthidwe kathu kosintha adalozera ku Amazon DNS. Tidatenga izi mopepuka ndipo kukhudzika kwa TTL yotsika pamasewera athu ndi ntchito za Amazon (monga DynamoDB) sizinawonekere.

Pamene timasamukira ku Kubernetes, tidapeza kuti DNS ikukonza zopempha 250 zikwi pa sekondi iliyonse. Zotsatira zake, mapulogalamu adayamba kukhala ndi nthawi yokhazikika komanso yofunikira pamafunso a DNS. Izi zidachitika ngakhale kuyesayesa kodabwitsa kukhathamiritsa ndikusintha wopereka DNS ku CoreDNS (yomwe pakuchulukira kwake idafikira ma pod 1000 omwe akuyenda pa 120 cores).

Pamene tikufufuza zifukwa zina zomwe zingatheke ndi zothetsera, tinapeza nkhani, kufotokoza mikhalidwe ya mpikisano yomwe ikukhudza chimango chosefera paketi netfilter mu Linux. Kutha kwa nthawi komwe tidawona, kuphatikiza ndi kauntala yowonjezera insert_zalephera mu mawonekedwe a Flannel anali ogwirizana ndi zomwe zapezeka m'nkhaniyi.

Vutoli limapezeka pagawo la Source and Destination Network Address Translation (SNAT ndi DNAT) ndi kulowa mu tebulo lotsatira. kugwilizana. Chimodzi mwazinthu zomwe zidakambidwa mkati ndikulangizidwa ndi anthu ammudzi chinali kusuntha DNS kupita kumalo ogwirira ntchito. Pamenepa:

SNAT siyofunika chifukwa magalimoto amakhala mkati mwa node. Sichiyenera kuyendetsedwa kudzera mu mawonekedwe eth0.
DNAT siyofunika chifukwa IP yopitako ili pafupi ndi node, osati pod yosankhidwa mwachisawawa malinga ndi malamulo. iptables.

Tinaganiza zokakamirabe njira imeneyi. CoreDNS idayikidwa ngati DaemonSet ku Kubernetes ndipo tidakhazikitsa seva ya DNS yakumaloko. chanza poto iliyonse poika mbendera --cluster-dns malamulo cubelet . Njira iyi idakhala yothandiza pakutha kwa DNS.

Komabe, tidawonabe kutayika kwa paketi komanso kuwonjezeka kwa kauntala insert_zalephera mu mawonekedwe a Flannel. Izi zidapitilira pambuyo poti ntchitoyo idakhazikitsidwa chifukwa tinatha kuchotsa SNAT ndi/kapena DNAT pamayendedwe a DNS okha. Mipikisano yamtundu idasungidwira mitundu ina yamagalimoto. Mwamwayi, mapaketi athu ambiri ndi TCP, ndipo ngati vuto lichitika amangotumizidwanso. Tikuyesabe kupeza njira yabwino yothetsera mitundu yonse yamagalimoto.

Kugwiritsa Ntchito Nthumwi Pakuwongolera Bwino Katundu

Pamene timasamukira ku Kubernetes, tinayamba kuvutika ndi katundu wosagwirizana pakati pa ma pod. Tidapeza kuti HTTP Keepalive idapangitsa kuti maulumikizidwe a ELB apachike pamapopu okonzeka amtundu uliwonse wotulutsidwa. Choncho, kuchuluka kwa magalimoto kunadutsa pang'ono peresenti ya mapodo omwe alipo. Yankho loyamba lomwe tidayesa linali kukhazikitsa MaxSurge ku 100% pamisonkhano yatsopano pazochitika zoyipa kwambiri. Zotsatira zake zidakhala zocheperako komanso zosadalirika potengera kutumizidwa kwakukulu.

Njira ina yomwe tidagwiritsa ntchito ndikuwonjezera mwachinyengo zopempha zothandizira pazantchito zofunika kwambiri. Pachifukwa ichi, makoko omwe amaikidwa pafupi amakhala ndi malo ochulukirapo poyerekeza ndi ma pod ena olemera. Sizikanagwira ntchito pakapita nthawi chifukwa kukanakhala kuwononga chuma. Kuphatikiza apo, mapulogalamu athu a Node anali ndi ulusi umodzi ndipo, motero, amatha kugwiritsa ntchito pachimake chimodzi. Njira yokhayo yothetsera vutoli inali kugwiritsa ntchito kusanja bwino katundu.

Takhala tikufuna kuyamikira kwathunthu nthumwi. Zomwe zikuchitika panopa zidatilola kuti tigwiritse ntchito mochepa kwambiri ndikupeza zotsatira mwamsanga. Mtumiki ndi wothandizira kwambiri, wotseguka, wosanjikiza-XNUMX wopangidwira ntchito zazikulu za SOA. Itha kugwiritsa ntchito njira zapamwamba zowongolera katundu, kuphatikiza zoyeserera zokha, zowononga ma circuit, ndi kuchepetsa mitengo yapadziko lonse. (Zindikirani. transl.: Mutha kuwerenga zambiri za izi mu nkhaniyi za Istio, zomwe zachokera ku Envoy.)

Tinabwera ndi masinthidwe awa: khalani ndi Evoy sidecar pa pod iliyonse ndi njira imodzi, ndikulumikiza gululo ku chidebe komweko kudzera padoko. Kuti tichepetse kuchulukira komanso kusunga ma radius ang'onoang'ono, tidagwiritsa ntchito gulu la Envoy front-proxy pods, imodzi pa Availability Zone (AZ) pa ntchito iliyonse. Adadalira injini yosavuta yodziwira ntchito yolembedwa ndi m'modzi mwa mainjiniya athu omwe adangobweza mndandanda wa ma pod mu AZ iliyonse kuti agwire ntchito.

Ma Service front-Envoys adagwiritsa ntchito njira yodziwira izi ndi gulu limodzi lokwera ndi njira. Tinakhazikitsa nthawi yokwanira, tinawonjezera makonda onse ophwanyira dera, ndikuwonjezera masinthidwe oyeseranso pang'ono kuti athandizire kulephera kumodzi ndikuwonetsetsa kutumizidwa bwino. Tidayika TCP ELB patsogolo pa aliyense wa Atumiki awa. Ngakhale zosungirako kuchokera ku gawo lathu lalikulu la projekiti zidakakamira pa ma pod ena a Evoy, adatha kunyamula katunduyo bwino kwambiri ndipo adakonzedwa kuti azitha kuwerengera mochepera_request kumbuyo.

Potumiza, tidagwiritsa ntchito mbedza ya preStop pamakapu onse ogwiritsira ntchito komanso ma pod am'mbali. Chingwecho chinayambitsa cholakwika poyang'ana mawonekedwe a admin omaliza omwe ali pa chidebe cham'mbali ndipo adagona kwakanthawi kuti alole kulumikizana kokhazikika kuthe.

Chimodzi mwa zifukwa zomwe tinatha kusuntha mofulumira ndi chifukwa chazitsulo zatsatanetsatane zomwe tinatha kuziphatikiza mosavuta ndikuyika kwa Prometheus. Izi zidatipangitsa kuwona zomwe zikuchitika pomwe tidasintha masinthidwe ndikugawanso magalimoto.

Zotsatira zake zinali zachangu komanso zowonekeratu. Tinayamba ndi mautumiki osagwirizana kwambiri, ndipo panthawiyi ikugwira ntchito patsogolo pa mautumiki 12 ofunika kwambiri mumagulu. Chaka chino tikukonzekera zosinthira kukhala ma mesh athunthu okhala ndi ntchito zotsogola kwambiri, kusokoneza ma circuit, kuzindikira zakunja, kuchepetsa mitengo ndi kufufuza.

Chithunzi 3-1. Kulumikizana kwa CPU kwa ntchito imodzi panthawi yosinthira kupita ku Envoy

Zotsatira zomaliza

Kupyolera muzochitikazi komanso kafukufuku wowonjezera, tapanga gulu lolimba la zomangamanga lomwe lili ndi luso lamphamvu pakupanga, kutumiza, ndi kugwiritsa ntchito magulu akuluakulu a Kubernetes. Mainjiniya onse a Tinder tsopano ali ndi chidziwitso komanso luso lopangira zida ndi kutumiza mapulogalamu ku Kubernetes.

Pomwe kufunikira kowonjezera mphamvu kudayamba pazitukuko zakale, tidadikirira mphindi zingapo kuti ma EC2 atsopano ayambike. Tsopano zotengera zimayamba kuyenda ndikuyamba kukonza kuchuluka kwa magalimoto mkati mwa masekondi m'malo mwa mphindi. Kukonza zotengera zingapo pamwambo umodzi wa EC2 kumathandizanso kukhazikika kopingasa. Zotsatira zake, tikulosera kutsika kwakukulu kwamitengo ya EC2019 mu 2 poyerekeza ndi chaka chatha.

Kusamukaku kunatenga pafupifupi zaka ziwiri, koma tidamaliza mu Marichi 2019. Pakadali pano, nsanja ya Tinder imagwira ntchito pagulu la Kubernetes lomwe lili ndi ntchito 200, node 1000, ma pod 15 ndi zotengera 000 zothamanga. Zomangamanga sizilinso gawo lokhalo lamagulu ogwirira ntchito. Mainjiniya athu onse amagawana udindowu ndikuwongolera njira yomanga ndi kutumiza mapulogalamu awo pogwiritsa ntchito ma code okha.

PS kuchokera kwa womasulira

Werenganinso nkhani zingapo pa blog yathu:

Source: www.habr.com

Kusintha kwa Tinder kupita ku Kubernetes