Qaphela. transl.: Abasebenzi besevisi ye-Tinder edumile emhlabeni basanda kwabelana ngemininingwane ethile yobuchwepheshe yokuthuthela ingqalasizinda yabo ku-Kubernetes. Le nqubo yathatha cishe iminyaka emibili futhi yaholela ekwethulweni kwenkundla enkulu kakhulu kuma-K8, ehlanganisa izinsiza ezingama-200 ezisingathwa kuma-container ayizinkulungwane ezingama-48. Yibuphi ubunzima obuthakazelisayo onjiniyela be-Tinder abahlangabezane nabo futhi imiphi imiphumela abafike kuyo? Funda lokhu kuhumusha.
Kungani?
Cishe eminyakeni emibili edlule, i-Tinder yanquma ukuthuthela inkundla yayo ku-Kubernetes. I-Kubernetes izovumela ithimba le-Tinder ukuthi lifake isitsha futhi lithuthele ekukhiqizeni ngomzamo omncane ngokusebenzisa ukuthunyelwa okungaguquki. (ukuthunyelwa okungaguquleki). Kulokhu, ukuhlanganiswa kwezinhlelo zokusebenza, ukuthunyelwa kwazo, nengqalasizinda ngokwayo kuzochazwa ngokuhlukile ngekhodi.
Besifuna nesixazululo enkingeni yokuqina nokuqina. Lapho ukukala kuba bucayi, ngokuvamile kwakudingeka silinde imizuzu embalwa ukuze izimo ezintsha ze-EC2 ziqhubeke. Umqondo wokwethula iziqukathi nokuqala ukuhlinzeka ngethrafikhi ngemizuzwana esikhundleni semizuzu ube ukhanga kakhulu kithi.
Inqubo iphenduke yaba nzima. Ngesikhathi sokufuduka kwethu ngasekuqaleni kuka-2019, iqoqo le-Kubernetes lafinyelela isisindo esibucayi futhi saqala ukubhekana nezinkinga ezihlukahlukene ngenxa yomthamo wethrafikhi, usayizi weqoqo, kanye ne-DNS. Endleleni, sixazulule izinkinga eziningi ezithakazelisayo ezihlobene nokufuduka kwezinsizakalo ezingu-200 nokugcina iqoqo le-Kubernetes elihlanganisa ama-node angu-1000, ama-pods angu-15000 kanye neziqukathi ezigijimayo ezingu-48000.
Kanjani?
Kusukela ngoJanuwari 2018, sidlule ezigabeni ezihlukahlukene zokufuduka. Siqale ngokuqukatha zonke izinsiza zethu futhi siwathumele ezindaweni zamafu zokuhlola i-Kubernetes. Kusukela ngo-Okthoba, siqale ukuthuthela zonke izinsiza ezikhona ku-Kubernetes. NgoMashi wonyaka olandelayo, saqeda ukuthutha futhi manje inkundla ye-Tinder isebenza ngokukhethekile ku-Kubernetes.
Ukwakha izithombe ze-Kubernetes
Sinezindawo zokugcina ikhodi yomthombo ezingaphezu kuka-30 zama-microservices asebenza kuqoqo le-Kubernetes. Ikhodi kula makhosombe ibhalwe ngezilimi ezahlukene (ngokwesibonelo, Node.js, Java, Scala, Go) enezindawo eziningi zesikhathi sokusebenza zolimi olufanayo.
Isistimu yokwakha yakhelwe ukuhlinzeka "ngomongo wokwakha" owenzeka ngokwezifiso we-microservice ngayinye. Imvamisa iqukethe i-Dockerfile kanye nohlu lwemiyalo yegobolondo. Okuqukethwe kwabo kungenziwa ngokwezifiso ngokuphelele, futhi ngasikhathi sinye, zonke lezi zimo zokwakha zibhalwa ngokwefomethi esezingeni. Ukulinganisa okuqukethwe kwesakhiwo kuvumela isistimu eyodwa yokwakha ukuthi iphathe wonke ama-microservices.
Umfanekiso 1-1. Inqubo yokwakha ejwayelekile ngesiqukathi Somakhi
Ukuze kuzuzwe ukuvumelana okuphezulu phakathi kwezikhathi zokusebenza (izindawo zesikhathi sokusebenza) inqubo efanayo yokwakha isetshenziswa ngesikhathi sokuthuthukiswa nokuhlolwa. Sibhekane nenselele ethakazelisa kakhulu: bekufanele sakhe indlela yokuqinisekisa ukuvumelana kwendawo yokwakha kuyo yonke inkundla. Ukufeza lokhu, zonke izinqubo zokuhlanganisa zenziwa ngaphakathi kwesitsha esikhethekile. Builder.
Ukuqaliswa kwakhe kwesiqukathi kudinga amasu e-Docker athuthukile. Umakhi uzuza njengefa i-ID yomsebenzisi wendawo nezimfihlo (ezifana nokhiye we-SSH, imininingwane ye-AWS, njll.) ezidingekayo ukuze ufinyelele amakhosombe e-Tinder ayimfihlo. Ikhweza inkomba yendawo equkethe imithombo ukuze igcine ngokwemvelo ama-artifact okwakhiwa. Le ndlela ithuthukisa ukusebenza ngoba iqeda isidingo sokukopisha ama-artifact okwakha phakathi kwesiqukathi Somakhi nomsingathi. Ama-artifact okwakha agciniwe angasetshenziswa kabusha ngaphandle kokucushwa okwengeziwe.
Kwamanye amasevisi, kudingeke ukuthi sakhe esinye isiqukathi ukuze senze imephu yendawo yokuhlanganisa iye endaweni yesikhathi sokusebenza (isibonelo, umtapo wezincwadi we-Node.js bcrypt ukhiqiza ama-artifacts aqondene nenkundla ethile phakathi nokufakwa). Phakathi nenqubo yokuhlanganisa, izidingo zingahluka phakathi kwezinsizakalo, futhi i-Dockerfile yokugcina ihlanganiswa ngokuphazima kweso.
I-Kubernetes cluster architecture kanye nokufuduka
Ukuphathwa kosayizi weqoqo
Sanquma ukusebenzisa kube-aw ukuthunyelwa kweqoqo okuzenzakalelayo kuzimo ze-Amazon EC2. Ekuqaleni, yonke into yayisebenza endaweni eyodwa evamile yama-node. Sisheshe sabona isidingo sokuhlukanisa imithwalo yomsebenzi ngosayizi kanye nohlobo lwesibonelo ukuze sisebenzise izinsiza ngokuphumelelayo. Umqondo wawuwukuthi ukusebenzisa ama-pod anemicu eminingi elayishiwe kubonakale kubikezeleka kakhulu ngokusebenza kunokuhlalisana kwawo nenani elikhulu lamaphodi anomucu owodwa.
Ekugcineni salungisa:
- m5.4 omkhulu - ukuqapha (Prometheus);
- c5.4okukhulu - ngomthwalo womsebenzi we-Node.js (umthwalo womsebenzi onentambo eyodwa);
- c5.2okukhulu - ye-Java ne-Go (umthwalo womsebenzi onemicu eminingi);
- c5.4okukhulu - Iphaneli yokulawula (amanodi ama-3).
Ukufuduka
Esinye sezinyathelo zokulungiselela zokufuduka kusuka engqalasizinda endala kuya e-Kubernetes kwakuwukuqondisa kabusha ukuxhumana okukhona okuqondile phakathi kwezinsizakalo kuzilinganisi ezintsha zomthwalo (Elastic Load Balancers (ELB). Zadalwa ku-subnet ethile yefu eliyimfihlo elibonakalayo (VPC). Le subnet ixhunywe ku-Kubernetes VPC. Lokhu kusivumele ukuthi sithuthe amamojula kancane kancane, ngaphandle kokucabangela ukuhleleka okuqondile kokuncika kwesevisi.
Lezi zindawo zokugcina zidalwe kusetshenziswa amasethi anesisindo amarekhodi e-DNS anama-CNAME akhomba ku-ELB entsha ngayinye. Ukuze sishintshe, sengeze okufakiwe okusha okukhomba ku-ELB entsha yesevisi ye-Kubernetes enesisindo esingu-0. Sibe sesibeka i-Time To Live (TTL) yokufakiwe esethelwe ku-0. Ngemva kwalokhu, izisindo ezindala nezintsha zase. ilungiswe kancane, futhi ekugcineni u-100% womthwalo uthunyelwe kuseva entsha. Ngemva kokuqedwa koshintsho, inani le-TTL libuyele ezingeni elanele.
Amamojula e-Java ebesinawo akwazi ukubhekana ne-TTL DNS ephansi, kodwa izinhlelo zokusebenza ze-Node azikwazanga. Omunye wonjiniyela ubhale kabusha ingxenye yekhodi yephuli yokuxhumana futhi wayisonga kumphathi obuyekeza amachibi njalo ngemizuzwana engama-60. Indlela ekhethiwe isebenze kahle kakhulu futhi ngaphandle kokwehliswa kokusebenza okubonakalayo.
Izifundo
Imikhawulo Yendwangu Yenethiwekhi
Ekuseni ngovivi kukaJanuwari 8, 2019, inkundla ye-Tinder yaphahlazeka kungazelelwe. Ekuphenduleni ukwanda okungahlobene kokubambezeleka kweplatifomu ekuqaleni kwalokho ekuseni, inani lama-pods namanodi kuqoqo lenyuka. Lokhu kubangele ukuthi inqolobane ye-ARP iphele kuwo wonke amanodi ethu.
Kunezinketho ezintathu ze-Linux ezihlobene nenqolobane ye-ARP:
()
gc_thresh3 - lokhu kuwumkhawulo onzima. Ukubonakala kokufakiwe "kokuchichima kwetafula likamakhelwane" kulogi kusho ukuthi ngisho nangemva kokuqoqwa kwemfucuza ehambisanayo (GC), sasingekho isikhala esanele kunqolobane ye-ARP ukuze sigcine isingeniso esingumakhelwane. Kulokhu, i-kernel imane ilahle iphakethe ngokuphelele.
Sisebenzisa njengendwangu yenethiwekhi e-Kubernetes. Amaphakethe asakazwa nge-VXLAN. I-VXLAN ingumhubhe we-L2 ophakanyiswe phezu kwenethiwekhi ye-L3. Ubuchwepheshe busebenzisa i-MAC-in-UDP (i-MAC Address-in-User Datagram Protocol) encapsulation futhi buvumela ukunwetshwa kwamasegimenti enethiwekhi Yesendlalelo sesi-2. Iphrothokholi yezokuthutha kunethiwekhi yesikhungo sedatha ebonakalayo yi-IP kanye ne-UDP.
Umfanekiso 2β1. Umdwebo we-Flannel ()
Umfanekiso 2-2. Iphakheji ye-VXLAN ()
I-Kubernetes ngayinye inodi yesisebenzi yabela indawo yekheli ebonakalayo enemaski engu/24 kusuka kubhulokhi enkulu/9. Ku-node ngayinye lokhu okukodwa okufakiwe kuthebula lomzila, okukodwa kuthebula le-ARP (ku-interface ye-flannel.1), kanye nokukodwa kuthebula lokushintsha (FDB). Zengezwa okokuqala lapho kuqaliswa i-node yesisebenzi noma isikhathi ngasinye lapho kutholwa indawo entsha.
Ukwengeza, ukuxhumana kwe-node-pod (noma i-pod-pod) ekugcineni kudlula esibonakalayo eth0 (njengoba kukhonjisiwe kumdwebo weFlaneli ngenhla). Lokhu kuphumela ekufakweni okwengeziwe kuthebula le-ARP kumthombo ngamunye ohambelanayo nomsingathi wendawo.
Endaweni yethu, lolu hlobo lokuxhumana luvame kakhulu. Ezintweni zesevisi ku-Kubernetes, i-ELB iyakhiwa futhi i-Kubernetes ibhalisa inodi ngayinye nge-ELB. I-ELB ayazi lutho ngama-pods futhi i-node ekhethiwe ingase ingabi indawo yokugcina yephakethe. Iphuzu liwukuthi lapho i-node ithola iphakethe ku-ELB, ibheka ukucabangela imithetho iptables ngesevisi ethile futhi akhethe ngokungahleliwe i-pod kwenye indawo.
Ngesikhathi sokwehluleka, bekunamanodi angama-605 kuqoqo. Ngenxa yezizathu ezishiwo ngenhla, lokhu bekwanele ukunqoba ukubaluleka gc_thresh3, okuyikhona okuzenzakalelayo. Uma lokhu kwenzeka, akugcini nje ukuthi amaphakethe aqale ukwehliswa, kodwa sonke isikhala sekheli le-Flannel esinemaski / 24 siyanyamalala etafuleni le-ARP. Ukuxhumana kwe-Node-pod kanye nemibuzo ye-DNS iphazamisekile (i-DNS isingathwe kuqoqo; funda kamuva kulesi sihloko ukuze uthole imininingwane).
Ukuxazulula le nkinga, udinga ukwandisa amanani gc_thresh1, gc_thresh2 ΠΈ gc_thresh3 bese uqala kabusha i-Flannel ukuze ubhalise kabusha amanethiwekhi angekho.
Ukukala kwe-DNS okungalindelekile
Phakathi nenqubo yokuthutha, sasebenzisa ngokugcwele i-DNS ukuze silawule ithrafikhi futhi kancane kancane sidlulisele izinsiza kusuka nengqalasizinda endala ukuya e-Kubernetes. Setha amanani aphansi e-TTL kumaRecordSets ahlobene ku-Route53. Ngenkathi ingqalasizinda endala isebenza ezimweni ze-EC2, ukucushwa kwethu kwesixazululi kukhomba ku-Amazon DNS. Sikuthathe kalula lokhu futhi umthelela we-TTL ephansi ezinsizeni zethu nasezinsizeni ze-Amazon (njenge-DynamoDB) awuzange unakwe.
Njengoba sithuthela izinsizakalo ku-Kubernetes, sithole ukuthi i-DNS ibicubungula izicelo eziyizinkulungwane ezingama-250 ngomzuzwana. Ngenxa yalokho, izinhlelo zokusebenza zaqala ukuzwa izikhathi zokuvala njalo nezibucayi zemibuzo ye-DNS. Lokhu kwenzeke naphezu kwemizamo emangalisayo yokwandisa nokushintsha umhlinzeki we-DNS ku-CoreDNS (okuyinto ekulayisheni okuphezulu okufinyelele ama-pods angu-1000 asebenza kuma-cores angu-120).
Ngenkathi sicwaninga ezinye izimbangela nezisombululo ezingaba khona, sathola , echaza izimo zomjaho ezithinta uhlaka lokuhlunga iphakethe netfilter ku-Linux. Ukuphela kwesikhathi esikubonile, kuhambisana nekhawunta ekhulayo Faka_yehlulekile kusixhumi esibonakalayo seFlaneli bekuhambisana nokutholwe yisihloko.
Inkinga yenzeka esigabeni Sokuhunyushwa Kwekheli Lenethiwekhi Yomthombo kanye Nendawo Ekuyo (i-SNAT ne-DNAT) kanye nokungena okulandelayo etafuleni. contrack. Enye yezindlela zokulungisa okuxoxwe ngazo ngaphakathi futhi ephakanyiswe umphakathi kwakuwukuhambisa i-DNS endaweni yokusebenza ngokwayo. Esimweni esinjalo:
- I-SNAT ayidingeki ngoba ithrafikhi ihlala ngaphakathi kwe-node. Akudingi ukuthi idluliselwe ku-interface eth0.
- I-DNAT ayidingeki njengoba i-IP okuyiwa kuyo ingeyendawo endaweni, futhi hhayi i-pod ekhethwe ngokungahleliwe ngokwemithetho. iptables.
Sinqume ukunamathela kule ndlela. I-CoreDNS yasetshenziswa njenge-DaemonSet ku-Kubernetes futhi sasebenzisa iseva yendawo ye-DNS isisombululo iphodi ngayinye ngokubeka ifulege --cluster-dns imiyalo kubeletβ. Lesi sixazululo sibonakale sisebenza ngokuvala isikhathi kwe-DNS.
Kodwa-ke, sisabona ukulahleka kwephakethe kanye nokwenyuka kwekhawunta Faka_yehlulekile ku-interface ye-Flannel. Lokhu kwaqhubeka ngemva kokuqaliswa kokusebenza ngoba sikwazile ukuqeda i-SNAT kanye/noma i-DNAT yethrafikhi ye-DNS kuphela. Izimo zomjaho zagcinelwa ezinye izinhlobo zethrafikhi. Ngenhlanhla, amaphakethe ethu amaningi ayi-TCP, futhi uma kwenzeka inkinga avele adluliselwe kabusha. Sisazama ukuthola isixazululo esifanelekile sazo zonke izinhlobo zethrafikhi.
Ukusebenzisa Inxusa Ukulinganisa Okungcono Komthwalo
Njengoba sithuthela izinsizakalo ezingemuva siye e-Kubernetes, saqala ukuhlushwa umthwalo ongalingani phakathi kwama-pods. Sithole ukuthi i-HTTP Keepalive ibangele ukuthi ukuxhumana kwe-ELB kulenga kumaphodi alungile okuqala okuthunyelwa ngakunye okukhishiwe. Ngakho-ke, ingxenye enkulu yethrafikhi yadlula iphesenti elincane lama-pods atholakalayo. Isixazululo sokuqala esisihlolile kwakuwukusetha iMaxSurge ibe ngu-100% ekusetshenzisweni okusha ezimweni ezimbi kakhulu. Umphumela ubonakale ungabalulekile futhi ungathembisi ngokuya ngokusetshenziswa okukhulu.
Esinye isixazululo esisisebenzisile kwakuwukwengeza izicelo zensiza ngokuzenzela zezinsizakalo ezibalulekile. Kulesi simo, ama-pods abekwe eduze angaba nendawo eyengeziwe yokuqondisa uma kuqhathaniswa namanye ama-pods asindayo. Nakhona bekungeke kusebenze ngokuhamba kwesikhathi ngoba kuzobe kuwukumosha izinsiza. Ngaphezu kwalokho, izinhlelo zethu zokusebenza zeNode zaziwuchungechunge olulodwa futhi, ngokufanelekile, zingasebenzisa umongo owodwa kuphela. Okuwukuphela kwesixazululo sangempela kwakuwukusebenzisa ukulinganisa okungcono komthwalo.
Sekuyisikhathi eside sifuna ukwazisa ngokugcwele . Isimo esikhona manje sisivumele ukuthi sikusebenzise ngendlela elinganiselwe futhi sithole imiphumela esheshayo. I-Envoy ingummeleli osebenza kahle kakhulu, umthombo ovulekile, ungqimba-XNUMX oklanyelwe izinhlelo zokusebenza ezinkulu ze-SOA. Ingasebenzisa izindlela ezithuthukisiwe zokulinganisa umthwalo, okuhlanganisa ukuzama kabusha okuzenzakalelayo, ama-circuit breaker, kanye nomkhawulo wezinga lomhlaba. (Qaphela. transl.: Ungafunda kabanzi ngalokhu ku mayelana ne-Istio, esekelwe ku-Envoy.)
Siqhamuke nokucushwa okulandelayo: yiba ne-sidecar yezithunywa ze-pod ngayinye kanye nomzila owodwa, bese uxhuma iqoqo kusiqukathi sasendaweni ngembobo. Ukuze kuncishiswe ukwehla okungaba khona nokugcina irediyasi encane eshayayo, sisebenzise uxhaxha lwamaphodi wommeleli we-Evoy, eyodwa ngeNdawo Yokutholakala (AZ) ngesevisi ngayinye. Bathembele enjinini yokutholwa kwesevisi elula ebhalwe omunye wonjiniyela bethu evele yabuyisela uhlu lwama-pods ku-AZ ngayinye ngesevisi enikeziwe.
I-service front-Computer bese isebenzisa le ndlela yokuthola isevisi ngeqoqo elilodwa elikhuphuka nomfula kanye nomzila. Setha izikhathi zokuvala ezanele, sandisa zonke izilungiselelo ze-circuit breaker, futhi sengeza ukucushwa okuncane kokuzama futhi ukusiza ngokuhluleka okukodwa kanye nokuqinisekisa ukuthunyelwa okushelelayo. Sibeke i-TCP ELB phambi kwezithunywa zesevisi ngayinye. Ngisho noma i-keevalive evela kusendlalelo sethu sommeleli oyinhloko ibinamathele kwamanye amaphodi e-Essay, asakwazi ukuphatha umthwalo kangcono kakhulu futhi alungiselelwa ukuthi abhalansise nge-least_request ku-backend.
Ukuze sisetshenziswe, sisebenzise ihhuku ye-preStop kukho kokubili ama-pods ohlelo nama-sidecar pod. Ihuku liqalise iphutha ekuhloleni isimo sephoyinti lokugcina elitholakala kusiqukathi senqola eseceleni yaphinde yalala isikhashana ukuze ivumele ukuxhumeka okusebenzayo ukuthi kunqamuke.
Esinye sezizathu esenze sakwazi ukuhamba ngokushesha kungenxa yamamethrikhi anemininingwane esikwazile ukuwahlanganisa kalula ekufakweni kwe-Prometheus okuvamile. Lokhu kusivumele ukuthi sibone ngqo ukuthi kwakwenzekani ngenkathi silungisa amapharamitha wokumisa futhi sisabalalisa kabusha ithrafikhi.
Imiphumela yayishesha futhi isobala. Siqale ngezinsizakalo ezingalingani kakhulu, futhi okwamanje isebenza phambi kwezinsizakalo eziyi-12 ezibaluleke kakhulu kuqoqo. Kulo nyaka sihlela ukushintshela kunethiwekhi yesevisi egcwele enokutholwa kwesevisi ethuthuke kakhulu, ukunqanyulwa kwesekethe, ukutholwa kwezinto ezingaphandle, ukukhawulela izinga kanye nokulandela umkhondo.
Umfanekiso 3β1. Ukuhlangana kwe-CPU yesevisi eyodwa ngesikhathi sokushintshela ku-Envoy
Umphumela wokugcina
Ngalokhu okuhlangenwe nakho kanye nocwaningo olwengeziwe, sakhe ithimba lengqalasizinda eliqinile elinamakhono aqinile ekuklameni, ekuthumeleni, nasekusebenziseni amaqoqo amakhulu e-Kubernetes. Bonke onjiniyela be-Tinder manje banolwazi nolwazi lokupakisha iziqukathi futhi bathumele izinhlelo zokusebenza ku-Kubernetes.
Lapho kuphakama isidingo somthamo owengeziwe engqalasizinda endala, kwadingeka silinde imizuzu embalwa ukuze kuqaliswe izimo ezintsha ze-EC2. Manje iziqukathi ziqala ukusebenza futhi ziqala ukucubungula ithrafikhi phakathi nemizuzwana esikhundleni semizuzu. Ukuhlela iziqukathi eziningi kusenzakalo esisodwa se-EC2 futhi kunikeza ukugxila okuvundlile okuthuthukisiwe. Ngenxa yalokho, sibikezela ukwehla okukhulu kwezindleko ze-EC2019 ngo-2 uma kuqhathaniswa nonyaka odlule.
Ukufuduka kwathatha cishe iminyaka emibili, kodwa sikuqede ngoMashi 2019. Njengamanje, inkundla ye-Tinder isebenza ngokukhethekile kuqoqo le-Kubernetes elihlanganisa izinsiza ezingu-200, ama-node angu-1000, ama-pods angu-15 kanye neziqukathi ezigijimayo ezingu-000. Ingqalasizinda ayisesona sodwa isizinda samaqembu okusebenza. Bonke onjiniyela bethu babelana ngalo mthwalo wemfanelo futhi balawula inqubo yokwakha nokuthumela izinhlelo zabo zokusebenza besebenzisa ikhodi kuphela.
I-PS evela kumhumushi
Funda futhi uchungechunge lwezihloko kubhulogi yethu:
- Β«".
- Β«".
- Β«".
- Β«".
- Β«".
- Β«".
- Β«".
- Β«".
- Β«".
- Β«".
Source: www.habr.com