Phawula. transl.: Abasebenzi benkonzo ye-Tinder edumileyo yehlabathi mva nje babelane ngeenkcukacha ezithile zobuchwepheshe zokufudusela iziseko zabo kwi-Kubernetes. Inkqubo yathatha phantse iminyaka emibini kwaye yakhokelela ekusungulweni kweqonga elikhulu kakhulu kwi-K8s, eliquka iinkonzo ezingama-200 ezibanjwe kwiikhonteyina ezingamawaka angama-48. Zeziphi iingxaki ezibangela umdla eziye zadibana neenjineli zeTinder kwaye zeziphi iziphumo ezifike kuzo? Funda le nguqulelo.
Kutheni?
Phantse kwiminyaka emibini eyadlulayo, uTinder wagqiba kwelokuba afudukele iqonga lakhe eKubernetes. I-Kubernetes iya kuvumela iqela le-Tinder ukuba lifake isikhongozeli kwaye lihambe liye kwimveliso ngomzamo omncinci ngokusasazwa okungaguqukiyo. (ubeko olungaguqukiyo). Kule meko, ukuhlanganiswa kwezicelo, ukuthunyelwa kwazo, kunye neziseko zophuhliso ngokwazo ziya kuchazwa ngokukodwa ngekhowudi.
Sikwajonge nesisombululo kwingxaki yokuqina nozinzo. Xa ukulinganisa kwaba nzima, kuye kwafuneka ukuba silinde imizuzu emininzi ukuze iimeko ezintsha ze-EC2 ziqhubeke. Umbono wokusungula izikhongozeli kunye nokuqalisa ukukhonza itrafikhi ngemizuzwana endaweni yemizuzu ibe nomtsalane kuthi.
Inkqubo yajika yaba nzima. Ngexesha lokufuduka kwethu ekuqaleni kuka-2019, iqela le-Kubernetes lafikelela kubunzima obukhulu kwaye saqala ukujongana neengxaki ezahlukeneyo ngenxa yomthamo wezithuthi, ubungakanani beqela, kunye ne-DNS. Endleleni, sisombulule iingxaki ezininzi ezinomdla ezinxulumene nokufuduka kweenkonzo ezingama-200 kunye nokugcina i-Kubernetes cluster ebandakanya ii-node eziyi-1000, ii-pods eziyi-15000 kunye nezikhongozeli ezibalekayo ezingama-48000.
Njani?
Ukusukela ngoJanuwari 2018, siye sadlula kumanqanaba ahlukeneyo okufuduka. Siqale ngokufaka zonke iinkonzo zethu kwaye sizithumele kwiindawo zovavanyo lwe-Kubernetes. Ukuqala ngo-Okthobha, siqale ngokuhambisa zonke iinkonzo ezikhoyo eKubernetes. NgoMatshi wonyaka olandelayo, sagqiba ukufuduka kwaye ngoku iqonga le-Tinder lisebenza ngokukodwa kwi-Kubernetes.
Ukwakha imifanekiso yeKubernetes
Singaphezulu kwe-30 yemithombo yekhowudi yokugcina yee-microservices ezisebenza kwiqela le-Kubernetes. Ikhowudi kwezi ndawo zokugcina ibhalwe ngeelwimi ezahlukeneyo (umzekelo, iNode.js, Java, Scala, Go) eneendawo ezininzi zexesha lokusebenza kulwimi olufanayo.
Inkqubo yokwakha yenzelwe ukubonelela ngokwezifiso "zokwakha umxholo" kwi-microservice nganye. Ihlala iqulathe iDockerfile kunye noluhlu lwemiyalelo yeqokobhe. Umxholo wabo unokwenziwa ngokwezifiso ngokupheleleyo, kwaye kwangaxeshanye, zonke ezi meko zakhayo zibhalwa ngokwefomathi esemgangathweni. Ukulinganisa iimeko zokwakha kuvumela inkqubo enye yokwakha ukuba iphathe zonke iinkonzo ezincinci.
Umfanekiso 1-1. Inkqubo yokwakha esemgangathweni ngokusebenzisa isikhongozeli soMakhi
Ukufezekisa ukuhambelana okuphezulu phakathi kwamaxesha okuqhuba (iindawo zexesha lokusebenza) inkqubo yokwakha efanayo isetyenziswa ngexesha lophuhliso kunye novavanyo. Siye sajongana nomceli mngeni onomdla kakhulu: kwafuneka siphuhlise indlela yokuqinisekisa ukuhambelana kwendawo yokwakha kulo lonke iqonga. Ukufezekisa oku, zonke iinkqubo zendibano zenziwa ngaphakathi kwisitya esikhethekileyo. umakhi.
Ukuphunyezwa kwesikhongozeli sakhe kufuna iindlela eziphambili zeDocker. Umakhi ufumana i-ID yomsebenzisi wendawo kunye neemfihlo (ezifana nesitshixo se-SSH, iziqinisekiso ze-AWS, njl. njl.) ezifunekayo ukufikelela kwiindawo zokugcina ze-Tinder zabucala. Ixhoma oovimba bolawulo basekuhlaleni abaqulathe imithombo yokugcina izinto ngokwemvelo zokwakha. Le ndlela iphucula ukusebenza kuba iphelisa imfuneko yokukhuphela izinto zakudala phakathi kwesikhongozeli soMakhi kunye nomninimzi. Izinto zokwakha ezigciniweyo zingaphinda zisetyenziswe ngaphandle koqwalaselo olongezelelweyo.
Kwiinkonzo ezithile, kuye kwafuneka ukuba senze esinye isikhongozeli ukwenza imephu yemeko-bume yokuhlanganiswa kwimeko-bume yexesha lokusebenza (umzekelo, ithala leencwadi le-bcrypt leNode.js livelisa iqonga elithile lezinto zakudala zokubini ngexesha lofakelo). Ngexesha lenkqubo yokuhlanganiswa, iimfuno zinokwahluka phakathi kweenkonzo, kwaye i-Dockerfile yokugqibela ihlanganiswe ngokubhabha.
Uyilo lweqela leKubernetes kunye nokufuduka
Ulawulo lobungakanani beqela
Sagqiba ekubeni sisebenzise kube-aws ukusasazwa kweqela elizenzekelayo kwi-Amazon EC2 iimeko. Ekuqaleni, yonke into yayisebenza kwindawo enye yeendawo zokuhlala. Sikhawuleze sayiqonda imfuneko yokwahlula imithwalo yemisebenzi ngokobungakanani kunye nohlobo lomzekelo ukwenza ukusetyenziswa kwezibonelelo ngokufanelekileyo. Ingqiqo yayikukuba ukuqhuba iipods ezininzi ezilayishiweyo ezinemisonto emininzi ziye zaqikelelwa ngakumbi ngokwemigaqo yokusebenza kunokuhlalisana kwazo kunye nenani elikhulu leepods ezinemisonto enye.
Ekugqibeleni salungisa:
- m5.4 enkulu - ukubeka iliso (Prometheus);
- c5.4 enkulu - yeNode.js umthwalo womsebenzi (umthwalo omnye womsebenzi);
- c5.2 enkulu - yeJava kunye neGo (umthwalo womsebenzi omninzi);
- c5.4 enkulu — kwiphaneli yokulawula (i-3 nodes).
Ukufuduka
Elinye lamanyathelo okulungiselela ukufuduka ukusuka kwisiseko esidala ukuya eKubernetes yayikukubuyisela unxibelelwano olukhoyo oluthe ngqo phakathi kweenkonzo kwiibalancers zomthwalo omtsha (Elastic Load Balancers (ELB). Zadalwa kwi-subnet ethile yelifu eliyimfihlo labucala (VPC). Le subnet iqhagamshelwe kwi-Kubernetes VPC. Oku kusivumele ukuba sifuduke iimodyuli ngokuthe ngcembe, ngaphandle kokuqwalasela ulandelelwano oluthile lokuxhomekeka kwenkonzo.
Ezi siphelo zidalwe kusetyenziswa iiseti ezinobunzima beerekhodi ze-DNS ezazine-CNAME ezikhomba kwi-ELB nganye entsha. Ukutshintshela ngaphaya, songeze i-entry entsha ekhomba i-ELB entsha yenkonzo ye-Kubernetes kunye nobunzima be-0. Emva koko sibeka i-Time to Live (TTL) yokungena efakwe kwi-0. Emva koko, ubunzima obudala kunye namatsha ilungiswe kancinci, kwaye ekugqibeleni i-100% yomthwalo ithunyelwe kumncedisi omtsha. Emva kokugqitywa kokutshintshwa, ixabiso le-TTL libuyele kwinqanaba elaneleyo.
Iimodyuli zeJava esinazo zinokuhlangabezana ne-TTL DNS ephantsi, kodwa izicelo zeNode azikwazanga. Enye yeenjineli yabhala kwakhona inxalenye yekhowudi yokudibanisa i-pool kwaye yayisonga kumphathi ohlaziye amachibi rhoqo ngemizuzwana engama-60. Indlela ekhethiweyo isebenze kakuhle kakhulu kwaye ngaphandle kokuthotywa kokubonakalayo okubonakalayo.
Izifundo
Imida yeFabric yeNethiwekhi
Ngentseni kaJanuwari 8, 2019, iqonga leTinder lantlitheka ngokungalindelekanga. Ekuphenduleni ukunyuka okungahambelaniyo kwi-platform latency ekuqaleni kwaloo ntsasa, inani lee-pods kunye ne-nodes kwi-cluster yanda. Oku kubangele ukuba i-cache ye-ARP iphelelwe kuzo zonke iindawo zethu.
Kukho iinketho ezintathu zeLinux ezinxulumene necache yeARP:
()
gc_thresh3 - lo ngumda onzima. Ukubonakala kokungena "kwetafile yommelwane kuphuphuma" kwilogi kwakuthetha ukuba nangemva kokuqokelelwa kwenkunkuma ye-synchronous (GC), kwakungekho ndawo eyaneleyo kwi-cache ye-ARP yokugcina i-entry engummelwane. Kule meko, i-kernel ilahle nje ipakethi ngokupheleleyo.
Sisebenzisa njengelaphu lenethiwekhi eKubernetes. Iipakethi zithunyelwa ngeVXLAN. I-VXLAN yitonela ye-L2 ephakanyiswe phezu kwenethiwekhi ye-L3. Ithekhnoloji isebenzisa i-MAC-in-UDP (i-MAC Address-in-User Datagram Protocol) i-encapsulation kwaye ivumela ukwandiswa kwamacandelo e-Layer 2 network. Iprotocol yezothutho kwinethiwekhi yeziko ledatha ebonakalayo yi-IP kunye ne-UDP.
Umfanekiso 2–1. Umzobo weflaneli ()
Umfanekiso 2-2. Iphakheji ye-VXLAN ()
Indawo nganye yabasebenzi baseKubernetes yabela indawo enedilesi ene-/24 imaski ukusuka kwibhloko enkulu/yesi-9. Kwinodi nganye le enye ingeniso kwitafile yomzila, enye ingeniso kwitafile ye-ARP (kwi-interface ye-flannel.1), kunye nokungena kwitafile yokutshintsha (FDB). Zongezwa okokuqala ngqa i-node yabasebenzi iqalwa okanye ngalo lonke ixesha kufunyanwa indawo entsha.
Ukongeza, unxibelelwano lwe-node-pod (okanye i-pod-pod) ekugqibeleni luya kujongano I-eth0 (njengoko kubonisiwe kumzobo weFlaneli ongentla). Oku kubangela ukungena okongeziweyo kwitheyibhile ye-ARP kumthombo ngamnye ohambelanayo kunye nendawo yokufikela.
Kwindawo esihlala kuyo, olu hlobo lonxibelelwano luxhaphake kakhulu. Ngezinto zenkonzo kwi-Kubernetes, i-ELB yenziwe kwaye i-Kubernetes ibhalisa i-node nganye nge-ELB. I-ELB ayazi nto malunga neepods kwaye i-node ekhethiweyo ayinakuba yindawo yokugqibela yepakethi. Ingongoma kukuba xa i-node ifumana ipakethi kwi-ELB, icinga ukuba ithathela ingqalelo imigaqo iptables kwinkonzo ethile kwaye ukhetha ngokungakhethiyo i-pod kwenye indawo.
Ngexesha lokungaphumeleli, kwakukho iindawo ze-605 kwiqela. Ngenxa yezizathu ezichazwe ngasentla, oku kwakwanele ukoyisa ukubaluleka gc_thresh3, engagqibekanga. Xa oku kusenzeka, kungekhona kuphela iipakethi eziqala ukuwiswa, kodwa yonke indawo yedilesi yeFlannel ene-mask / 24 iyanyamalala kwitafile ye-ARP. Unxibelelwano lwe-Node-pod kunye nemibuzo ye-DNS iphazamisekile (i-DNS ibanjwe kwiqela; funda kamva kweli nqaku ngeenkcukacha).
Ukusombulula le ngxaki, kufuneka ukwandise amaxabiso gc_thresh1, gc_thresh2 и gc_thresh3 kwaye uqalise kwakhona iFlaneli ukubhalisa kwakhona iinethiwekhi ezingekhoyo.
Ukukalwa kwe-DNS engalindelekanga
Ngethuba lenkqubo yokufuduka, sasebenzisa ngokusebenzayo i-DNS ukulawula i-traffic kunye nokudlulisa ngokuthe ngcembe iinkonzo ukusuka kwisiseko esidala ukuya kwi-Kubernetes. Siseta amaxabiso aphantsi e-TTL kwiiRekhodi ezihambelanayo kwi-Route53. Xa isiseko esidala sasisebenza kwiimeko ze-EC2, ulungelelwaniso lwethu lomxazululi lwalatha kwi-Amazon DNS. Sikuthathele phezulu oku kwaye impembelelo ye-TTL ephantsi kwiinkonzo zethu kunye neenkonzo ze-Amazon (ezifana neDynamoDB) azizange ziqatshelwe.
Njengoko sasifuduka kwiinkonzo kwi-Kubernetes, sifumene ukuba i-DNS yayilungisa izicelo ezingamawaka angama-250 ngomzuzwana. Ngenxa yoko, izicelo zaqala ukufumana amaxesha arhoqoyo nanzima kwimibuzo ye-DNS. Oku kwenzeka nangona iinzame ezimangalisayo zokwandisa kunye nokutshintsha umnikezeli we-DNS kwi-CoreDNS (eyayifikelela kwi-peak load ifikelele kwii-pods ze-1000 ezisebenza kwii-cores ze-120).
Ngelixa siphanda abanye oonobangela kunye nezisombululo ezinokwenzeka, siye safumanisa , echaza iimeko zobuhlanga ezichaphazela isakhelo sokucoca ipakethi isihluzi senethiwekhi kwi Linux. Amaxesha esiwabonileyo, adityaniswa nekhawuntari eyandayo Faka_akuphumelelanga kwi-interface ye-Flannel yayihambelana neziphumo zenqaku.
Ingxaki yenzeka kwinqanaba loMthombo kunye neNdawo yokuGuqulwa kweDilesi yeNethiwekhi (i-SNAT kunye ne-DNAT) kunye nokungena okulandelayo kwitheyibhile. contrack. Enye yeendlela zokusebenza ezixoxwe ngaphakathi kwaye zicetyiswe luluntu yayikukuhambisa i-DNS kwi-node yabasebenzi ngokwayo. Kule meko:
- I-SNAT ayidingeki kuba i-traffic ihlala ngaphakathi kwi-node. Akufuneki ukuba ihanjiswe kwi-interface I-eth0.
- I-DNAT ayidingeki ekubeni i-IP yendawo yokuya kwindawo yendawo, kwaye ayiyiyo i-pod ekhethiweyo ngokulandela imigaqo. iptables.
Sagqiba kwelokuba sibambelele kule ndlela. I-CoreDNS yasetyenziswa njenge-DaemonSet e-Kubernetes kwaye siphumeze iseva yendawo ye-DNS kwindawo isisombululo ipod nganye ngokuseta iflegi --iqela-dns imiyalelo cubelet . Esi sisombululo siye sasebenza kumaxesha e-DNS.
Nangona kunjalo, sabona ilahleko yepakethe kunye nokunyuka kwekhawuntara Faka_akuphumelelanga kujongano lweFlaneli. Oku kwaqhubeka emva kokuba i-workaround iphunyezwe ngenxa yokuba sakwazi ukuphelisa i-SNAT kunye / okanye i-DNAT ye-DNS traffic kuphela. Iimeko zobuhlanga zagcinelwa ezinye iintlobo zezithuthi. Ngethamsanqa, uninzi lweepakethi zethu zi-TCP, kwaye ukuba ingxaki iyenzeka zithunyelwa kwakhona. Sisazama ukufumana isisombululo esifanelekileyo kuzo zonke iintlobo zetrafikhi.
Ukusebenzisa uMthunywa ukuLinganisa uLawulo olungcono
Njengoko sasifuduka kwiinkonzo zokubuyela umva ukuya eKubernetes, saqala ukubandezeleka kumthwalo ongalungelelananga phakathi kweepod. Sifumene ukuba i-HTTP Keepalive ibangele uxhulumaniso lwe-ELB luxhonywe kwiipods zokuqala ezilungele ukuhanjiswa ngalunye. Ngaloo ndlela, ubuninzi bezithuthi buhamba ngepesenti encinci yeepods ezikhoyo. Isisombululo sokuqala esisivavanyileyo yayikukubeka iMaxSurge ukuya kwi-100% ekufakweni okutsha kwezona meko zimbi kakhulu. Umphumo wabonakala ungabalulekanga kwaye awuthembisi ngokwemiqathango yokuthunyelwa okukhulu.
Esinye isisombululo esisisebenzisileyo ibikukwandisa ngokungeyomfuneko izicelo zezibonelelo zeenkonzo ezibalulekileyo. Kule meko, ii-pods ezibekwe kufutshane ziya kuba negumbi elininzi lokuhamba xa kuthelekiswa nezinye iipods ezinzima. Ayizukusebenza ekuhambeni kwexesha kuba ibiya kuba yinkcitha-xesha yobutyebi. Ukongeza, izicelo zethu zeNode zazinomsonto omnye kwaye, ngokufanelekileyo, zinokusebenzisa undoqo omnye kuphela. Ekuphela kwesisombululo sokwenyani yayikukusebenzisa ulungelelwaniso lomthwalo olungcono.
Kudala sifuna ukwazisa ngokupheleleyo . Imeko yangoku yasivumela ukuba siyisebenzise ngendlela elinganiselwe kakhulu kwaye sifumane iziphumo ngokukhawuleza. Umthunywa ngumsebenzi ophezulu, umthombo ovulekileyo, umaleko-XNUMX we-proxy eyenzelwe izicelo ezinkulu ze-SOA. Inokusebenzisa ubuchule obuphambili bokulinganisa umthwalo, kubandakanywa ukuzama kwakhona okuzenzekelayo, i-breakers, kunye nokunciphisa izinga lehlabathi. (Phawula. transl.: Unokufunda ngakumbi ngale nto kwi malunga ne-Istio, esekwe kuMthunywa.)
Size nolu lungelelwaniso lulandelayo: yiba ne-sidecar yomthunywa kwipod nganye kunye nendlela enye, kwaye uqhagamshele iqela kwisikhongozeli sasekhaya ngezibuko. Ukunciphisa i-cascading enokubakho kunye nokugcina iradiyasi encinci yokubetha, sisebenzise iqela lee-proxy pods zangaphambili, enye ngeNdawo yokuFumana (AZ) ngenkonzo nganye. Bathembele kwi-injini yokufumanisa inkonzo elula ebhalwe ngenye yeenjineli zethu ezibuyisela nje uluhlu lweepods kwi-AZ nganye ngenkonzo enikiweyo.
Abathunywa benkonzo abangaphambili baye basebenzisa le ndlela yokufumanisa inkonzo kunye neqela elinyukayo kunye nendlela. Sibeka ixesha elaneleyo lokuphuma, sandise zonke izicwangciso ze-breaker circuit, kwaye songeza uhlengahlengiso oluncinci lokuzama kwakhona ukunceda ngokusilela okukodwa kunye nokuqinisekisa ukuhanjiswa okulungile. Sibeke i-TCP ELB phambi kwenkonzo nganye yaBathunywa. Nokuba i-keevalive evela kumaleko wethu oyintloko we-proxy ibibambekile kwezinye iipod ze-Envoy, bebesakwazi ukusingatha umthwalo ngcono kakhulu kwaye zilungiselelwe ukuba zilungelelane nge-least_request kwi-backend.
Ukusasazwa, sisebenzise i-hook ye-preStop kuzo zombini iipodi zesicelo kunye ne-sidecar pods. Ikhonkco libangele impazamo ekujongeni ubume bendawo yolawulo ebekwe kwisikhongozeli semoto esecaleni yaze yalala ixeshana ukuvumela uqhagamshelo olusebenzayo ukuba lupheliswe.
Esinye sezizathu zokuba sakwazi ukuhamba ngokukhawuleza kungenxa yeemetriki ezineenkcukacha esikwazile ukuzidibanisa ngokulula kufakelo lwePrometheus oluqhelekileyo. Oku kusivumele ukuba sibone kanye ukuba kwenzeka ntoni ngelixa sihlengahlengisa iiparamitha zoqwalaselo kunye nokusasazwa ngokutsha kwetrafikhi.
Iziphumo zazikhawuleza kwaye zazicacile. Saqala ngezona nkonzo zingalinganiyo, kwaye okwangoku zisebenza phambi kweenkonzo ze-12 ezibalulekileyo kwiqela. Kulo nyaka siceba ukutshintshela kumnatha ogcweleyo wenkonzo enokufunyanwa kwenkonzo ekumgangatho ophezulu, ukwaphulwa kwesekethe, ukufumanisa okungaphandle, ukucutha izinga kunye nokulandela umkhondo.
Umfanekiso 3-1. Ukudityaniswa kweCPU yenkonzo enye ngexesha lokutshintshela kuMthunywa
Iziphumo zokugqibela
Ngala mava kunye nophando olongezelelweyo, sakhe iqela eliqinileyo leziseko ezingundoqo elinezakhono ezomeleleyo zokuyila, ukuhambisa, kunye nokusebenza kwamaqela amakhulu e-Kubernetes. Zonke iinjineli zeTinder ngoku zinolwazi kunye namava okupakisha izikhongozeli kunye nokuthumela izicelo eKubernetes.
Xa imfuno yomthamo owongezelelweyo yavela kwiziseko ezingundoqo ezidala, kuye kwafuneka ukuba silinde imizuzu emininzi ukulungiselela iimeko ezintsha ze-EC2 ukusungulwa. Ngoku izikhongozeli ziqala ukusebenza kwaye ziqalise ukusetyenzwa kwetrafikhi ngaphakathi kwemizuzwana endaweni yemizuzu. Ukucwangcisa izikhongozeli ezininzi kumzekelo omnye we-EC2 kukwabonelela ngogxininiso oluphuculweyo oluthe tye. Ngenxa yoko, siqikelela ukuncitshiswa okukhulu kweendleko ze-EC2019 ngo-2 xa kuthelekiswa nonyaka ophelileyo.
Ukufuduka kwathatha phantse iminyaka emibini, kodwa sayigqiba ngoMatshi ka-2019. Okwangoku, iqonga le-Tinder lisebenza ngokukodwa kwiqela le-Kubernetes elibandakanya iinkonzo ezingama-200, iindawo eziyi-1000, ii-pods eziyi-15 kunye nezikhongozeli ezibalekayo ezingama-000. Iziseko zophuhliso ayiseyiyo yodwa i-domain yamaqela asebenzayo. Zonke iinjineli zethu zabelana ngolu xanduva kwaye zilawula inkqubo yokwakha kunye nokuthunyelwa kwezicelo zabo ngokusebenzisa ikhowudi kuphela.
PS evela kumguquleli
Funda kwakhona uthotho lwamanqaku kwibhlog yethu:
- «».
- «».
- «».
- «».
- «».
- «».
- «».
- «».
- «».
- «».
umthombo: www.habr.com