Cherechedza. transl.: Vashandi vepasi rose vane mukurumbira Tinder sevhisi nguva pfupi yadarika vakagovana humwe hutekinoroji hwekutamisa zvivakwa zvavo kuKubernetes. Kuita kwacho kwakatora makore anoda kusvika maviri uye kwakakonzera kutangwa kwepuratifomu yakakura kwazvo paK8s, ine masevhisi mazana maviri anogarwa pa200 zviuru zvemidziyo. Ndeapi matambudziko anonakidza akasangana nemainjiniya eTinder uye ndeapi mhedzisiro yavakasvika? Verenga shanduro iyi.
Sei?
Anenge makore maviri apfuura, Tinder yakafunga kutamisa chikuva chayo kuKubernetes. Kubernetes yaizobvumira timu yeTinder kuti igadzikane uye ifambe mukugadzira nekuita kushoma kuburikidza nekutumirwa kusingachinji. (kutumirwa kusingachinji). Muchiitiko ichi, kuungana kwezvikumbiro, kutumirwa kwavo, uye zvivakwa pachazvo zvaizotsanangurwa zvakasarudzika nekodhi.
Takanga tichitsvagawo mhinduro kudambudziko re scalability uye kugadzikana. Kana kuyera kwave kunetsa, taiwanzomirira maminetsi akati wandei kuti zviitiko zvitsva zveEC2 zvifambe. Pfungwa yekuvhura midziyo uye kutanga kushandira traffic mumasekonzi panzvimbo yemaminetsi yakava yakanaka kwatiri.
Kuita kwacho kwakava kwakaoma. Munguva yekutama kwedu mukutanga kwa2019, iyo Kubernetes cluster yakasvika kune yakaoma uye takatanga kusangana nematambudziko akasiyana nekuda kwehuwandu hwetraffic, saizi yemasumbu, uye DNS. Tiri munzira, takagadzirisa matambudziko akawanda anonakidza ane chekuita nekutama masevhisi mazana maviri uye kuchengetedza Kubernetes cluster ine 200 nodes, 1000 pods uye 15000 midziyo inomhanya.
Sei?
Kubva muna Ndira 2018, takapfuura nematanho akasiyana-siyana ekutama. Takatanga nekuisa masevhisi edu ese uye nekuaendesa kuKubernetes test Cloud nharaunda. Kutanga muna Gumiguru, takatanga kufambisa masevhisi ese aripo kuKubernetes. Pakazosvika Kurume wegore rinotevera, takapedza kutama uye ikozvino Tinder chikuva chinomhanya chete paKubernetes.
Kugadzira mifananidzo yeKubernetes
Isu tine anopfuura makumi matatu sosi kodhi kodhi yemamicroservices anomhanya pane Kubernetes cluster. Iyo kodhi mune idzi repositori yakanyorwa mumitauro yakasiyana (semuenzaniso, Node.js, Java, Scala, Go) ine akawanda ekumhanya nharaunda yemutauro mumwe chete.
Iyo yekuvaka sisitimu yakagadzirirwa kupa yakazara inogoneka "kuvaka mamiriro" kune yega microservice. Iyo inowanzo ine Dockerfile uye runyorwa rwemirairo yegomba. Zviri mukati mavo zvinogoneka zvachose, uye panguva imwechete, ese aya anovaka mamiriro anonyorwa zvinoenderana neyakajairwa fomati. Kumisa mamiriro ekuvaka kunobvumira imwechete kuvaka sisitimu yekubata ese mamicroservices.
Mufananidzo 1-1. Yakamisikidzwa kuvaka maitiro kuburikidza neBuilder mudziyo
Kuti uwane kuwirirana kwakanyanya pakati penguva dzekumhanya (nzvimbo dzekumhanya) iyo imwechete yekuvaka maitiro inoshandiswa panguva yekuvandudza uye kuyedzwa. Takatarisana nedambudziko rinonakidza: isu taifanira kugadzira nzira yekuona kuenderana kwenzvimbo yekuvaka papuratifomu yese. Kuti uite izvi, maitiro ese egungano anoitwa mukati memudziyo unokosha. muvaki.
Kuitwa kwake kwemudziyo kwaida hunyanzvi hweDocker. Muvaki anogara nhaka yemuno mushandisi ID uye zvakavanzika (seSSH kiyi, AWS zvitupa, nezvimwewo) zvinodiwa kuti uwane yakavanzika Tinder repositori. Inokwirisa madhairekitori emunharaunda ane masosi ekuchengetedza masikirwo ekuvaka zvigadzirwa. Iyi nzira inonatsiridza mashandiro nekuti inobvisa kukosha kwekukopa kuvaka zvigadzirwa pakati peMuvaki mudziyo nemugamuchiri. Zvigadzirwa zvekuvaka zvakachengetwa zvinogona kushandiswa zvakare pasina imwe gadziriso.
Kune mamwe masevhisi, taifanira kugadzira chimwe chigadziko chekumisikidza nharaunda yekubatanidza kune inomhanya nharaunda (semuenzaniso, iyo Node.js bcrypt raibhurari inoburitsa chikuva-chaiwo mabhinari artifacts panguva yekuisa). Munguva yekuunganidza maitiro, zvinodiwa zvinogona kusiyana pakati pemasevhisi, uye iyo yekupedzisira Dockerfile inounganidzwa panhunzi.
Kubernetes cluster architecture uye kutama
Cluster size management
Takasarudza kushandisa kube-aws kune otomatiki cluster deployment paAmazon EC2 zviitiko. Pakutanga, zvese zvakashanda mune imwechete dziva remanodhi. Isu takakurumidza kuona kukosha kwekuparadzanisa mitoro yebasa nehukuru uye muenzaniso mhando kuti tishandise zvakanyanya zviwanikwa. Mhedziso yaive yekuti kumhanya akati wandei akaremerwa mapodhi ane shinda dzakazove dzinofanotaurwa maererano nekuita pane kugarisana kwavo nenhamba huru yemapodhi ane tambo imwe chete.
Pakupedzisira takagadzirisa:
- m5.4xlarge - yekutarisa (Prometheus);
- c5.4 yakakura - yeNode.js basa rekushanda (rimwe-tambo yebasa);
- c5.2 yakakura - yeJava uye Go (yakawanda yakapetwa basa);
- c5.4 yakakura - yepaneru yekutonga (3 nodes).
Kutama
Imwe yematanho ekugadzirira kutama kubva kune yekare zvivakwa kuenda Kubernetes yaive yekudzosera kutaurirana kwakatwasuka pakati pemasevhisi kune mitsva yekutakura mitoro (Elastic Load Balancers (ELB). Ivo vakagadzirwa pane chaiyo subnet yeyakavanzika yakavanzika gore (VPC). Iyi subnet yakabatana neKubernetes VPC. Izvi zvakatibvumira kutamisa mamodule zvishoma nezvishoma, tisingatarise kurongeka chaiko kwekutsamira kwesevhisi.
Aya mamagumo akagadzirwa pachishandiswa huremu seti yeDNS marekodhi aive nemaCNAME anongedza kune yega yega ELB itsva. Kuti tishandure, takawedzera hutsva hutsva hunonongedza kune ELB itsva yebasa reKubernetes ine uremu hwe 0. Takazogadzirisa Nguva Yekurarama (TTL) yekupinda yakaiswa ku 0. Mushure meizvi, zviremu zvekare uye zvitsva zvaive. yakagadziridzwa zvishoma nezvishoma, uye pakupedzisira 100% yemutoro yakatumirwa kune sevha itsva. Mushure mekuchinja kwapera, kukosha kweTTL kwakadzokera kune imwe nhanho yakakwana.
Iwo maJava modules ataive nawo aigona kurarama neakaderera TTL DNS, asi maNode maapplication aisagona. Mumwe wemainjiniya anonyora patsva chikamu chekubatanidza dziva kodhi ndokuiputira mune maneja anovandudza madziva ese makumi matanhatu masekonzi. Iyo yakasarudzwa nzira yakashanda zvakanaka kwazvo uye pasina chero inocherekedza maitiro ekuderera.
Zvidzidzo
Iyo Limits yeNetwork Fabric
Mangwanani-ngwanani aNdira 8, 2019, chikuva cheTinder chakadonha zvisingatarisirwi. Mukupindura kukuwedzera kusingawirirani kwepuratifomu latency mangwanani iwayo, nhamba yemapodhi nemanodhi musumbu yakawedzera. Izvi zvakaita kuti cache yeArP ive yapera simba pamanode edu ese.
Pane matatu eLinux sarudzo ane chekuita neArP cache:
()
gc_thresh3 - iyi yakaoma muganhu. Kuonekwa kwe "tafura yevavakidzani inofashukira" murogi yaireva kuti kunyangwe mushure mekuunganidzwa kwemarara esynchronous (GC), pakanga pasina nzvimbo yakakwana muARP cache yekuchengetedza yekupinda yevavakidzani. Muchiitiko ichi, kernel yakangorasa pakiti zvachose.
Tinoshandisa semucheka wetiweki muKubernetes. Mapaketi anofambiswa pamusoro peVXLAN. VXLAN inzira yeL2 yakasimudzwa pamusoro peL3 network. Iyo tekinoroji inoshandisa MAC-mu-UDP (MAC Kero-mu-Mushandisi Datagram Protocol) encapsulation uye inobvumira kuwedzera kweLayer 2 network zvikamu. Iyo yekufambisa protocol pane yepanyama data centre network ndeye IP pamwe neUDP.
Mufananidzo 2-1. Flannel diagram ()
Mufananidzo 2-2. VXLAN package ()
Imwe neimwe Kubernetes mushandi node inogovera chaiyo kero nzvimbo ine / 24 mask kubva kune yakakura / 9 block. Kune imwe neimwe node izvi ndizvo imwe yekupinda mutafura yekufambisa, imwe yekupinda mutafura yeARP (pane flannel.1 interface), uye imwe yekupinda mutafura yekuchinjisa (FDB). Vanowedzerwa kekutanga apo node yevashandi inotangwa kana pese panowanikwa node itsva.
Pamusoro pezvo, node-pod (kana pod-pod) kutaurirana kunopedzisira kwaenda kuburikidza neiyo interface eth0 (sezvinoratidzwa mumufananidzo weFlannel pamusoro). Izvi zvinoguma nekuwedzera kwekupinda mutafura yeARP kune yega yega inoenderana sosi uye kwekuenda.
Munharaunda yedu, rudzi urwu rwekutaurirana rwakanyanya. Zvezvinhu zvesevhisi muKubernetes, ELB inogadzirwa uye Kubernetes inonyoresa node imwe neimwe neELB. Iyo ELB haina chainoziva nezvepods uye node yakasarudzwa inogona kunge isiri iyo yekupedzisira yekuenda kwepakiti. Pfungwa ndeyokuti kana node inogamuchira pakiti kubva kuELB, inoiona ichifunga nezvemitemo iptables kune imwe sevhisi uye inosarudza podhi pane imwe node.
Panguva yekutadza, pakanga paine 605 node musumbu. Nokuda kwezvikonzero zvataurwa pamusoro apa, izvi zvakanga zvakakwana kukurira kukosha gc_thresh3, inova iyo yakasarudzika. Kana izvi zvikaitika, kwete chete mapaketi anotanga kudonhedzwa, asi iyo yese Flannel chaiyo kero nzvimbo ine / 24 mask inonyangarika kubva patafura yeARP. Node-pod kutaurirana uye DNS mibvunzo inokanganiswa (DNS inotambirwa musumbu; verenga gare gare muchinyorwa chino kuti uwane rumwe ruzivo).
Kuti ugadzirise dambudziko iri, unofanirwa kuwedzera maitiro gc_thresh1, gc_thresh2 ΠΈ gc_thresh3 uye tangazve Flannel kuti unyore zvakare ma network asipo.
Zvisingatarisirwi DNS kuyera
Munguva yekutama, isu takashinga kushandisa DNS kubata traffic uye zvishoma nezvishoma kutamisa masevhisi kubva kune yekare zvivakwa kuenda Kubernetes. Isu tinoseta yakaderera TTL kukosha kune yakabatana RecordSets muRoute53. Apo zvivakwa zvekare zvaimhanya paEC2 zviitiko, kugadzirisa kwedu kugadzirisa kwakanongedza kuAmazon DNS. Isu takatora izvi sezvisina basa uye kukanganisa kweiyo yakaderera TTL pamasevhisi edu neAmazon masevhisi (akadai seDynamoDB) haana kunyanya kucherechedzwa.
Sezvo isu takatama masevhisi kuenda Kubernetes, takaona kuti DNS yaigadzirisa zviuru mazana maviri nemakumi mashanu zvikumbiro pasekondi. Nekuda kweizvozvo, zvikumbiro zvakatanga kusangana nguva dzose uye yakakomba nguva yekubuda kweDNS mibvunzo. Izvi zvakaitika zvisinei nekuedza kunoshamisa kwekugadzirisa uye kushandura mupi weDNS kuCoreDNS (iyo pakuremerwa kwakasvika zana mapodhi achimhanya pa250 cores).
Tichiri kutsvagira zvimwe zvinokonzeresa nemhinduro, takawana , inotsanangura mamiriro emujaho anokanganisa pakiti yekusefa net sefa muLinux. Nguva dzekubuda dzatakaona, pamwe nekaunda iri kuwedzera insert_foiled muFlannel interface yaienderana nezvakawanikwa pachinyorwa.
Dambudziko rinoitika padanho reKwakabva uye Destination Network Kero Dudziro (SNAT neDNAT) uye kunotevera kupinda mutafura. kubvumirana. Imwe yemaworkaround akakurukurwa mukati uye akakurudzirwa nenharaunda yaive yekufambisa iyo DNS kune yevashandi node pachayo. Muchiitiko ichi:
- SNAT haidiwe nekuti traffic inogara mukati me node. Izvo hazvidi kufambiswa kuburikidza neiyo interface eth0.
- DNAT haidiwi sezvo nzvimbo yekuenda IP inzvimbo kune node, uye kwete podhi yakasarudzwa yakasarudzwa maererano nemitemo. iptables.
Takasarudza kuomerera nenzira iyi. CoreDNS yakaiswa seDaemonSet muKubernetes uye isu takaita yemuno node DNS server mu. chisimba.conf podhi imwe neimwe nekuisa mureza --cluster-dns mirairo cubeletβ. Iyi mhinduro yakave inoshanda kune DNS nguva yekubuda.
Nekudaro, isu tichiri kuona kurasikirwa kwepaketi uye kuwedzera kwekaunda insert_foiled muFlannel interface. Izvi zvakaenderera mberi mushure mekunge workaround yaitwa nekuti isu takakwanisa kubvisa SNAT uye / kana DNAT yeDNS traffic chete. Mamiriro emakwikwi akachengetedzwa kune mamwe marudzi emotokari. Sezvineiwo, mazhinji emapaketi edu iTCP, uye kana dambudziko rikaitika anongotumirwazve. Tichiri kuedza kutsvaga mhinduro yakakodzera kune ese marudzi e traffic.
Kushandisa Envoy kune Zvirinani Load Bancing
Sezvo isu takatama masevhisi ekumashure kuenda kuKubernetes, takatanga kutambura nemutoro usina kuenzana pakati pemapods. Isu takaona kuti HTTP Keepalive yakakonzera ELB kubatana kusungirira pamapodhi ekutanga akagadzirira ega ega kutumirwa. Nekudaro, iyo yakawanda yetraffic yakapfuura nepadiki muzana yemapodhi anowanikwa. Mhinduro yekutanga yatakaidza yaive yekumisikidza MaxSurge ku100% pane nyowani deployments yeakanyanya kesi mamiriro. Mhedzisiro yacho yakave isingakoshi uye isingavimbisi maererano nekutumirwa kukuru.
Imwe mhinduro yatakashandisa yaive yekuwedzera zvikumbiro zvezvishandiso zvemasevhisi akakosha. Muchiitiko ichi, mapods akaiswa padhuze angave ane nzvimbo yakawanda yekufambisa kana ichienzaniswa nemamwe mapodhi anorema. Zvaisazoshanda mukufamba kwenguva nekuti kwaizova kutambisa zviwanikwa. Pamusoro pezvo, maNode edu ekushandisa aive ega-tambo uye, nekudaro, aingogona kushandisa imwechete musimboti. Mhinduro chaiyo yaive yekushandisa zvirinani kuyera mutoro.
Tagara tichida kuonga zvizere . Mamiriro ezvinhu aripo akatibvumira kuti tiiise nenzira shoma uye tiwane mhinduro pakarepo. Envoy ndeyepamusoro-inoshanda, yakavhurika-sosi, layer-XNUMX proxy yakagadzirirwa yakakura SOA application. Iyo inogona kushandisa epamberi mitoro yekuyera matekiniki, anosanganisira otomatiki retries, macircuit breakers, uye pasirese chiyero. (Cherechedza. transl.: Unogona kuverenga zvakawanda nezve izvi mukati nezve Istio, iyo yakavakirwa paNhume.)
Isu takauya neiyo inotevera gadziriso: iva neEnvoy sidecar yepodhi yega yega uye nzira imwechete, uye batanidza sumbu kune mudziyo munharaunda kuburikidza nechiteshi. Kuti tideredze kudonha uye kuchengetedza diki inorova radius, takashandisa boka reEnvoy front-proxy pods, imwe paAvailability Zone (AZ) yesevhisi yega yega. Ivo vaivimba neinjini yakapusa yekuwana sevhisi yakanyorwa neimwe yeinjiniya yedu yaingodzosa runyoro rwemapodhi muAZ yega yega yebasa rakapihwa.
Sevhisi kumberi-Nhume dzakabva dzashandisa iyi sevhisi yekuwana sevhisi nerukova sumbu uye nzira. Isu takaisa nguva dzakakwana, takawedzera ese edunhu breaker marongero, uye takawedzera kushoma kuyedza kugadzirisa kuti tibatsire nekutadza kumwe chete uye kuve nechokwadi chekutumira. Takaisa TCP ELB pamberi peimwe neimwe yeaya masevhisi pamberi-Nhume. Kunyangwe kana iyo keepalive kubva kune yedu main proxy layer yakanamira pane mamwe Evoy pods, vaive vachiri kukwanisa kubata mutoro zvirinani uye vakagadziridzwa kuti vaenzanise kuburikidza nediki_request mubackend.
Kuendesa, takashandisa iyo preStop hook pane ese mapodhi ekushandisa uye sidecar pods. Hoko yakakonzeresa chikanganiso mukutarisa mamiriro eiyo admin endpoint iri padivi remota chigaba ndokuenda kunorara kwekanguva kuti ibvumire inoshanda ma connections kupera.
Chimwe chezvikonzero zvatakakwanisa kufamba nekukurumidza imhaka yemametriki akadzama atakakwanisa kubatanidza nyore nyore mune yakajairwa Prometheus kuisirwa. Izvi zvakatibvumira kuti tinyatsoona zvaiitika isu tichigadzirisa maparamita ekugadzirisa uye kugoverazve traffic.
Migumisiro yacho yakanga iri pakarepo uye yakajeka. Takatanga nemasevhisi asina kuenzana, uye panguva ino inoshanda pamberi peiyo 12 inonyanya kukosha masevhisi musumbu. Gore rino tiri kuronga shanduko kuenda kune yakazara sevhisi mesh ine yakawedzera sevhisi kuwanikwa, kutyora kwedunhu, kuona kunze, kumisa chiyero uye kutsvaga.
Mufananidzo 3-1. CPU convergence yeimwe sevhisi panguva yekuchinja kuenda kuEnvoy
Mhedzisiro yekupedzisira
Kuburikidza nechiitiko ichi uye nekuwedzera tsvagiridzo, takavaka timu yakasimba yezvivakwa ine hunyanzvi hwakasimba mukugadzira, kutumira, nekushandisa masumbu makuru eKubernetes. Vese mainjiniya eTinder parizvino vane ruzivo uye ruzivo rwekurongedza midziyo uye kuendesa zvikumbiro kuKubernetes.
Pakamuka kudiwa kwekuwedzera pane zvivakwa zvekare, taifanira kumirira maminetsi akati wandei kuti zviitiko zvitsva zveEC2 zvitange. Iye zvino midziyo inotanga kumhanya uye inotanga kugadzirisa traffic mukati memasekondi pane maminetsi. Kuronga midziyo yakawanda pane imwe chete EC2 muenzaniso inopawo yakagadziridzwa yakachinjika kutarisisa. Nekuda kweizvozvo, isu tinofanotaura kudzikiswa kwakanyanya mumitengo yeEC2019 muna 2 zvichienzaniswa negore rapfuura.
Kutama kwacho kwakatora makore maviri, asi takaipedza munaKurume 2019. Parizvino, iyo Tinder chikuva inomhanya chete paKubernetes cluster ine mazana maviri masevhisi, chiuru nodes, 200 pods uye 1000 midziyo inomhanya. Infrastructure haisisiri iyo yega nzvimbo yezvikwata zvekushanda. Vese mainjiniya edu vanogovana basa iri uye vanodzora maitiro ekuvaka uye kutumira maapplication avo vachishandisa kodhi chete.
PS kubva kumushanduri
Verenga zvakare nhevedzano yezvinyorwa pane yedu blog:
- Β«".
- Β«".
- Β«".
- Β«".
- Β«".
- Β«".
- Β«".
- Β«".
- Β«".
- Β«".
Source: www.habr.com