"Kubernetes yakawedzera latency nekagumi": ndiani ane mhosva pane izvi?

Cherechedza. transl.: Ichi chinyorwa, chakanyorwa naGalo Navarro, uyo ane nzvimbo yePrincipal Software Engineer paEuropean company Adevinta, inonakidza uye inodzidzisa "kuongorora" mumunda wekushanda kwezvivakoti. Musoro waro wepakutanga wakawedzerwa zvishoma mukududzira nechikonzero icho munyori anotsanangura pakutanga.

Cherechedza kubva kumunyori: Inotaridzika senge iyi post kukwezvwa kutarisisa zvakanyanya kupfuura zvaitarisirwa. Ndichiri kushatirwa nemashoko okuti musoro wenyaya yacho unorasisa uye kuti vamwe vaverengi vanosuwa. Ndinonzwisisa zvikonzero zvezviri kuitika, saka, pasinei nengozi yekuparadza mhirizhonga yose, ndinoda kukurumidza kukuudza kuti chinyorwa ichi chii. Chinhu chinoshamisa chandaona zvikwata zvichitamira kuKubernetes ndechekuti pese panoitika dambudziko (sekuwedzera latency mushure mekutama), chinhu chekutanga chinopihwa mhosva ndiKubernetes, asi zvinozoitika kuti orchestrator haasi kunyatsoita. mhosva. Ichi chinyorwa chinotaura nezveimwe nyaya yakadaro. Zita rayo rinodzokorora kudanidzira kwemumwe wevagadziri vedu (gare gare iwe uchaona kuti Kubernetes haana chekuita nazvo). Iwe hauwane chero zvakazarurwa zvinokatyamadza nezve Kubernetes pano, asi iwe unogona kutarisira akati wandei ezvidzidzo zvakanaka nezve masisitimu akaomarara.

Masvondo mashoma apfuura, timu yangu yanga ichitama imwe microservice kuenda kupuratifomu yakakosha yaisanganisira CI/CD, Kubernetes-based runtime, metrics, uye zvimwe zvinonaka. Kutama kwacho kwaive kwemuyedzo: takaronga kuitora sehwaro toendesa mamwe masevhisi angangoita zana nemakumi mashanu mumwedzi iri kuuya. Vese vane basa rekushanda kwemamwe makuru makuru epamhepo mapuratifomu muSpain (Infojobs, Fotocasa, nezvimwewo).

Mushure mekunge taendesa chikumbiro kuKubernetes uye tadzosera mamwe traffic kwairi, chishamiso chinotyisa chakatimirira. Kunonoka (kunonoka) zvikumbiro muKubernetes zvaive zvakapetwa ka10 kupfuura muEC2. Kazhinji, zvaive zvakafanira kuwana mhinduro kudambudziko iri, kana kusiya kutama kweiyo microservice (uye, pamwe, iyo yese purojekiti).

Nei latency yakakwira zvakanyanya muKubernetes kupfuura muEC2?

Kuti tiwane bhodhoro, takaunganidza metrics munzira yese yekukumbira. Kuvaka kwedu kuri nyore: API gedhi (Zuul) proxies inokumbira kune microservice zviitiko muEC2 kana Kubernetes. MuKubernetes tinoshandisa NGINX Ingress Controller, uye kumashure zvinhu zvakajairika senge Deployment neJVM application paChitubu chikuva.

                                  EC2
                            +---------------+
                            |  +---------+  |
                            |  |         |  |
                       +-------> BACKEND |  |
                       |    |  |         |  |
                       |    |  +---------+  |                   
                       |    +---------------+
             +------+  |
Public       |      |  |
      -------> ZUUL +--+
traffic      |      |  |              Kubernetes
             +------+  |    +-----------------------------+
                       |    |  +-------+      +---------+ |
                       |    |  |       |  xx  |         | |
                       +-------> NGINX +------> BACKEND | |
                            |  |       |  xx  |         | |
                            |  +-------+      +---------+ |
                            +-----------------------------+

Dambudziko rakaita kunge rine chekuita nekutanga latency mumashure (ndakamaka nzvimbo ine dambudziko pagirafu se "xx"). PaEC2, mhinduro yechikumbiro yakatora anenge 20ms. MuKubernetes, iyo latency yakawedzera kusvika 100-200 ms.

Isu takakurumidza kudzinga vangangofungirwa zvine chekuita nekumhanya kwekuchinja. Iyo JVM vhezheni inoramba yakafanana. Matambudziko eContainerization zvakare aive asina chekuita nazvo: application yanga yave kushanda zvinobudirira mumidziyo paEC2. Loading? Asi isu takaona kunonoka kwakanyanya kunyangwe pachikumbiro chimwe pasekondi. Kumbomira kwekuunganidza marara kunogonawo kuregeredzwa.

Mumwe wedu Kubernetes admins akashamisika kana chikumbiro chaive nekutsamira kwekunze nekuti DNS mibvunzo yakakonzera nyaya dzakafanana kare.

Hypothesis 1: DNS zita rekugadzirisa

Pachikumbiro chega chega, application yedu inowana AWS Elasticsearch muenzaniso kamwechete kusvika katatu mudura senge elastic.spain.adevinta.com. Mukati memidziyo yedu pane goko, saka tinokwanisa kutarisa kana kutsvaga domain kunotora nguva yakareba.

DNS mibvunzo kubva mumudziyo:

[root@be-851c76f696-alf8z /]# while true; do dig "elastic.spain.adevinta.com" | grep time; sleep 2; done
;; Query time: 22 msec
;; Query time: 22 msec
;; Query time: 29 msec
;; Query time: 21 msec
;; Query time: 28 msec
;; Query time: 43 msec
;; Query time: 39 msec

Zvikumbiro zvakafanana kubva kune imwe yeEC2 zviitiko apo application iri kushanda:

bash-4.4# while true; do dig "elastic.spain.adevinta.com" | grep time; sleep 2; done
;; Query time: 77 msec
;; Query time: 0 msec
;; Query time: 0 msec
;; Query time: 0 msec
;; Query time: 0 msec

Tichifunga nezvekuti kutariswa kwacho kwakatora makumi matatu, zvakava pachena kuti DNS resolution pakuwana Elasticsearch yaive ichibatsira mukuwedzera kwekunonoka.

Zvisinei, izvi zvaishamisa nokuda kwezvikonzero zviviri:

1. Isu tatova netoni yeKubernetes maapplication ayo anodyidzana neAWS zviwanikwa pasina kutambura kubva kune yakakwira latency. Chero chikonzero, inonyatsoenderana nenyaya iyi.
2. Isu tinoziva kuti JVM inoita mu-ndangariro DNS caching. Mumifananidzo yedu, kukosha kweTTL kwakanyorwa mukati $JAVA_HOME/jre/lib/security/java.security uye isa kumasekonzi gumi: networkaddress.cache.ttl = 10. Mune mamwe mazwi, iyo JVM inofanirwa cache yese DNS mibvunzo kwemasekonzi gumi.

Kuti tisimbise fungidziro yekutanga, takasarudza kurega kufonera DNS kwechinguva uye kuona kana dambudziko raenda. Chekutanga, takasarudza kugadzirisa zvakare chishandiso kuti chitaure zvakanangana neElasticsearch neIP kero, kwete kuburikidza nezita rezita. Izvi zvinoda shanduko yekodhi uye kutumirwa kutsva, saka isu takangoita mepu iyo domain kune yayo IP kero mukati /etc/hosts:

34.55.5.111 elastic.spain.adevinta.com

Iye zvino mudziyo wakagamuchira IP kanenge ipapo ipapo. Izvi zvakaguma nekumwe kuvandudzwa, asi isu taingove pedyo zvishoma kune yaitarisirwa latency mazinga. Kunyangwe DNS resolution yakatora nguva yakareba, chikonzero chaicho chichiri kutitiza.

Diagnostics kuburikidza netiweki

Isu takasarudza kuongorora traffic kubva mumudziyo tichishandisa tcpdumpkuti uone kuti chii chaizvo chiri kuitika pane network:

[root@be-851c76f696-alf8z /]# tcpdump -leni any -w capture.pcap

Takazotumira zvikumbiro zvakati wandei uye tikadhawunirodha kubata kwavo (kubectl cp my-service:/capture.pcap capture.pcap) kuti uwedzere kuongororwa mukati Wireshark.

Pakanga pasina chinhu chekufungira nezve DNS mibvunzo (kunze kwechinhu chidiki chimwe chete chandichataura nezvacho gare gare). Asi paiva nezvimwe zvisinganzwisisike munzira iyo sevhisi yedu yaibata nayo chikumbiro chimwe nechimwe. Pazasi pane skrini yekutora inoratidza chikumbiro chichigamuchirwa mhinduro isati yatanga:

Nhamba dzepakeji dzinoratidzwa muchikamu chekutanga. Kuti zvive pachena, ndakanyora-mavara akasiyana TCP hova.

Rukova rwegirini runotanga nepakiti 328 inoratidza kuti mutengi (172.17.22.150) akamisa sei TCP yekubatanidza kune mudziyo (172.17.36.147). Mushure mekutanga kubata maoko (328-330), package 331 yakaunzwa HTTP GET /v1/.. - chikumbiro chinouya kubasa redu. The whole process yakatora 1 ms.

Iyo grey rukova (kubva packet 339) inoratidza kuti sevhisi yedu yakatumira chikumbiro cheHTTP kumuenzaniso weElasticsearch (hapana TCP kubata maoko nekuti iri kushandisa chinongedzo chiripo). Izvi zvakatora 18ms.

Parizvino zvese zvakanaka, uye nguva dzinoenderana nekunonoka kunotarisirwa (20-30 ms kana kuyerwa kubva kumutengi).

Zvisinei, chikamu chebhuruu chinotora 86ms. Chii chiri kuitika mairi? Nepacket 333, sevhisi yedu yakatumira chikumbiro cheHTTP GET ku /latest/meta-data/iam/security-credentials, uye pakarepo mushure mayo, pamusoro peiyo TCP yekubatanidza, chimwe chikumbiro cheGET ku /latest/meta-data/iam/security-credentials/arn:...

Isu takaona kuti izvi zvakadzokororwa nechikumbiro chese mukutsvaga. DNS resolution inonyatso kunonoka mumidziyo yedu (tsananguro yechiitiko ichi inonakidza, asi ini ndichaichengeta kune yakaparadzana chinyorwa). Zvakazoitika kuti chikonzero chekunonoka kwenguva refu kwaive kufona kuAWS Instance Metadata sevhisi pachikumbiro chega chega.

Hypothesis 2: kufona kusingakoshi kuAWS

Magumo ese ari maviri ndee AWS Instance Metadata API. Yedu microservice inoshandisa iyi sevhisi tichimhanyisa Elasticsearch. Mafoni ese ari maviri chikamu cheyakakosha mvumo maitiro. Mhedziso iyo inowanikwa pachikumbiro chekutanga inobata basa reIAM rine chekuita nemuenzaniso.

/ # curl http://169.254.169.254/latest/meta-data/iam/security-credentials/
arn:aws:iam::<account_id>:role/some_role

Chikumbiro chechipiri chinobvunza magumo echipiri emvumo yenguva pfupi yechiitiko ichi:

/ # curl http://169.254.169.254/latest/meta-data/iam/security-credentials/arn:aws:iam::<account_id>:role/some_role`
{
    "Code" : "Success",
    "LastUpdated" : "2012-04-26T16:39:16Z",
    "Type" : "AWS-HMAC",
    "AccessKeyId" : "ASIAIOSFODNN7EXAMPLE",
    "SecretAccessKey" : "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    "Token" : "token",
    "Expiration" : "2017-05-17T15:09:54Z"
}

Mutengi anogona kuzvishandisa kwenguva pfupi uye anofanira kugara achiwana zvitupa zvitsva (vasati vadaro Expiration) Iyo modhi iri nyore: AWS inotenderedza makiyi echinguvana kazhinji nekuda kwezvikonzero zvekuchengetedza, asi vatengi vanogona kuachengeta kwemaminetsi mashoma kutsiva chirango chekushanda chine chekuita nekuwana zvitupa zvitsva.

Iyo AWS Java SDK inofanirwa kutora basa rekuronga maitiro aya, asi nekuda kwechimwe chikonzero izvi hazviitike.

Mushure mekutsvaga nyaya paGitHub, takasangana nedambudziko #1921. Akatibatsira kuziva kwataizo“chera” mberi.

Iyo AWS SDK inogadziridza zvitupa kana imwe yeanotevera mamiriro ikaitika:

• Zuva rekupera (Expiration) Kuwira mukati EXPIRATION_THRESHOLD, hardcode kusvika kumaminitsi gumi nemashanu.
• Nguva yakawanda yapfuura kubva pakuedza kwekupedzisira kuvandudza zvitupa pane REFRESH_THRESHOLD, hardcode kwemaminitsi makumi matanhatu.

Kuti uone zuva chairo rekupera kwezvitupa zvatinogashira, isu takamhanyisa pamusoro cURL mirairo kubva kune ese mugaba uye EC2 muenzaniso. Nguva yechokwadi yechitupa yakagamuchirwa kubva mumudziyo yakave ipfupi kwazvo: chaizvo maminetsi gumi nemashanu.

Iye zvino zvinhu zvose zvave zvakajeka: nokuda kwechikumbiro chekutanga, sevhisi yedu yakagamuchira zvitupa zvenguva pfupi. Sezvo dzanga dzisiri kushanda kweanopfuura maminetsi gumi nemashanu, iyo AWS SDK yaizofunga kuvagadziridza pachikumbiro chaizotevera. Uye izvi zvakaitika nechikumbiro chese.

Sei nguva yechokwadi yezvitupa yave ipfupi?

AWS Instance Metadata yakagadzirirwa kushanda neEC2 zviitiko, kwete Kubernetes. Kune rimwe divi, isu takanga tisingadi kushandura chimiro chekushandisa. Nokuda kweizvi takashandisa KIAM - chishandiso icho, uchishandisa maajenti pane yega Kubernetes node, inobvumira vashandisi (mainjiniya anotumira zvikumbiro kuboka) kupa mabasa eIAM kumidziyo mumapodhi sekunge aive eEC2 zviitiko. KIAM inobata mafoni kuAWS Instance Metadata sevhisi uye inoagadzirisa kubva kucache yayo, yakamboagamuchira kubva kuAWS. Kubva pamaonero ekushandisa, hapana chinochinja.

KIAM inopa zvitupa zvenguva pfupi kumapods. Izvi zvine musoro tichifunga kuti avhareji yehupenyu hwepodhi ipfupi pane EC2 muenzaniso. Default yechokwadi nguva yezvitupa zvakaenzana nemaminitsi gumi nemashanu.

Nekuda kweizvozvo, kana iwe ukafukidza ese ari maviri default kukosha pamusoro peumwe neumwe, dambudziko rinomuka. Chitupa chega chega chinopihwa chikumbiro chinopera mushure memaminitsi gumi nemashanu. Nekudaro, iyo AWS Java SDK inomanikidza kuvandudzwa kwechero chitupa chine isingasviki maminetsi gumi nemashanu asara zuva rekupera risati rasvika.

Nekuda kweizvozvo, chitupa chenguva pfupi chinomanikidzwa kuvandudzwa nechikumbiro chega chega, izvo zvinosanganisira mafoni akati wandei kuAWS API uye zvichikonzera kuwedzera kwakanyanya mukunonoka. MuAWS Java SDK takawana chimiro chikumbiro, iyo inotaura nezvedambudziko rakafanana.

Mhinduro yakazova nyore. Isu takangogadzirisazve KIAM kukumbira zvitupa zvine nguva yakareba yechokwadi. Kana izvi zvangoitika, zvikumbiro zvakatanga kuyerera pasina kubatanidzwa kweAWS Metadata sevhisi, uye latency yakadonha kusvika kunyange yakaderera mazinga kupfuura muEC2.

zvakawanikwa

Zvichienderana neruzivo rwedu nekutama, imwe yeanonyanya kukonzeresa matambudziko haisi mabhugi muKubernetes kana zvimwe zvinhu zvepuratifomu. Izvo zvakare hazvigadzirise chero zvikanganiso zvakakosha mumamicroservices atiri kutakura. Matambudziko anowanzomuka nekuda kwekuti tinoisa zvinhu zvakasiyana pamwechete.

Isu tinosanganisa pamwe chete masisitimu akaomarara asina kumbobvira adyidzana nemumwe, tichitarisira kuti pamwe chete ivo vachagadzira imwechete, yakakura sisitimu. Maiwe, iyo yakawanda zvinhu, iyo yakawanda nzvimbo yezvikanganiso, inokwira iyo entropy.

Kwatiri, iyo yakakwira latency yakanga isiri mhedzisiro yebhugi kana sarudzo dzakaipa muKubernetes, KIAM, AWS Java SDK, kana microservice yedu. Yakanga iri mhedzisiro yekubatanidza maviri akazvimirira ega zvigadziriso: imwe muKIAM, imwe muAWS Java SDK. Zvichitorwa zvakaparadzana, ese maparamendi ane musoro: iyo inoshanda setifiketi yekuvandudza mutemo muAWS Java SDK, uye ipfupi nguva yechokwadi yezvitupa muKAIM. Asi kana ukaaisa pamwe chete, mibairo inova isingatarisirwi. Mhinduro mbiri dzakazvimirira uye dzine musoro hadzifanire kuita zvine musoro kana dzasanganiswa.

PS kubva kumushanduri

Iwe unogona kudzidza zvakawanda nezve dhizaini yeKIAM utility yekubatanidza AWS IAM neKubernetes pa. ichi chinyorwa kubva kuvagadziri vayo.

Uyewo verenga pa blog yedu:

• «3 nyaya dzeKubernetes kutadza mukugadzira: anti-hukama, kuvharika kwenyasha, webhook";
• «Kukosheswa kwepod muKubernetes kwakakonzera kuderera kuGrafana Labs";
• «6 inovaraidza system bugs mukushanda kweKubernetes [uye mhinduro yavo]";
• «6 nyaya dzinoshanda kubva kuSRE yedu yemazuva ese hupenyu".

Source: www.habr.com