"Kubernetes adachulukitsa latency nthawi 10": ndani ali ndi mlandu pa izi?

Zindikirani. transl.: Nkhaniyi, yolembedwa ndi Galo Navarro, yemwe ali ndi udindo wa Principal Software Engineer ku kampani ya ku Ulaya ya Adevinta, ndi "kafukufuku" wochititsa chidwi komanso wophunzitsa pa ntchito ya zomangamanga. Mutu wake woyambirira udakulitsidwa pang'ono pomasulira pazifukwa zomwe wolemba akufotokoza pachiyambi penipeni.

Chidziwitso kuchokera kwa wolemba: Zikuwoneka ngati izi anakopeka chidwi chochuluka kuposa momwe amayembekezera. Ndimamvabe mawu okwiya akuti mutu wa nkhaniyo ndi wosocheretsa komanso kuti owerenga ena ndi achisoni. Ndikumvetsa zifukwa zomwe zikuchitika, choncho, ngakhale kuti pali chiopsezo chowononga chiwembu chonsecho, ndikufuna ndikuuzeni nthawi yomweyo zomwe nkhaniyi ikunena. Chochititsa chidwi chomwe ndachiwona ngati magulu akusamukira ku Kubernetes ndikuti nthawi iliyonse pakabuka vuto (monga kuchuluka kwa latency pambuyo pa kusamuka), chinthu choyamba chomwe chimaimbidwa mlandu ndi Kubernetes, koma kenako zimakhala kuti woyimbayo sakuyenera. mlandu. Nkhaniyi ikufotokoza za nkhani ina yotero. Dzina lake limabwereza kufuula kwa m'modzi mwa omwe amatipanga (pambuyo pake mudzawona kuti Kubernetes alibe chochita nazo). Simupeza mavumbulutso odabwitsa okhudza Kubernetes pano, koma mutha kuyembekezera maphunziro angapo okhudza machitidwe ovuta.

Masabata angapo apitawa, gulu langa linali kusamuka kamphindi kakang'ono kupita ku nsanja yayikulu yomwe imaphatikizapo CI/CD, Kubernetes-based runtime, metrics, ndi zabwino zina. Kusunthaku kunali koyesa: tidakonza zozitenga ngati maziko ndikusamutsa ntchito zina pafupifupi 150 m'miyezi ikubwerayi. Onsewa ali ndi udindo wogwiritsa ntchito nsanja zazikulu kwambiri zapaintaneti ku Spain (Infojobs, Fotocasa, etc.).

Titatumiza ku Kubernetes ndikuwongolera kuchuluka kwa magalimoto komweko, zodabwitsa zowopsa zidatiyembekezera. Kuchedwa (kuchedwa) zopempha ku Kubernetes zinali zochulukirapo ka 10 kuposa mu EC2. Kawirikawiri, kunali koyenera kupeza njira yothetsera vutoli, kapena kusiya kusamuka kwa microservice (ndipo, mwinamwake, polojekiti yonse).

Chifukwa chiyani latency ili yokwera kwambiri ku Kubernetes kuposa mu EC2?

Kuti tipeze cholepheretsa, tinasonkhanitsa ma metrics panjira yonse yofunsira. Zomangamanga zathu ndizosavuta: ma proxies a API (Zuul) amapempha ku microservice zochitika mu EC2 kapena Kubernetes. Ku Kubernetes timagwiritsa ntchito NGINX Ingress Controller, ndipo kumbuyo ndi zinthu wamba ngati Kutumizidwa ndi ntchito ya JVM pa nsanja ya Spring.

                                  EC2
                            +---------------+
                            |  +---------+  |
                            |  |         |  |
                       +-------> BACKEND |  |
                       |    |  |         |  |
                       |    |  +---------+  |                   
                       |    +---------------+
             +------+  |
Public       |      |  |
      -------> ZUUL +--+
traffic      |      |  |              Kubernetes
             +------+  |    +-----------------------------+
                       |    |  +-------+      +---------+ |
                       |    |  |       |  xx  |         | |
                       +-------> NGINX +------> BACKEND | |
                            |  |       |  xx  |         | |
                            |  +-------+      +---------+ |
                            +-----------------------------+

Vutoli likuwoneka kuti likukhudzana ndi kuchedwa koyambirira kumbuyo (ndinalemba malo ovuta pa graph ngati "xx"). Pa EC2, kuyankha kwa pempho kunatenga pafupifupi 20ms. Ku Kubernetes, latency idakwera mpaka 100-200 ms.

Tidachotsa mwachangu omwe akuwakayikira okhudzana ndi kusintha kwa nthawi yoyendetsa. Mtundu wa JVM umakhalabe womwewo. Mavuto a Containerization nawonso analibe chochita ndi izi: pulogalamuyo inali ikuyenda bwino m'mitsuko ya EC2. Mukutsegula? Koma tidawona kuchedwa kwambiri ngakhale pempho limodzi pamphindikati. Kuima kotolera zinyalala kukhozanso kunyalanyazidwa.

M'modzi mwa oyang'anira athu a Kubernetes adadabwa ngati pulogalamuyi ili ndi zodalira zakunja chifukwa mafunso a DNS adayambitsanso zofanana m'mbuyomu.

Hypothesis 1: Kusintha kwa dzina la DNS

Pa pempho lililonse, ntchito yathu imapeza chitsanzo cha AWS Elasticsearch kamodzi kapena katatu mu domain ngati elastic.spain.adevinta.com. Mkati mwa zotengera zathu pali chipolopolo, kotero titha kuyang'ana ngati kusaka domain kumatenga nthawi yayitali.

Mafunso a DNS kuchokera pachidebe:

[root@be-851c76f696-alf8z /]# while true; do dig "elastic.spain.adevinta.com" | grep time; sleep 2; done
;; Query time: 22 msec
;; Query time: 22 msec
;; Query time: 29 msec
;; Query time: 21 msec
;; Query time: 28 msec
;; Query time: 43 msec
;; Query time: 39 msec

Zopempha zofanana ndi zina za EC2 pomwe pulogalamuyo ikugwira ntchito:

bash-4.4# while true; do dig "elastic.spain.adevinta.com" | grep time; sleep 2; done
;; Query time: 77 msec
;; Query time: 0 msec
;; Query time: 0 msec
;; Query time: 0 msec
;; Query time: 0 msec

Poganizira kuti kuyang'anaku kudatenga pafupifupi 30ms, zidawonekeratu kuti kusamvana kwa DNS mukamapeza Elasticsearch kunalidi kumathandizira pakuwonjezeka kwa latency.

Komabe, izi zinali zachilendo pazifukwa ziwiri:

1. Tili kale ndi matani a mapulogalamu a Kubernetes omwe amalumikizana ndi zothandizira za AWS popanda kuvutika ndi latency yayikulu. Ziribe chifukwa chake, zikukhudzana makamaka ndi nkhaniyi.
2. Tikudziwa kuti JVM imachita kukumbukira DNS caching. Pazithunzi zathu, mtengo wa TTL walembedwamo $JAVA_HOME/jre/lib/security/java.security ndikukhazikitsa masekondi 10: networkaddress.cache.ttl = 10. Mwanjira ina, JVM iyenera kusungitsa mafunso onse a DNS kwa masekondi 10.

Kuti titsimikizire lingaliro loyamba, tinaganiza zosiya kuyimba DNS kwakanthawi ndikuwona ngati vutolo latha. Choyamba, tidaganiza zokonzanso pulogalamuyo kuti ilumikizane mwachindunji ndi Elasticsearch ndi adilesi ya IP, osati kudzera mu dzina lachidziwitso. Izi zingafunike kusintha kachidindo ndi kutumizidwa kwatsopano, chifukwa chake tidangoyika malowa ku adilesi yake ya IP /etc/hosts:

34.55.5.111 elastic.spain.adevinta.com

Tsopano chidebecho chinalandira IP pafupifupi nthawi yomweyo. Izi zinapangitsa kusintha kwina, koma tinali pafupi pang'ono ndi milingo ya latency yomwe ikuyembekezeka. Ngakhale kusamvana kwa DNS kunatenga nthawi yayitali, chifukwa chenicheni sichinatipezebe.

Diagnostics kudzera pa netiweki

Tinaganiza zosanthula kuchuluka kwa magalimoto kuchokera ku chidebecho pogwiritsa ntchito tcpdumpkuti muwone zomwe zikuchitika pa intaneti:

[root@be-851c76f696-alf8z /]# tcpdump -leni any -w capture.pcap

Kenako tidatumiza zopempha zingapo ndikutsitsa zida zawo (kubectl cp my-service:/capture.pcap capture.pcap) kuti muwunikenso mu Wireshark.

Panalibe kalikonse kokayikitsa pazofunsa za DNS (kupatula kanthu kakang'ono kamene ndilankhula mtsogolo). Koma panali zovuta zina mmene utumiki wathu unkachitira pempho lililonse. Pansipa pali chithunzi chojambulidwa chomwe chikuwonetsa kuti pempho likuvomerezedwa yankho lisanayambe:

Nambala zamaphukusi zikuwonetsedwa mugawo loyamba. Kuti zimveke bwino, ndayika mitsinje yosiyanasiyana ya TCP.

Mtsinje wobiriwira woyambira ndi paketi 328 umasonyeza momwe kasitomala (172.17.22.150) adakhazikitsira mgwirizano wa TCP ku chidebe (172.17.36.147). Pambuyo pa kugwirana chanza koyamba (328-330), phukusi 331 linabweretsa HTTP GET /v1/.. - pempho lobwera ku ntchito yathu. Njira yonse idatenga 1 ms.

Mtsinje wotuwa (kuchokera pa paketi 339) ukuwonetsa kuti ntchito yathu idatumiza pempho la HTTP ku chitsanzo cha Elasticsearch (palibe TCP kugwirana chanza chifukwa ikugwiritsa ntchito kulumikizana komwe kulipo). Izi zidatenga 18ms.

Pakalipano zonse zili bwino, ndipo nthawizo zimagwirizana ndi kuchedwa komwe kumayembekezeredwa (20-30 ms poyesedwa kuchokera kwa kasitomala).

Komabe, gawo la buluu limatenga 86ms. Kodi chikuchitika ndi chiyani mmenemo? Ndi paketi 333, ntchito yathu idatumiza pempho la HTTP GET ku /latest/meta-data/iam/security-credentials, ndipo zitatha izi, pa kulumikizana komweku kwa TCP, pempho lina la GET ku /latest/meta-data/iam/security-credentials/arn:...

Tidapeza kuti izi zikubwerezedwa ndi pempho lililonse pakutsata. Kusamvana kwa DNS ndikocheperako pang'ono muzotengera zathu (mafotokozedwe a izi ndi osangalatsa, koma ndisungira nkhani ina). Zinapezeka kuti chifukwa cha kuchedwa kwanthawi yayitali chinali kuyimbira foni ku AWS Instance Metadata service pa pempho lililonse.

Hypothesis 2: mafoni osafunikira ku AWS

Mapeto onse ndi a AWS Instance Metadata API. Microservice yathu imagwiritsa ntchito ntchitoyi poyendetsa Elasticsearch. Mafoni onsewa ndi gawo lazovomerezeka zoyambira. Mapeto omwe amapezeka pa pempho loyamba amakhudza udindo wa IAM wokhudzana ndi chitsanzocho.

/ # curl http://169.254.169.254/latest/meta-data/iam/security-credentials/
arn:aws:iam::<account_id>:role/some_role

Pempho lachiwiri limafunsa kumapeto kwachiwiri kwa zilolezo zosakhalitsa za nthawi iyi:

/ # curl http://169.254.169.254/latest/meta-data/iam/security-credentials/arn:aws:iam::<account_id>:role/some_role`
{
    "Code" : "Success",
    "LastUpdated" : "2012-04-26T16:39:16Z",
    "Type" : "AWS-HMAC",
    "AccessKeyId" : "ASIAIOSFODNN7EXAMPLE",
    "SecretAccessKey" : "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    "Token" : "token",
    "Expiration" : "2017-05-17T15:09:54Z"
}

Wothandizira atha kuzigwiritsa ntchito kwakanthawi kochepa ndipo nthawi ndi nthawi ayenera kupeza ziphaso zatsopano (zisanachitike Expiration). Mtunduwu ndi wosavuta: AWS imatembenuza makiyi osakhalitsa pafupipafupi pazifukwa zachitetezo, koma makasitomala amatha kuwasunga kwa mphindi zingapo kuti alipire chiwongola dzanja chokhudzana ndi kupeza ziphaso zatsopano.

AWS Java SDK iyenera kutenga udindo wokonza ndondomekoyi, koma pazifukwa zina izi sizichitika.

Pambuyo pofufuza nkhani pa GitHub, tidakumana ndi vuto #1921. Anatithandiza kudziwa njira yopitirako.

AWS SDK imasintha satifiketi ikachitika chimodzi mwazinthu izi:

• Tsiku lothera ntchito (Expiration) Kugwa mu EXPIRATION_THRESHOLD, yokhazikika mpaka mphindi 15.
• Nthawi yochulukirapo yadutsa kuyambira kuyesa komaliza kukonzanso ziphaso kuposa REFRESH_THRESHOLD, hardcode kwa mphindi 60.

Kuti tiwone tsiku lenileni lotha ntchito ya ziphaso zomwe timalandira, tidatsata malamulo a cURL pamwambapa kuchokera ku chidebe ndi chitsanzo cha EC2. Nthawi yovomerezeka ya satifiketi yomwe idalandilidwa kuchokera pachidebe idakhala yayifupi kwambiri: ndendende mphindi 15.

Tsopano zonse zamveka bwino: pempho loyamba, utumiki wathu unalandira ziphaso zosakhalitsa. Popeza sizinali zovomerezeka kwa mphindi zopitilira 15, AWS SDK ingaganize zowasinthira pazopempha zotsatila. Ndipo izi zidachitika ndi pempho lililonse.

Chifukwa chiyani nthawi yovomerezeka ya ziphaso yakhala yayifupi?

AWS Instance Metadata idapangidwa kuti izigwira ntchito ndi zochitika za EC2, osati Kubernetes. Kumbali ina, sitinafune kusintha mawonekedwe ogwiritsira ntchito. Kwa izi tinagwiritsa ntchito KIAM - chida chomwe, pogwiritsa ntchito othandizira pa node iliyonse ya Kubernetes, chimalola ogwiritsa ntchito (mainjiniya omwe amatumiza mapulogalamu kumagulu) kuti agawire maudindo a IAM kuzinthu zomwe zili m'matumba ngati kuti zinali zochitika za EC2. KIAM imadula mafoni ku ntchito ya AWS Instance Metadata ndikuwasintha kuchokera ku cache yake, atawalandira kale kuchokera ku AWS. Kuchokera pamalingaliro ogwiritsira ntchito, palibe chomwe chimasintha.

KIAM imapereka satifiketi kwakanthawi kochepa ku ma pod. Izi ndizomveka poganizira kuti moyo wapakati wa pod ndi waufupi kuposa chitsanzo cha EC2. Nthawi yovomerezeka ya ziphaso zofanana ndi mphindi khumi ndi zisanu.

Zotsatira zake, ngati mutaphimba zonse ziwiri zosasinthika pamwamba pa wina ndi mzake, pamakhala vuto. Satifiketi iliyonse yoperekedwa ku ntchitoyo imatha pakadutsa mphindi 15. Komabe, AWS Java SDK imakakamiza kukonzanso satifiketi iliyonse yomwe yatsala ndi mphindi zosakwana 15 tsiku lake lisanathe.

Zotsatira zake, satifiketi yakanthawiyo imakakamizika kukonzedwanso ndi pempho lililonse, lomwe limaphatikizapo kuyimba mafoni angapo ku AWS API ndikupangitsa kuwonjezeka kwakukulu kwa latency. Mu AWS Java SDK tapeza pempho lazinthu, lomwe limatchula vuto lofananalo.

Yankho lake linakhala losavuta. Tidangosinthanso KIAM kuti ipemphe satifiketi yokhala ndi nthawi yayitali yovomerezeka. Izi zitachitika, zopempha zinayamba kuyenda popanda kutenga nawo mbali pa ntchito ya AWS Metadata, ndipo latency inatsikira kumagulu otsika kwambiri kuposa EC2.

anapezazo

Kutengera zomwe takumana nazo pakusamuka, chimodzi mwazinthu zomwe zimayambitsa zovuta si nsikidzi ku Kubernetes kapena zinthu zina papulatifomu. Komanso sichithana ndi zolakwika zilizonse mu ma microservices omwe tikuyenda. Nthawi zambiri mavuto amadza chifukwa chophatikiza zinthu zosiyanasiyana.

Timasakaniza pamodzi machitidwe ovuta omwe sanagwirizanepo kale, kuyembekezera kuti palimodzi adzapanga dongosolo limodzi, lalikulu. Kalanga, zinthu zambiri, malo ochulukirapo amalakwitsa, ndiye kuti entropy imakwera.

Kwa ife, kuchedwa kwambiri sikunali chifukwa cha nsikidzi kapena zisankho zoipa ku Kubernetes, KIAM, AWS Java SDK, kapena microservice yathu. Zinali zotsatira za kuphatikiza zosintha ziwiri zodziyimira pawokha: imodzi ku KIAM, ina mu AWS Java SDK. Kutengera padera, magawo onsewa ndi omveka: mfundo yotsitsimutsa satifiketi mu AWS Java SDK, ndi nthawi yayifupi yovomerezeka ya ziphaso ku KAIM. Koma mukawaphatikiza, zotsatira zake zimakhala zosayembekezereka. Njira ziwiri zodziyimira pawokha komanso zomveka siziyenera kukhala zomveka zikaphatikizidwa.

PS kuchokera kwa womasulira

Mutha kudziwa zambiri zamamangidwe a KIAM utility kuphatikiza AWS IAM ndi Kubernetes pa nkhaniyi kuchokera kwa omwe adalilenga.

Komanso werengani pa blog yathu:

• «Nkhani 3 zakulephera kwa Kubernetes pakupanga: anti-affinity, shutdown mwachisomo, webhook";
• «Momwe zinthu zofunika kwambiri ku Kubernetes zidayambitsa kutsika ku Grafana Labs";
• «Zosangalatsa za 6 pakugwira ntchito kwa Kubernetes [ndi yankho lawo]";
• «Nkhani 6 zothandiza pa moyo wathu watsiku ndi tsiku wa SRE".

Source: www.habr.com