Cherechedza. transl.: Iyi nhoroondo inovhura ziso yeOmio β muunganidzi wekufamba wekuEurope β inotora vaverengi kubva padzidziso yekutanga kuenda kune inonakidza inoshanda intricacies yeKubernetes kumisikidza. Kujairana nezviitiko zvakadaro hakungobatsiri chete kuwedzera maonero ako, asiwo kudzivirira matambudziko asiri madiki.
Wakambove nechikumbiro chakamira panzvimbo, rega kupindura kuongororwa kwehutano, uye kutadza kufunga kuti nei? Imwe tsananguro inogoneka ine chekuita neCPU resource quota miganho. Izvi ndizvo zvatichataura nezvazvo munyaya ino.
TL; DR:
Isu tinokurudzira zvakasimba kudzima CPU miganhu muKubernetes (kana kudzima CFS quotas muKubelet) kana uri kushandisa shanduro yeLinux kernel ine CFS quota bug. Mukati mepakati serious uye tsikidzi inotungamira mukukwenya zvakanyanya uye kunonoka.
MuOmio zvese zvivakwa zvinotungamirwa naKubernetes. Ese edu ekare uye asingaverengeki mabasa anomhanya chete paKubernetes (tinoshandisa Google Kubernetes Injini). Mumwedzi mitanhatu yapfuura, takatanga kuona zvishoma nezvishoma. Zvishandiso zvinomisa kana kumira kupindura kune hutano cheki, kurasikirwa nekubatanidza kunetiweki, nezvimwe. Unhu uhwu hwakatishamisa kwenguva refu, uye pakupedzisira takasarudza kukoshesa dambudziko racho.
Pfupiso yechinyorwa:
- Mazwi mashoma nezve midziyo uye Kubernetes;
- Maitiro eCPU zvikumbiro uye miganho zvinoitwa;
- Mashandisiro anoita CPU muganho munzvimbo dzakawanda-musimboti;
- Maitiro ekutevera CPU throttling;
- Dambudziko rekugadzirisa uye nuances.
Mazwi mashoma nezve midziyo uye Kubernetes
Kubernetes ndiyo yazvino chiyero munyika yezvivakwa. Basa rayo guru ndiro orchestration yemidziyo.
Containers
Munguva yakapfuura, taifanira kugadzira zvigadzirwa seJava JARs/WARs, Python Mazai, kana zvinotemerwa kuti zvimhanye pamaseva. Zvisinei, kuti dziite kuti dzishande, rimwe basa raifanira kuitwa: kuisa nharaunda yenguva yekumhanya (Java/Python), kuisa mafaira anodiwa munzvimbo dzakakodzera, kuve nechokwadi chekuenderana neshanduro chaiyo yekushandisa system, nezvimwe. Mune mamwe mazwi, kutarisisa kwaifanira kubhadharwa kune manejimendi manejimendi (iyo yaiwanzo konzera kukakavara pakati pevagadziri uye system administrator).
Macontainer akachinja zvese. Iye zvino chigadzirwa chacho mufananidzo wemudziyo. Inogona kumiririrwa semhando yefaira rekuwedzera rinogoneka riine kwete chete chirongwa, asiwo yakazara-yakazara kuuraya nharaunda (Java/Python/...), pamwe chete nemafaira anodiwa/mapakeji, akafanoiswa uye akagadzirira kuita. run. Containers inogona kuiswa uye kumhanya pane akasiyana maseva pasina mamwe matanho ekuwedzera.
Mukuwedzera, midziyo inoshanda munzvimbo yavo yesandbox. Vane yavo chaiyo chaiyo network adapta, yavo faira hurongwa ine shoma kuwana, yavo pachavo hierarchy of maitiro, avo vasingakwanisi pamusoro CPU uye ndangariro, etc. Zvose izvi zvinoitwa nokuda inokosha subsystem yeLinux kernel - namespaces.
Kubernetes
Sezvambotaurwa, Kubernetes imudziyo orchestrator. Inoshanda seizvi: unoipa dziva remakina, wobva wati: "Hei, Kubernetes, ngatitangei zviitiko gumi zvegaba rangu ne 2 processors uye 3 GB yekuyeuka imwe neimwe, uye ichengetedze ichimhanya!" Kubernetes achachengeta zvakasara. Ichawana yemahara huwandu, vhura midziyo uye wozvitangazve kana zvichidikanwa, buritsa chigadziriso kana uchichinja shanduro, nezvimwe. Chaizvoizvo, Kubernetes inokutendera kuti ubvise chikamu chehardware uye inogadzira akasiyana masisitimu akakodzera kuendesa uye kuita maapplication.
Kubernetes kubva pakuona kwevasina basa
Ndezvipi zvikumbiro uye miganhu muKubernetes
Zvakanaka, takavhara midziyo uye Kubernetes. Isu tinoziva zvakare kuti midziyo yakawanda inogona kugara pamushini mumwe chete.
Enzaniso inogona kudhonzwa nefurati revanhu vose. Nzvimbo yakapamhama (michina/mayuniti) inotorwa nekurendwa kune akati wandei maroja (midziyo). Kubernetes anoita se realtor. Mubvunzo unomuka, sei kuchengetedza maroja kubva mukukakavadzana kune mumwe nemumwe? Zvakadini kana mumwe wavo, akati, akasarudza kukwereta imba yekugezera kwehafu yezuva?
Apa ndipo panopinda zvikumbiro nemiganhu. CPU chikumbiro inodiwa chete nokuda kwezvinangwa zvokuronga. Ichi chimwe chinhu chakaita se "chishuwo chinyorwa" chemudziyo, uye chinoshandiswa kusarudza iyo inonyanya kukodzera node. Panguva imwecheteyo CPU muganhu inogona kufananidzwa nechibvumirano chekurenda - nekukurumidza patinosarudza unit yemudziyo, iyo handikwanise pfuura miganhu yakatarwa. Uye apa ndipo panomuka dambudziko ...
Maitirwo ezvikumbiro nemiganhu zvinoitwa muKubernetes
Kubernetes inoshandisa throttling mechanism (kusvetuka wachi cycles) yakavakirwa mukernel kuita CPU miganhu. Kana application ikadarika muganho, throttling inogoneswa (kureva inogamuchira mashoma eCPU cycles). Zvikumbiro nemiganhu yekuyeuka zvakarongwa zvakasiyana, saka zviri nyore kuona. Kuti uite izvi, ingotarisa chekupedzisira restart chimiro chepodhi: ingave iri "OOMKilled". CPU throttling haisi nyore, sezvo K8s inongoita metrics kuwanikwa nekushandisa, kwete nemapoka.
CPU Chikumbiro
Maitiro eCPU chikumbiro chinoitwa sei
Kuti zvive nyore, ngatitarisei maitiro ekushandisa muchina une 4-musimboti CPU semuenzaniso.
K8s inoshandisa nzira yeboka rekutonga (cgroups) kudzora kugoverwa kwezviwanikwa (ndangariro uye processor). Muenzaniso we hierarchical unowanikwa kune iyo: mwana anogara nhaka miganhu yeboka revabereki. Mashoko ekugovera anochengetwa mune chaiyo faira system (/sys/fs/cgroup) Panyaya ye processor izvi ndizvo /sys/fs/cgroup/cpu,cpuacct/*.
K8s inoshandisa faira cpu.share kugovera zviwanikwa zve processor. Kwatiri, iyo midzi cgroup inowana 4096 migove yeCPU zviwanikwa - 100% yeiyo iripo processor simba (1 musimboti = 1024; iyi yakakosha kukosha). Chikwata chemidzi chinogovera zviwanikwa zvakaenzanirana zvichienderana nezvikamu zvevazukuru vakanyoreswa cpu.share, uye ivowo vanoita zvimwe chetezvo kuvazukuru vavo, zvichingodaro. Pane yakajairwa Kubernetes node, iro boka remidzi rine vana vatatu: system.slice, user.slice ΠΈ kubepods. Iwo maviri ekutanga mapoka madiki anoshandiswa kugovera zviwanikwa pakati peakakosha system mitoro uye mapurogiramu evashandisi kunze kweK8s. Wekupedzisira - kubepods - yakagadzirwa naKubernetes kugovera zviwanikwa pakati pemapodhi.
Dhiagiramu iri pamusoro inoratidza kuti mapoka madiki ekutanga neechipiri akagamuchira rimwe nerimwe 1024 migove, neboka rekuberpod rakagoverwa 4096 shares Izvi zvinogoneka sei: mushure mezvose, boka remidzi rinokwanisa kuwana chete 4096 migove, uye uwandu hwemigove yevana vake hunodarika iyi nhamba (6144)? Iyo poindi ndeyekuti kukosha kunoita zvine musoro, saka iyo Linux scheduler (CFS) inoshandisa iyo kugovera zvakaenzanirana CPU zviwanikwa. Muchiitiko chedu, mapoka maviri okutanga anogamuchira 680 migove chaiyo (16,6% ye4096), uye kubepod inogamuchira yakasara 2736 shares Kana nguva yekupera, mapoka maviri ekutanga haazoshandisi zviwanikwa zvakagoverwa.
Neraki, mugadziri ane maitiro ekudzivirira kutambisa isina kushandiswa CPU zviwanikwa. Inotamisa "isina basa" kune dziva repasi rose, kubva kwairi kugoverwa kumapoka anoda imwe processor simba (kuchinjisa kunoitika mumabheti kudzivirira kurasikirwa kwekutenderera). Mutoo wakafanana unoshandiswa kuvazukuru vose vedzinza.
Iyi meshini inovimbisa kugovaniswa kwakaringana kwesimba re processor uye inova nechokwadi chekuti hapana munhu anoita "kuba" zviwanikwa kubva kune vamwe.
CPU Limit
Pasinei nenyaya yekuti zvigadziriso zvemiganhu uye zvikumbiro muK8s zvinotaridzika zvakafanana, kuita kwavo kwakasiyana zvakanyanya: izvi. zvakanyanya kurasisa uye chikamu chidiki chakanyorwa.
K8s inobata kushandisa miganhu. Zvirongwa zvavo zvinotsanangurwa mumafaira cfs_period_us ΠΈ cfs_quota_us mu cgroup directory (iyo faira iriwo ipapo cpu.share).
Kusiyana cpu.share, iyo quota yakavakirwa pa nguva, uye kwete pane iripo processor simba. cfs_period_us inotsanangura nguva yenguva (epoch) - inogara iri 100000 ΞΌs (100 ms). Pane sarudzo yekushandura kukosha uku muK8s, asi inongowanikwa mualpha ikozvino. Iye anoronga anoshandisa epoch kutangazve quotas dzakashandiswa. Chechipiri faira cfs_quota_us, inotsanangura nguva iripo (quota) muchikamu chega chega. Ziva kuti inotsanangurwa zvakare mumamicroseconds. Iyo quota inogona kudarika epoch kureba; nemamwe mazwi, inogona kunge yakakura kupfuura 100 ms.
Ngatitarisei zviitiko zviviri pamichina gumi nematanhatu (yakajairika rudzi rwekombuta yatinayo kuOmio):
Scenario 1: 2 shinda uye 200 ms muganho. No throttling
Scenario 2: 10 shinda uye 200 ms muganho. Kurova kunotanga mushure memakumi maviri ms, kuwana kune processor zviwanikwa kunotangazve mushure meimwe 20 ms.
Ngatiti iwe unoisa iyo CPU muganho kune 2 kernels; Kubernetes ichashandura kukosha uku ku200 ms. Izvi zvinoreva kuti mudziyo unogona kushandisa huwandu hwe200ms yeCPU nguva pasina kupuruzira.
Uye apa ndipo panotangira mafaro. Sezvambotaurwa pamusoro, iyo inowanikwa quota ndeye 200 ms. Kana uri kushanda zvakafanana gumi shinda pamushini we12-core (ona mufananidzo wechiitiko 2), nepo mamwe mapodhi ese asina chaanoita, iyo quota inopera ne20 ms chete (sezvo 10 * 20 ms = 200 ms), uye tambo dzese dzepodhi iyi dzicharembera. Β» (mutsindo) kwe80 ms inotevera. Zvatotaurwa , nekuda kweiyo yakanyanya throttling inoitika uye mudziyo haugone kuzadzisa iyo iripo quota.
Nzira yekuongorora sei throttling mumapods?
Ingo pinda kune iyo pod uye uite cat /sys/fs/cgroup/cpu/cpu.stat.
-
nr_periods- iyo yakazara nhamba yezvirongwa zvenguva; -
nr_throttled- nhamba ye throttled nguva mukuumbwanr_periods; -
throttled_time- yakawedzera throttled nguva muma nanoseconds.
Chii chaizvo chiri kuitika?
Nekuda kweizvozvo, isu tinowana yakakwirira throttling mune ese maapplication. Dzimwe nguva anenge apinda nguva imwe nehafu yakasimba kudarika yakaverengerwa!
Izvi zvinotungamira kuzvikanganiso zvakasiyana-siyana - kugadzirira cheki kutadza, kutonhora kwemudziyo, network yekubatanidza mabreak, nguva yekubuda mukati mesevhisi mafoni. Izvi pakupedzisira zvinoguma nekuwedzera latency uye yakakwira yekukanganisa mitengo.
Sarudzo nemhedzisiro
Zvose zviri nyore pano. Isu takasiya miganhu yeCPU ndokutanga kugadzirisa OS kernel mumasumbu kune yazvino vhezheni, umo bug yakagadziriswa. Huwandu hwekukanganisa (HTTP 5xx) mumasevhisi edu hwakabva hwadonha zvakanyanya:
HTTP 5xx kukanganisa
HTTP 5xx zvikanganiso zveimwe yakakosha sevhisi
Nguva yekupindura p95
Yakakosha sevhisi chikumbiro latency, 95th percentile
Operating cost
Nhamba yemuenzaniso maawa akashandiswa
Kubata chii?
Sezvakataurwa pakutanga kwechinyorwa:
Enzaniso inogona kudhonzwa neimba inoungana vanhu... Kubernetes anoita semutengesi. Asi sei kuchengetedza maroja kubva mukukakavadzana kune mumwe nemumwe? Zvakadini kana mumwe wavo, akati, akasarudza kukwereta imba yekugezera kwehafu yezuva?
Heino kubata. Chigaba chimwe chisina hanya chinogona kudya zvese zviripo zveCPU zviwanikwa pamushini. Kana iwe uine smart application stack (somuenzaniso, JVM, Go, Node VM yakanyatsogadzirirwa), saka iyi haisi dambudziko: unogona kushanda mumamiriro ezvinhu akadaro kwenguva yakareba. Asi kana zvikumbiro zvisina kunyatsogadziriswa kana kusagadziriswa zvachose (FROM java:latest), mamiriro ezvinhu anogona kubuda kunze kwekutonga. KuOmio tine otomatiki base Dockerfiles ane akakwana default marongero eiyo huru mitauro stack, saka iyi nyaya yanga isipo.
Tinokurudzira kuongorora ma metrics (kushandiswa, saturation uye zvikanganiso), API kunonoka uye kukanganisa mitengo. Ita shuwa kuti zvawanikwa zvinozadzisa zvinotarisirwa.
nezvakanyorwa
Iyi ndiyo nyaya yedu. Zvinyorwa zvinotevera zvakabatsira zvikuru kunzwisisa zvaiitika:
- ;
- ;
- ;
- ;
- - tsvaga "cpu throttling".
Kubernetes bug inoshuma:
- ;
- ;
- .
Wakambosangana nematambudziko akafanana mukuita kwako here kana uine ruzivo rwakabatana nekudonha munzvimbo dzekugadzira midziyo? Govera nyaya yako mumashoko!
PS kubva kumushanduri
Verenga zvakare pablog yedu:
- Β«";
- Β«";
- Β«".
Source: www.habr.com