Lumelang bohle! Lebitso la ka ke Oleg Sidorenkov, ke sebetsa ho DomClick ke le moetapele oa sehlopha sa litšebeletso tsa motheo. Re 'nile ra sebelisa Cube ho rekisoa ka lilemo tse fetang tse tharo,' me nakong ena re bile le linako tse ngata tse fapaneng tse thahasellisang ka eona. Kajeno ke tla u joetsa hore na, ka mokhoa o nepahetseng, o ka pepeta ts'ebetso e eketsehileng ho tsoa ho vanilla Kubernetes bakeng sa sehlopha sa hau. Itokisetse ho tsamaea!
Kaofela le tseba hantle hore Kubernetes ke sistimi e bulehileng ea mohloli o bulehileng oa 'mino oa lijana; hantle, kapa li-binary tse 5 tse etsang boselamose ka ho laola potoloho ea bophelo ea li-microservices tsa hau sebakeng sa seva. Ntle le moo, sena ke sesebelisoa se feto-fetohang se ka bokelloang joalo ka sehahi sa Lego bakeng sa ho ikamahanya le maemo bakeng sa mesebetsi e fapaneng.
'Me ntho e' ngoe le e 'ngoe e bonahala e le ntle: lahlela li-server ka har'a sehlopha, joalo ka patsi ka har'a lebokose la mollo,' me ha u tsebe masoabi. Empa haeba u le bakeng sa tikoloho, joale u tla nahana: "Nka boloka mollo ka setofo joang 'me ka ikoahlaela moru?". Ka mantsoe a mang, mokhoa oa ho fumana mekhoa ea ho ntlafatsa litšebeletso tsa motheo le ho fokotsa litšenyehelo.
1. Boloka tlaleho ea sehlopha le lisebelisoa tsa kopo
E 'ngoe ea mekhoa e thibelang haholo empa e sebetsang hantle ke ho kenyelletsa likopo / meeli. Arola lits'ebetso ka libaka tsa mabitso, le libaka tsa mabitso ka lihlopha tsa nts'etsopele. Beha sesebelisoa pele u sebelisa boleng ba tšebeliso ea nako ea processor, memori, polokelo ea ephemeral.
resources:
requests:
memory: 2Gi
cpu: 250m
limits:
memory: 4Gi
cpu: 500mKa boiphihlelo, re fihletse qeto ea hore: ha hoa lokela ho phahamisa likopo tse tsoang meeling ka makhetlo a fetang a mabeli. Boholo ba lihlopha bo baloa ho latela likōpo, 'me haeba u beha kopo ho phapang ea lisebelisoa, mohlala, ka makhetlo a 5-10, joale nahana hore na ho tla etsahala'ng ka node ea hau ha e tletse li-pods' me ka tšohanyetso e amohela mojaro. Ha ho letho le molemo. Bonyane, throttling, 'me e le palo e kahodimodimo, bua hantle ho mosebeletsi' me u fumane mojaro oa cyclic holim'a li-node tse ling ka mor'a hore li-pods li qale ho tsamaea.
Ho feta moo, ka thuso limitranges o ka beha boleng ba lisebelisoa bakeng sa setshelo qalong - bonyane, boholo le kamehla:
➜ ~ kubectl describe limitranges --namespace ops
Name: limit-range
Namespace: ops
Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio
---- -------- --- --- --------------- ------------- -----------------------
Container cpu 50m 10 100m 100m 2
Container ephemeral-storage 12Mi 8Gi 128Mi 4Gi -
Container memory 64Mi 40Gi 128Mi 128Mi 2Hopola ho fokotsa lisebelisoa tsa sebaka sa mabitso e le hore taelo e le 'ngoe e se ke ea nka lisebelisoa tsohle tsa sehlopha:
➜ ~ kubectl describe resourcequotas --namespace ops
Name: resource-quota
Namespace: ops
Resource Used Hard
-------- ---- ----
limits.cpu 77250m 80
limits.memory 124814367488 150Gi
pods 31 45
requests.cpu 53850m 80
requests.memory 75613234944 150Gi
services 26 50
services.loadbalancers 0 0
services.nodeports 0 0Joalokaha u ka bona ho tsoa ho tlhaloso resourcequotas, haeba taelo ea ops e batla ho kenya li-pods tse tla ja 10 cpu e 'ngoe, joale mohlophisi a ke ke a lumella hore e etsoe mme o tla fana ka phoso:
Error creating: pods "nginx-proxy-9967d8d78-nh4fs" is forbidden: exceeded quota: resource-quota, requested: limits.cpu=5,requests.cpu=5, used: limits.cpu=77250m,requests.cpu=53850m, limited: limits.cpu=10,requests.cpu=10Ho rarolla bothata bo tšoanang, o ka ngola sesebelisoa, mohlala, joalo ka , e ka bolokang le ho fana ka boemo ba lisebelisoa tsa taelo.
2. Khetha molemo ka ho fetisisa faele polokelo
Mona ke rata ho ama sehlooho sa li-volumes tse tsitsitseng le tsamaiso ea disk ea Kubernetes worker node. Ke tšepa hore ha ho motho ea sebelisang "Cube" ho HDD tlhahiso, empa ka linako tse ling esita le SSD e tloaelehileng e se e ntse e sa lekana. Re ile ra tobana le bothata bo joalo hoo mapolanka a neng a bolaea disk ka ts'ebetso ea I / O, 'me ha ho na tharollo e ngata mona:
-
Sebelisa li-SSD tse sebetsang hantle haholo kapa u fetohele ho NVMe (haeba u laola lisebelisoa tsa hau).
-
Fokotsa boemo ba ho rema lifate.
-
Etsa "bohlale" ho leka-lekanya li-pods tse betang disk (
podAntiAffinity).
Setšoantšo se ka holimo se bontša se etsahalang tlas'a nginx-ingress-controller ka disk ha access_logs e nolofalitsoe (~ 12k logs/sec). Boemo bo joalo, ha e le hantle, bo ka lebisa ho senyeha ha likopo tsohle ho node ena.
Ha e le PV, oho, ha kea leka tsohle. Meqolo e phehellang. Sebelisa khetho e molemohali e u tšoanelang. Ho 'nile ha etsahala historing naheng ea rona hore karolo e nyenyane ea litšebeletso e hloka meqolo ea RWX,' me khale ba ile ba qala ho sebelisa polokelo ea NFS bakeng sa mosebetsi ona. Theko e tlaase le ... e lekaneng. Ha e le hantle, re ne re ja shit le eena - ho phela hantle, empa re ithutile ho mo fetola, 'me hlooho ea hae ha e sa le bohloko. 'Me haeba ho khoneha, fetohela ho polokelo ea ntho ea S3.
3. Haha Litšoantšo tse ntlafalitsoeng
Ho molemo ho sebelisa litšoantšo tse ntlafalitsoeng ka har'a setshelo e le hore Kubernetes a ka li lata kapele le ho li etsa hantle.
Ntlafatso e bolela hore litšoantšo:
-
e na le ts'ebeliso e le 'ngoe feela kapa e etsa mosebetsi o le mong feela;
-
boholo bo nyane, hobane litšoantšo tse kholo li fetisoa hampe ho feta marang-rang;
-
ho ba le lintlha tsa bophelo bo botle le ho itokisetsa tseo Kubernetes a ka li sebelisang ho nka khato nakong ea ho theoha;
-
sebelisa lisebelisoa tse sebetsang hantle tsa lisebelisoa (joaloka Alpine kapa CoreOS) tse hanyetsanang le liphoso tsa tlhophiso;
-
sebelisa mekhoa e mengata ea ho haha mehato e le hore u ka sebelisa lisebelisoa tse hlophisitsoeng feela eseng mehloli e tsamaeang le eona.
Ho na le lisebelisoa le lits'ebeletso tse ngata tse u lumellang hore u hlahlobe le ho ntlafatsa litšoantšo ha u fofa. Ho bohlokoa ho lula u li boloka li le morao-rao ebile li bolokehile. Ka lebaka leo, u fumana:
-
Fokotsa mojaro oa marang-rang sehlopheng sohle.
-
Nako e fokotsehileng ea ho qala setshelo.
-
Boholo bo nyane ba ngoliso eohle ea hau ea Docker.
4. Sebelisa cache ea DNS
Haeba re bua ka meroalo e phahameng, ntle le ho lokisa sistimi ea DNS ea sehlopha, bophelo bo monate haholo. Ka nako e 'ngoe, bahlahisi ba Kubernetes ba ile ba tšehetsa tharollo ea bona ea kube-dns. E ile ea boela ea kenngoa ts'ebetsong naheng ea rona, empa software ena ha ea ka ea mamela ka ho khetheha 'me ha ea ka ea fana ka ts'ebetso e hlokahalang, leha ho le joalo, ho bonahala eka mosebetsi o bonolo. Eaba ho hlaha li-coredns, tseo re ileng ra fetohela ho tsona 'me re sa tsebe ho sareloa, hamorao e ile ea e-ba tšebeletso ea kamehla ea DNS ho K8s. Ka nako e 'ngoe, re ile ra hōla ho fihlela ho 40 tse likete ho tsamaiso ea DNS,' me tharollo ena le eona e ne e sa lekana. Empa, ka monyetla oa lehlohonolo, Nodelocaldns o ile a tsoa, aka node local cache, aka .
Ke hobane'ng ha re e sebelisa? Ho na le kokoanyana ho Linux kernel eo, ha makhetlo a mangata a fihlella ka conntrack NAT holim'a UDP, e lebisang boemong ba peiso ea ho ngolla litafole tsa conntrack, 'me karolo ea sephethephethe ka NAT e lahlehile (leeto le leng le le leng ka Tšebeletso ke NAT). Nodelocaldns e rarolla bothata bona ka ho tlosa NAT le ho ntlafatsa khokahanyo ea TCP ho ea holimo DNS, hammoho le ho boloka lipotso tsa DNS sebakeng sa heno (ho kenyeletsoa le cache e khutšoanyane ea 5 ea bobeli).
5. Eketsa li-pods ka tsela e iketsang
A na u ka bua ka kholiseho hore li-microservices tsohle tsa hau li se li loketse ho eketseha ka makhetlo a mabeli ho isa ho a mararo? Mokhoa oa ho abela lisebelisoa hantle lits'ebetsong tsa hau? Ho boloka li-pods tse 'maloa tse fetang boima ba mosebetsi ho ka ba boima,' me ho li boloka morao nakong ea ho fokotsa likotsi ho tloha keketseho ea tšohanyetso ea sephethephethe ho ea tšebeletsong. Moelelo oa khauta o thusa ho fihlela spell ea ho atisa litšebeletso tse kang и .
VPA e o lumella ho phahamisa likopo / meeli ea lijana tsa hau ka bo eona ka har'a pod e ipapisitseng le ts'ebeliso ea 'nete. E ka ba molemo joang? Haeba u na le li-Pods tseo ka lebaka le itseng li ke keng tsa fokotsoa ka mokhoa o otlolohileng (o sa tšepahaleng ka ho feletseng), joale u ka leka ho tšepa VPA ho fetola mehloli ea eona. Karolo ea eona ke sistimi ea likhothaletso e ipapisitseng le datha ea nalane le ea hajoale ho tsoa ho metric-server, kahoo haeba u sa batle ho fetola likopo / meeli ka bo eona, o ka beha leihlo lisebelisoa tse khothaletsoang bakeng sa lijana tsa hau le ho ntlafatsa litlhophiso ho boloka CPU le memori. sehlopheng.
Setšoantšo se nkiloe ho https://levelup.gitconnected.com/kubernetes-autoscaling-101-cluster-autoscaler-horizontal-pod-autoscaler-and-vertical-pod-2a441d9ad231
Sehlophisi ho Kubernetes se lula se ipapisitse le likopo. Ho sa tsotellehe hore na u beha boleng bofe moo, mohlophisi o tla batla node e loketseng e thehiloeng ho eona. Boleng ba meeli bo hlokoa ke kublet e le hore u tsebe nako ea ho phunya kapa ho bolaea pod. 'Me kaha parameter ea bohlokoa feela ke boleng ba likopo, VPA e tla sebetsa le eona. Nako le nako ha u lekanya kopo ea hau ho ea holimo, u hlalosa hore na likopo li lokela ho ba life. Mme ho tla etsahalang ka meedi ka nako eo? Paramethara ena le eona e tla lekanngoa ka ho lekana.
Ka mohlala, mona ke litlhophiso tse tloaelehileng tsa pod:
resources:
requests:
memory: 250Mi
cpu: 200m
limits:
memory: 500Mi
cpu: 350mEnjene ea likhothaletso e etsa qeto ea hore kopo ea hau e hloka 300m CPU le 500Mi hore e sebetse hantle. U tla fumana li-setting tsena:
resources:
requests:
memory: 500Mi
cpu: 300m
limits:
memory: 1000Mi
cpu: 525mJoalo ka ha ho boletsoe ka holimo, sena ke tekanyo e lekanang ho ipapisitsoe le likopo/meeli ea karo-karolelano ho ponaletso:
-
CPU: 200m → 300m: karo-karolelano ea 1: 1.75;
-
Mehopolo: 250Mi → 500Mi: karolelano ea 1:2.
Mabapi le HPA, joale mochine oa ts'ebetso o hlakile haholoanyane. Liteko li behiloe bakeng sa metrics tse kang processor le memori, 'me haeba karolelano ea replicas kaofela e feta moeli, joale sekala sa kopo ka +1 pod ho fihlela boleng bo oela ka tlas'a monyako, kapa ho fihlela palo e kholo ea likopi e fihletsoe.
Setšoantšo se nkiloe ho https://levelup.gitconnected.com/kubernetes-autoscaling-101-cluster-autoscaler-horizontal-pod-autoscaler-and-vertical-pod-2a441d9ad231
Ntle le metrics e tloaelehileng joalo ka CPU le Memory, o ka beha menyako ho metrics ea hau ea tloaelo ea Prometheus mme oa sebetsa le bona haeba u ikutloa hore ena ke eona tsela e nepahetseng ka ho fetesisa ea ho tseba hore na u ka eketsa kopo ea hau neng. Hang ha ts'ebeliso e tsitsitse ka tlase ho tekanyo e boletsoeng ea metric, HPA e tla qala ho theola li-pods ho ea ho palo e tlase ea replica kapa ho fihlela mojaro o kopana le moeli o boletsoeng.
6. U se ke Ua Lebala ka Node Affinity le Pod Affinity
Ha se li-node tsohle tse tsamaeang ka thepa e tšoanang, 'me ha se li-pods tsohle tse hlokang ho sebelisa lisebelisoa tse matla haholo. Kubernetes e u lumella ho hlakisa boits'oaro ba li-node le li-pods u sebelisa Node Affinity и Pod Affinity.
Haeba u na le li-node tse loketseng ts'ebetso e matla ea compute, joale bakeng sa katleho e kholo, ho molemo ho tlama likopo ho li-node tse loketseng. Ho etsa sena, sebelisa nodeSelector e nang le node label.
Ha re re u na le li-node tse peli: e 'ngoe e nang le CPUType=HIGHFREQ le palo e kholo ea li-cores tse potlakileng, e 'ngoe e nang le MemoryType=HIGHMEMORY memori e eketsehileng le tshebetso e potlakileng. Tsela e bonolo ka ho fetisisa ke ho abela ho tsamaisoa ha pod ho node HIGHFREQka ho eketsa karolo spec khetho e kang ena:
…
nodeSelector:
CPUType: HIGHFREQTsela e theko e boima le e tobileng ea ho etsa sena ke ho e sebelisa nodeAffinity tšimong affinity razdela spec. Ho na le likhetho tse peli:
-
requiredDuringSchedulingIgnoredDuringExecution: ho beha ka thata (mohlophisi o tla kenya li-pods feela ka li-node tse itseng ('me ha ho kae kapa kae)); -
preferredDuringSchedulingIgnoredDuringExecution: tlhophiso e bonolo (mohlophisi o tla leka ho isa ho li-node tse itseng, 'me haeba e hlōleha, e tla leka ho fetisetsa ho node e latelang e fumanehang).
O ka hlakisa syntax e khethehileng bakeng sa ho laola mabitso a node, mohlala, In, NotIn, Exists, DoesNotExist, Gt kapa Lt. Leha ho le joalo, hopola hore mekhoa e rarahaneng lethathamong le lelelele la lileibole e tla liehisa ho etsa liqeto maemong a mahlonoko. Ka mantsoe a mang, u se ke ua fetelletsa lintho.
Joalokaha ho boletsoe ka holimo, Kubernetes e u lumella ho beha tlamo ea li-pods tsa hajoale. Ka mantsoe a mang, u ka etsa hore li-pods tse itseng li sebetse 'moho le li-pods tse ling sebakeng se le seng (se loketseng maru) kapa li-node.
В podAffinity masimo affinity razdela spec masimo a tšoanang a fumaneha joalo ka tabeng ea nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution и preferredDuringSchedulingIgnoredDuringExecution. Phapang feela ke hore matchExpressions e tla tlama li-pods ho node e seng e ntse e tsamaisa pod e nang le label eo.
Kubernetes e eketsehileng e fana ka tšimo podAntiAffinity, eo, ka lehlakoreng le leng, e sa tlamang pod ho node e nang le li-pods tse khethehileng.
Mabapi le lipolelo nodeAffinity Keletso e tšoanang e ka fanoa: leka ho boloka melao e le bonolo le e utloahalang, u se ke ua leka ho jarisa tlhaloso ea pod ka melao e rarahaneng. Ho bonolo haholo ho theha molao o sa lumellaneng le maemo a sehlopha, ho beha mojaro o eketsehileng ho sehlophisi le ho nyenyefatsa ts'ebetso ea kakaretso.
7. Taints & Tolerations
Ho na le mokhoa o mong oa ho laola kemiso. Haeba u na le sehlopha se seholo se nang le li-node tse makholo le likete tsa microservices, ho thata haholo ho thibela li-pods tse itseng hore li se ke tsa tšoaroa ke li-node tse itseng.
Mochine oa litšila - melao e thibelang - e thusa ka sena. Mohlala, o ka thibela li-node tse itseng ho tsamaisa li-pods maemong a itseng. Ho kenya litšila sebakeng se itseng, sebelisa khetho taint ka kubectl. Hlalosa senotlolo le boleng 'me u silafatse joalo ka NoSchedule kapa NoExecute:
$ kubectl taint nodes node10 node-role.kubernetes.io/ingress=true:NoScheduleHape ke habohlokoa ho hlokomela hore mochine oa litšila o tšehetsa liphello tse tharo tse kholo: NoSchedule, NoExecute и PreferNoSchedule.
-
NoSchedulee bolela hore ho fihlela ho na le keno e tsamaellanang le tlhaloso ea podtolerations, e ke ke ea romelloa ho node (mohlala onanode10). -
PreferNoSchedule- mofuta o nolofalitsoengNoSchedule. Tabeng ena, mohlophisi o tla leka ho se fane ka li-pods tse se nang mokhoa o ts'oanang.tolerationska node, empa sena ha se moeli o thata. Haeba ho se na lisebelisoa sehlopheng, joale li-pods li tla qala ho sebelisoa sebakeng sena. -
NoExecute- phello ena e baka ho ntšoa hang-hang ha li-pods tse se nang mokhoa o ts'oanangtolerations.
Hoa makatsa hore ebe boitšoaro bona bo ka etsolloa ka mokhoa oa ho mamella. Sena se loketse ha ho na le node "e thibetsoeng" 'me u hloka ho beha litšebeletso tsa motheo feela ho eona. Joang ho e etsa? Lumella feela li-pods tseo ho tsona ho nang le mamello e loketseng.
Mona ke hore na pod spec e ka shebahala joang:
spec:
tolerations:
- key: "node-role.kubernetes.io/ingress"
operator: "Equal"
value: "true"
effect: "NoSchedule"Sena ha se bolele hore nakong ea redeploy e latelang, pod e tla otla node ena hantle, sena ha se mochine oa Node Affinity le. nodeSelector. Empa ka ho kopanya likarolo tse 'maloa, o ka fihlela setaele se feto-fetohang haholo.
8. Beha Pod Deployment ea Bohlokoa
Hobane feela u lokiselitse litlamo tsa pod-to-node ha ho bolele hore li-pods tsohle li lokela ho tšoaroa ka mokhoa o ts'oanang. Mohlala, o kanna oa batla ho tsamaisa li-Pods pele ho tse ling.
Kubernetes e fana ka mekhoa e fapaneng ea ho beha Pod Priority le Preemption. Tlhophiso e na le likarolo tse 'maloa: ntho PriorityClass le litlhaloso tsa sebaka priorityClassName ka sebopeho sa pod. Nahana ka mohlala ona:
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: high-priority
value: 99999
globalDefault: false
description: "This priority class should be used for very important pods only"Rea bopa PriorityClass, e fane ka lebitso, tlhaloso le boleng. E phahameng value, e le ea bohlokoa ka ho fetisisa. Theko e ka ba palo efe kapa efe ea 32-bit e ka tlase ho kapa e lekana le 1. Litefiso tse holimo li boloketsoe li-pods tsa sistimi tse bohlokoa haholo, tseo hangata li ke keng tsa qojoa. Ho lelekoa ho tla etsahala feela haeba pod ea pele-pele e se na moo e ka retelehelang teng, joale tse ling tsa li-pods tse tsoang node e itseng li tla tlosoa. Haeba mochine ona o le thata haholo bakeng sa hau, joale o ka eketsa khetho preemptionPolicy: Never, 'me joale ho ke ke ha e-ba le preemption, pod e tla ba ea pele moleng' me e emetse hore mohlophisi a fumane lisebelisoa tsa mahala bakeng sa eona.
Ka mor'a moo, re theha pod, eo ho eona re hlalosang lebitso priorityClassName:
apiVersion: v1
kind: Pod
metadata:
name: static-web
labels:
role: myrole
spec:
containers:
- name: web
image: nginx
ports:
- name: web
containerPort: 80
protocol: TCP
priorityClassName: high-priority
U ka theha litlelase tse ngata tse tlang pele kamoo u ratang, leha ho khothaletsoa hore u se ke oa ts'oaroa ke sena (e re, ipehele maemo a tlase, a mahareng le a holimo).
Kahoo, haeba ho hlokahala, o ka eketsa bokhoni ba ho tsamaisa lits'ebeletso tsa bohlokoa, tse kang nginx-ingress-controller, coredns, joalo-joalo.
9. Ntlafatsa sehlopha sa hau sa ETCD
ETCD e ka bitsoa boko ba sehlopha sohle. Ho bohlokoa haholo ho boloka ts'ebetso ea database ena ka boemo bo phahameng, kaha lebelo la ts'ebetso ho "Cube" le itšetlehile ka eona. Tekanyetso e nepahetseng, 'me ka nako e ts'oanang, tharollo e ntle e tla ba ho boloka sehlopha sa ETCD ho li-node tse kholo e le hore ho be le tieho e nyane ho kube-apiserver. Haeba sena se sa khonehe, joale beha ETCD haufi le moo u ka khonang, ka bandwidth e ntle pakeng tsa barupeluoa. Hape ela hloko hore na ke li-node tse kae tse tsoang ETCD tse ka oelang ntle le kotsi ho sehlopha.
Hopola hore ho eketseha ho feteletseng ha palo ea barupeluoa sehlopheng ho ka eketsa mamello ea liphoso ka litšenyehelo tsa ts'ebetso, ntho e 'ngoe le e' ngoe e lokela ho ba ka mokhoa o itekanetseng.
Haeba re bua ka ho theha tšebeletso, joale ho na le litlhahiso tse fokolang:
-
E-ba le lisebelisoa tse ntle, tse ipapisitseng le boholo ba sehlopha (o ka bala ).
-
Fetola li-parameter tse 'maloa haeba u kentse sehlopha pakeng tsa li-DC kapa marang-rang a hau le li-disks li siea lintho tse ngata tseo u li batlang (o ka li bala. ).
fihlela qeto e
Sengoliloeng sena se hlalosa lintlha tseo sehlopha sa rona se lekang ho li latela. Sena ha se tlhaloso ea mohato ka mohato ea liketso, empa likhetho tse ka thusang ho ntlafatsa bokaholimo ba sehlopha. Ho hlakile hore sehlopha ka seng se ikhethile ka tsela ea sona, 'me litharollo tsa tokiso li ka fapana haholo, kahoo ho ka ba monate ho fumana maikutlo ho tsoa ho uena: u beha leihlo sehlopha sa hau sa Kubernetes joang, u ntlafatsa ts'ebetso ea sona joang. Arolelana phihlelo ea hau ho litlhaloso, ho tla ba monate ho e tseba.
Source: www.habr.com