Hlokomela phetolelo: Ena ke phetolelo ea tlhahlobo ea lefu la sechaba ho tsoa ho blog ea boenjiniere ea k'hamphani . E hlalosa bothata ba ho kena-kenana le sehlopha sa Kubernetes, se lebisitseng ho fokotseheng ha litšebeletso tse ling tsa tlhahiso.
Sengoliloeng sena se ka ba molemo ho ba batlang ho ithuta ho eketsehileng ka postmortems kapa ho thibela mathata a mang a ka bang teng a DNS nakong e tlang.
Sena ha se DNS
E ka se be DNS
E ne e le DNS
Hanyane ka li-postmortem le lits'ebetso ho Preply
Postmortem e hlalosa ho se sebetse hantle kapa ketsahalo e itseng tlhahisong. The postmortem e kenyelletsa nako ea liketsahalo, phello ea basebelisi, sesosa sa motheo, liketso tse nkiloeng, le lithuto tse ithutoang.
Likopanong tsa beke le beke le pizza, har'a sehlopha sa tekheniki, re arolelana lintlha tse fapaneng. E 'ngoe ea likarolo tsa bohlokoa tsa liboka tse joalo ke post-mortems, eo hangata e tsamaeang le nehelano e nang le li-slide le tlhahlobo e tebileng ea ketsahalo eo. Leha re sa opa liatla ka mor'a ho hlahlojoa ha lefu, re leka ho theha moetlo oa "ho hloka molato" (). Re lumela hore ho ngola le ho hlahisa li-postmortems ho ka re thusa (le ba bang) ho thibela liketsahalo tse tšoanang nakong e tlang, ke kahoo re li arolelanang.
Batho ba amehang ketsahalong e itseng ba lokela ho ikutloa hore ba ka bua ka botlalo ba sa tšabe kotlo kapa kotlo. Ha ho molato! Ho ngola postmortem ha se kotlo, empa ke monyetla oa ho ithuta bakeng sa k'hamphani eohle.
Mathata le DNS ho Kubernetes. Postmortem
Letsatsi: 28.02.2020
Bangoli: Amet U., Andrey S., Igor K., Alexey P.
Boemo: E felile
Ka bokhutšoanyane: Ho se fumanehe ka mokhoa o itseng oa DNS (26 min) bakeng sa lits'ebeletso tse ling sehlopheng sa Kubernetes
Tšusumetso: Liketsahalo tse 15000 tse lahlehileng bakeng sa litšebeletso tsa A, B le C
sesosa sa ho qala: Kube-proxy ha e khone ho tlosa ka nepo keno ea khale ho tsoa tafoleng ea contrack, kahoo lits'ebeletso tse ling li ne li ntse li leka ho hokela ho li-pods tse seng teng.
E0228 20:13:53.795782 1 proxier.go:610] Failed to delete kube-system/kube-dns:dns endpoint connections, error: error deleting conntrack entries for UDP peer {100.64.0.10, 100.110.33.231}, error: conntrack command returned: ...Qalisa: Ka lebaka la mojaro o tlase ka har'a sehlopha sa Kubernetes, CoreDNS-autoscaler e ile ea fokotsa palo ea li-pods ho tloha ho tse tharo ho isa ho tse peli.
tharollo: Ts'ebetso e latelang ea ts'ebeliso e ile ea qala ho theha li-node tse ncha, CoreDNS-autoscaler e kentse li-pods tse ling ho sebeletsa sehlopha, e leng se ileng sa etsa hore ho ngoloe bocha tafole ea conntrack.
Ho fumanoa: Tlhokomelo ea Prometheus e fumane palo e kholo ea liphoso tsa 5xx bakeng sa lits'ebeletso A, B le C mme ea qala mohala ho lienjineri tse sebetsang.
5xx liphoso ho Kibana
Liketso
e le hantle
Tšoaea
Ho ikarabella
Sepheo
Tlosa autoscaler bakeng sa CoreDNS
thibetsoe
Amet U.
DEVOPS-695
Hlophisa seva ea caching ea DNS
fokotseha
Max V.
DEVOPS-665
Beha leihlo la contrack
thibetsoe
Amet U.
DEVOPS-674
Lithuto Tse Ithutiloeng
Se tsamaileng hantle:
- Tlhokomelo e sebelitse hantle. Karabelo e ne e potlakile ebile e hlophisehile
- Ha rea ka ra otla meeli leha e le efe ho li-node
Se ne se fosahetse:
- Ho ntse ho sa tsejoe sesosa sa 'nete, se ts'oanang le ka contrack
- Ketso tsohle li lokisa litlamorao feela, eseng sesosa (bug)
- Re ne re tseba hore haufinyane re ka 'na ra ba le mathata ka DNS, empa ha rea ka ra etelletsa mesebetsi pele
Moo re bileng lehlohonolo:
- Phallo e latelang e ile ea hlahisoa ke CoreDNS-autoscaler, e ileng ea hlakola tafole ea contrack.
- Phoso ena e amme litšebeletso tse ling feela
Lenane la nako (EET)
Nako
e le hantle
22:13
CoreDNS-autoscaler e fokolitse palo ea li-pods ho tloha ho tse tharo ho isa ho tse peli
22:18
Lienjineri tse neng li le mosebetsing li ile tsa qala ho fumana mehala ho tsoa ho sistimi e behang leihlo
22:21
Baenjiniere ba mosebetsing ba ile ba qala ho fumana sesosa sa liphoso.
22:39
Baenjiniere ba mosebetsing ba ile ba qala ho khutlisetsa e 'ngoe ea litšebeletso tsa morao-rao ho mofuta o fetileng
22:40
5xx liphoso li ile tsa khaotsa ho hlaha, boemo bo tsitsitse
- Nako ea ho fumana: Metsotso ea 4
- Nako pele ho ts'ebetso: Metsotso ea 21
- Nako ea ho lokisa: Metsotso ea 1
boitsebiso bo eketsehileng
- CoreDNS logs:
I0228 20:13:53.507780 1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"coredns", UID:"2493eb55-3dc0-11ea-b3a2-02bb48f8c230", APIVersion:"apps/v1", ResourceVersion:"132690686", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down replica set coredns-6cbb6646c9 to 2 - Lihokela ho Kibana (sehiloeng), Grafana (sehiloeng)
Ho fokotsa ts'ebeliso ea CPU, kernel ea Linux e sebelisa ntho e bitsoang contrack. Ka bokhutšoanyane, sena ke sesebelisoa se nang le lethathamo la lirekoto tsa NAT tse bolokiloeng tafoleng e khethehileng. Ha pakete e latelang e fihla ho tloha pod e le 'ngoe ho ea ho pod e tšoanang le pele, aterese ea ho qetela ea IP e ke ke ea baloa hape, empa e tla nkoa tafoleng ea contrack.
Kamoo contrack e sebetsang kateng
Liphello
Ona e ne e le mohlala oa e 'ngoe ea li-postmortem tsa rona tse nang le lihokelo tse ling tsa bohlokoa. Haholo-holo sehloohong sena, re arolelana lintlha tse ka bang molemo ho lik'hamphani tse ling. Ke ka hona re sa tšabeng ho etsa liphoso ke ka hona re phatlalatsang e 'ngoe ea li-postmortem tsa rona. Mona ke tse ling tse khahlisang tsa postmortems ea sechaba:
- GitLab:
- Dropbox:
- Spotify:
- Ba bang ba bangata ho tloha le polokelo
- Hape postmortem ea sechaba ka SRE Book
Source: www.habr.com