Mavuto ndi DNS ku Kubernetes. Public postmortem

Zindikirani kumasulira: Uku ndi kumasulira kwa postmortem yapagulu yochokera kubulogu yakampani ya engineering Konzekerani. Imalongosola vuto la kulumikizana kwa gulu la Kubernetes, zomwe zidapangitsa kuti ntchito zina zopanga zizichepa pang'ono.

Nkhaniyi ikhoza kukhala yothandiza kwa iwo omwe akufuna kuphunzira zambiri za postmortems kapena kupewa zovuta zina za DNS m'tsogolomu.

Izi si DNS
Sizingakhale DNS
Inali DNS

Zambiri za postmortems ndi njira mu Preply

Postmortem imatanthawuza kusagwira ntchito bwino kapena chochitika china pakupanga. The postmortem imaphatikizapo ndondomeko ya nthawi ya zochitika, kufotokozera za zotsatira za wogwiritsa ntchito, zomwe zimayambitsa, zomwe zachitika, ndi maphunziro omwe aphunziridwa.

Kufunafuna SRE

Pamisonkhano yamlungu ndi mlungu ndi pizza, pakati pa gulu laukadaulo, timagawana zambiri. Chimodzi mwa zinthu zofunika kwambiri pamisonkhano yotereyi ndi post-mortems, yomwe nthawi zambiri imatsagana ndi kuwonetsera ndi zithunzi komanso kusanthula mozama za zochitikazo. Ngakhale sitiombera m'manja tikamwalira, timayesetsa kukhala ndi chikhalidwe cha "osalakwa" (chikhalidwe chosalakwa). Timakhulupirira kuti kulemba ndi kuwonetsa postmortems kungatithandize ife (ndi ena) kupewa zochitika zofanana mtsogolomu, chifukwa chake tikugawana nawo.

Anthu amene akhudzidwa ndi chochitikacho ayenera kuganiza kuti angathe kufotokoza mwatsatanetsatane popanda kuopa chilango kapena chilango. Palibe mlandu! Kulemba postmortem si chilango, koma mwayi wophunzira kwa kampani yonse.

Sungani CALMS & DevOps: S ndi Yogawana

Mavuto ndi DNS ku Kubernetes. Postmortem

Tsiku: 28.02.2020

Olemba: Amet U., Andrey S., Igor K., Alexey P.

Chikhalidwe: Zatha

Mwachidule: Kusapezeka kwapang'ono kwa DNS (26 min) pazinthu zina mugulu la Kubernetes

Zotsatira: Zochitika 15000 zomwe zidatayika chifukwa cha ntchito A, B ndi C

Choyambitsa: Kube-proxy sinathe kuchotsa cholowera chakale kuchokera patebulo la contrack, kotero mautumiki ena anali kuyesa kulumikiza ma pod omwe sanalipo.

E0228 20:13:53.795782       1 proxier.go:610] Failed to delete kube-system/kube-dns:dns endpoint connections, error: error deleting conntrack entries for UDP peer {100.64.0.10, 100.110.33.231}, error: conntrack command returned: ...

Choyambitsa: Chifukwa chochepa kwambiri mkati mwa gulu la Kubernetes, CoreDNS-autoscaler idachepetsa kuchuluka kwa ma pods omwe amatumizidwa kuchokera pa atatu mpaka awiri.

yankho; Kutumizidwa kotsatira kwa pulogalamuyi kunayambitsa kupanga ma node atsopano, CoreDNS-autoscaler anawonjezera ma pods kuti agwiritse ntchito gululo, zomwe zinayambitsa kulembedwanso kwa tebulo la contrack.

Kuzindikira: Kuwunika kwa Prometheus kudapeza zolakwika zambiri za 5xx pazantchito A, B ndi C ndikuyitanitsa mainjiniya omwe ali pantchito.

5xx zolakwika ku Kibana

Zochita

zotsatira
mtundu
Wodalirika
Cholinga

Zimitsani autoscaler ya CoreDNS
oletsedwa
Amet U.
DEVOPS-695

Khazikitsani seva ya DNS yosungira
kuchepa
Max V.
DEVOPS-665

Khazikitsani kuwunika kwa contrack
oletsedwa
Amet U.
DEVOPS-674

Maphunziro

Zomwe zidayenda bwino:

• Kuwunika kunagwira ntchito bwino. Yankho linali lofulumira komanso lokonzekera
• Sitinadutse malire aliwonse pama node

Chinalakwika ndi chiyani:

• Palibe chomwe chimayambitsa zenizeni, chofanana ndi zenizeni cholakwika mu kutsutsana
• Zochita zonse zimakonza zotulukapo zake, osati zomwe zimayambitsa (bug)
• Tidadziwa kuti posachedwa titha kukhala ndi vuto ndi DNS, koma sitinayike patsogolo ntchitozo

Kumene tidachita mwayi:

• Kutumiza kotsatira kudayambitsidwa ndi CoreDNS-autoscaler, yomwe idalemba patebulo la contrack.
• Vutoli lidakhudza ntchito zina zokha

Nthawi (EET)

Nthawi
zotsatira

22:13
CoreDNS-autoscaler idachepetsa kuchuluka kwa nyemba kuchokera pa atatu mpaka awiri

22:18
Mainjiniya omwe anali pantchito adayamba kulandira mafoni kuchokera ku monitoring system

22:21
Mainjiniya omwe anali pantchitoyo adayamba kudziwa chomwe chayambitsa zolakwikazo.

22:39
Mainjiniya omwe anali pantchito adayamba kubweza imodzi mwazinthu zaposachedwa kwambiri ku mtundu wakale

22:40
Zolakwa za 5xx zidasiya kuwonekera, zinthu zakhazikika

• Nthawi yozindikira: 4 min
• Nthawi yochitapo kanthu: 21 min
• Nthawi yokonza: 1 min

zina zambiri

• Zolemba za CoreDNS:

  I0228 20:13:53.507780       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"coredns", UID:"2493eb55-3dc0-11ea-b3a2-02bb48f8c230", APIVersion:"apps/v1", ResourceVersion:"132690686", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down replica set coredns-6cbb6646c9 to 2

• Maulalo ku Kibana (kudula), Grafana (kudula)
• Kumene Linux contrack sialinso bwenzi lanu
• kube-proxy Subtleties: Kuthetsa Kukhazikitsanso Kulumikizana Kwapakatikati
• Racy contrack ndi DNS kuyang'ana nthawi

Kuti muchepetse kugwiritsidwa ntchito kwa CPU, kernel ya Linux imagwiritsa ntchito chinthu chotchedwa contrack. Mwachidule, ichi ndi chida chomwe chili ndi mndandanda wa zolemba za NAT zomwe zimasungidwa patebulo lapadera. Paketi yotsatira ikafika kuchokera ku pod imodzimodziyo kupita ku pod yomweyi monga kale, adilesi yomaliza ya IP sidzawerengedwanso, koma idzachotsedwa patebulo la contrack.

Momwe contrack imagwirira ntchito

Zotsatira

Ichi chinali chitsanzo cha imodzi mwama postmortem athu okhala ndi maulalo othandiza. Makamaka m'nkhaniyi, tikugawana zambiri zomwe zingakhale zothandiza kwa makampani ena. Ndiye chifukwa chake sitiopa kulakwitsa ndipo ndichifukwa chake timapanga imodzi mwama postmortems athu poyera. Nazi zina zochititsa chidwi za postmortems zapagulu:

• GitLab: Postmortem ya database yatha pa Januware 31
• Dropbox: Kuwonongeka kwa post-mortem
• Spotify: Ubale wa Spotify wa Chikondi / Chidani ndi DNS
• Ena ambiri ochokera nkhani iyi ndi posungira Kubernetes Kulephera Nkhani
• komanso chitsanzo postmortem yapagulu ndi SRE Book

Source: www.habr.com