{"id":80532,"date":"2020-05-07T01:42:17","date_gmt":"2020-05-06T23:42:17","guid":{"rendered":"https:\/\/prohoster.info\/blog\/administrirovanie\/problemy-s-dns-v-kubernetes-publichnyj-postmortem"},"modified":"2020-05-07T01:42:17","modified_gmt":"2020-05-06T23:42:17","slug":"problemy-s-dns-v-kubernetes-publichnyj-postmortem","status":"publish","type":"post","link":"https:\/\/prohoster.info\/sk\/blog\/administrirovanie\/problemy-s-dns-v-kubernetes-publichnyj-postmortem","title":{"rendered":"Probl\u00e9my s DNS v Kubernetes. Verejn\u00fd postmortem","gt_translate_keys":[{"key":"rendered","format":"text"}]},"content":{"rendered":"

Pozn. prekl.:<\/b> Toto je preklad verejn\u00e9ho postmortemu z in\u017einierskeho blogu spolo\u010dnosti Preply<\/a><\/noindex>. Popisuje probl\u00e9m s conntrack v Kubernetes klastri, ktor\u00fd viedol k \u010diasto\u010dn\u00e9mu v\u00fdpadku niektor\u00fdch produk\u010dn\u00fdch slu\u017eieb.<\/i><\/p>\n

Tento \u010dl\u00e1nok m\u00f4\u017ee by\u0165 u\u017eito\u010dn\u00fd pre t\u00fdch, ktor\u00ed sa chc\u00fa dozvedie\u0165 viac o postmortemoch alebo pred\u00eds\u0165 niektor\u00fdm potenci\u00e1lnym probl\u00e9mom s DNS v bud\u00facnosti.<\/p>\n

\"Probl\u00e9my
\nTo nie je DNS
\nNem\u00f4\u017ee to by\u0165 DNS
\nTo bol DNS<\/i>
\n<\/a><\/noindex><\/p>\n

Trocha o postmortemoch a procesoch v Preply<\/h2>\n

<\/p>\n

V postmorteme je pop\u00edsan\u00fd v\u00fdpadok v prev\u00e1dzke alebo nejak\u00e1 udalos\u0165 v produkcii. Postmortem obsahuje chronol\u00f3giu udalost\u00ed, popis vplyvu na pou\u017e\u00edvate\u013eov, prvotn\u00fa pr\u00ed\u010dinu, opatrenia a z\u00edskan\u00e9 ponau\u010denia.<\/p>\n

Seeking SRE<\/a><\/noindex><\/i><\/p><\/blockquote>\n

\nNa na\u0161ich t\u00fd\u017edenn\u00fdch stretnutiach s pizzou, v kruhu technick\u00e9ho t\u00edmu, zdie\u013eame r\u00f4zne inform\u00e1cie. Jednou z najd\u00f4le\u017eitej\u0161\u00edch \u010dast\u00ed tak\u00fdchto stretnut\u00ed s\u00fa postmortemy, ktor\u00e9 zvy\u010dajne sprev\u00e1dza prezent\u00e1cia so sn\u00edmkami a h\u013abkovou anal\u00fdzou udalosti. Napriek tomu, \u017ee po postmortemoch neapplaudujeme, sna\u017e\u00edme sa podporova\u0165 kult\u00faru bez obvi\u0148ovania (blameless culture<\/a><\/noindex>). Ver\u00edme, \u017ee p\u00edsanie a prezent\u00e1cia postmortemov n\u00e1m (a nielen n\u00e1m) m\u00f4\u017ee pom\u00f4c\u0165 predch\u00e1dza\u0165 podobn\u00fdm incidentom v bud\u00facnosti, a preto sa o ne del\u00edme.<\/p>\n

Osoby zapojen\u00e9 do incidentu by mali c\u00edti\u0165, \u017ee m\u00f4\u017eu podrobne hovori\u0165 o \u0148om bez strachu z trestu alebo odvety. \u017diadne odmietanie! P\u00edsanie postmortemu nie je trest, ale mo\u017enos\u0165 vzdel\u00e1vania pre cel\u00fa spolo\u010dnos\u0165.<\/p>\n

Keep CALMS & DevOps: S is for Sharing<\/a><\/noindex><\/i><\/p><\/blockquote>\n

<\/p>\n

Probl\u00e9my s DNS v Kubernetes. Postmortem<\/h2>\n

\nD\u00e1tum:<\/b> 28.02.2020<\/p>\n

Autori:<\/b> Amet U., Andrej S., Igor K., Alexej P.<\/p>\n

Stav:<\/b> Ukon\u010den\u00e9<\/p>\n

Stru\u010dne:<\/b> \u010ciasto\u010dn\u00e1 nedostupnos\u0165 DNS (26 min) pre niektor\u00e9 slu\u017eby v Kubernetes klastri<\/p>\n

Vplyv:<\/b> 15 000 udalost\u00ed stratilo pre slu\u017eby A, B a C<\/p>\n

Prvotn\u00e1 pr\u00ed\u010dina:<\/b> Kube-proxy sa nedok\u00e1zal spr\u00e1vne zbavi\u0165 starej polo\u017eky z tabu\u013eky conntrack, a preto sa niektor\u00e9 slu\u017eby st\u00e1le pok\u00fa\u0161ali pripoji\u0165 k neexistuj\u00facim podom.<\/p>\n

E0228 20:13:53.795782       1 proxier.go:610] Failed to delete kube-system\/kube-dns:dns endpoint connections, error: error deleting conntrack entries for UDP peer {100.64.0.10, 100.110.33.231}, error: conntrack command returned: ...<\/code><\/pre>\n
\nSp\u00fa\u0161\u0165a\u010d:<\/b> V d\u00f4sledku n\u00edzkej z\u00e1\u0165a\u017ee vn\u00fatri Kubernetes klastra, CoreDNS-autoscaler zn\u00ed\u017eil po\u010det podov v nasaden\u00ed z troch na dva.<\/p>\n
Rie\u0161enie:<\/b> Nov\u00e9 nasadenie aplik\u00e1cie iniciovalo vytvorenie nov\u00fdch uzlov, CoreDNS-autoscaler pridal viac podov na obsluhu klastra, \u010do vyvolalo prepisovanie tabu\u013eky conntrack<\/p>\n
Zistenie:<\/b> Monitoring Prometheus zistil vysok\u00fd po\u010det 5xx ch\u00fdb pre slu\u017eby A, B a C a inicioval v\u00fdzvu pre pohotovostn\u00fdch in\u017einierov<\/p>\n
\"Probl\u00e9my
\n5xx chyby v Kibana<\/i><\/p>\n
Akcie<\/h3>\n
<\/p>\n
Akcia
\nTyp
\nZodpovedn\u00fd
\n\u00daloha<\/p>\n
Deaktivova\u0165 autoskaler pre CoreDNS
\nprevencia
\nAmet U.
\nDEVOPS-695<\/p>\n
Nastavi\u0165 caching DNS server
\nzn\u00ed\u017ei\u0165
\nMax V.
\nDEVOPS-665<\/p>\n
Nastavi\u0165 monitoring conntrack
\nprevencia
\nAmet U.
\nDEVOPS-674<\/p>\n
<\/p>\n
Z\u00edskan\u00e9 ponau\u010denia<\/h3>\n
\n\u010co i\u0161lo dobre:<\/b><\/p>\n