Izinkinga nge-DNS ku-Kubernetes. Ukuhlolwa kwesidumbu esidlangalaleni

Qaphela ukuhumusha: Lokhu ukuhunyushwa kokuhlolwa kwesidumbu esidlangalaleni okuvela kubhulogi yenkampani yobunjiniyela Ukulungiselela. Ichaza inkinga nge-conntrack eqoqweni le-Kubernetes, okuholele ekunciphiseni kancane kwezinye izinsiza zokukhiqiza.

Lesi sihloko singase sibe usizo kulabo abafuna ukufunda okwengeziwe mayelana nokuhlolwa kwezidumbu noma ukuvimbela ezinye izinkinga ezingaba khona ze-DNS esikhathini esizayo.

Lena akuyona i-DNS
Ngeke kube yi-DNS
Bekuyi-DNS

Okuncane mayelana nama-postmortems nezinqubo ku-Preply

I-postmortem ichaza ukungasebenzi kahle noma umcimbi othile ekukhiqizeni. I-postmortem ihlanganisa umugqa wesikhathi wezenzakalo, umthelela wabasebenzisi, imbangela eyinhloko, izinyathelo ezithathiwe, kanye nezifundo ezifundiwe.

Ifuna i-SRE

Emihlanganweni yamasonto onke ne-pizza, phakathi kwethimba lezobuchwepheshe, sabelana ngolwazi oluhlukahlukene. Enye yezingxenye ezibaluleke kakhulu zemihlangano enjalo ukuxilongwa kwezidumbu, okuvame ukuphelezelwa isethulo esinama-slides kanye nokuhlaziywa okujulile kwesigameko. Noma singashayi izandla ngemva kokuhlolwa kwezidumbu, sizama ukuthuthukisa isiko "lokungasoli" (isiko elingasoleki). Sikholelwa ukuthi ukubhala nokwethula ukuhlolwa kwezidumbu kungasisiza (kanye nabanye) ukuvimbela izigameko ezifanayo esikhathini esizayo, yingakho sabelana ngazo.

Abantu abahilelekile esehlakalweni kufanele babe nomuzwa wokuthi bangakhuluma ngokuningiliziwe ngaphandle kokwesaba ukujeziswa noma ukujeziswa. Akunacala! Ukubhala i-postmortem akusona isijeziso, kodwa yithuba lokufunda lenkampani yonke.

Gcina i-CALMS & DevOps: I-S ingeyokwabelana

Izinkinga nge-DNS ku-Kubernetes. I-Postmortem

Usuku: 28.02.2020

Ababhali: Amet U., Andrey S., Igor K., Alexey P.

Isimo: Kuqediwe

Kafushane: Ukungatholakali kwe-DNS ingxenye (26 min) kwamanye amasevisi kuqoqo le-Kubernetes

Umthelela: Imicimbi engu-15000 elahlekile ngezinkonzo A, B kanye no-C

Imbangela: I-Kube-proxy ayikwazanga ukususa ngokufanelekile okufakiwe okudala kusukela kuthebula le-conntrack, ngakho-ke amanye amasevisi ayesazama ukuxhuma kuma-pods angekho.

E0228 20:13:53.795782       1 proxier.go:610] Failed to delete kube-system/kube-dns:dns endpoint connections, error: error deleting conntrack entries for UDP peer {100.64.0.10, 100.110.33.231}, error: conntrack command returned: ...

Qalisa: Ngenxa yomthwalo ophansi ngaphakathi kweqoqo le-Kubernetes, i-CoreDNS-autoscaler yehlise inani lama-pods ekusetshenzisweni ukusuka kokuthathu kuya kokubili.

Isixazululo: Ukuthunyelwa okulandelayo kohlelo lokusebenza kuqalise ukwakhiwa kwama-node amasha, i-CoreDNS-autoscaler yengeza ama-pod amaningi ukuze isebenze iqoqo, okubangele ukubhalwa kabusha kwetafula lokungena.

Ukutholwa: Ukuqapha kuka-Prometheus kuthole inani elikhulu lamaphutha angu-5xx kumasevisi A, B kanye no-C futhi kwaqala ucingo konjiniyela abasemsebenzini.

5xx amaphutha e-Kibana

Действия

Isenzo
Thayipha
Unomthwalo wemfanelo
Inhloso

Khubaza isithwebuli esizenzakalelayo se-CoreDNS
kuvinjelwe
Amet U.
I-DEVOPS-695

Setha iseva ye-DNS yenqolobane
nciphisa
UMax V.
I-DEVOPS-665

Setha ukuqapha kwe-contrack
kuvinjelwe
Amet U.
I-DEVOPS-674

Izifundo Ezifundiwe

Yini ehambe kahle:

• Ukuqapha kusebenze kahle. Impendulo yayishesha futhi ihlelekile
• Asizange sifinyelele noma yimiphi imikhawulo kumanodi

Yini ebingalungile:

• Isizathu sangempela asikaziwa, esifana ne bug ethile ekunqandeni
• Zonke izenzo zilungisa imiphumela kuphela, hhayi imbangela (isiphazamisi)
• Sasazi ukuthi ngokuhamba kwesikhathi singase sibe nezinkinga nge-DNS, kodwa asizange sibeke phambili imisebenzi

Lapho saba nenhlanhla khona:

• Ukuthunyelwa okulandelayo kucushwe yi-CoreDNS-autoscaler, evala ithebula lokungena.
• Lesi siphazamisi sithinte kuphela amanye amasevisi

Umugqa wesikhathi (EET)

Isikhathi
Isenzo

22:13
I-CoreDNS-autoscaler yehlise inani lama-pods ukusuka kokuthathu kuya kokubili

22:18
Onjiniyela ababesemsebenzini baqala ukuthola izingcingo ezivela ohlelweni lokuqapha

22:21
Onjiniyela ababesemsebenzini baqala ukuthola imbangela yamaphutha.

22:39
Onjiniyela abasemsebenzini baqale ukuhlehlisa enye yezinkonzo zakamuva enguqulweni yangaphambilini

22:40
5xx amaphutha ayeke ukuvela, isimo sizinzile

• Isikhathi sokutholwa: 4 imizuzu
• Isikhathi ngaphambi kwesenzo: 21 imizuzu
• Isikhathi sokulungisa: 1 imizuzu

ulwazi olwengeziwe

• Amalogi we-CoreDNS:

  I0228 20:13:53.507780       1 event.go:221] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"coredns", UID:"2493eb55-3dc0-11ea-b3a2-02bb48f8c230", APIVersion:"apps/v1", ResourceVersion:"132690686", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled down replica set coredns-6cbb6646c9 to 2

• Izixhumanisi eziya ku-Kibana (cut), Grafana (cut)
• Lapho i-Linux contrack ingaseyena umngane wakho
• I-kube-proxy Subtleties: Ukulungisa iphutha Ukusetha Kabusha Uxhumano Lwesikhashana
• I-Racy contrack kanye nezikhathi zokubheka i-DNS

Ukunciphisa ukusetshenziswa kwe-CPU, i-Linux kernel isebenzisa into ebizwa ngokuthi i-contrack. Ngamafuphi, lolu uhlelo oluqukethe uhlu lwamarekhodi e-NAT agcinwe kuthebula elikhethekile. Lapho iphakethe elilandelayo lifika lisuka kuphod efanayo liya ku-pod efanayo nangaphambili, ikheli le-IP lokugcina ngeke libalwe kabusha, kodwa lizothathwa kuthebula le-conntrack.

I-contrack isebenza kanjani

Imiphumela

Lesi bekuyisibonelo sokunye kokufa kwethu okunezixhumanisi eziwusizo. Ngokukhethekile kulesi sihloko, sabelana ngolwazi olungase lube usizo kwezinye izinkampani. Yingakho singakwesabi ukwenza amaphutha yingakho sibeka omunye wabashonile emphakathini. Nawa amanye ama-postmortem omphakathi athokozisayo:

• I-GitLab: Ukuphela kwe-Postmortem yokuphela kwesizindalwazi ngoJanuwari 31
• I-Dropbox: Ukuphela kokuhlolwa kwesidumbu
• Spotify: Ubudlelwano Bothando/Nenzondo bukaSpotify ne-DNS
• Abanye abaningi kusuka le ngqikithi kanye nenqolobane I-Kubernetes Failure Stories
• Futhi isibonelo i-postmortem yomphakathi nge-SRE Book

Source: www.habr.com