🥇Kuongorora kwehupenyu muKubernetes kunogona kuve nengozi

Cherechedza. transl.: Injiniya anotungamira kubva kuZalando, Henning Jacobs, akadzokorora kuona matambudziko pakati pevashandisi veKubernetes mukunzwisisa chinangwa chehupenyu (uye kugadzirira) probes uye kushandisa kwavo chaiko. Naizvozvo, akaunganidza pfungwa dzake muchinyorwa ichi chine simba, chinozopedzisira chave chikamu chezvinyorwa zveK8s.

Macheki ehutano, anozivikanwa muKubernetes se lifeness probes (kureva,, chaizvo, "bvunzo dzekugoneka" - approx. transl.), zvinogona kuva nengozi zvikuru. Ini ndinokurudzira kuvanzvenga kana zvichibvira: izvo chete zvinosarudzika kana zvichinyatsodikanwa uye iwe unonyatsoziva nezve chaiyo uye mhedzisiro yekushandiswa kwavo. Ichi chinyorwa chichataura nezve hupenyu uye kugadzirira cheki, uye zvakare ichakuudza iwe mune zvipi zviitiko zvakakodzera uye haufanire kuashandisa.

Wandaishanda naye Sandor nguva pfupi yadarika akagovera paTwitter zvikanganiso zvinowanzoitika zvaanosangana nazvo, kusanganisira izvo zvine chekuita nekushandiswa kwekugadzirira/kurarama maprobes:

Kurongwa zvisina kururama livenessProbe inogona kuwedzera mamiriro ekuremerwa (snowball shutdown + inogona kureba chigaba / application yekutanga nguva) uye zvinotungamira kune mamwe mhedzisiro yakaipa senge kutsamira madonhwe. (onawo chinyorwa changu chazvino nezve kudzikisira huwandu hwezvikumbiro muK3s + ACME musanganiswa). Zvakatonyanya kuipa kana iyo liveness probe yakasanganiswa neyehutano cheki, inova yekunze dhatabhesi: imwe DB kutadza kunotangazve midziyo yako yese!

General message "Usashandise lifeness probes" mune iyi kesi hazvibatsiri zvakanyanya, saka ngatitarisei kuti kugadzirira uye kurarama cheki ndeyei.

Ongorora: Yakawanda yebvunzo iri pazasi yakatanga kuverengerwa muzvinyorwa zveZalando zvemukati zvekuvandudza.

Readiness and Liveness macheki

Kubernetes inopa nzira mbiri dzakakosha dzinodanwa hupenyu hunoongorora uye kugadzirira kuongorora. Ivo nguva nenguva vanoita chimwe chiito-chakadai sekutumira chikumbiro cheHTTP, kuvhura chinongedzo cheTCP, kana kuita murairo mumudziyo-kusimbisa kuti chikumbiro chiri kushanda sezvaitarisirwa.

Kubernetes inoshandisa kugadzirira probeskuti unzwisise kana mudziyo wagadzirira kugamuchira traffic. Ipodhi inonzi yakagadzirira kushandiswa kana midziyo yayo yose yakagadzirira. Kumwe kushandiswa kweiyi mashandiro kudzora kuti ndeapi mapodhi anoshandiswa semashure eKubernetes masevhisi (uye kunyanya Ingress).

Upenyu hunoongorora batsira Kubernetes kunzwisisa kana yave nguva yekumisikidza mudziyo. Semuyenzaniso, cheki yakadaro inokutendera kuti utore dheti kana application yanamira panzvimbo imwechete. Kutangazve mudziyo mune ino mamiriro kunobatsira kubvisa application pasi kunyangwe paine zvikanganiso, asi zvinogona zvakare kutungamira mukutadza kudonha (ona pazasi).

Kana iwe ukayedza kuendesa iyo application yekuvandudza iyo inotadza iyo liveness / kugadzirira cheki, kuburitswa kwayo kunomiswa sezvo Kubernetes akamirira chimiro. Ready kubva kumativi ese.

Muenzaniso:

Heino muenzaniso wekugadzirira kuongorora nzira /health kuburikidza neHTTP ine default marongero (chinguva: 10 masekonzi, nguva yapera: 1 sekondi, kubudirira pachikumbaridzo: 1, kukundikana chikumbaridzo: 3):

# часть общего описания deployment'а/стека
podTemplate:
  spec:
    containers:
    - name: my-container
      # ...
      readinessProbe:
        httpGet:
          path: /health
          port: 8080

kurumbidza

Kune mamicroservices ane HTTP endpoint (REST, nezvimwewo) nguva dzose tsanangura chirongwa chekugadzirira, iyo inotarisa kana application (pod) yakagadzirira kugamuchira traffic.
Ita shuwa kugadzirira probe inovhara kuwanikwa kweiyo chaiyo web server port:
- kushandisa zviteshi zvezvinangwa zvekutonga, zvinodaidzwa kuti "admin" kana "management" (semuenzaniso, 9090), ye readinessProbe, ita shuwa kuti iyo yekupedzisira inongodzoka OK kana yekutanga HTTP port (se8080) yakagadzirira kugamuchira traffic*;
  *Ndinoziva pane imwe nyaya kuZalando uko zvisina kuitika izvi, kureva. readinessProbe Ndakatarisa chiteshi che "management", asi sevha pachayo haina kutanga kushanda nekuda kwematambudziko ekurodha cache.
- kubatanidza kugadzirira probe kune imwe chiteshi chengarava kunogona kutungamira kune chokwadi chekuti kuwandisa pachiteshi chikuru hakuzoratidzike mukutariswa kwehutano (kureva kuti, dziva reshinda paseva rakazara, asi cheki yehutano ichiri kuratidza kuti zvese zvakanaka. )
Ita shuwa kuti kugadzirira probe inogonesa dhatabhesi kutanga / kutama;
- Nzira iri nyore yekuita izvi ndeyekubata sevha yeHTTP chete mushure mekutanga kwapera (semuenzaniso, kutama dhatabhesi kubva Flyway zvichingoenda zvakadaro.); ndiko kuti, pachinzvimbo chekushandura mamiriro ekutarisa hutano, usangotanga sevha yewebhu kusvika kutama kwe database kwapera *.
  * Iwe unogona zvakare kumhanya dhatabhesi kutama kubva kune init midziyo kunze kwepodhi. Ndichiri fan yekuzvimiririra zvikumbiro, ndiko kuti, idzo mune mudziyo wekushandisa anoziva nzira yekuunza dhatabhesi munzvimbo inodiwa pasina kubatana kwekunze.
Shandisa httpGet kugadzirira macheki kuburikidza neyakajairwa hutano cheki yekupedzisira (semuenzaniso, /health).
Nzwisisa iyo default check parameters (interval: 10s, timeout: 1s, successThreshold: 1, failureThreshold: 3):
- iyo yakasarudzika sarudzo inoreva iyo pod ichave hazvisati zvaita mushure memasekonzi makumi matatu (30 akakundikana macheki ehungwaru).
Shandisa chiteshi chakasiyana che "admin" kana "management" kana tekinoroji stack (semuenzaniso Java/Spring) ichibvumira, kupatsanura hutano uye metric manejimendi kubva kune yakajairika traffic:
- asi usakanganwa nezve point 2.
Kana zvichidikanwa, probe yekugadzirira inogona kushandiswa kudziya / kurodha cache uye kudzosera 503 chimiro kodhi kudzamara mudziyo wadziya.
- Ndinokurudzirawo kuti uverenge cheki itsva startupProbe, yakaonekwa muvhezheni 1.16 (takanyora nezvazvo muchiRussia pano - approx. transl.).

Caveats

Usavimba nekunze kwekutsamira (zvakadai sematura edatha) paunenge uchimhanyisa kugadzirira / kurarama bvunzo - izvi zvinogona kutungamira mukutadza kukanganisa:
- Semuenzaniso, ngatitorei sevhisi yeREST ine mapodhi gumi zvichienderana nedhatabhesi rePostgres: kana cheki ichienderana nekubatana kwekushanda kuDB, ese gumi mapodhi anogona kutadza kana paine kunonoka kunetiweki/DB side - kazhinji zvose zvinopera zvakaipa kupfuura zvaigona;
- Ndokumbira utarise kuti Spring Data inotarisa dhatabhesi yekubatanidza nekukasira *;
  * Aya ndiwo maitiro ekutanga eSpring Data Redis (kanenge yaive nguva yekupedzisira yandakatarisa), izvo zvakakonzera kukundikana kwe "njodzi": apo Redis yakanga isipo kwenguva pfupi, mapodhi ese "akarovera".
- "zvekunze" mupfungwa iyi zvinogona zvakare kureva mamwe mapodhi echishandiso chimwe chete, ndiko kuti, cheki haifanire kuenderana nemamiriro emamwe mapodhi esumbu rimwe chete kudzivirira kuparara kwegwatidziro:
  - mibairo inogona kusiyanisa kune maapplication ane yakagoverwa nyika (semuenzaniso, mu-memory caching mumapods).
Usashandise kuongorora hupenyu zvemapods (kunze kwezviitiko kana zvichinyatsodikanwa uye iwe unonyatsoziva nezve chaiyo nemhedzisiro yekushandiswa kwavo):
- Iyo liveness probe inogona kubatsira kudzoreredza midziyo yakaturikwa, asi sezvo iwe uine kutonga kwakazara pamusoro pechikumbiro chako, zvinhu zvakaita seyakaturikwa maitiro uye deadlocks hazvifanirwe kuitika: yakanakisa imwe nzira ndeyekupaza nemaune application nekuidzosera kune yakare yakadzikama;
- kutadza kwehupenyu kuferefeta kuchaita kuti mudziyo utangezve, zvichibva zvawedzera mhedzisiro yekurodha-zvine chekuita nezvikanganiso: kutangazve mudziyo kunozokonzera kuderera (panguva yenguva yekutanga application, taura 30-odd masekonzi), zvichikonzera zvikanganiso zvitsva. , kuwedzera mutoro pane mamwe midziyo uye kuwedzera mukana wekukundikana kwavo, nezvimwewo;
- macheki ehupenyu akasanganiswa nekutsamira kwekunze ndiko kusanganisa kwakaipisisa, kutyisidzira kutadza kutadza: kunonoka zvishoma padivi redatabase kunotungamira kutangwazve kwemidziyo yako yese!
Parameters yehupenyu uye kugadzirira cheki inofanira kunge yakasiyana:
- iwe unogona kushandisa liveness probe neyakafanana cheki yehutano, asi yepamusoro yekupindura chikumbaridzo (failureThreshold), semuenzaniso, ipa chimiro hazvisati zvaita mushure mekuedza 3 uye funga kuti liveness probe yakundikana mushure mekuedza gumi;
Usashandise exec cheki, sezvo zvakabatana nematambudziko anozivikanwa anotungamira mukuonekwa kwe zombie maitiro:
- tsanangudzo: ona mharidzo neDatadog nyanzvi.

Summary

Shandisa probes yekugadzirira kuona kana pod yagadzirira kugamuchira traffic.
Shandisa zvipenyu zvehupenyu chete kana zvichinyatsodiwa.
Kusashandisa kwakakodzera kwekugadzirira / hupenyu hwekuferefeta kunogona kutungamira mukuderedzwa kuwanikwa uye kutadza kudonha.

Zvimwe zvekuwedzera pamusoro wenyaya

Kubernetes docs: Gadzira Kurarama uye Kugadzirira Probes;
Kubernetes Kurarama uye Kugadzirira Probes Kudzokororwa: Maitiro eKudzivirira Kuzvipfura Iwe Mune Imwe Rutsoka;
NRE Labs Outage Post-Mortem (anotaurawo nezve livenessProbe).

Kwidziridzo Nhamba 1 kubva 2019-09-29

Nezve init midziyo yekutamisa database: Mashoko emuzasi akawedzerwa.

EJ akandiyeuchidza nezvePDB: rimwe rematambudziko ane liveness cheki kushaikwa kwekubatana pakati pepods. Kubernetes has Pod Disruption Budgets (PDB) kudzikamisa huwandu hwekutadza kwakafanana kunogona kuitika kune application, asi macheki haatarise iyo PDB. Zvakanakisa, tinogona kuudza K8s kuti "Tangisazve imwe pod kana bvunzo yayo ikatadza, asi usavatangazve ese kuti arege kuita kuti zvinhu zvinyanye."

Bryan akanyatso taura: “Shandisa hupenyu hwekuongorora kana uchinyatsoziva kuti chii chinhu chakanakisa kuita kuuraya chikumbiro"(zvakare, usatakurwa).

Kwidziridzo Nhamba 2 kubva 2019-09-29

Nezvekuverenga zvinyorwa zvisati zvashandiswa: Ndakagadzira chikumbiro chinoenderana (chimiro chikumbiro) kuwedzera zvinyorwa nezve liveness probes.

PS kubva kumushanduri

Verenga zvakare pablog yedu:

Source: www.habr.com

Hupenyu probes muKubernetes inogona kuve nengozi