🥇Kufufuza kwa moyo ku Kubernetes kungakhale koopsa

Zindikirani. transl.: Katswiri wotsogola wochokera ku Zalando, Henning Jacobs, wawona mobwerezabwereza zovuta pakati pa ogwiritsa ntchito Kubernetes pakumvetsetsa cholinga cha moyo (ndi kukonzekera) kafukufuku ndikugwiritsa ntchito moyenera. Chifukwa chake, adasonkhanitsa malingaliro ake muzolemba zazikuluzikuluzi, zomwe pamapeto pake zidzakhala gawo la zolemba za K8s.

Macheke azaumoyo, omwe amadziwika ku Kubernetes monga moyo amafufuza (i.e., kwenikweni, “mayeso a kuthekera” - pafupifupi transl.), zingakhale zoopsa kwambiri. Ndikupangira kuwapewa ngati kuli kotheka: kupatulapo kokha pamene kuli kofunikira ndipo mumadziwa bwino zachindunji ndi zotsatira za ntchito yawo. Bukuli lidzalankhula za moyo ndi kukonzekera macheke, komanso kukuuzani muzochitika ziti ndi ofunika ndipo simuyenera kuzigwiritsa ntchito.

Mnzanga Sandor posachedwapa adagawana nawo pa Twitter zolakwika zomwe amakumana nazo, kuphatikiza zomwe zikugwirizana ndi kugwiritsa ntchito zofufuza zokonzekera / kuchitapo kanthu:

Zosanjidwa molakwika livenessProbe imatha kukulitsa zinthu zolemetsa kwambiri (kutsekeka kwa mpira wa chipale chofewa + chidebe chomwe chingakhale chachitali / nthawi yoyambira pulogalamu) ndikubweretsa zotsatira zina zoyipa monga kutsika kwa kudalira. (onaninso nkhani yanga yaposachedwa za kuchepetsa kuchuluka kwa zopempha mu kuphatikiza kwa K3s + ACME). Ndizoipa kwambiri pamene kafukufuku wamoyo akuphatikizidwa ndi cheke chaumoyo, chomwe ndi nkhokwe yakunja: kulephera kumodzi kwa DB kudzayambitsanso zotengera zanu zonse!

Uthenga wamba "Osagwiritsa ntchito ma probes" pamenepa sizikuthandizira kwambiri, kotero tiyeni tiwone zomwe kukonzekera ndi kuwunika kwa moyo kuli.

Zindikirani: Mayeso ambiri omwe ali pansipa adaphatikizidwa muzolemba za Zalando zamkati.

Macheke a Readiness and Liveness

Kubernetes amapereka njira ziwiri zofunika zomwe zimatchedwa zofufuza moyo ndi kukonzekera kufufuza. Nthawi ndi nthawi amachitapo kanthu - monga kutumiza pempho la HTTP, kutsegula kulumikiza kwa TCP, kapena kupereka lamulo mumtsuko - kutsimikizira kuti ntchitoyo ikugwira ntchito monga momwe amayembekezera.

Kubernetes amagwiritsa ntchito kukonzekera kufufuzakuti mumvetse pamene chidebecho chakonzeka kuvomereza magalimoto. Mtsuko umatengedwa kuti ndi wokonzeka kugwiritsidwa ntchito ngati zotengera zake zonse zakonzeka. Kugwiritsiridwa ntchito kumodzi kwa makinawa ndikuwongolera ma pod omwe amagwiritsidwa ntchito ngati kumbuyo kwa ntchito za Kubernetes (makamaka Ingress).

Moyo umafufuza thandizani Kubernetes kumvetsetsa ikafika nthawi yoti muyambitsenso chidebecho. Mwachitsanzo, cheke choterechi chimakulolani kuti muchepetse kutsekeka pomwe pulogalamuyo ikanikira pamalo amodzi. Kuyambitsanso chidebe m'chigawochi kumathandiza kuti pulogalamuyo ichoke pansi ngakhale pali zolakwika, koma kungayambitsenso kulephera (onani pansipa).

Ngati muyesa kutumiza zosintha zomwe zikulephera kuwunika kwa moyo / kukonzekera, kutulutsidwa kwake kudzayimitsidwa pomwe Kubernetes akudikirira Ready kuchokera pamitundu yonse.

Chitsanzo:

Pano pali chitsanzo cha kafukufuku wokonzekera kufufuza njira /health kudzera pa HTTP yokhala ndi zosintha zosasintha (mpata: 10 masekondi, lekeza panjira: 1 mphindi, Chipambano: 1, malire olephera3):

# часть общего описания deployment'а/стека
podTemplate:
  spec:
    containers:
    - name: my-container
      # ...
      readinessProbe:
        httpGet:
          path: /health
          port: 8080

ayamikira

Kwa ma microservices okhala ndi HTTP endpoint (REST, etc.) nthawi zonse amatanthauzira kafukufuku wokonzekera, yomwe imayang'ana ngati ntchito (pod) ili wokonzeka kuvomereza magalimoto.
Onetsetsani kukonzekera kafukufuku imakhudza kupezeka kwa doko lenileni la seva:
- kugwiritsa ntchito madoko pazoyang'anira, zotchedwa "admin" kapena "management" (mwachitsanzo, 9090), kwa readinessProbe, onetsetsani kuti mapeto amangobwerera OK ngati doko lalikulu la HTTP (monga 8080) ndilokonzeka kuvomereza traffic*;
  *Ndikudziwa nkhani imodzi ku Zalando pomwe izi sizinachitike, mwachitsanzo. readinessProbe Ndinayang'ana doko la "kasamalidwe", koma seva yokhayo sinayambe kugwira ntchito chifukwa cha mavuto otsegula cache.
- kuyika kafukufuku wokonzekera ku doko lapadera kungayambitse kuti kuchulukitsitsa pa doko lalikulu sikudzawonetsedwa mu cheke chaumoyo (ndiko kuti, dziwe la ulusi pa seva ladzaza, koma cheke chaumoyo chikuwonetsabe kuti zonse zili bwino. ).
Onetsetsani kuti Kukonzekera kumathandizira kukhazikitsidwa kwa database/kusamuka;
- Njira yosavuta yokwaniritsira izi ndikulumikizana ndi seva ya HTTP pokhapokha kukhazikitsidwa kwatha (mwachitsanzo, kusamutsa database kuchokera ku Flyway ndi zina zotero.); ndiye kuti, m'malo mosintha momwe mungayang'anire thanzi lanu, musayambitse seva yapaintaneti mpaka kusamutsidwa kwa database kumalize *.
  * Mutha kuyendetsanso kusamuka kwa database kuchokera pazotengera za init kunja kwa pod. Ndikadali wokonda zodziyimira pawokha, ndiye kuti, zomwe chidebe chogwiritsira ntchito chimadziwa kubweretsa database pamalo omwe mukufuna popanda kulumikizana kwakunja.
Gwiritsani ntchito httpGet kuyezetsa kokonzekera kudzera m'magawo omwe ali ndi thanzi labwino (mwachitsanzo, /health).
Mvetsetsani magawo a chekeni (interval: 10s, timeout: 1s, successThreshold: 1, failureThreshold: 3):
- zosankha zosasinthika zikutanthauza kuti pod idzakhala osakonzeka patatha pafupifupi masekondi 30 (macheke atatu adalephera kuchita bwino).
Gwiritsani ntchito doko lapadera la "admin" kapena "management" ngati ukadaulo (monga Java/Spring) ulola, kulekanitsa kasamalidwe kaumoyo ndi ma metrics ndi kuchuluka kwa magalimoto nthawi zonse:
- koma osayiwala za point 2.
Ngati ndi kotheka, kafukufuku wokonzekera atha kugwiritsidwa ntchito kutenthetsa / kutsitsa posungira ndikubwezeretsa nambala 503 mpaka chidebecho chitenthe:
- Ndikupangiranso kuti muwerenge cheke chatsopano startupProbe, adawonekera mu mtundu 1.16 (tidalemba za izi mu Chirasha apa - pafupifupi. transl.).

Mipango

Osadalira kudalira kwakunja (monga malo osungiramo data) poyesa kukonzekera / kukhala ndi moyo - izi zitha kubweretsa kulephera kwakukulu:
- Mwachitsanzo, tiyeni titenge ntchito yabwino ya REST yokhala ndi ma pod 10 kutengera nkhokwe imodzi ya Postgres: cheke chikatengera kulumikizana kogwira ntchito ndi DB, ma pod onse 10 amatha kulephera ngati pakhala kuchedwa pa netiweki / DB mbali - nthawi zambiri zimatero. zonse zimatha moyipa kuposa momwe zingakhalire;
- Chonde dziwani kuti Spring Data imayang'ana kulumikizidwa kwa database mosakhazikika *;
  * Ili ndilo khalidwe losasinthika la Spring Data Redis (osachepera inali nthawi yotsiriza yomwe ndinayang'ana), zomwe zinapangitsa kuti "tsoka" lilephereke: pamene Redis sanapezeke kwa kanthawi kochepa, ma pod onse "anagwa".
- "zakunja" m'lingaliro limeneli angatanthauzenso makoko ena a ntchito yomweyi, ndiye kuti, cheke sichiyenera kudalira momwe ma pod ena agulu lomwelo alili kuti apewe kuwonongeka:
  - zotsatira zitha kusiyanasiyana pamapulogalamu omwe ali ndi gawo logawika (mwachitsanzo, kusungitsa mu-memory mu ma pod).
Osagwiritsa ntchito kafukufuku wamoyo kwa ma pod (kupatulapo ndizochitika zomwe zili zofunikadi ndipo mumadziwa bwino za zomwe azigwiritsa ntchito):
- Kufufuza kwa moyo kungathandize kubwezeretsa zotengera zomwe zidapachikidwa, koma popeza muli ndi mphamvu zonse pakugwiritsa ntchito kwanu, zinthu monga njira zopachikika ndi zotsekera siziyenera kuchitika: njira ina yabwino ndikuphwanya mwadala pulogalamuyo ndikuyibwezeretsa momwe idakhazikika kale;
- kufufuza kwa moyo komwe kulephera kupangitsa kuti chidebecho chiyambitsenso, zomwe zitha kukulitsa zotsatira za zolakwika zokhudzana ndi boot: kuyambitsanso chidebe kumabweretsa nthawi yocheperako (osachepera nthawi yoyambira, nenani masekondi 30+), kubweretsa zolakwika zatsopano, kuonjezera katundu pazitsulo zina ndikuwonjezera mwayi wa kulephera kwawo, ndi zina zotero;
- cheke chamoyo chophatikizidwa ndi kudalira kwakunja ndichophatikizira choyipa kwambiri, ndikuwopseza kulephera kwapang'onopang'ono: kuchedwa pang'ono kumbali yakusungirako kumabweretsa kuyambiranso kwa zotengera zanu zonse!
Magawo a moyo ndi kukonzekera macheke ziyenera kukhala zosiyana:
- mutha kugwiritsa ntchito kafukufuku wamoyo ndi kuyezetsa thanzi komweko, koma kuyankha kwakukulu (failureThreshold), mwachitsanzo, perekani udindo osakonzeka pambuyo poyesera 3 ndikuganizira kuti kafukufuku wamoyo walephera pambuyo poyesera 10;
Musagwiritse ntchito ma exec checks, popeza amalumikizidwa ndi zovuta zodziwika zomwe zimapangitsa kuti njira za zombie ziwonekere:
- zambiri: onani zoperekedwa ndi akatswiri a Datadog.

Chidule

Gwiritsani ntchito ma probes okonzeka kuti mudziwe nthawi yomwe pod yakonzeka kulandira magalimoto.
Gwiritsani ntchito ma probes amoyo pokhapokha ngati akufunikiradi.
Kugwiritsa ntchito molakwika zowerengera zokonzekera / kukhala ndi moyo kungayambitse kuchepa kwa kupezeka ndi kulephera kwapang'onopang'ono.

Zowonjezera pamutuwu

Kubernetes docs: Konzani Liveness and Readiness Probes;
Kubernetes Liveness and Readiness Probes Adabwerezedwanso: Momwe Mungapewere Kudziwombera Pamapazi Ena;
NRE Labs Outage Post-Mortem (amakambanso za livenessProbe).

Kusintha nambala 1 kuyambira 2019-09-29

Za zotengera za init zosamukira ku database: Mawu a m'munsi awonjezeredwa.

EJ adandikumbutsa za PDB: Vuto limodzi loyang'ana moyo ndi kusowa kwa mgwirizano pakati pa ma pod. Kubernetes ali Pod Disruption Budgets (PDB) kuchepetsa kuchuluka kwa zolephera zomwe pulogalamu ingakumane nayo, komabe macheke samaganizira za PDB. Momwemo, tikhoza kuuza K8s kuti "Yambitsaninso pod imodzi ngati mayesero ake alephera, koma musayambitsenso zonse kuti mupewe kuipiraipira."

Bryan anaziyika izo mwangwiro: "Gwiritsani ntchito kufufuza kwa moyo mukamadziwa bwino lomwe chinthu chabwino kuchita ndi kupha ntchito"(kachiwiri, musatengeke).

Kusintha nambala 2 kuyambira 2019-09-29

Ponena za kuwerenga zolembedwa musanagwiritse ntchito: Ndinapanga pempho lofanana (pempho lazinthu) kuti muwonjezere zolembedwa zokhudzana ndi kafukufuku wamoyo.

PS kuchokera kwa womasulira

Werenganinso pa blog yathu:

Source: www.habr.com

Kufufuza kwa moyo ku Kubernetes kungakhale koopsa