Kubernetes matipi & matipi: maficha enyasha kuvharika muNGINX uye PHP-FPM

Mamiriro akajairika paunenge uchiita CI/CD muKubernetes: chikumbiro chinofanira kukwanisa kubvuma zvikumbiro zvemutengi usati wamira zvachose, uye zvakanyanya kukosha, kubudirira kupedzisa zviripo.

Kuteerera nemamiriro ezvinhu aya kunobvumira iwe kuti uwane zero downtime panguva yekuendesa. Nekudaro, kunyangwe uchishandisa mabundle akakurumbira (senge NGINX uye PHP-FPM), unogona kusangana nematambudziko anozotungamira mukuwanda kwezvikanganiso nekutumirwa kwega kwega ...

Dzidziso. Iyo pod inorarama sei

Takatoburitsa zvakadzama nezve life cycle yepod ichi chinyorwa. Muchirevo chechinyorwa chiri kutariswa, isu tinofarira zvinotevera: panguva iyo pod inopinda muhurumende Kugumisa, zvikumbiro zvitsva zvinomira kutumirwa kwairi (pod kubviswa kubva pane rondedzero yemagumo esevhisi). Nekudaro, kudzivirira kuderera panguva yekuendesa, zvakaringana kuti isu tigadzirise dambudziko rekumisa application nemazvo.

Iwe unofanirwawo kuyeuka kuti nguva yenyasha yekusagadzikana ndeye 30 seconds: mushure meizvi, iyo pod ichamiswa uye chikumbiro chinofanira kuva nenguva yekugadzirisa zvikumbiro zvese izvi zvisati zvaitika. taura pfungwa: kunyangwe chero chikumbiro chinotora anopfuura 5-10 masekonzi chatova nedambudziko, uye nenyasha kudzima hakuchabatsiri ...

Kuti unzwisise zviri nani zvinoitika kana podhi yapera, ingotarisa dhayagiramu inotevera:

A1, B1 - Kugamuchira shanduko nezve mamiriro enzvimbo
A2 - Kubva SIGTERM
B2 - Kubvisa pod kubva kumagumo
B3 - Kugamuchira shanduko (iyo rondedzero yemagumo yachinja)
B4 - Gadziridza iptables mitemo

Ndokumbira utarise: kudzima podhi yekupedzisira uye kutumira SIGTERM hazviitike zvakatevedzana, asi zvakafanana. Uye nekuda kwekuti Ingress haingogamuchire rondedzero yakagadziridzwa yeEndpoints, zvikumbiro zvitsva kubva kune vatengi zvinotumirwa kune pod, izvo zvinokonzeresa kukanganisa kwe500 panguva yekumisa pod. (kuti uwane mamwe mashoko pamusoro penyaya iyi, isu rakashandurwa). Dambudziko iri rinoda kugadziriswa nenzira dzinotevera:

• Tumira Kubatanidza: vhara mumusoro wemhinduro (kana izvi zvine chekuita neHTTP application).
• Kana zvisingabviri kuita shanduko kune kodhi, zvino chinyorwa chinotevera chinotsanangura mhinduro ichakubvumidza iwe kugadzirisa zvikumbiro kusvika pakupera kwenguva yenyasha.

Dzidziso. Sei NGINX uye PHP-FPM inomisa maitiro avo

NGINX

Ngatitangei neNGINX, sezvo zvese zviri pachena kana zvishoma nazvo. Kunyura mune dzidziso, tinodzidza kuti NGINX ine imwe master process uye akati wandei "vashandi" - aya maitiro evana anogadzirisa zvikumbiro zvevatengi. Sarudzo iri nyore inopihwa: kushandisa murairo nginx -s <SIGNAL> kumisa maitiro kungave nekukasira kudzima kana nenyasha kudzima mode. Zviripachena, ndiyo yekupedzisira sarudzo inotifarira.

Ipapo zvese zviri nyore: iwe unofanirwa kuwedzera kune preStop-hook murairo unozotumira chiratidzo chakanaka chekudzima. Izvi zvinogona kuitwa muDeployment, mumudziyo block:

       lifecycle:
          preStop:
            exec:
              command:
              - /usr/sbin/nginx
              - -s
              - quit

Zvino, kana pod yavhara, tichaona zvinotevera mumatanda emidziyo yeNGINX:

2018/01/25 13:58:31 [notice] 1#1: signal 3 (SIGQUIT) received, shutting down
2018/01/25 13:58:31 [notice] 11#11: gracefully shutting down

Uye izvi zvichareva zvatinoda: NGINX inomirira kuti zvikumbiro zvizadziswe, yobva yauraya maitiro. Nekudaro, pazasi isu tichafunga zvakare dambudziko rinowanzoitika nekuda kweiyo, kunyangwe nemurairo nginx -s quit nzira yacho inopera zvisiri izvo.

Uye panguva ino isu taitwa ne NGINX: zvishoma kubva pamatanda unogona kunzwisisa kuti zvinhu zvose zviri kushanda sezvazvinofanira.

Chii chakaitika nePHP-FPM? Inobata sei kudzima kwakanaka? Ngatizvionei.

PHP-FPM

Panyaya ye PHP-FPM, pane zvishoma zvishoma ruzivo. Kana iwe uchitarisa pa official manual maererano PHP-FPM, ichataura kuti zvinotevera POSIX zvikwangwani zvinogamuchirwa:

1. SIGINT, SIGTERM - fast shutdown;
2. SIGQUIT - kuvharika kwenyasha (zvatinoda).

Zviratidzo zvakasara hazvidiwi mubasa iri, saka tichasiya kuongorora kwavo. Kuti umise maitiro nemazvo, iwe unozofanirwa kunyora inotevera preStop hook:

        lifecycle:
          preStop:
            exec:
              command:
              - /bin/kill
              - -SIGQUIT
              - "1"

Pakutanga kuona, izvi ndizvo zvese zvinodikanwa kuita kuvharika kwakanaka mumidziyo miviri. Zvisinei, basa racho rakaoma kupfuura zvarinoratidzika. Pazasi pane zviitiko zviviri umo kuvharika kwakanaka hakuna kushanda uye kwakakonzera kusawanikwa kwenguva pfupi kweprojekiti panguva yekuendeswa.

Dzidzira. Matambudziko anogona kuitika nekudzima kwakanaka

NGINX

Chokutanga pane zvose, zvinobatsira kuyeuka: kunze kwekuita murairo nginx -s quit Pane imwezve nhanho yakakodzera kutarisisa. Takasangana nenyaya apo NGINX yaizoramba ichitumira SIGTERM pachinzvimbo cheSIGQUIT chiratidzo, zvichiita kuti zvikumbiro zvisapedze nemazvo. Mhosva dzakafanana dzinogona kuwanikwa, semuenzaniso, pano. Zvinosuruvarisa, hatina kukwanisa kuziva chikonzero chaicho chemuitiro uyu: pakanga pane kusava nechokwadi pamusoro peiyo NGINX version, asi haina kusimbiswa. Chiratidzo chaive chekuti mameseji akaonekwa mumatanda emidziyo yeNGINX: "vhura socket #10 yasara mukubatana 5", mushure mezvo podhi yakamira.

Isu tinogona kuona dambudziko rakadaro, semuenzaniso, kubva kumhinduro paIngress yatinoda:

Zviratidzo zvemaitiro emakodhi panguva yekuendeswa

Muchiitiko ichi, tinongogamuchira 503 kukanganisa kodhi kubva kuIngress pachayo: haigone kuwana iyo NGINX mudziyo, sezvo isingachasvikike. Kana iwe ukatarisa matanda emidziyo ane NGINX, ane zvinotevera:

[alert] 13939#0: *154 open socket #3 left in connection 16
[alert] 13939#0: *168 open socket #6 left in connection 13

Mushure mekushandura chiratidzo chekumira, mudziyo unotanga kumira zvakanaka: izvi zvinosimbiswa nekuti iyo 503 kukanganisa haichaonekwi.

Kana iwe ukasangana nedambudziko rakafanana, zvine musoro kufunga kuti chii chinomira chiratidzo chinoshandiswa mumudziyo uye kuti preStop hoko inotaridzika sei chaizvo. Zvinogoneka kuti chikonzero chiripo mune izvi.

PHP-FPM ... uye nezvimwe

Dambudziko nePHP-FPM rinotsanangurwa nenzira shoma: haimirire kupedzwa kwemaitiro emwana, inovagumisa, ndicho chikonzero kukanganisa kwe502 kunoitika panguva yekuendesa uye mamwe mabasa. Pane akati wandei mabug report pa bugs.php.net kubvira 2005 (eg pano и pano), inotsanangura dambudziko iri. Asi iwe unogona kunge usingaone chero chinhu mumatanda: PHP-FPM ichazivisa kupedzwa kwemaitiro ayo pasina zvikanganiso kana yechitatu-bato zviziviso.

Zvakakosha kujekesa kuti dambudziko racho pacharo rinogona kutsamira kune zvishoma kana hukuru pane chikumbiro pachacho uye chinogona kusazviratidza, semuenzaniso, mukutarisa. Kana iwe ukasangana nazvo, nyore workaround inouya mupfungwa kutanga: wedzera preStop hoko ne sleep(30). Ichakubvumidza kuti upedze zvese zvikumbiro zvaive kare (uye isu hatigamuchire zvitsva, sezvo pod kare anokwanisa ku Kugumisa), uye mushure memasekondi makumi matatu pod pachayo inopera nechiratidzo SIGTERM.

Icho chinopera icho lifecycle nekuti mudziyo uchataridzika seizvi:

    lifecycle:
      preStop:
        exec:
          command:
          - /bin/sleep
          - "30"

Nekudaro, nekuda kweiyo 30-yechipiri sleep isu tiri zvakawanda isu tichawedzera nguva yekuendesa, sezvo pod yega yega ichamiswa zvishoma 30 seconds, izvo zvakaipa. Chii chingaitwa pamusoro peizvi?

Ngatitendeukire kune bato rine chekuita nekuita kwakananga kwechikumbiro. Muchiitiko chedu ndizvo PHP-FPM, iyo nekusarudzika haitarise maitirwo emwana wayo maitiro: Iyo master process inogumiswa nekukurumidza. Unogona kushandura maitiro aya uchishandisa rairo process_control_timeout, iyo inotsanangura miganhu yenguva yemaitiro emwana kumirira masaini kubva kuna tenzi. Kana iwe ukaisa kukosha kumasekonzi makumi maviri, izvi zvinovhara yakawanda yemibvunzo inomhanya mumudziyo uye inomisa iyo master process kana yapera.

Neruzivo urwu, ngatidzokere kudambudziko redu rekupedzisira. Sezvambotaurwa, Kubernetes haisi monolithic chikuva: kutaurirana pakati pezvikamu zvayo zvakasiyana kunotora nguva. Izvi zvinonyanya kuitika kana tichifunga nezvekushanda kweIngresses nezvimwe zvinhu zvine hukama, sezvo nekuda kwekunonoka kwakadaro panguva yekutumirwa zviri nyore kuwana kuwedzera kwe500 kukanganisa. Semuenzaniso, kukanganisa kunogona kuitika padanho rekutumira chikumbiro kune kumusoro, asi iyo "nguva lag" yekudyidzana pakati pezvikamu ipfupi - isingasviki sekondi.

Naizvozvo, Pakazara nemurairo watotaurwa process_control_timeout unogona kushandisa chivakwa chinotevera lifecycle:

lifecycle:
  preStop:
    exec:
      command: ["/bin/bash","-c","/bin/sleep 1; kill -QUIT 1"]

Muchiitiko ichi, tichabhadhara kunonoka nemurairo sleep uye usawedzera zvakanyanya nguva yekuendesa: pane musiyano unooneka pakati pemasekonzi makumi matatu neimwe?.. Muchokwadi, ndiyo process_control_timeoutuye lifecycle inoshandiswa chete se "mambure ekuchengetedza" kana paine lag.

Kazhinji kutaura iyo yakatsanangurwa maitiro uye inoenderana workaround inoshanda kwete chete kune PHP-FPM. Mamiriro akafanana anogona kuitika neimwe nzira kana imwe nzira kana uchishandisa mimwe mitauro/mafuremu. Kana iwe usingakwanise kugadzirisa kuvharika kwenyasha nedzimwe nzira - semuenzaniso, nekunyorazve kodhi kuitira kuti application iite nemazvo masaini ekumisa - unogona kushandisa yakatsanangurwa nzira. Inogona kunge isiri iyo yakanyanya kunaka, asi inoshanda.

Dzidzira. Kuyedzwa kwemutoro kutarisa kushanda kwepodhi

Kuyedzwa kwemutoro ndiyo imwe yenzira dzekutarisa kuti mudziyo unoshanda sei, sezvo maitiro aya achiiunza padyo nemamiriro chaiwo ekurwa kana vashandisi vakashanyira saiti. Kuti uedze mazano ari pamusoro apa, unogona kushandisa Yandex.Tankom: Inovhara zvese zvatinoda zvakakwana. Aya anotevera mazano uye mazano ekuita kuedza nemuenzaniso wakajeka kubva pane zvakaitika kwatiri nekuda kwemagrafu eGrafana neYandex.Tank pachayo.

Chinonyanya kukosha apa ndechekuti tarisa kuchinja nhanho nhanho. Mushure mekuwedzera gadziriso nyowani, mhanyisa bvunzo uye uone kana zvabuda zvachinja zvichienzaniswa neyekupedzisira kumhanya. Kana zvisina kudaro, zvichange zvakaoma kuziva zvigadziriswe zvisingabatsiri, uye nekufamba kwenguva zvinogona kukuvadza chete (somuenzaniso, kuwedzera nguva yekuendesa).

Imwe nuance ndeyekutarisa matanda emidziyo panguva yekugumiswa kwayo. Ruzivo rwekuvharwa kwakanaka kwakanyorwa ipapo here? Pane here kukanganisa mumatanda kana uchiwana zvimwe zviwanikwa (semuenzaniso, kune yakavakidzana PHP-FPM mudziyo)? Zvikanganiso mukushandisa pachako (sezvakaita neNGINX yakatsanangurwa pamusoro)? Ndinovimba kuti ruzivo rwekutanga kubva kuchinyorwa chino ruchakubatsira iwe kunzwisisa zviri nani zvinoitika kumudziyo panguva yekugumiswa kwayo.

Saka, yekutanga bvunzo kumhanya yakaitika pasina lifecycle uye pasina mamwe mirairo ye server application (process_control_timeout mu PHP-FPM). Chinangwa chebvunzo iyi chaive chekuona huwandu hwemhosho (uye kana paine chero). Zvakare, kubva kune rumwe ruzivo, iwe unofanirwa kuziva kuti avhareji yenguva yekuendesa kune yega yega pod yaive masekonzi mashanu kusvika gumi kusvika yagadzirira zvakakwana. Mibairo yacho ndeiyi:

Iyo Yandex.Tank information panel inoratidza spike ye502 kukanganisa, kwakaitika panguva yekutumirwa uye yakagara paavhareji kusvika kumasekonzi mashanu. Sezvingabvira izvi zvaive nekuti zvikumbiro zvaivepo kune yekare pod zvaive zvichigumiswa pazvaimiswa. Mushure meizvi, zvikanganiso zve5 zvakaonekwa, izvo zvaive mugumisiro wegaba rakamiswa NGINX, iro rakadonhedzawo zvisungo nekuda kwekumashure (izvo zvakadzivirira Ingress kubva pairi).

Ngationei kuti sei process_control_timeout muPHP-FPM ichatibatsira kumirira kupedzwa kwemaitiro emwana, i.e. gadzirisa zvikanganiso zvakadaro. Isa zvakare uchishandisa rairo iyi:

Hapasisina zvikanganiso panguva ye500th deployment! Kuiswa kwacho kunobudirira, kudzima kwakanaka kunoshanda.

Nekudaro, zvakakosha kurangarira nyaya neIngress midziyo, chikamu chidiki chezvikanganiso zvatingagashira nekuda kwekunonoka kwenguva. Kuti udzivise ivo, chasara kuwedzera chimiro ne sleep uye dzokorora kutumira. Nekudaro, mune yedu chaiyo, hapana shanduko dzakaonekwa (zvakare, hapana zvikanganiso).

mhedziso

Kumisa maitiro nenyasha, isu tinotarisira maitiro anotevera kubva pachikumbiro:

1. Mirira masekondi mashoma uye wobva wamira kugamuchira zvitsva zvinongedzo.
2. Mirira kuti zvikumbiro zvese zvipedze uye uvhare zvese zvinongedzo zvekuchengetedza izvo zvisiri kuita zvikumbiro.
3. Pedzisa maitiro ako.

Nekudaro, hazvisi zvese zvinoshandiswa zvinogona kushanda nenzira iyi. Imwe mhinduro kudambudziko muKubernetes realities ndeiyi:

• kuwedzera pre-stop hook iyo inomirira mashoma mashoma;
• kudzidza iyo faira yekumisikidza yedu backend kune yakakodzera paramita.

Muenzaniso ne NGINX unojekesa kuti kunyange chikumbiro chinofanira kutanga kugadzirisa zviratidzo zvekugumisa nenzira yakarurama chingasadaro, saka zvakakosha kutarisa kukanganisa kwe500 panguva yekushandiswa kwekushandiswa. Izvi zvinokubvumirawo kuti utarise dambudziko racho zvakanyanya uye kwete kutarisa pane imwe pod kana chigadziko, asi tarisa hurongwa hwose huzere.

Sechishandiso chekuedza, unogona kushandisa Yandex.Tank pamwe chete neipi yekuongorora system (munyaya yedu, data yakatorwa kubva kuGrafana nePrometheus backend yekuedzwa). Matambudziko ane nyasha kuvharika anooneka zvakajeka pasi pemitoro inorema iyo bhenji inogona kugadzira, uye kutarisa kunobatsira kuongorora mamiriro acho zvakadzama panguva kana mushure mebvunzo.

Mukupindura mhinduro pachinyorwa: zvakakosha kutaura kuti matambudziko nemhinduro zvinotsanangurwa pano maererano neNGINX Ingress. Kune zvimwe zviitiko, kune zvimwe zvigadziriso, izvo zvatingafunga nezvazvo mune zvinotevera zvigadzirwa zvenhevedzano.

PS

Zvimwe kubva kuK8s matipi & matipi akateedzana:

• «Tsika Yekukanganisa Mapeji muNGINX Ingress";
• «Nezve node kugoverwa uye mitoro pawebhu application";
• «Kuwana kune saiti dze dev";
• «Kumhanyisa bootstrap kune yakakura dhatabhesi".

Source: www.habr.com