Kubernetes cov lus qhia & kev ua kom yuam kev: cov yam ntxwv ntawm kev kaw zoo hauv NGINX thiab PHP-FPM

Ib qho xwm txheej thaum siv CI / CD hauv Kubernetes: daim ntawv thov yuav tsum tsis tuaj yeem lees txais cov neeg thov tshiab ua ntej tso tseg, thiab qhov tseem ceeb tshaj, ua tiav cov uas twb muaj lawm.

Ua raws li qhov xwm txheej no tso cai rau koj ua tiav xoom downtime thaum lub sijhawm xa mus. Txawm li cas los xij, txawm tias thaum siv cov pob khoom nrov heev (xws li NGINX thiab PHP-FPM), koj tuaj yeem ntsib cov teeb meem uas yuav ua rau muaj qhov yuam kev nrog txhua qhov kev xa tawm ...

Kev xav. Cas pod nyob

Peb twb tau luam tawm nyob rau hauv kom meej txog lub neej voj voog ntawm ib tug pod kab lus no. Nyob rau hauv cov ntsiab lus ntawm lub ncauj lus nyob rau hauv kev txiav txim siab, peb txaus siab rau cov hauv qab no: thaum lub sij hawm lub pod nkag mus rau hauv lub xeev. Terminating, cov lus thov tshiab tsis raug xa mus rau nws (pod tshem tawm los ntawm cov npe ntawm cov ntsiab lus kawg rau kev pabcuam). Yog li, txhawm rau kom tsis txhob poob qis thaum lub sijhawm xa mus, nws yog qhov txaus rau peb los daws qhov teeb meem ntawm kev tso tseg daim ntawv thov kom raug.

Koj yuav tsum nco ntsoov tias lub sijhawm ua kev zoo siab yog lub sijhawm 30 vib nas this: Tom qab no, lub plhaub taum yuav raug txiav thiab daim ntawv thov yuav tsum muaj sijhawm los ua txhua qhov kev thov ua ntej lub sijhawm no. Примечание: Txawm hais tias ib qho kev thov uas siv ntau tshaj 5-10 vib nas this twb muaj teeb meem, thiab kev kaw zoo nkauj yuav tsis pab nws ...

Txhawm rau kom nkag siab zoo dua yuav tshwm sim thaum lub pod xaus, tsuas yog saib ntawm daim duab hauv qab no:

A1, B1 - Tau txais kev hloov pauv ntawm lub xeev ntawm lub hearth
A2 - Tawm SIGTERM
B2 - Tshem tawm cov pods ntawm qhov kawg
B3 - Tau txais kev hloov pauv (cov npe ntawm cov ntsiab lus kawg tau hloov pauv)
B4 - Hloov kho cov cai iptables

Thov nco ntsoov: rho tawm qhov kawg ntawm lub pod thiab xa SIGTERM tsis tshwm sim ua ntu zus, tab sis nyob rau tib lub sijhawm. Thiab vim qhov tseeb tias Ingress tsis tau txais cov npe tshiab ntawm Endpoints tam sim ntawd, cov lus thov tshiab los ntawm cov neeg siv khoom yuav raug xa mus rau hauv lub pod, uas yuav ua rau 500 qhov yuam kev thaum lub sij hawm pods xaus. (rau cov ntaub ntawv ntxaws ntxiv ntawm qhov teeb meem no, peb txhais). Qhov teeb meem no yuav tsum tau daws raws li hauv qab no:

• Xa Kev Sib Txuas: kaw hauv cov lus teb headers (yog tias qhov no cuam tshuam txog HTTP thov).
• Yog tias nws tsis tuaj yeem hloov pauv rau cov cai, cov lus hauv qab no piav qhia txog kev daws teeb meem uas yuav tso cai rau koj ua cov ntawv thov kom txog thaum kawg ntawm lub sijhawm zoo.

Kev xav. Yuav ua li cas NGINX thiab PHP-FPM txiav lawv cov txheej txheem

NGINX

Cia peb pib nrog NGINX, txij li txhua yam yog ntau dua lossis tsawg dua nrog nws. Kev dhia mus rau hauv txoj kev xav, peb kawm tias NGINX muaj ib tus tswv txheej txheem thiab ob peb "cov neeg ua haujlwm" - cov no yog cov txheej txheem menyuam yaus uas ua cov neeg thov. Ib qho kev xaiv yooj yim yog muab: siv cov lus txib nginx -s <SIGNAL> txiav tawm cov txheej txheem nyob rau hauv ceev kaw los yog graceful shutdown hom. Obviously, nws yog qhov kev xaiv tom kawg uas nyiam peb.

Ces txhua yam yog yooj yim: koj yuav tsum tau ntxiv rau preStop-nug ib tug hais kom ua yuav xa ib tug graceful shutdown teeb liab. Qhov no tuaj yeem ua tiav hauv Kev Tshaj Tawm, hauv lub thawv thaiv:

       lifecycle:
          preStop:
            exec:
              command:
              - /usr/sbin/nginx
              - -s
              - quit

Tam sim no, thaum lub plhaub kaw lawm, peb yuav pom cov hauv qab no hauv NGINX thawv cav:

2018/01/25 13:58:31 [notice] 1#1: signal 3 (SIGQUIT) received, shutting down
2018/01/25 13:58:31 [notice] 11#11: gracefully shutting down

Thiab qhov no yuav txhais tau tias peb xav tau dab tsi: NGINX tos qhov kev thov kom tiav, thiab tom qab ntawd tua cov txheej txheem. Txawm li cas los xij, hauv qab no peb tseem yuav xav txog qhov teeb meem tshwm sim vim qhov twg, txawm tias nrog cov lus txib nginx -s quit cov txheej txheem xaus tsis raug.

Thiab nyob rau theem no peb tau ua tiav nrog NGINX: tsawg kawg ntawm cov cav koj tuaj yeem nkag siab tias txhua yam ua haujlwm raws li nws yuav tsum tau ua.

Dab tsi yog qhov deal nrog PHP-FPM? Yuav ua li cas tswj kev kaw qhov zoo? Cia peb xav txog nws.

PHP-FPM

Nyob rau hauv rooj plaub ntawm PHP-FPM, muaj cov ntaub ntawv tsawg me ntsis. Yog koj tsom rau phau ntawv official raws li PHP-FPM, nws yuav hais tias POSIX cov cim hauv qab no tau txais:

1. SIGINT, SIGTERM - ceev kaw;
2. SIGQUIT - Kev kaw zoo nkauj (qhov peb xav tau).

Cov cim tseem ceeb tsis tas yuav tsum tau ua hauv txoj haujlwm no, yog li peb yuav tshem tawm lawv qhov kev tshuaj xyuas. Txhawm rau txiav cov txheej txheem kom raug, koj yuav tsum tau sau cov lus hauv qab no preStop nuv:

        lifecycle:
          preStop:
            exec:
              command:
              - /bin/kill
              - -SIGQUIT
              - "1"

Thaum xub thawj siab ib muag, qhov no yog txhua yam uas yuav tsum tau ua ib tug graceful kaw nyob rau hauv ob lub thawv. Txawm li cas los xij, txoj haujlwm nyuaj dua li qhov nws zoo li. Hauv qab no yog ob qhov xwm txheej uas qhov kev kaw zoo nkauj tsis ua haujlwm thiab ua rau lub sijhawm luv luv ntawm qhov project thaum lub sijhawm xa mus.

Xyaum. Muaj teeb meem nrog kev kaw zoo nkauj

NGINX

Ua ntej tshaj plaws, nws tseem ceeb heev kom nco ntsoov: ntxiv rau kev ua tiav cov lus txib nginx -s quit Muaj ib theem ntxiv uas tsim nyog them rau. Peb tau ntsib qhov teeb meem uas NGINX tseem yuav xa SIGTERM es tsis txhob ntawm SIGQUIT teeb liab, ua rau kev thov ua tsis tiav. Cov xwm txheej zoo sib xws tuaj yeem pom, piv txwv li, no. Hmoov tsis zoo, peb tsis tuaj yeem txiav txim siab qhov laj thawj tshwj xeeb rau tus cwj pwm no: muaj qhov tsis txaus ntseeg txog NGINX version, tab sis nws tsis tau lees paub. Cov tsos mob yog tias cov lus tau pom nyob rau hauv NGINX thawv cav: "qhib qhov (socket) # 10 sab laug hauv kev sib txuas 5", tom qab ntawd lub pod nres.

Peb tuaj yeem soj ntsuam qhov teeb meem no, piv txwv li, los ntawm cov lus teb ntawm Ingress peb xav tau:

Indicators ntawm xwm txheej codes thaum lub sij hawm deployment

Hauv qhov no, peb tsuas yog tau txais 503 qhov yuam kev code los ntawm Ingress nws tus kheej: nws tsis tuaj yeem nkag mus rau NGINX thawv, vim nws tsis tuaj yeem siv tau lawm. Yog tias koj saib lub thawv ntim nrog NGINX, lawv muaj cov hauv qab no:

[alert] 13939#0: *154 open socket #3 left in connection 16
[alert] 13939#0: *168 open socket #6 left in connection 13

Tom qab hloov lub teeb liab nres, lub thawv pib nres kom raug: qhov no tau lees paub los ntawm qhov tseeb tias qhov yuam kev 503 tsis pom lawm.

Yog tias koj ntsib qhov teeb meem zoo sib xws, nws ua rau kev txiav txim siab seb lub teeb liab nres yog siv nyob rau hauv lub thawv thiab dab tsi raws nraim qhov preStop nuv zoo li. Nws yog qhov ua tau heev uas yog vim li cas nyob hauv qhov no.

PHP-FPM ... thiab ntau dua

Qhov teeb meem nrog PHP-FPM tau piav qhia hauv qhov tsis tseem ceeb: nws tsis tos kom tiav cov txheej txheem menyuam yaus, nws txiav lawv, uas yog vim li cas 502 qhov yuam kev tshwm sim thaum xa mus thiab lwm yam haujlwm. Muaj ob peb kab lus ceeb toom ntawm bugs.php.net txij li xyoo 2005 (xws li no и no), uas piav txog qhov teeb meem no. Tab sis koj feem ntau yuav tsis pom dab tsi hauv cov cav: PHP-FPM yuav tshaj tawm qhov ua tiav ntawm nws cov txheej txheem yam tsis muaj qhov yuam kev lossis cov ntawv ceeb toom thib peb.

Nws tsim nyog qhia meej tias qhov teeb meem nws tus kheej yuav nyob ntawm qhov tsawg dua lossis ntau dua ntawm daim ntawv thov nws tus kheej thiab yuav tsis tshwm sim nws tus kheej, piv txwv li, hauv kev saib xyuas. Yog tias koj ntsib nws, ib qho yooj yim workaround los rau hauv siab ua ntej: ntxiv preStop nuv nrog sleep(30). Nws yuav tso cai rau koj ua kom tiav txhua qhov kev thov uas tau ua ntej (thiab peb tsis lees txais cov tshiab, txij li cov pod twb muaj peev xwm ntawm Terminating), thiab tom qab 30 vib nas this lub pod nws tus kheej yuav xaus nrog lub teeb liab SIGTERM.

Nws hloov tawm ntawd lifecycle rau lub thawv yuav zoo li no:

    lifecycle:
      preStop:
        exec:
          command:
          - /bin/sleep
          - "30"

Txawm li cas los xij, vim yog 30-thib ob sleep Kuv muaj zog heev peb yuav nce lub sij hawm xa tawm, txij li txhua lub pod yuav raug txiav yam tsawg 30 vib nas this, uas yog phem. Yuav ua li cas txog qhov no?

Cia peb tig mus rau tog neeg lub luag haujlwm rau kev ua tiav ntawm daim ntawv thov. Hauv peb qhov xwm txheej nws yog PHP-FPM, uas los ntawm lub neej ntawd tsis saib xyuas kev ua tiav ntawm nws cov txheej txheem menyuam yaus: Tus txheej txheem tswv raug txiav tam sim ntawd. Koj tuaj yeem hloov tus cwj pwm no siv cov lus qhia process_control_timeout, uas qhia txog lub sijhawm txwv rau cov txheej txheem menyuam yaus kom tos cov teeb liab los ntawm tus tswv. Yog tias koj teeb tsa tus nqi rau 20 vib nas this, qhov no yuav npog feem ntau ntawm cov lus nug uas khiav hauv lub thawv thiab yuav nres cov txheej txheem tswv thaum lawv ua tiav.

Nrog rau qhov kev paub no, cia peb rov qab mus rau peb qhov teeb meem kawg. Raws li tau hais, Kubernetes tsis yog lub platform monolithic: kev sib txuas lus ntawm nws cov khoom sib txawv yuav siv sij hawm qee lub sijhawm. Qhov no yog qhov tseeb tshwj xeeb tshaj yog thaum peb xav txog kev ua haujlwm ntawm Ingresses thiab lwm yam cuam tshuam, vim tias vim qhov kev ncua sij hawm ntawm kev xa tawm nws yog qhov yooj yim kom tau txais 500 qhov yuam kev. Piv txwv li, qhov yuam kev yuav tshwm sim nyob rau theem ntawm kev xa daim ntawv thov mus rau qhov siab, tab sis "lub sij hawm lag" ntawm kev sib cuam tshuam ntawm cov khoom yog luv luv - tsawg tshaj li ib ob.

Yog li ntawd, Nyob rau hauv tag nrho nrog rau cov lus qhia uas twb tau hais lawm process_control_timeout koj tuaj yeem siv cov kev tsim kho hauv qab no rau lifecycle:

lifecycle:
  preStop:
    exec:
      command: ["/bin/bash","-c","/bin/sleep 1; kill -QUIT 1"]

Hauv qhov no, peb yuav them rov qab rau qhov kev ncua nrog cov lus txib sleep thiab tsis txhob nce lub sij hawm xa mus: puas muaj qhov sib txawv ntawm 30 vib nas this thiab ib qho? ... Qhov tseeb, nws yog process_control_timeoutthiab lifecycle tsuas yog siv los ua "kev nyab xeeb net" thaum muaj kev lag luam.

Feem ntau hais lus tus cwj pwm piav qhia thiab cov kev sib raug zoo siv tsis yog rau PHP-FPM nkaus xwb. Ib qho xwm txheej zoo sib xws yuav tshwm sim ib txoj hauv kev lossis lwm qhov tshwm sim thaum siv lwm yam lus / lub hauv paus. Yog tias koj tsis tuaj yeem kho qhov zoo nkauj kaw hauv lwm txoj hauv kev - piv txwv li, los ntawm kev rov sau cov lej kom daim ntawv thov raug ua tiav cov cim txiav - koj tuaj yeem siv txoj kev piav qhia. Tej zaum nws yuav tsis yog qhov zoo nkauj tshaj plaws, tab sis nws ua haujlwm.

Xyaum. Load test los xyuas qhov kev ua haujlwm ntawm lub plhaub

Kev ntsuas kev thauj khoom yog ib txoj hauv kev los xyuas seb lub thawv ua haujlwm li cas, txij li cov txheej txheem no coj nws los ze zog rau qhov kev sib ntaus sib tua tiag tiag thaum cov neeg siv tuaj xyuas lub xaib. Txhawm rau kuaj cov lus pom zoo saum toj no, koj tuaj yeem siv Yandex.Tankom: Nws npog tag nrho peb cov kev xav tau zoo kawg nkaus. Cov hauv qab no yog cov lus qhia thiab cov lus pom zoo rau kev sim nrog cov piv txwv meej los ntawm peb qhov kev paub ua tsaug rau cov duab ntawm Grafana thiab Yandex.Tank nws tus kheej.

Qhov tseem ceeb tshaj plaws ntawm no yog xyuas cov kev hloov pauv ib kauj ruam. Tom qab ntxiv qhov kho tshiab, khiav qhov kev sim thiab saib seb cov txiaj ntsig tau hloov pauv li cas piv rau qhov kawg khiav. Txwv tsis pub, nws yuav nyuaj rau txheeb xyuas cov kev daws teeb meem tsis zoo, thiab nyob rau lub sijhawm ntev nws tsuas tuaj yeem ua kev puas tsuaj (piv txwv li, nce lub sijhawm xa tawm).

Lwm qhov nuance yog saib cov thawv ntim khoom thaum nws txiav tawm. Puas yog cov ntaub ntawv hais txog kev kaw zoo nkauj kaw nyob ntawd? Puas muaj qhov yuam kev hauv cov cav thaum nkag mus rau lwm yam kev pabcuam (piv txwv li, rau lub thawv PHP-FPM uas nyob sib ze)? Qhov yuam kev hauv daim ntawv thov nws tus kheej (xws li NGINX tau piav qhia saum toj no)? Kuv vam tias cov ntaub ntawv qhia los ntawm tsab xov xwm no yuav pab koj nkag siab zoo dua tias muaj dab tsi tshwm sim rau lub thawv thaum nws txiav tawm.

Yog li, thawj qhov kev sim khiav tau tshwm sim yam tsis muaj lifecycle thiab tsis muaj cov lus qhia ntxiv rau daim ntawv thov server (process_control_timeout hauv PHP-FPM). Lub hom phiaj ntawm qhov kev xeem no yog txhawm rau txheeb xyuas qhov kwv yees ntawm qhov yuam kev (thiab seb puas muaj). Tsis tas li ntawd, los ntawm cov ntaub ntawv ntxiv, koj yuav tsum paub tias qhov nruab nrab xa tawm lub sijhawm rau txhua lub pod yog li 5-10 vib nas this kom txog thaum nws npaj txhij. Cov txiaj ntsig yog:

Lub vaj huam sib luag Yandex.Tank qhia tau hais tias muaj qhov tsis zoo ntawm 502 qhov yuam kev, uas tshwm sim thaum lub sijhawm xa mus thiab siv sijhawm ntev txog li 5 vib nas this. Xav tias qhov no yog vim qhov kev thov uas twb muaj lawm rau lub plhaub qub tau raug txiav tawm thaum nws raug txiav tawm. Tom qab qhov no, 503 qhov yuam kev tshwm sim, uas yog qhov tshwm sim ntawm lub thawv NGINX nres, uas tseem poob kev sib txuas vim qhov backend (uas tiv thaiv Ingress los ntawm kev txuas mus rau nws).

Cia peb saib yuav ua li cas process_control_timeout hauv PHP-FPM yuav pab peb tos kom tiav cov txheej txheem menyuam yaus, piv txwv li. kho tej yam yuam kev. Rov ua dua siv cov lus qhia no:

Tsis muaj qhov yuam kev ntxiv thaum lub sijhawm 500th xa tawm! Kev xa tawm yog ua tiav, kev kaw haujlwm zoo.

Txawm li cas los xij, nws tsim nyog nco ntsoov qhov teeb meem nrog Ingress ntim, feem pua ​​​​ntawm qhov ua yuam kev me me uas peb yuav tau txais vim yog lub sijhawm lag luam. Txhawm rau zam lawv, txhua yam uas tseem tshuav yog ntxiv cov qauv nrog sleep thiab rov ua qhov kev xa tawm. Txawm li cas los xij, hauv peb qhov tshwj xeeb, tsis muaj kev hloov pauv tau pom (dua, tsis muaj qhov yuam kev).

xaus

Txhawm rau tshem tawm cov txheej txheem zoo, peb cia siab tias yuav coj tus cwj pwm hauv qab no los ntawm daim ntawv thov:

1. Tos ob peb vib nas this thiab tom qab ntawd tso tseg tsis lees txais kev sib txuas tshiab.
2. Tos rau txhua qhov kev thov kom ua tiav thiab kaw tag nrho cov kev sib txuas khaws cia uas tsis ua raws li kev thov.
3. Xaus koj cov txheej txheem.

Txawm li cas los xij, tsis yog txhua daim ntawv thov tuaj yeem ua haujlwm li no. Ib qho kev daws teeb meem hauv Kubernetes qhov tseeb yog:

• ntxiv ib qho kev sib tw ua ntej uas yuav tos ob peb feeb;
• kawm cov ntaub ntawv configuration ntawm peb lub backend rau cov tsis tsim nyog.

Qhov piv txwv nrog NGINX ua kom pom tseeb tias txawm tias daim ntawv thov uas yuav tsum pib ua cov ntawv txiav tawm kom raug yuav tsis ua li ntawd, yog li nws yog ib qho tseem ceeb uas yuav tau kuaj xyuas 500 qhov yuam kev thaum lub sij hawm xa daim ntawv thov. Qhov no kuj tso cai rau koj saib qhov teeb meem ntau dua thiab tsis tsom mus rau ib qho pod lossis thawv, tab sis saib tag nrho cov kev tsim kho vaj tse tag nrho.

Raws li cov cuab yeej ntsuas, koj tuaj yeem siv Yandex.Tank ua ke nrog txhua qhov kev soj ntsuam (hauv peb cov ntaub ntawv, cov ntaub ntawv raug coj los ntawm Grafana nrog Prometheus backend rau kev xeem). Cov teeb meem nrog kev kaw zoo nkauj tau pom meej meej nyob rau hauv cov khoom hnyav uas lub ntsuas ntsuas tuaj yeem tsim tau, thiab kev saib xyuas pab txhawm rau txheeb xyuas qhov xwm txheej kom ntxaws ntxiv thaum lossis tom qab kuaj.

Hauv kev teb rau cov lus tawm tswv yim ntawm tsab xov xwm: nws tsim nyog hais tias cov teeb meem thiab kev daws teeb meem tau piav qhia ntawm no nyob rau hauv kev cuam tshuam rau NGINX Ingress. Rau lwm qhov xwm txheej, muaj lwm yam kev daws teeb meem, uas peb yuav xav txog hauv cov ntaub ntawv hauv qab no ntawm koob.

PS

Lwm yam los ntawm K8s lub tswv yim & tricks series:

• «Custom Error Pages hauv NGINX Ingress»;
• «Hais txog kev faib cov nodes thiab cov loads ntawm lub vev xaib thov»;
• «Nkag mus rau dev sites»;
• «Txhim kho bootstrap rau cov ntaub ntawv loj".

Tau qhov twg los: www.hab.com