Kuv lub npe yog Viktor Yagofarov, thiab kuv tab tom tsim lub Kubernetes platform ntawm DomClick ua tus thawj tswj kev txhim kho hauv pab pawg Ops (kev ua haujlwm). Kuv xav tham txog cov qauv ntawm peb cov Dev <-> Ops txheej txheem, cov yam ntxwv ntawm kev khiav hauj lwm ib qho ntawm cov loj tshaj plaws k8s pawg nyob rau hauv Russia, nrog rau cov kev coj ntawm DevOps / SRE uas peb pab neeg siv.
Pab neeg Ops
Pab neeg Ops tam sim no muaj 15 tus neeg. Peb ntawm lawv yog lub luag haujlwm rau lub chaw ua haujlwm, ob qho haujlwm nyob rau hauv ib cheeb tsam sib txawv thiab muaj, suav nrog thaum hmo ntuj. Yog li, ib tug neeg ntawm Ops yeej ib txwm nyob ntawm saib thiab npaj los teb rau qhov xwm txheej ntawm txhua qhov nyuaj. Peb tsis muaj kev ua haujlwm hmo ntuj, uas ua rau peb lub siab xav thiab muab sijhawm rau txhua tus kom tau pw txaus thiab siv sijhawm so tsis yog ntawm lub computer xwb.
Txhua tus muaj peev xwm sib txawv: networkers, DBAs, ELK pawg kws tshwj xeeb, Kubernetes admins/developers, saib xyuas, virtualization, hardware specialists, thiab lwm yam. Ib qho sib koom ua ke txhua tus - txhua tus tuaj yeem hloov pauv ib qho ntawm peb rau qee qhov: piv txwv li, qhia cov nodes tshiab rau hauv pawg k8s, hloov tshiab PostgreSQL, sau CI / CD + Ansible pipeline, automate ib yam dab tsi hauv Python / Bash / Mus, txuas kho vajtse rau Data center. Kev muaj peev xwm muaj zog hauv txhua cheeb tsam tsis tiv thaiv koj los ntawm kev hloov koj cov kev taw qhia ntawm kev ua si thiab pib txhim kho hauv qee qhov chaw. Piv txwv li, kuv tau koom nrog ib lub tuam txhab ua tus kws tshaj lij PostgreSQL, thiab tam sim no kuv lub luag haujlwm tseem ceeb yog Kubernetes pawg. Hauv pab pawg, txhua qhov siab txais tos thiab kev nkag siab ntawm leverage yog tsim heev.
Los ntawm txoj kev, peb mus yos hav zoov. Cov kev xav tau rau cov neeg sib tw yog tus qauv zoo heev. Rau kuv tus kheej, nws yog ib qho tseem ceeb uas ib tug neeg haum rau hauv pab neeg, tsis yog kev sib cav, tab sis kuj paub yuav ua li cas tiv thaiv nws qhov kev xav, xav tsim thiab tsis ntshai ua tej yam tshiab, muab nws cov tswv yim. Tsis tas li ntawd, kev txawj sau ntawv hauv cov lus sau ntawv, kev paub txog cov hauv paus ntawm Linux thiab lus Askiv yuav tsum muaj. Lus Askiv yog xav tau yooj yim kom ib tug neeg, nyob rau hauv qhov kev tshwm sim ntawm ib tug fakap, muaj peev xwm google daws qhov teeb meem nyob rau hauv 10 vib nas this, thiab tsis nyob rau hauv 10 feeb. Tam sim no nws nyuaj heev los nrhiav cov kws tshaj lij nrog kev paub tob ntawm Linux: nws lom zem, tab sis ob ntawm peb tus neeg sib tw tsis tuaj yeem teb cov lus nug "Dab tsi yog Load Average? Nws yog dab tsi?", thiab lo lus nug "Yuav ua li cas sib sau ua ke ib tug tseem ceeb pob tseg los ntawm ib tug C program" yog suav hais tias ib yam dab tsi los ntawm lub ntiaj teb no ntawm superhumans ... los yog dinosaurs. Peb yuav tsum tso qhov no, vim feem ntau tib neeg tau tsim lwm yam kev txawj ntse, tab sis peb yuav qhia Linux. Cov lus teb rau lo lus nug "vim li cas DevOps engineer yuav tsum paub tag nrho cov no nyob rau hauv lub ntiaj teb no niaj hnub ntawm huab" yuav tsum tau tawm sab nraum lub Scope ntawm tsab xov xwm, tab sis nyob rau hauv peb lo lus: tag nrho cov no yog xav tau.
Cov cuab yeej pab neeg
Pab Pawg Cov cuab yeej ua lub luag haujlwm tseem ceeb hauv automation. Lawv lub luag haujlwm tseem ceeb yog los tsim cov duab yooj yim thiab CLI cov cuab yeej rau cov neeg tsim khoom. Piv txwv li, peb qhov kev txhim kho sab hauv Confer tso cai rau koj kom tso tawm ib daim ntawv thov rau Kubernetes nrog ob peb nas nias, teeb tsa nws cov peev txheej, cov yuam sij los ntawm lub vault, thiab lwm yam. Yav dhau los, muaj Jenkins + Helm 2, tab sis kuv yuav tsum tau txhim kho kuv tus kheej cov cuab yeej los tshem tawm cov ntawv luam tawm thiab ua kom zoo ib yam rau software lifecycle.
Pab pawg Ops tsis sau cov kav dej rau cov neeg tsim khoom, tab sis tuaj yeem qhia txog txhua yam teeb meem hauv lawv cov ntawv sau (qee tus neeg tseem muaj Helm 3).
DevOps
Raws li rau DevOps, peb pom nws zoo li no:
Dev teams sau code, dov tawm ntawm Confer to dev -> qa/stage -> prod. Kev lav phib xaub kom ntseeg tau tias txoj cai tsis qeeb thiab tsis muaj qhov yuam kev nyob nrog Dev thiab Ops pab pawg. Thaum nruab hnub, tus neeg ua haujlwm ntawm pab pawg Ops yuav tsum xub teb rau qhov xwm txheej nrog nws daim ntawv thov, thiab thaum yav tsaus ntuj thiab hmo ntuj, tus thawj coj ntawm lub luag haujlwm (Ops) yuav tsum tsa tus tsim tawm ntawm lub luag haujlwm yog tias nws paub txog. paub tseeb tias qhov teeb meem tsis yog nyob rau hauv lub infrastructure. Tag nrho cov kev ntsuas thiab kev ceeb toom hauv kev saib xyuas tshwm sim tau los yog ib nrab-automatically.
Ops thaj tsam ntawm lub luag haujlwm pib txij li lub sijhawm daim ntawv thov raug dov tawm mus rau hauv kev tsim khoom, tab sis Dev lub luag haujlwm tsis xaus rau qhov ntawd - peb ua tib yam thiab nyob hauv tib lub nkoj.
Cov neeg tsim tawm qhia cov neeg saib xyuas yog tias lawv xav tau kev pab sau tus tswj hwm microservice (piv txwv li, Mus rov qab + HTML5), thiab cov thawj coj qhia cov neeg tsim tawm ntawm cov teeb meem kev tsim kho lossis teeb meem ntsig txog k8s.
Los ntawm txoj kev, peb tsis muaj monolith txhua, tsuas yog microservices. Lawv tus lej tam sim no hloov pauv ntawm 900 thiab 1000 hauv pawg prod k8s, yog ntsuas los ntawm tus lej kev xa khoom. Tus naj npawb ntawm cov pods fluctuates ntawm 1700 thiab 2000. Tam sim no muaj txog 2000 pods nyob rau hauv pawg prod.
Kuv tsis tuaj yeem muab cov lej tseeb, txij li peb saib xyuas cov microservices uas tsis tsim nyog thiab txiav lawv tawm ib nrab. K8s pab peb taug qab cov chaw tsis tsim nyog , uas txuag tau ntau yam kev pab thiab nyiaj txiag.
Kev tswj cov peev txheej
Saib xyuas
Cov txheej txheem zoo thiab kev saib xyuas cov ntaub ntawv dhau los ua lub hauv paus hauv kev ua haujlwm ntawm pawg loj. Peb tseem tsis tau pom qhov kev daws teeb meem thoob ntiaj teb uas yuav npog 100% ntawm txhua qhov kev saib xyuas xav tau, yog li peb niaj hnub tsim cov kev cai sib txawv hauv ib puag ncig no.
- Zabbix. Kev saib xyuas zoo qub, uas yog tsim los saib xyuas tag nrho cov xwm txheej ntawm cov txheej txheem. Nws qhia peb thaum lub node tuag nyob rau hauv cov nqe lus ntawm kev ua, nco, disks, network, thiab hais txog. Tsis muaj dab tsi supernatural, tab sis peb kuj muaj cais DaemonSet ntawm cov neeg sawv cev, nrog kev pab los ntawm cov uas, piv txwv li, peb saib xyuas lub xeev ntawm DNS nyob rau hauv pawg: peb saib rau ruam coredns pods, peb xyuas qhov muaj ntawm sab nraud hosts. Nws yuav zoo li yog vim li cas thab nrog qhov no, tab sis nrog cov tsheb loj loj ntawm cov khoom siv no yog qhov tseem ceeb ntawm kev ua tsis tiav. kuv xa , cas kuv tawm tsam nrog DNS kev ua tau zoo hauv pawg.
- Prometheus Operator. Ib txheej ntawm cov exporters sib txawv muab cov ntsiab lus loj ntawm tag nrho cov khoom ntawm pawg. Tom ntej no, peb pom tag nrho cov no ntawm cov dashboards loj hauv Grafana, thiab siv alertmanager rau kev ceeb toom.
Lwm yam cuab yeej pab tau rau peb yog . Peb tau sau nws tom qab ob peb zaug peb tau ntsib qhov xwm txheej uas ib pab pawg sib tshooj lwm pab pawg Ingress txoj hauv kev, ua rau 50x yuam kev. Tam sim no ua ntej xa mus rau kev tsim khoom, cov neeg tsim khoom tshawb xyuas tias tsis muaj leej twg yuav cuam tshuam, thiab rau kuv pab neeg no yog ib qho cuab yeej zoo rau kev kuaj thawj zaug ntawm cov teeb meem nrog Ingresses. Nws yog funny uas thaum xub thawj nws tau sau rau admins thiab nws zoo nkaus li "clumsy", tab sis tom qab dev teams poob rau hauv kev hlub nrog lub cuab tam, nws tau hloov ntau heev thiab pib zoo li tsis zoo li "ib tug admin ua lub web ntsej muag rau admins. β Tsis ntev peb yuav tso tseg cov cuab yeej no thiab cov xwm txheej zoo li no yuav raug lees paub txawm tias ua ntej lub raj xa dej tau dov tawm.
Pab neeg muaj peev xwm hauv Cube
Ua ntej peb nkag mus rau hauv cov piv txwv, nws tsim nyog piav qhia seb peb faib cov peev txheej li cas rau microservices.
Kom nkag siab tias pab pawg twg thiab siv lawv li cas peev txheej (processor, nco, SSD hauv zos), peb faib txhua qhov lus txib nws tus kheej npe nyob rau hauv "Cube" thiab txwv nws qhov siab tshaj plaws peev xwm ntawm processor, nco thiab disk, yav tas los tham txog cov kev xav tau ntawm pab pawg. Raws li, ib qho lus txib, feem ntau, yuav tsis thaiv tag nrho pawg rau kev xa tawm, faib ntau txhiab tus cores thiab terabytes ntawm lub cim xeeb. Kev nkag mus rau namespace tau tso cai los ntawm AD (peb siv RBAC). Namespaces thiab lawv cov kev txwv tau ntxiv los ntawm kev thov rub mus rau GIT chaw cia khoom, thiab tom qab ntawd txhua yam tau txiav tawm los ntawm Ansible pipeline.
Ib qho piv txwv ntawm kev faib cov peev txheej rau pab pawg:
namespaces:
chat-team:
pods: 23
limits:
cpu: 11
memory: 20Gi
requests:
cpu: 11
memory: 20Gi
Thov thiab txwv
Kub" thov yog tus naj npawb ntawm guaranteed reserved kev pab rau plhaub taum pauv (ib lossis ntau lub thawv docker) hauv ib pawg. Limit yog qhov siab tshaj plaws uas tsis tau lees paub. Feem ntau koj tuaj yeem pom ntawm daim duab qhia tias qee pab pawg tau teeb tsa nws tus kheej ntau dhau ntawm kev thov rau tag nrho nws cov ntawv thov thiab tsis tuaj yeem xa daim ntawv thov mus rau "Cube", vim tias txhua qhov kev thov nyob rau hauv lawv lub npe chaw twb tau "siv".
Txoj hauv kev kom raug ntawm qhov teeb meem no yog saib qhov kev siv cov peev txheej tiag tiag thiab muab piv nrog cov nyiaj tau thov (Request).
Nyob rau hauv cov screenshots saum toj no koj tuaj yeem pom tias "Thov" CPUs tau phim rau cov xov tooj tiag tiag, thiab Kev txwv tuaj yeem tshaj li cov xov tooj ntawm CPU tiag =)
Tam sim no cia peb saib qee qhov chaw nyob hauv kev nthuav dav (Kuv xaiv namespace kube-system - qhov system namespace rau cov khoom ntawm "Cube" nws tus kheej) thiab pom qhov piv ntawm qhov siv tau siv lub sijhawm thiab nco rau qhov thov:
Nws yog qhov pom tseeb tias ntau lub cim xeeb thiab CPU tau tshwj tseg rau cov kev pabcuam system dua li siv tau tiag tiag. Nyob rau hauv cov ntaub ntawv ntawm kube-system, qhov no yog justified: nws tshwm sim hais tias nginx ingress maub los los yog nodelocaldns ntawm lawv ncov ntaus CPU thiab noj ntau RAM, yog li ntawm no xws li ib tug cia yog justified. Tsis tas li ntawd, peb tsis tuaj yeem tso siab rau cov kab kos rau 3 teev dhau los: nws yog qhov xav pom cov kev ntsuas keeb kwm dhau lub sijhawm loj.
Ib qho system ntawm "cov lus pom zoo" tau tsim. Piv txwv li, ntawm no koj tuaj yeem pom cov peev txheej twg yuav zoo dua los ntawm kev nce qhov "tso cai" (lub siab tso cai bar) kom "throttling" tsis tshwm sim: lub sijhawm thaum cov peev txheej tau siv CPU lossis nco hauv lub sijhawm faib thiab tos kom txog thaum nws yuav "tsis khov":
Thiab ntawm no yog cov pods uas yuav tsum txwv tsis pub lawv qab los noj mov:
rau throttling + Kev saib xyuas cov peev txheej, koj tuaj yeem sau ntau dua ib kab lus, yog li nug cov lus nug hauv cov lus. Hauv ob peb lo lus, kuv tuaj yeem hais tias txoj haujlwm ntawm kev siv cov kev ntsuas no nyuaj heev thiab yuav tsum tau siv sijhawm ntau thiab ntsuas kev ua haujlwm nrog "lub qhov rais" thiab "CTE" Prometheus / VictoriaMetrics (cov ntsiab lus no yog nyob rau hauv quotes, txij li thaum muaj yuav luag. tsis muaj dab tsi zoo li no hauv PromQL, thiab koj yuav tsum faib cov lus nug txaus ntshai rau hauv ntau lub vijtsam ntawm cov ntawv nyeem thiab ua kom lawv zoo).
Raws li qhov tshwm sim, cov neeg tsim khoom muaj cov cuab yeej los saib xyuas lawv lub npe chaw nyob hauv Cube, thiab lawv tuaj yeem xaiv rau lawv tus kheej nyob qhov twg thiab lub sijhawm twg cov ntawv thov tuaj yeem muaj lawv cov peev txheej "txiav," thiab cov servers twg tuaj yeem muab tag nrho CPU txhua hmo.
Cov txheej txheem
Nyob rau hauv lub tuam txhab raws li nws yog tam sim no zam, peb ua raws li DevOps- thiab SRE- tus kws kho mob Thaum lub tuam txhab muaj 1000 microservices, txog 350 tus tsim tawm thiab 15 tus thawj tswj hwm rau tag nrho cov txheej txheem, koj yuav tsum "ua kom zoo nkauj": tom qab tag nrho cov "baswords" no yog qhov yuav tsum tau ceev kom automate txhua yam thiab txhua tus, thiab cov thawj coj yuav tsum tsis txhob muaj lub hauv paus. hauv cov txheej txheem.
Raws li Ops, peb muab ntau yam kev ntsuas thiab cov dashboards rau cov neeg tsim khoom ntsig txog cov kev pabcuam teb cov nqi thiab qhov yuam kev.
Peb siv methodologies xws li: , ΠΈ los ntawm kev sib koom ua ke. Peb sim txo tus naj npawb ntawm cov dashboards kom pom tseeb tias qhov kev pabcuam twg tam sim no degrading (piv txwv li, cov lej teb ib ob, lub sijhawm teb los ntawm 99 feem pua), thiab lwm yam. Thaum qee qhov kev ntsuas tshiab tsim nyog rau cov dashboards dav dav, peb tam sim kos thiab ntxiv rau lawv.
Kuv tsis tau kos duab rau ib hlis. Qhov no tej zaum yog ib qho cim zoo: nws txhais tau tias feem ntau ntawm "xav tau" twb tau pom lawm. Nws tshwm sim tias lub lim tiam kuv yuav kos ib co duab tshiab tsawg kawg ib zaug ib hnub.
Cov txiaj ntsig tau txais txiaj ntsig zoo vim tias tam sim no cov neeg tsim khoom tsis tshua mus rau admins nrog cov lus nug "qhov twg los saib qee yam kev ntsuas."
Kev nqis tes ua Kev Pabcuam Mesh tsuas yog nyob ib ncig ntawm lub ces kaum thiab yuav tsum ua kom lub neej yooj yim dua rau txhua tus, cov npoj yaig los ntawm Cov Cuab Yeej twb tau nyob ze rau kev siv qhov paub daws teeb meem "Istio ntawm tus neeg noj qab haus huv": lub neej voj voog ntawm txhua qhov kev thov HTTP yuav pom hauv kev saib xyuas, thiab nws ib txwm muaj peev xwm nkag siab "nyob rau theem twg txhua yam tawg" thaum lub sijhawm kev pabcuam (thiab tsis yog) kev sib cuam tshuam. Sau npe yuav tau txais xov xwm los ntawm DomClick hub. =)
Kubernetes kev txhawb nqa infrastructure
Keeb kwm, peb siv lub patched version Kubespray - Lub luag haujlwm tseem ceeb rau kev xa tawm, txuas ntxiv thiab hloov kho Kubernetes. Qee lub sij hawm, kev txhawb nqa rau cov khoom siv tsis yog kubeadm raug txiav los ntawm cov ceg tseem ceeb, thiab cov txheej txheem ntawm kev hloov mus rau kubeadm tsis tau npaj. Yog li ntawd, lub tuam txhab Southbridge tau ua nws tus kheej diav rawg (nrog kubeadm kev txhawb nqa thiab kho sai rau cov teeb meem tseem ceeb).
Cov txheej txheem rau kev hloov kho txhua pawg k8s zoo li no:
- Noj Kubespray los ntawm Southbridge, kos nrog peb cov xov, Merjim.
- Peb tab tom dov tawm qhov hloov tshiab rau Kev nyuaj siab- "Cub".
- Peb yob tawm qhov hloov tshiab ib ntawm ib zaug (hauv Ansible qhov no yog "serial: 1") hauv dev- "Cub".
- Peb hloov tshiab chob Hnub Saturday yav tsaus ntuj ib lub ntawm ib lub sijhawm.
Muaj cov phiaj xwm los hloov nws yav tom ntej Kubespray rau ib yam dab tsi sai dua thiab mus rau kub em.
Nyob rau hauv tag nrho peb muaj peb "Cubes": Stress, Dev thiab Prod. Peb tab tom npaj yuav tshaj tawm lwm qhov (kub standby) Prod-"Cube" nyob rau hauv lub thib ob cov ntaub ntawv chaw. Kev nyuaj siab ΠΈ dev nyob hauv "virtual machines" (oVirt rau Stress thiab VMWare huab rau Dev). chob- "Lub voos xwmfab" nyob ntawm "liab liab qab": cov no yog tib yam ntawm 32 CPU threads, 64-128 GB ntawm lub cim xeeb thiab 300 GB SSD RAID 10 - muaj 50 ntawm lawv tag nrho. Peb "nyias" nodes tau mob siab rau "masters" chob- "Cuba": 16 GB ntawm lub cim xeeb, 12 CPU xov.
Rau kev muag khoom, peb nyiam siv "liab hlau" thiab zam cov txheej tsis tsim nyog xws li OpenStack: peb tsis xav tau "cov neeg zej zog nrov" thiab CPU nyiag lub sij hawm. Thiab qhov nyuaj ntawm kev tswj hwm kwv yees li ob npaug ntawm cov ntaub ntawv hauv tsev OpenStack.
Rau CI / CD "Cubic" thiab lwm yam khoom siv hauv vaj tse peb siv GIT server cais, Helm 3 (nws yog qhov kev hloov pauv ntawm Helm 2, tab sis peb zoo siab heev nrog cov kev xaiv. piv rau ib qho me me ais), Jenkins, Ansible thiab Docker. Peb nyiam cov ceg ntoo thiab xa mus rau qhov chaw sib txawv ntawm ib qho chaw cia khoom.
xaus
Qhov no yog, nyob rau hauv dav dav, dab tsi cov txheej txheem DevOps zoo li ntawm DomClick los ntawm kev xav ntawm tus kws ua haujlwm ua haujlwm. Cov kab lus tau dhau los ua cov txuj ci tsawg dua qhov kuv xav tau: yog li ntawd, ua raws li DomClick xov xwm ntawm HabrΓ©: yuav muaj ntau cov lus "hardcore" txog Kubernetes thiab ntau dua.
Tau qhov twg los: www.hab.com