Dzina langa ndine Viktor Yagofarov, ndipo ndikupanga nsanja ya Kubernetes ku DomClick monga woyang'anira chitukuko chaukadaulo mu gulu la Ops (ntchito). Ndikufuna kulankhula za dongosolo lathu la Dev <-> Ops, zomwe zimagwira ntchito imodzi mwamagulu akuluakulu a k8s ku Russia, komanso machitidwe a DevOps / SRE omwe gulu lathu limagwiritsa ntchito.
Timu ya Ops
Gulu la Ops pakadali pano lili ndi anthu 15. Atatu mwa iwo ali ndi udindo wa ofesi, awiri amagwira ntchito m'madera osiyanasiyana ndipo amapezeka, kuphatikizapo usiku. Chifukwa chake, wina wochokera ku Ops amakhala nthawi zonse poyang'anira ndipo ali wokonzeka kuyankha pazochitika zovuta zilizonse. Sitikhala ndi maulendo ausiku, omwe amateteza maganizo athu ndikupatsa aliyense mwayi wogona mokwanira komanso nthawi yopuma osati pa kompyuta.
Aliyense ali ndi luso losiyana: ma network, ma DBA, akatswiri a stack ELK, Kubernetes admins/madivelopa, kuyang'anira, kuona, akatswiri a hardware, ndi zina zotero. Chinthu chimodzi chimagwirizanitsa aliyense - aliyense akhoza kulowa m'malo mwa aliyense wa ife kumlingo wina: mwachitsanzo, yambitsani ma node atsopano mumagulu a k8s, sinthani PostgreSQL, lembani mapaipi a CI/CD + Ansible, sinthani china chake mu Python/Bash/Go, lumikizani zida ku. Data center. Luso lamphamvu m'dera lililonse silikulepheretsani kusintha momwe mumagwirira ntchito ndikuyamba kuchita bwino mbali ina. Mwachitsanzo, ndidalowa nawo kampani ngati katswiri wa PostgreSQL, ndipo tsopano gawo langa lalikulu ndi magulu a Kubernetes. Mu timu, kutalika kulikonse kumalandiridwa ndipo lingaliro lachidziwitso limapangidwa kwambiri.
Mwa njira, tikusaka. Zofunikira kwa ofuna kusankha ndizokhazikika. Kwa ine ndekha, ndikofunika kuti munthu agwirizane ndi gululo, osati kutsutsana, komanso amadziwa momwe angatetezere maganizo ake, akufuna kukulitsa ndipo saopa kuchita chinachake chatsopano, amapereka malingaliro ake. Komanso, luso lopanga mapulogalamu m'zilankhulo zolembera, kudziwa zoyambira za Linux ndi Chingerezi ndikofunikira. Chingerezi chimangofunika kuti munthu akakhala ndi fakap athe google yankho lavutoli mumasekondi 10, osati mphindi 10. Tsopano ndizovuta kwambiri kupeza akatswiri omwe ali ndi chidziwitso chozama cha Linux: ndizoseketsa, koma awiri mwa atatu omwe akufuna kuyankha sangathe kuyankha funso lakuti "Kodi Average Yonyamula N'chiyani? Zimapangidwa ndi chiyani?", Ndipo funso lakuti "Momwe mungasonkhanitsire dambo lalikulu kuchokera ku pulogalamu ya C" limatengedwa kuti ndi chinachake kuchokera ku dziko la supermen ... kapena dinosaurs. Tiyenera kupirira izi, popeza nthawi zambiri anthu amakhala ndi luso lina, koma tidzaphunzitsa Linux. Yankho la funso lakuti "chifukwa chiyani injiniya wa DevOps ayenera kudziwa zonsezi m'dziko lamakono la mitambo" ayenera kusiyidwa kunja kwa nkhaniyo, koma m'mawu atatu: zonsezi ndizofunikira.
Zida Zamagulu
Gulu la Zida limagwira ntchito yofunika kwambiri pakupanga makina. Ntchito yawo yayikulu ndikupanga zida zosavuta zojambulira ndi CLI za opanga. Mwachitsanzo, Confer yathu yachitukuko chamkati imakupatsani mwayi wotulutsa pulogalamu ku Kubernetes ndikungodina pang'ono mbewa, sinthani zomwe zili, makiyi ochokera kumalo osungira, ndi zina zambiri. M'mbuyomu, panali Jenkins + Helm 2, koma ndidayenera kupanga chida changa chochotsera kukopera ndikubweretsa kufanana kwa pulogalamu yamoyo.
Gulu la Ops silimalemba mapaipi kwa omanga, koma limatha kulangiza pazolemba zawo (anthu ena akadali ndi Helm 3).
DevOps
Ponena za DevOps, tikuwona motere:
Magulu a Dev amalemba khodi, tulutsani kudzera pa Confer to dev -> qa/stage -> prod. Udindo wowonetsetsa kuti codeyo sichedwa ndipo ilibe zolakwika ili ndi magulu a Dev ndi Ops. Masana, munthu yemwe ali pantchito kuchokera ku gulu la Ops ayenera choyamba kuyankha zomwe zachitika ndi pempho lawo, ndipo madzulo ndi usiku, woyang'anira ntchito (Ops) ayenera kudzutsa wopanga ntchitoyo ngati akudziwa. otsimikiza kuti vuto siliri mu zomangamanga. Ma metrics ndi zidziwitso zonse pakuwunika zimawonekera zokha kapena modzidzimutsa.
Dera laudindo la Ops limayamba kuyambira pomwe ntchitoyo idatulutsidwa, koma udindo wa Dev sumathera pamenepo - timachita zomwezo ndipo tili m'bwato lomwelo.
Madivelopa amalangiza ma admin ngati akufuna kuthandizidwa polemba microservice ya admin (mwachitsanzo, Go backend + HTML5), ndipo ma admins amalangiza opanga mapulogalamu pazovuta zilizonse zokhudzana ndi ma k8s.
Mwa njira, tilibe monolith konse, ma microservices okha. Chiwerengero chawo mpaka pano chikusinthasintha pakati pa 900 ndi 1000 mugulu la prod k8s, ngati ayesedwa ndi nambala. ntchito. Chiwerengero cha makoko chimasinthasintha pakati pa 1700 ndi 2000. Pakali pano pali pafupifupi 2000 pods mu prod cluster.
Sindingathe kupereka ziwerengero zenizeni, chifukwa timawunika ma microservices osafunikira ndikuzidula zokha. Ma K8 amatithandiza kuti tizitsatira zinthu zosafunikira , zomwe zimapulumutsa chuma ndi ndalama zambiri.
Kasamalidwe kazinthu
Kuwunikira
Kuyang'anira kokonzedwa bwino komanso kodziwitsa bwino kumakhala mwala wapangodya pakugwira ntchito kwa gulu lalikulu. Sitinapeze yankho lapadziko lonse lapansi lomwe lingakhudze 100% yazofunikira zonse zowunikira, kotero nthawi ndi nthawi timapanga njira zosiyanasiyana zothanirana ndi vutoli.
- Zabbix. Kuyang'anira kwakale kwabwino, komwe kumayang'ana makamaka kuyang'anira momwe zimakhalira. Imatiuza pamene node imafa potengera kukonza, kukumbukira, ma disks, network, ndi zina zotero. Palibe chauzimu, koma tilinso ndi DaemonSet yosiyana ya othandizira, mothandizidwa ndi zomwe, mwachitsanzo, timayang'anira dziko la DNS mgululi: timayang'ana ma pods opusa a coredns, timayang'ana kupezeka kwa makamu akunja. Zikuwoneka kuti chifukwa chiyani mukuvutikira ndi izi, koma ndi kuchuluka kwa magalimoto ambiri gawo ili ndilolephera kwambiri. Poyamba ndinali kale , momwe ndimavutikira ndi magwiridwe antchito a DNS mgulu.
- Wothandizira Prometheus. Gulu la otumiza kunja osiyanasiyana limapereka chithunzithunzi chachikulu cha zigawo zonse za gululo. Kenako, tikuwona zonsezi pamadeshibodi akulu ku Grafana, ndikugwiritsa ntchito alertmanager pazidziwitso.
Chida china chothandiza kwa ife chinali . Tidalemba pambuyo kangapo tidakumana ndi vuto lomwe gulu lina lidadutsa njira za gulu lina, zomwe zidabweretsa zolakwika 50x. Tsopano musanatumize kupanga, omanga ayang'ane kuti palibe amene angakhudzidwe, ndipo kwa gulu langa ichi ndi chida chabwino chodziwira matenda oyambirira a Ingresses. Ndizoseketsa kuti poyamba zidalembedwera ma admins ndipo zimawoneka ngati "zopusa", koma magulu a dev atakonda chidacho, zidasintha kwambiri ndipo zidayamba kuoneka osati ngati "admin adapanga nkhope ya ma admin. β Posachedwapa tidzasiya chida ichi ndipo zinthu zoterezi zidzatsimikiziridwa ngakhale payipi isanatulutsidwe.
Zida zamagulu mu Cube
Tisanalowe mu zitsanzo, ndi bwino kufotokoza momwe timagawira zothandizira microservices.
Kuti mumvetsetse ndi magulu ati omwe amagwiritsa ntchito zothandizira (purosesa, kukumbukira, SSD yakomweko), timagawa lamulo lililonse lake malo okhala mu "Cube" ndikuchepetsa kuthekera kwake kwakukulu potengera purosesa, kukumbukira ndi diski, atakambirana kale zofunikira zamagulu. Chifukwa chake, lamulo limodzi, nthawi zambiri, silingatseke gulu lonse kuti liyimitsidwe, kugawa masauzande a ma cores ndi ma terabytes a kukumbukira. Kufikira kumalo a mayina kumaperekedwa kudzera mu AD (timagwiritsa ntchito RBAC). Malo a mayina ndi malire awo amawonjezedwa kudzera popempha kukoka kumalo osungirako a GIT, ndiyeno zonse zimangotulutsidwa kudzera paipi ya Ansible.
Chitsanzo cha kugawa zothandizira ku gulu:
namespaces:
chat-team:
pods: 23
limits:
cpu: 11
memory: 20Gi
requests:
cpu: 11
memory: 20Gi
Zopempha ndi malire
Cube" pempho ndi chiwerengero cha zinthu zotsimikiziridwa zosungidwa pod (chidebe chimodzi kapena zingapo za docker) mgulu. Malire ndi kuchuluka kosatsimikizika. Nthawi zambiri mumatha kuwona pamagrafu momwe gulu lina ladzipangira zopempha zambiri pazogwiritsa ntchito zonse ndipo silingathe kuyika pulogalamuyi ku "Cube", popeza zopempha zonse pansi pa dzina lawo "zagwiritsidwa kale ntchito".
Njira yolondola yotulutsira izi ndikuyang'ana kugwiritsa ntchito kwenikweni kwazinthu ndikufanizira ndi kuchuluka komwe kwafunsidwa (Pempho).
Pazithunzi pamwambapa mutha kuwona kuti ma CPU "Ofunsidwa" akufanana ndi kuchuluka kwa ulusi, ndipo Malire amatha kupitilira nambala yeniyeni ya ulusi wa CPU =)
Tsopano tiyeni tiwone malo ena mwatsatanetsatane (ndinasankha namespace kube-system - malo amazina a zigawo za "Cube" palokha) ndikuwona chiΕ΅erengero cha nthawi yogwiritsidwa ntchito ndi purosesa yogwiritsidwa ntchito ndi yofunsidwa:
Ndizodziwikiratu kuti kukumbukira zambiri ndi CPU zimasungidwa ntchito zamakina kuposa zomwe zimagwiritsidwa ntchito. Pankhani ya kube-system, izi ndizoyenera: zidachitika kuti nginx ingress controller kapena nodelocaldns pachimake adagunda CPU ndikuwononga RAM yambiri, kotero pano kusungitsa koteroko kuli koyenera. Kuphatikiza apo, sitingadalire ma chart a maola atatu omaliza: ndikofunikira kuwona ma metric am'mbiri pa nthawi yayikulu.
Dongosolo la "zolimbikitsa" linapangidwa. Mwachitsanzo, apa mutha kuwona kuti ndi zinthu ziti zomwe zingakhale bwino kukweza "malire" (malo ololedwa apamwamba) kuti "kugwedezeka" kusachitike: nthawi yomwe chida chatha kale CPU kapena kukumbukira mugawo lanthawi lomwe laperekedwa ndi akudikirira mpaka "kusazizira":
Ndipo apa pali mapoto omwe amayenera kuchepetsa chilakolako chawo:
pa kugwedeza + kuwunika kwazinthu, mutha kulemba zolemba zingapo, chifukwa chake funsani mafunso mu ndemanga. M'mawu ochepa, nditha kunena kuti ntchito yodzipangira ma metrics ndizovuta kwambiri ndipo imafuna nthawi yochulukirapo komanso kusanja zinthu ndi "zenera" ntchito ndi "CTE" Prometheus / VictoriaMetrics (mawu awa ali m'mawu, popeza pali pafupifupi palibe chonga ichi mu PromQL, ndipo muyenera kugawa mafunso owopsa m'mawu angapo ndikuwongolera).
Chotsatira chake, opanga ali ndi zida zowunikira mayina awo ku Cube, ndipo amatha kusankha okha kuti ndi nthawi yanji yomwe mapulogalamuwo angakhale nawo "kudula," ndi ma seva omwe angapereke CPU yonse usiku wonse.
Njira
Pakampani monga momwe zilili pano zapamwamba, timatsatira DevOps- ndi SONA-wothandizira Kampani ikakhala ndi ma microservices 1000, opanga pafupifupi 350 ndi ma admins 15 pazomangamanga zonse, muyenera kukhala "owoneka bwino": kuseri kwa "mabaswords" onsewa pakufunika mwachangu chilichonse ndi aliyense, ndipo ma admins sayenera kukhala otsekereza. mu ndondomeko.
Monga Ops, timapereka ma metric ndi ma dashboard osiyanasiyana kwa omanga okhudzana ndi kuchuluka kwa mayankho a ntchito ndi zolakwika.
Timagwiritsa ntchito njira monga: , ΠΈ powaphatikiza pamodzi. Timayesa kuchepetsa chiwerengero cha ma dashboards kuti pang'onopang'ono ziwonekere kuti ndi ntchito iti yomwe ikunyozetsa panopa (mwachitsanzo, zizindikiro zoyankhira pa sekondi imodzi, nthawi yoyankha ndi 99th percentile), ndi zina zotero. Ma metrics ena atsopano akakhala ofunikira pamadeshibodi wamba, timajambula ndikuwonjezera nthawi yomweyo.
Kwa mwezi umodzi sindinajambule ma graph. Ichi mwina ndi chizindikiro chabwino: zikutanthauza kuti zambiri za "zofuna" zakwaniritsidwa kale. Zinachitika kuti mkati mwa mlungu ndimajambula graph yatsopano kamodzi patsiku.
Zotsatira zake ndizofunika chifukwa tsopano opanga samakonda kupita kwa admins ndi mafunso "komwe angayang'ane mtundu wina wa metric."
Kukhazikitsa Service Mesh yangotsala pang'ono ndipo iyenera kupangitsa moyo kukhala wosavuta kwa aliyense, ogwira nawo ntchito ku Zida ali kale pafupi kukhazikitsa "Istio ya munthu wathanzi": moyo wa pempho lililonse la HTTP (s) liziwoneka pakuwunika, ndipo nthawi zonse zimakhala zotheka kumvetsetsa "panthawi yomwe zonse zidasweka" panthawi yolumikizana (osati kokha). Lembetsani ku nkhani za DomClick hub. =)
Kubernetes thandizo la zomangamanga
M'mbiri, timagwiritsa ntchito mtundu wa zigamba Kubespray - Udindo wofunikira pakutumiza, kukulitsa ndikusintha Kubernetes. Panthawi ina, kuthandizira kwa makhazikitsidwe osakhala a kubeadm kudadulidwa kuchokera kunthambi yayikulu, ndipo njira yosinthira kubeadm sinakonzedwe. Zotsatira zake, kampani yaku Southbridge idapanga foloko yake (yothandizidwa ndi kubeadm komanso kukonza mwachangu zovuta).
Njira yosinthira magulu onse a k8s ikuwoneka motere:
- Tengani Kubespray kuchokera ku Southbridge, fufuzani ndi ulusi wathu, Merjim.
- Tikutulutsa zosintha ku kupanikizika- "Cube".
- Timatulutsa zosinthazo nthawi imodzi (mu Ansible iyi ndi "serial: 1") mkati Dev- "Cube".
- Timasintha Prod Loweruka madzulo nodi imodzi panthawi.
Pali mapulani oti adzalowe m'malo mwake mtsogolomu Kubespray kwa china chake mwachangu ndikupita kubeadm.
Pazonse tili ndi "Cubes" zitatu: Kupsinjika, Dev ndi Prod. Tikukonzekera kuyambitsa ina (yotentha standby) Prod-"Cube" mu data center yachiwiri. kupanikizika ΠΈ Dev khalani mu "makina enieni" (oVirt for Stress ndi VMWare mtambo wa Dev). Prod- "Cube" imakhala pa "chitsulo chopanda kanthu": awa ndi mfundo zofanana ndi ulusi wa 32 CPU, 64-128 GB ya kukumbukira ndi 300 GB SSD RAID 10 - pali 50 yonse. Node zitatu "zoonda" zimaperekedwa kwa "masters" Prod- "Cuba": 16 GB ya kukumbukira, 12 CPU ulusi.
Pogulitsa, timakonda kugwiritsa ntchito "zitsulo zopanda kanthu" ndikupewa zigawo zosafunikira monga OpenStack: sitifunikira "oyandikana nawo aphokoso" ndi CPU kuba nthawi. Ndipo zovuta za kayendetsedwe kazinthu pafupifupi kawiri pankhani ya OpenStack mkati.
Kwa CI/CD "Cubic" ndi zida zina zogwirira ntchito timagwiritsa ntchito seva yosiyana ya GIT, Helm 3 (kunali kusintha kowawa kuchokera ku Helm 2, koma ndife okondwa kwambiri ndi zomwe mungasankhe. atomiki), Jenkins, Ansible ndi Docker. Timakonda mawonekedwe a nthambi ndikutumizidwa kumadera osiyanasiyana kuchokera kunkhokwe imodzi.
Pomaliza
Izi ndizomwe, momwe machitidwe a DevOps amawonekera ku DomClick kuchokera pamalingaliro a injiniya wa ntchito. Nkhaniyi idakhala yocheperako kuposa momwe ndimayembekezera: chifukwa chake, tsatirani nkhani za DomClick pa HabrΓ©: padzakhala zolemba zambiri za "hardcore" za Kubernetes ndi zina zambiri.
Source: www.habr.com