Sunana Viktor Yagofarov, kuma ina haɓaka dandalin Kubernetes a DomClick a matsayin mai sarrafa ci gaban fasaha a cikin ƙungiyar Ops (aiki). Ina so in yi magana game da tsarin tsarin mu na Dev <-> Ops, fasali na aiki ɗaya daga cikin manyan k8s gungu a Rasha, da kuma ayyukan DevOps / SRE wanda ƙungiyarmu ke aiki.
Ƙungiyar Ops
A halin yanzu ƙungiyar Ops tana da mutane 15. Uku daga cikinsu suna da alhakin ofishin, biyu suna aiki a wani yanki na daban kuma suna samuwa, ciki har da dare. Don haka, wani daga Ops koyaushe yana kan saka idanu kuma yana shirye don amsa wani lamari na kowane rikitarwa. Ba mu da lokutan dare, wanda ke adana tunaninmu kuma yana ba kowa damar samun isasshen barci da kuma ciyar da lokacin hutu ba kawai a kwamfutar ba.
Kowane mutum yana da ƙwarewa daban-daban: masu sadarwar yanar gizo, DBAs, ƙwararrun tari na ELK, Kubernetes admins/masu haɓakawa, saka idanu, haɓakawa, ƙwararrun kayan masarufi, da sauransu. Abu ɗaya ya haɗa kowa da kowa - kowa zai iya maye gurbin kowane ɗayanmu zuwa ɗan lokaci: misali, gabatar da sabbin nodes a cikin gungu na k8s, sabunta PostgreSQL, rubuta bututun CI / CD + Mai yiwuwa, sarrafa wani abu a Python/Bash/Go, haɗa kayan aiki zuwa Cibiyar bayanai. Ƙarfin ƙarfi a kowane yanki ba zai hana ku canza alkiblar ayyukanku da fara ingantawa a wani yanki ba. Misali, na shiga kamfani a matsayin kwararre na PostgreSQL, kuma yanzu babban yankin da nake da shi shine Kubernetes clusters. A cikin ƙungiyar, kowane tsayi yana maraba kuma an haɓaka ma'anar amfani sosai.
Af, muna farauta. Abubuwan da ake buƙata don ƴan takara suna da daidaitattun daidaito. A gare ni da kaina, yana da mahimmanci cewa mutum ya shiga cikin tawagar, ba rikici ba ne, amma kuma ya san yadda za a kare ra'ayinsa, yana so ya ci gaba kuma ba ya jin tsoron yin wani sabon abu, yana ba da ra'ayoyinsa. Hakanan, ana buƙatar ƙwarewar shirye-shirye a cikin harsunan rubutu, sanin tushen tushen Linux da Ingilishi. Ana buƙatar turanci ne kawai ta yadda mutum idan aka yi fakap zai iya google maganin matsalar cikin daƙiƙa 10, ba cikin mintuna 10 ba. Yanzu yana da matukar wahala a sami ƙwararru masu zurfin ilimin Linux: abin ban dariya ne, amma biyu daga cikin 'yan takara uku ba za su iya amsa tambayar "Mene ne Matsakaicin Load ba? Menene aka yi da shi?", da kuma tambayar "Yadda za a tara core juji daga shirin C" ana daukar wani abu daga duniyar supermen ... ko dinosaur. Dole ne mu haƙura da wannan, tunda yawanci mutane sun haɓaka wasu ƙwarewa sosai, amma za mu koyar da Linux. Amsar tambayar "me yasa injiniyan DevOps ke buƙatar sanin duk wannan a cikin duniyar zamani na girgije" dole ne a bar shi a waje da iyakokin labarin, amma a cikin kalmomi uku: duk wannan ana buƙata.
Kayan aikin kungiya
Ƙungiyar Kayan aikin tana taka muhimmiyar rawa a sarrafa kansa. Babban aikin su shine ƙirƙirar kayan aikin hoto masu dacewa da CLI don masu haɓakawa. Misali, ci gaban mu na cikin gida Confer yana ba ku damar fitar da aikace-aikace a zahiri zuwa Kubernetes tare da danna maballin linzamin kwamfuta kaɗan, saita albarkatun sa, maɓallan daga vault, da sauransu. A baya can, akwai Jenkins + Helm 2, amma dole ne in haɓaka kayan aikina don kawar da kwafin-manna da kawo daidaito ga tsarin rayuwar software.
Ƙungiyar Ops ba ta rubuta bututun mai don masu haɓakawa, amma suna iya ba da shawara kan kowane batu a cikin rubutun su (wasu mutane har yanzu suna da Helm 3).
DevOps
Dangane da DevOps, muna ganin shi kamar haka:
Ƙungiyoyin Dev suna rubuta lamba, fitar da shi ta hanyar Confer to dev -> qa/stage -> prod. Alhakin tabbatar da cewa lambar ba ta raguwa kuma baya ƙunshe da kurakurai yana tare da ƙungiyoyin Dev da Ops. Da rana, wanda ke aiki daga tawagar Ops ya kamata da farko ya amsa wani abu da ya faru da aikace-aikacensa, kuma da yamma da dare, mai gudanarwa da ke aiki (Ops) ya ta da mai haɓakawa yana aiki idan ya san tabbata cewa matsalar ba a cikin kayayyakin more rayuwa. Duk awo da faɗakarwa a cikin saka idanu suna bayyana ta atomatik ko Semi-atomatik.
Yankin Ops na alhakin yana farawa daga lokacin da aka fitar da aikace-aikacen zuwa samarwa, amma alhakin Dev bai ƙare a can ba - muna yin abu ɗaya kuma muna cikin jirgin ruwa ɗaya.
Masu haɓakawa suna ba da shawara ga admins idan suna buƙatar taimako don rubuta microservice mai gudanarwa (misali, Go backend + HTML5), kuma masu gudanarwa suna ba da shawara ga masu haɓakawa akan duk wani al'amurran kayan more rayuwa ko batutuwan da suka shafi k8s.
Af, ba mu da monolith kwata-kwata, kawai microservices. Adadin su ya zuwa yanzu yana canzawa tsakanin 900 zuwa 1000 a cikin gungu na samfur k8s, idan an auna ta lamba. tura abubuwa. Adadin kwas ɗin yana canzawa tsakanin 1700 zuwa 2000. A halin yanzu akwai kusan 2000 pods a cikin gungun samfuran.
Ba zan iya ba da takamaiman lambobi ba, tunda muna saka idanu kan ƙananan sabis ɗin da ba dole ba kuma muna yanke su ta atomatik. K8s yana taimaka mana kiyaye abubuwan da ba dole ba , wanda ke adana albarkatu da kuɗi da yawa.
Gudanar da albarkatun
Kulawa
Kyakkyawan tsari da saka idanu mai ba da labari ya zama ginshiƙan ginshiƙan aikin babban tari. Har yanzu ba mu sami mafita na duniya wanda zai rufe 100% na duk buƙatun saka idanu ba, don haka lokaci-lokaci muna ƙirƙirar mafita na al'ada daban-daban a cikin wannan yanayin.
- Zabbix. Kyakkyawan tsohon sa ido, wanda aka yi niyya da farko don bin diddigin yanayin gabaɗayan abubuwan more rayuwa. Yana gaya mana lokacin da kumburi ya mutu ta fuskar sarrafawa, ƙwaƙwalwa, diski, hanyar sadarwa, da sauransu. Babu wani abu na allahntaka, amma muna da ma'auni daban-daban na DaemonSet, tare da taimakon wanda, alal misali, muna saka idanu akan yanayin DNS a cikin gungu: muna neman kullun coredns marasa hankali, muna duba samuwa na runduna na waje. Zai yi kama da dalilin da yasa ya damu da wannan, amma tare da yawan zirga-zirgar zirga-zirgar wannan bangaren babban batu ne na gazawa. Na riga , yadda na yi fama da aikin DNS a cikin tari.
- Prometheus Operator. Saitin masu fitar da kayayyaki daban-daban yana ba da babban bayyani na duk sassan tari. Bayan haka, muna ganin duk wannan akan manyan dashboards a Grafana, kuma muna amfani da mai sarrafa faɗakarwa don faɗakarwa.
Wani kayan aiki mai amfani a gare mu shine . Mun rubuta shi bayan sau da yawa mun ci karo da wani yanayi inda ƙungiya ɗaya ta mamaye hanyoyin Ingress na wata ƙungiya, wanda ya haifar da kurakurai 50x. Yanzu kafin turawa zuwa samarwa, masu haɓakawa suna duba cewa babu wanda zai shafa, kuma ga ƙungiyara wannan kayan aiki ne mai kyau don ganewar farko na matsalolin da Ingresses. Abin ban dariya ne cewa da farko an rubuta shi don admins kuma ya yi kama da "mai ban sha'awa", amma bayan ƙungiyoyin dev sun ƙaunaci kayan aikin, ya canza da yawa kuma ya fara kama da "wani admin ya yi fuskar yanar gizo ga admins. ” Ba da daɗewa ba za mu yi watsi da wannan kayan aiki kuma za a tabbatar da irin waɗannan yanayi tun kafin a fitar da bututun.
Albarkatun kungiya a cikin Cube
Kafin mu shiga cikin misalan, yana da kyau mu bayyana yadda muke ware albarkatu don microservices.
Don fahimtar ko wane ƙungiyoyi ne kuma a cikin wane adadi suke amfani da su albarkatun (processor, memory, SSD na gida), muna ware kowane umarni nasa namespace a cikin "Cube" kuma yana iyakance iyakar ƙarfinsa dangane da na'ura mai sarrafawa, ƙwaƙwalwar ajiya da faifai, tun da farko an tattauna bukatun ƙungiyoyin. Saboda haka, umarni ɗaya, gabaɗaya, ba zai toshe dukkan gungu don turawa ba, wanda ke ware dubban muryoyi da terabyte na ƙwaƙwalwar ajiya. Ana ba da dama ga sararin suna ta AD (muna amfani da RBAC). Ana ƙara wuraren suna da iyakokin su ta hanyar buƙatun ja zuwa ma'ajiyar GIT, sannan ana fitar da komai ta atomatik ta bututun mai yiwuwa.
Misali na rarraba albarkatu ga ƙungiya:
namespaces:
chat-team:
pods: 23
limits:
cpu: 11
memory: 20Gi
requests:
cpu: 11
memory: 20Gi
Bukatu da iyaka
Cubed" request shine adadin albarkatun da aka tanada don kwafsa (kwangon docker ɗaya ko fiye) a cikin gungu. Iyaka shine mafi girman rashin garanti. Sau da yawa kuna iya gani akan jadawali yadda wasu ƙungiyar suka saita kansu buƙatun da yawa don duk aikace-aikacenta kuma ba za su iya tura aikace-aikacen zuwa “Cube” ba, tunda duk buƙatun da ke ƙarƙashin sunan su an riga an “kashe su”.
Madaidaicin hanyar fita daga wannan yanayin shine duba ainihin yadda ake amfani da albarkatu kuma kwatanta shi da adadin da ake buƙata (Buƙatar).
A cikin hotunan kariyar kwamfuta da ke sama zaku iya ganin cewa "An buƙata" CPUs an daidaita su zuwa ainihin adadin zaren, kuma Iyakoki na iya wuce ainihin adadin zaren CPU =)
Yanzu bari mu dubi wasu sunaye daki-daki (Na zaɓi sunan kube-tsarin - tsarin sunan tsarin don abubuwan da ke cikin "Cube" kanta) kuma mu ga rabon lokacin da aka yi amfani da processor da ƙwaƙwalwar ajiya ga wanda ake nema:
A bayyane yake cewa an tanadi ƙarin ƙwaƙwalwar ajiya da CPU don ayyukan tsarin fiye da yadda ake amfani da su. Game da tsarin kube, wannan ya cancanta: ya faru ne cewa mai sarrafa nginx ko nodelocaldns a saman su ya bugi CPU kuma ya cinye RAM da yawa, don haka a nan irin wannan ajiyar ya cancanta. Bugu da ƙari, ba za mu iya dogara ga ginshiƙi na sa'o'i 3 na ƙarshe ba: yana da kyawawa don ganin ma'auni na tarihi a cikin babban lokaci.
An haɓaka tsarin "shawarwari" Alal misali, a nan za ku iya ganin wadanne albarkatun zai fi dacewa da haɓaka "iyaka" (masanin da aka ba da izini) don kada "zubawa" ya faru: lokacin da kayan aiki ya riga ya kashe CPU ko ƙwaƙwalwar ajiya a cikin lokacin da aka ƙayyade kuma yana jira har sai ya zama "ba a daskarewa":
Kuma ga kwas ɗin da yakamata su hana sha'awar su:
a kan srotting + saka idanu akan albarkatu, zaku iya rubuta labarin sama da ɗaya, don haka kuyi tambayoyi a cikin sharhi. A cikin 'yan kalmomi, zan iya cewa aikin sarrafa irin waɗannan ma'auni yana da wuyar gaske kuma yana buƙatar lokaci mai yawa da daidaita aiki tare da ayyukan "taga" da "CTE" Prometheus / VictoriaMetrics (waɗannan sharuɗɗan suna cikin ƙididdiga, tun da kusan akwai kusan. Babu kamar wannan a cikin PromQL, kuma dole ne ku raba tambayoyin ban tsoro zuwa fuskokin rubutu da yawa kuma ku inganta su).
A sakamakon haka, masu haɓakawa suna da kayan aiki don saka idanu akan sunayensu a cikin Cube, kuma suna iya zaɓar wa kansu inda kuma a wane lokaci aikace-aikacen za su iya "yanke" albarkatun su, da kuma waɗanne sabobin za a iya ba da dukan CPU dukan dare.
Hanyoyi
A cikin kamfani kamar yadda yake a yanzu na gaye, Muna bin DevOps- da SRE-ma'aikaci Lokacin da kamfani yana da microservices 1000, game da masu haɓaka 350 da admins 15 don duk abubuwan more rayuwa, dole ne ku kasance "gaye": bayan duk waɗannan "baswords" akwai buƙatar gaggawa don sarrafa komai da kowa, kuma admins bai kamata ya zama ƙwanƙwasa ba. a cikin matakai.
A matsayin Ops, muna samar da ma'auni daban-daban da dashboards don masu haɓakawa masu alaƙa da ƙimar amsawar sabis da kurakurai.
Muna amfani da hanyoyin kamar: , и ta hanyar hada su tare. Muna ƙoƙarin rage adadin dashboards ta yadda a kallo za a iya bayyana a fili wane sabis ne ke ɓarna a halin yanzu (misali, lambobin amsawa a sakan daya, lokacin amsawa ta kashi 99), da sauransu. Da zaran wasu sabbin ma'auni suka zama larura don manyan dashboards, nan da nan za mu zana mu ƙara su.
Ban zana jadawali ba tsawon wata guda. Wannan tabbas alama ce mai kyau: yana nufin yawancin "buƙatun" an riga an gane su. Ya faru cewa a cikin mako zan zana wasu sabon jadawali akalla sau ɗaya a rana.
Sakamakon sakamako yana da mahimmanci saboda yanzu masu haɓakawa ba safai suke zuwa wurin admins tare da tambayoyi "inda za a kalli wani nau'in awo."
Aiwatarwa Sabis Mesh yana kusa da kusurwa kuma ya kamata ya sa rayuwa ta fi sauƙi ga kowa da kowa, abokan aiki daga Kayan aiki sun riga sun kusa aiwatar da "Istio na mutum mai lafiya" m: yanayin rayuwar kowane buƙatun HTTP (s) zai kasance a bayyane a cikin saka idanu, kuma shi koyaushe zai yiwu a fahimci "a wane mataki komai ya karye" yayin hulɗar tsakanin sabis (kuma ba kawai) ba. Biyan kuɗi zuwa labarai daga cibiyar DomClick. =)
Kubernetes goyon bayan kayayyakin more rayuwa
A tarihi, muna amfani da sigar patched Kubespray - Matsayi mai yiwuwa don turawa, haɓakawa da sabunta Kubernetes. A wani lokaci, an yanke goyon baya ga abubuwan da ba kubeadm ba daga babban reshe, kuma ba a ba da shawarar tsarin canzawa zuwa kubeadm ba. A sakamakon haka, kamfanin Southbridge ya yi nasa cokali mai yatsa (tare da goyon bayan kubeadm da sauri don magance matsaloli masu mahimmanci).
Tsarin sabunta duk gungu na k8s yayi kama da haka:
- Dauka Kubespray daga Southbridge, duba da zaren mu, Merjim.
- Muna fitar da sabuntawa zuwa danniya- "Kube".
- Muna fitar da sabuntawar kulli ɗaya a lokaci guda (a cikin Mai yiwuwa wannan shine "serial: 1") a ciki Dev- "Kube".
- Muna sabuntawa Prod ranar Asabar da maraice guda daya.
Akwai shirye-shiryen maye gurbinsa a nan gaba Kubespray ga wani abu da sauri kuma ku tafi kubeadm.
A cikin duka muna da "Cubes" guda uku: damuwa, Dev da Prod. Muna shirin kaddamar da wani (zafi jiran aiki) Prod-"Cube" a cikin na biyu data cibiyar. danniya и Dev rayuwa a cikin "injuna na gani" (oVirt don Damuwa da girgije na VMWare don Dev). Prod- "Cube" yana rayuwa akan "ƙarfe mara ƙarfi": waɗannan nau'ikan nodes iri ɗaya ne tare da zaren CPU 32, 64-128 GB na ƙwaƙwalwar ajiya da 300 GB SSD RAID 10 - duka 50 daga cikinsu. An sadaukar da nodes na “sihiri” guda uku ga “masu iyalai” Prod- "Cuba": 16 GB na ƙwaƙwalwar ajiya, 12 CPU zaren.
Don tallace-tallace, mun fi son yin amfani da "ƙarfe mara ƙarfi" kuma mu guje wa yadudduka marasa mahimmanci kamar OpenStack: ba ma buƙatar "maƙwabta masu hayaniya" da CPU sata lokaci. Kuma wahalar gudanarwa kusan ninki biyu a yanayin OpenStack na cikin gida.
Don CI / CD "Cubic" da sauran abubuwan abubuwan more rayuwa muna amfani da uwar garken GIT daban, Helm 3 (ya kasance sauyi mai raɗaɗi daga Helm 2, amma muna farin ciki da zaɓuɓɓukan. kwayar zarra), Jenkins, Mai yiwuwa kuma Docker. Muna son rassan fasali da turawa zuwa wurare daban-daban daga wurin ajiya guda.
ƙarshe
Wannan shine, a cikin sharuddan gabaɗaya, menene tsarin DevOps yayi kama da DomClick daga hangen injiniyan ayyuka. Labarin ya zama ƙasa da fasaha fiye da yadda nake tsammani: sabili da haka, bi labarai na DomClick akan Habré: za a sami ƙarin labaran "hardcore" game da Kubernetes da ƙari.
source: www.habr.com