Qaphela. transl.: Ukutholwa kuka-Kubernetes kwa-GitLab kuthathwa njengenye yezinto ezimbili eziyinhloko ezinomthelela ekukhuleni kwenkampani. Kodwa-ke, kuze kube muva nje, ingqalasizinda yensizakalo ye-inthanethi ye-GitLab.com yakhiwe emishinini ebonakalayo, futhi cishe unyaka odlule ukuthuthela kwayo kuma-K8 kwaqala, okungakaqedwa namanje. Siyajabula ukwethula ukuhunyushwa kwesihloko sakamuva sikanjiniyela we-GitLab SRE mayelana nokuthi lokhu kwenzeka kanjani nokuthi onjiniyela ababamba iqhaza kuphrojekthi benza iziphetho zini.
Isikhathi esingangonyaka manje, uphiko lwethu lwengqalasizinda luthutha zonke izinsiza ezisebenza ku-GitLab.com ziye ku-Kubernetes. Ngalesi sikhathi, sihlangabezane nezinselele ezingahlobene nje kuphela nokuthutha izinsiza ziye e-Kubernetes, kodwa futhi nokuphatha ukuthunyelwa okuxubile ngesikhathi soshintsho. Izifundo ezibalulekile esizitholile kuzoxoxwa ngazo kulesi sihloko.
Kusukela ekuqaleni kwe-GitLab.com, amaseva ayo agijima efwini emishinini ebonakalayo. Le mishini ebonakalayo iphethwe nguChef futhi ifakwe kusetshenziswa eyethu . uma ngabe uhlelo lokusebenza ludinga ukubuyekezwa, luhlanganisa nokumane ubuyekeze imikhumbi yeseva ngendlela ehlelekile, elandelanayo kusetshenziswa ipayipi le-CI. Le ndlela - nakuba ihamba kancane futhi kancane - iqinisekisa ukuthi i-GitLab.com isebenzisa ukufakwa nokumisa okufanayo njengabasebenzisi abangaxhunyiwe ku-inthanethi (uyazilawula) Ukufakwa kwe-GitLab kusetshenziswa amaphakheji ethu e-Linux kulokhu.
Sisebenzisa le ndlela ngoba kubaluleke kakhulu ukuzwa lonke usizi nenjabulo amalungu avamile omphakathi aba nayo lapho efaka futhi elungiselela amakhophi awo e-GitLab. Le ndlela yasebenza kahle isikhathi esithile, kodwa lapho inani lamaphrojekthi ku-GitLab lidlula izigidi eziyishumi, sabona ukuthi alisahlangabezani nezidingo zethu zokukala nokusatshalaliswa.
Izinyathelo zokuqala eziya ku-Kubernetes kanye ne-cloud-native GitLab
Iphrojekthi yasungulwa ngo-2017 ukulungiselela i-GitLab ukuthunyelwa kwamafu, nokwenza abasebenzisi bakwazi ukufaka i-GitLab kumaqoqo e-Kubernetes. Sasazi ukuthi ukuhambisa i-GitLab iye ku-Kubernetes kuzokhuphula izinga leplathifomu ye-SaaS, kube lula ukuthunyelwa, futhi kuthuthukise ukusebenza kahle kwezinsiza zekhompuyutha. Ngasikhathi sinye, imisebenzi eminingi yohlelo lwethu lokusebenza incike kuma-partitions e-NFS akhweziwe, abambezela ukushintshwa kwemishini ebonakalayo.
Ukusunduza okubheke emvelweni wamafu kanye ne-Kubernetes kuvumele onjiniyela bethu ukuthi bahlele inguquko kancane kancane, lapho siye sashiya okunye ukuncika kohlelo lokusebenza kusitoreji senethiwekhi ngenkathi siqhubeka nokuthuthukisa izici ezintsha. Selokhu saqala ukuhlela ukufuduka ehlobo lika-2019, eminingi yale mikhawulo isixazululiwe, futhi inqubo yokuthuthela i-GitLab.com iye e-Kubernetes manje isiqhubeka kahle!
Izici ze-GitLab.com ku-Kubernetes
Ku-GitLab.com, sisebenzisa iqoqo le-GKE lesifunda elilodwa eliphatha yonke ithrafikhi yohlelo lokusebenza. Ukunciphisa inkimbinkimbi yokufuduka (okuvele kukhohlisayo), sigxila kumasevisi angathembele kusitoreji sasendaweni noma i-NFS. I-GitLab.com isebenzisa ikakhulukazi i-monolithic Rails codebase, futhi sihambisa ithrafikhi ngokusekelwe ezicini zomthwalo womsebenzi ezindaweni ezihlukene ezibekwe zodwa kuma-node pool azo.
Endabeni ye-frontend, lezi zinhlobo zihlukaniswe ngezicelo kuwebhu, i-API, i-Git SSH/HTTPS kanye ne-Registry. Endabeni ye-backend, sihlukanisa imisebenzi emgqeni ngokuya ngezici ezihlukahlukene kuye ngokuthi , okusivumela ukuthi sihlele Izinjongo Zezinga Lesevisi (ama-SLO) emithwalweni ehlukahlukene yomsebenzi.
Zonke lezi zinsizakalo ze-GitLab.com zilungiswa kusetshenziswa ishadi le-GitLab Helm elingashintshiwe. Ukucushwa kwenziwa ngamashadi amancane, anganikwa amandla ngokukhetha njengoba sithutha kancane kancane amasevisi siye kuqoqo. Noma sinqume ukungafaki ezinye zezinsizakalo zethu ezisezingeni eliphezulu ekufudukeni, njengeRedis, Postgres, GitLab Pages kanye ne-Gitaly, ukusebenzisa i-Kubernetes kusivumela ukuthi sehlise kakhulu inani lama-VM aphethwe nguChef njengamanje.
Ukubonakala Nokuphathwa Kokucushwa kwe-Kubernetes
Zonke izilungiselelo ziphethwe yi-GitLab ngokwayo. Kulokhu, amaphrojekthi amathathu wokumisa asuselwa ku-Terraform ne-Helm asetshenziswa. Sizama ukusebenzisa i-GitLab ngokwayo noma nini uma kungenzeka ukuze siqhube i-GitLab, kodwa emisebenzini yokusebenza sinokufaka okuhlukile kwe-GitLab. Lokhu kuyadingeka ukuze uqinisekise ukuthi awuncikile ekutholakaleni kwe-GitLab.com lapho wenza ukuthunyelwa kwe-GitLab.com nezibuyekezo.
Nakuba amapayipi ethu eqoqo le-Kubernetes esebenza ekufakweni okuhlukile kwe-GitLab, kunezibuko zamakhosombe ekhodi ezitholakala esidlangalaleni kulawa makheli alandelayo:
- - Uhlaka lokucushwa kwe-GitLab.com lweshadi le-GitLab Helm;
- - Iqukethe ukulungiselelwa kwezinsizakalo ezingahlobene ngokuqondile nohlelo lokusebenza lwe-GitLab. Lokhu kufaka phakathi ukulungiselelwa kokugawulwa kwemithi nokuqapha kweqoqo, kanye namathuluzi adidiyelwe njenge-PlantUML;
- - Ukucushwa kwe-Terraform ye-Kubernetes nengqalasizinda ye-VM yefa. Lapha ulungiselela zonke izinsiza ezidingekayo ukuze usebenzise iqoqo, okuhlanganisa iqoqo ngokwalo, amachibi ama-node, ama-akhawunti wesevisi, nokubhukha ikheli le-IP.
Ukubuka komphakathi kuboniswa uma kwenziwa izinguquko. ngesixhumanisi sokuhluka okuningiliziwe i-SRE ehlaziyayo ngaphambi kokwenza izinguquko kuqoqo.
Ku-SRE, isixhumanisi siholela ekwehlukeni okuningiliziwe ekufakweni kwe-GitLab, esetshenziselwa ukukhiqiza nokufinyelela okukhawulelwe. Lokhu kuvumela abasebenzi nomphakathi, ngaphandle kokufinyelela kuphrojekthi yokusebenza (evulekele kuphela ama-SRE), ukubuka izinguquko ezihlongozwayo zokumisa. Ngokuhlanganisa isibonelo se-GitLab yomphakathi sekhodi nesenzakalo esiyimfihlo samapayipi e-CI, sigcina ukugeleza komsebenzi okukodwa ngenkathi siqinisekisa ukuzimela ku-GitLab.com ukuze uthole izibuyekezo zokucushwa.
Esikutholile ngesikhathi sokufuduka
Ngesikhathi sokuthutha, kutholwe umuzwa wokuthi sisebenza ekufudukeni okusha nasekusetshenzisweni e-Kubernetes.
1. Ukwenyuka kwezindleko ngenxa yethrafikhi phakathi kwezindawo ezitholakalayo
Izibalo zansuku zonke zokuphuma (amabhayithi ngosuku) zemikhumbi yenqolobane ye-Git ku-GitLab.com
I-Google ihlukanisa inethiwekhi yayo ngezifunda. Lawo-ke, ahlukaniswe ngezindawo zokufinyeleleka (AZ). Ukusingathwa kwe-Git kuhlotshaniswa nenani elikhulu ledatha, ngakho-ke kubalulekile kithi ukulawula ukuphuma kwenethiwekhi. Kuthrafikhi yangaphakathi, ukuphuma kumahhala kuphela uma kuhlala ngaphakathi kwendawo yokutholakala efanayo. Ngalokhu kubhalwa, sisebenzisa cishe i-100 TB yedatha ngosuku lokusebenza olujwayelekile (futhi lokho okwezinqolobane ze-Git). Izinsizakalo ezazihlala emishinini efanayo ye-virtual topology yethu endala esekwe ku-VM manje sezisebenza kumaphodi e-Kubernetes ahlukene. Lokhu kusho ukuthi ithrafikhi ethile eyayisendaweni ngaphambilini eya ku-VM ingase ihambe ngaphandle kwezindawo ezitholakalayo.
Amaqoqo e-GKE esifunda akuvumela ukuthi uvule Izindawo Ezitholakalayo eziningi ukuze uphelelwe amandla. Sicabangela okungenzeka ngezinsizakalo ezikhiqiza umthamo omkhulu wethrafikhi. Lokhu kuzonciphisa izindleko zokuphuma ngenkathi kugcinwa ukunganakwa kweleveli yeqoqo.
2. Imikhawulo, izicelo zezinsiza kanye nokukala
Inombolo ye-replicas ecubungula ithrafikhi yokukhiqiza ku-registry.gitlab.com. Inani eliphakeme lethrafikhi ngo-~15:00 UTC.
Indaba yethu yokufuduka yaqala ngo-Agasti 2019, lapho sithutha isevisi yethu yokuqala, i-GitLab Container Registry, siya e-Kubernetes. Le nsizakalo ebaluleke kakhulu, enethrafikhi ephezulu bekuyisinqumo esihle ekufudukeni kokuqala ngoba iwuhlelo lokusebenza olungenasimo olunokuncika okumbalwa kwangaphandle. Inkinga yokuqala esihlangabezane nayo kwakuyinani elikhulu lama-pods akhishiwe ngenxa yokuntula inkumbulo kuma-node. Ngenxa yalokhu, kwadingeka sishintshe izicelo nemingcele.
Kutholwe ukuthi esimweni sohlelo lokusebenza lapho ukusetshenziswa kwememori kukhula ngokuhamba kwesikhathi, amanani aphansi ezicelo (ukugcina inkumbulo ye-pod ngayinye) kuhlanganiswe nomkhawulo oqinile "ovulekile" wokusetshenziswa kuholela ekugcwalisweni. (saturation) izindawo zokuhlala kanye nezinga eliphezulu lokuxoshwa. Ukubhekana nale nkinga, kwaba . Lokhu kususe ingcindezi kuma-node futhi kwaqinisekisa ukuthi ama-pods ane-lifecycle engazange ifake ingcindezi enkulu ku-node. Manje siqala ukufuduka ngesicelo esivulekile (futhi esicishe sifane) futhi sikhawule amanani, siwalungise njengoba kudingeka.
3. Amamethrikhi namalogi
Uphiko lwengqalasizinda lugxile ekubambezelekeni, amazinga amaphutha kanye nokugcwaliswa kwesikhala ngokufakiwe (I-SLO) ixhunywe ku .
Onyakeni odlule, esinye sezehlakalo ezibalulekile ophikweni lwengqalasizinda kube yintuthuko ekuqapheni nasekusebenzeni nama-SLO. Ama-SLO asivumele ukuthi sizibekele imigomo yezinsizakalo ezingazodwana esiziqaphe ngokucophelela ngesikhathi sokufuduka. Kodwa ngisho nalokhu kubonakala okuthuthukisiwe, akwenzeki ngaso sonke isikhathi ukubona izinkinga ngokushesha usebenzisa amamethrikhi nezixwayiso. Isibonelo, ngokugxila ekulinganiseni ukubambezeleka kanye namaphutha, asibambi ngokugcwele zonke izimo zokusetshenziswa kwesevisi eyenziwa ngokufuduka.
Lolu daba lwatholwa cishe ngokushesha ngemva kokuthutha eminye imithwalo yemisebenzi iye kuqoqo. Kube kubi kakhulu lapho kufanele sihlole imisebenzi lapho inani lezicelo lalincane, kodwa eyayinokuncika kokucushwa okuqondile. Esinye sezifundo ezibalulekile ekufudukeni kwaba isidingo sokungabheki amamethrikhi kuphela lapho uqapha, kodwa nezingodo kanye “nomsila omude”. (lokhu kumayelana eshadini - cishe. transl.) amaphutha. Manje ekufudukeni ngakunye sifaka uhlu oluningiliziwe lwemibuzo yelogi (imibuzo yelogi) futhi uhlele izinqubo ezicacile zokuhlehlisa ezingadluliswa ukusuka kwesinye ishifu kuye kwesilandelayo uma kuphakama izinkinga.
Ukunikeza izicelo ezifanayo ngokuhambisana nengqalasizinda ye-VM endala nengqalasizinda entsha esekwe ku-Kubernetes yethule inselele eyingqayizivele. Ngokungafani nokuhamba kwe-lift-and-shift (ukudluliselwa ngokushesha kwezicelo “njengoba zinjalo” kwingqalasizinda entsha; imininingwane eyengeziwe ingafundwa, isibonelo, - cishe. transl.), umsebenzi ofanayo kuma-VM “amadala” kanye ne-Kubernetes udinga ukuthi amathuluzi okuqapha ahambisane nazo zombili izindawo futhi akwazi ukuhlanganisa amamethrikhi ekubukeni okukodwa. Kubalulekile ukuthi sisebenzise amadeshibhodi afanayo kanye nemibuzo yokungena ukuze sifinyelele ukubonakala okungaguquki ngesikhathi soshintsho.
4. Ukushintsha ithrafikhi ibe yiqoqo elisha
Ku-GitLab.com, ingxenye yamaseva inikezelwe . I-Canary Park inikeza amaphrojekthi ethu angaphakathi futhi ingasebenza . Kodwa yakhelwe ngokuyinhloko ukuhlola izinguquko ezenziwe kwingqalasizinda kanye nokusetshenziswa. Isevisi yokuqala ethuthiwe iqale ngokwamukela inani elilinganiselwe lethrafikhi yangaphakathi, futhi siyaqhubeka sisebenzisa le ndlela ukuze siqinisekise ukuthi ama-SLO ayahlangatshezwana ngaphambi kokuthumela yonke ithrafikhi kuqoqo.
Endabeni yokufuduka, lokhu kusho ukuthi izicelo zamaphrojekthi angaphakathi zithunyelwa ku-Kubernetes kuqala, bese kancane kancane sishintsha yonke ithrafikhi siye kuqoqo ngokushintsha isisindo se-backend nge-HAProxy. Phakathi nokufuduka kusuka ku-VM kuya e-Kubernetes, kwacaca ukuthi kwakuzuzisa kakhulu ukuba nendlela elula yokuqondisa kabusha ithrafikhi phakathi kwengqalasizinda endala nentsha futhi, ngokufanele, ukugcina ingqalasizinda endala ilungele ukubuyiselwa emuva ezinsukwini ezimbalwa zokuqala ngemva kokufuduka.
5. Gcina amandla ama-pods kanye nokusetshenziswa kwawo
Cishe ngokushesha inkinga elandelayo yabonakala: ama-pods ensizakalo Yokubhalisa aqala ngokushesha, kodwa ukwethulwa kwe-pods ye-Sidekiq kwathatha isikhathi eside. . Isikhathi eside sokuqalisa sama-pods e-Sidekiq saba yinkinga lapho siqala ukuthuthela imithwalo yemisebenzi e-Kubernetes yabasebenzi ababedinga ukucubungula imisebenzi ngokushesha futhi bakhule ngokushesha.
Kulesi simo, isifundo sasiwukuthi nakuba i-Kubernetes 'Horizontal Pod Autoscaler (HPA) iphatha kahle ukukhula kwethrafikhi, kubalulekile ukucabangela izici zemithwalo yomsebenzi futhi unikeze amandla ayisipele kuma-pods (ikakhulukazi lapho isidingo sisakazwa ngokungalingani). Kithina, kube nokwanda okungazelelwe kwemisebenzi, okuholele ekukhulisweni okusheshayo, okuholele ekugcwalisweni kwezinsiza ze-CPU ngaphambi kokuba sibe nesikhathi sokukala i-node pool.
Kuhlala kunesilingo sokuminyanisa okuningi ngangokunokwenzeka kuqoqo, nokho, njengoba siqale sahlangabezana nezinkinga zokusebenza, manje sesiqala ngesabelomali esivulekile se-pod futhi siyasinciphisa ngokuhamba kwesikhathi, sibhekisisa ama-SLO. Ukwethula ama-pods esevisi ye-Sidekiq kukhule kakhulu futhi manje kuthatha cishe imizuzwana engama-40 ngokwesilinganiso. iwine kokubili i-GitLab.com kanye nabasebenzisi bethu bokufakwa okuzilawulayo abasebenza neshadi elisemthethweni le-GitLab Helm.
isiphetho
Ngemva kokuthutha isevisi ngayinye, sijabulele izinzuzo zokusebenzisa i-Kubernetes ekukhiqizeni: ukuthunyelwa kwesicelo esisheshayo nesiphephile, ukukala, kanye nokwabiwa kwensiza okusebenza kahle kakhulu. Ngaphezu kwalokho, izinzuzo zokufuduka zidlula isevisi ye-GitLab.com. Konke ukuthuthukiswa kweshadi le-Helm elisemthethweni kuzuzisa abasebenzisi balo.
Ngethemba ukuthi uyijabulele indaba ye-Kubernetes migration adventures. Siyaqhubeka nokuthuthela zonke izinsiza ezintsha kuqoqo. Ulwazi olwengeziwe lungatholakala ezincwadini ezilandelayo:
- «";
- «";
- .
I-PS evela kumhumushi
Funda futhi kubhulogi yethu:
- «";
- «";
- «";
- «".
Source: www.habr.com