Ukwenza iqonga le-kubernetes kwi-Pinterest

Ukutyhubela iminyaka, abasebenzisi be-Pinterest abazizigidi ezingama-300 benze ngaphezulu kwe-200 yeebhiliyoni zezikhonkwane kwiibhodi ezingaphezulu kwe-4 yeebhiliyoni. Ukusebenzela lo mkhosi wabasebenzisi kunye nesiseko esikhulu somxholo, i-portal iphuhlise amawaka eenkonzo, ukusuka kwii-microservices ezinokuphathwa zii-CPU ezimbalwa, ukuya kwii-monoliths ezinkulu ezisebenza kuzo zonke iinqanawa zoomatshini. Kwaye kwafika ixesha apho amehlo enkampani awela kwii-k8s. Kutheni le nto i-"cube" ibonakala ilungile kwi-Pinterest? Uya kufunda malunga noku kwinguqulelo yethu yenqaku lamva nje ukusuka blog Pinterest engeneering.

Ke, amakhulu ezigidi zabasebenzisi kunye namakhulu eebhiliyoni zeepini. Ukusebenzela lo mkhosi wabasebenzisi kunye nesiseko esikhulu somxholo, siye saphuhlisa amawaka eenkonzo, ukusuka kwii-microservices ezinokuphathwa zii-CPU ezimbalwa, ukuya kwii-monoliths ezinkulu ezisebenza kuzo zonke iinqwelo zoomatshini. Ukongeza, sineendlela ezahlukeneyo zesakhelo ezinokufuna kwakhona i-CPU, imemori, okanye ukufikelela kwi-I/O.

Ukugcina lo myezo wezilwanyana wezixhobo, iqela lophuhliso lijongene nemingeni emininzi:

• Akukho ndlela ifanayo yokuba iinjineli ziqhube imeko yemveliso. Iinkonzo ezingenammiselo, iinkonzo ezisemthethweni kunye neeprojekthi eziphantsi kophuhliso olusebenzayo zisekwe kwinqwaba yetekhnoloji eyahluke ngokupheleleyo. Oku kukhokelele ekudalweni kwekhosi yoqeqesho yeenjineli, kwaye kuwenza ube nzima kakhulu umsebenzi weqela lethu leziseko zophuhliso.
• Abaphuhlisi abaneenqwelo zabo zoomatshini ababonakalayo benza umthwalo omkhulu kubalawuli bangaphakathi. Ngenxa yoko, imisebenzi elula njengokuhlaziya i-OS okanye i-AMI ithatha iiveki kunye neenyanga. Oku kukhokelela ekwandeni komthwalo womsebenzi kwiimeko ezibonakala ngathi zemihla ngemihla.
• Ubunzima ekudaleni izixhobo zolawulo lweziseko ezingundoqo zehlabathi phezu kwezisombululo ezikhoyo. Le meko inzima ngakumbi kukuba ukufumana abanini beematshini ezibonakalayo akukho lula. Oko kukuthi, asazi ukuba lo mthamo unokukhutshwa ngokukhuselekileyo ukuze usebenze kwezinye iindawo zeziseko zophuhliso zethu.

Iinkqubo zokubethelela iikhonteyina ziyindlela yokumanyanisa ulawulo lomthwalo womsebenzi. Bavula umnyango wokwandisa isantya sophuhliso kunye nokwenza lula ulawulo lweziseko zophuhliso, kuba zonke izixhobo ezibandakanyekayo kwiprojekthi zilawulwa yinkqubo enye.

Umzobo 1: Izinto eziphambili kwiziseko zophuhliso (ukuthembeka, imveliso yomphuhlisi, kunye nokusebenza kakuhle).

Iqela le-Cloud Management Platform e-Pinterest lifumene ii-K8s ngo-2017. Ngesiqingatha sokuqala se-2017, sasibhale uninzi lwezakhono zethu zokuvelisa, kubandakanya i-API kunye nazo zonke iiseva zethu zewebhu. Emva koko, senze uvavanyo olucokisekileyo lweenkqubo ezahlukeneyo zokulungisa izisombululo zeekhonteyina, ukwakha amaqela kunye nokusebenza nazo. Ngasekupheleni kuka-2017, sagqiba ekubeni sisebenzise iKubernetes. Yayibhetyebhetye kwaye ixhaswa ngokubanzi kuluntu lwabaphuhlisi.

Ukuza kuthi ga ngoku, sizakhele ezethu izixhobo ze-boot yeqela esekwe kwi-Kops kwaye safudusa amacandelo akhoyo akhoyo afana nothungelwano, ukhuseleko, iimetrics, ukuloga, ulawulo lwesazisi, kunye netrafikhi ukuya eKubernetes. Siphinde sasebenzisa inkqubo yokulinganisa umthwalo womsebenzi kwisixhobo sethu, ubunzima obufihliweyo kubaphuhlisi. Ngoku sigxile ekuqinisekiseni ukuzinza kweqela, ukukala kunye nokudibanisa abathengi abatsha.

Kubernetes: Indlela yePinterest

Ukuqalisa ngeKubernetes kwisikali sePinterest njengeqonga iinjineli zethu ezinokuthi zilithande zize nemingeni emininzi.

Njengenkampani enkulu, sityale imali eninzi kwizixhobo zeziseko ezingundoqo. Imizekelo ibandakanya izixhobo zokhuseleko eziphethe ukusetyenzwa kwesatifikethi kunye nokuhanjiswa okungundoqo, amacandelo olawulo lwendlela, iinkqubo zokufunyanwa kwenkonzo, amacandelo okubonakala, kunye necandelo lokuthunyelwa kwelogi kunye neemetrics. Konke oku kwaqokelelwa ngesizathu: siye sahamba ngendlela eqhelekileyo yovavanyo kunye nephutha, kwaye ngoko sifuna ukudibanisa zonke ezi zixhobo kwiziseko ezitsha kwi-Kubernetes endaweni yokubuyisela ivili elidala kwiqonga elitsha. Le ndlela yokwenza ngokubanzi yenze lula ukufuduka, kuba yonke inkxaso yesicelo sele ikhona kwaye ayifuni ukudalwa ukusuka ekuqaleni.

Ngakolunye uhlangothi, iimodeli zokubikezela umthwalo kwi-Kubernetes ngokwayo (ezifana nokuthunyelwa, imisebenzi, kunye neeseti ze-Daemon) ayanele kwiprojekthi yethu. Le miba yokusebenziseka yimiqobo emikhulu yokufudukela eKubernetes. Umzekelo, sive abaphuhlisi benkonzo bekhalaza ngokungabikho okanye imimiselo yokungena engalunganga. Siphinde sadibana nokusetyenziswa okungalunganga kweenjini zetemplate, xa amakhulu eekopi enziwe ngokucaciswa okufanayo kunye nomsebenzi, obangele iingxaki zokulungiswa kwephupha.

Kwakunzima kakhulu ukugcina iinguqulelo ezahlukeneyo kwiqela elinye. Khawucinge ngobunzima benkxaso yabathengi ukuba ufuna ukusebenza ngaxeshanye kwiinguqulelo ezininzi zemo yendalo efanayo, kunye nazo zonke iingxaki zabo, iibhugi kunye nohlaziyo.

IiPropati zoMsebenzisi wePinterest kunye nabalawuli

Ukwenza kube lula kwiinjineli zethu ukuphumeza i-Kubernetes, kunye nokwenza lula kunye nokukhawulezisa iziseko zethu zeziseko ezingundoqo, siye saphuhlisa ezethu iinkcazelo zemithombo yesiko (CRDs).

Ii-CRD zibonelela ngokusebenza kulandelayo:

1. Ukudibanisa izixhobo ezahlukeneyo zeKubernetes zomthonyama ukuze zisebenze njengomthwalo omnye womsebenzi. Umzekelo, isixhobo sePinterestService sibandakanya ukuthunyelwa, inkonzo yokungena, kunye nemephu yoqwalaselo. Oku kuvumela abaphuhlisi bangabinaxhala malunga nokucwangcisa iDNS.
2. Sebenzisa inkxaso yesicelo esiyimfuneko. Umsebenzisi kufuneka agxininise kuphela kwinkcazo yesikhongozeli ngokwengqiqo yeshishini labo, ngelixa umlawuli weCRD ephumeza zonke izikhongozeli ze-init eziyimfuneko, izinto eziguquguqukayo zokusingqongileyo kunye neenkcukacha zepod. Oku kubonelela ngenqanaba elahlukileyo lokuthuthuzela abaphuhlisi.
3. Abalawuli beCRD bakwalawula umjikelo wobomi bemithombo yendalo kunye nokuphucula ukufumaneka kolungiso. Oku kubandakanya ukulungelelanisa izinto ezifunwayo nezichanekileyo, ukuhlaziya ubume beCRD kunye nokugcina iilogi zeminyhadala, nokunye. Ngaphandle kwe-CRD, abaphuhlisi baya kunyanzeleka ukuba balawule izixhobo ezininzi, nto leyo eya kwandisa kuphela ukubakho kwempazamo.

Nanku umzekelo wePinterestService kunye nesixhobo sangaphakathi esilawulwa ngumlawuli wethu:

Njengoko ubona ngasentla, ukuxhasa isikhongozeli sesiqhelo kufuneka sidibanise isikhongozeli se-init kunye nezongezo ezininzi ukubonelela ngokhuseleko, ukubonakala, kunye nokugcwala kwenethiwekhi. Ukongeza, senze iitemplates zemephu yoqwalaselo kwaye siphumeze inkxaso yeetemplates zePVC zemisebenzi yebhetshi, kunye nokulandelwa kwezinto ezininzi eziguquguqukayo zokusingqongileyo ukulandelela ubuni, ukusetyenziswa kwezixhobo, kunye nokuqokelela inkunkuma.

Kunzima ukucinga ukuba abaphuhlisi banokufuna ukubhala ezi fayile zoqwalaselo ngesandla ngaphandle kwenkxaso yeCRD, singasathethi ke ngokugcina nokucoca uqwalaselo.

Ukuhamba komsebenzi wokusasazwa kwesicelo

Lo mfanekiso ungasentla ubonisa indlela yokufaka isincedisi sePinterest kwiqela leKubernetes:

1. Abaphuhlisi banxibelelana neqela lethu leKubernetes ngeCLI kunye nojongano lomsebenzisi.
2. Izixhobo ze-CLI/UI zibuyisela uqwalaselo lokuhamba komsebenzi iifayile ze-YAML kunye nezinye iipropathi zokwakha (i-ID yoguqulelo olufanayo) kwi-Artifactory kwaye emva koko uzingenise kwiNkonzo yokuThunyelwa kweMisebenzi. Eli nyathelo liqinisekisa ukuba iinguqulelo zemveliso kuphela zihanjiswa kwiqela.
3. I-JSS lisango lamaqonga ahlukeneyo, kuquka iKubernetes. Apha umsebenzisi uqinisekisiwe, izabelo zikhutshiwe kwaye uqwalaselo lweCRD yethu lujongiwe ngokuyinxenye.
4. Emva kokujonga i-CRD kwicala le-JSS, ulwazi luthunyelwa kwi-k8s platform API.
5. Umlawuli wethu weCRD ubeka iliso kwimisitho kuzo zonke izixhobo zabasebenzisi. Iguqula ii-CRs zibe yimithombo ye-k8s yendalo, yongeza iimodyuli eziyimfuneko, imisela iimeko eziguquguqukayo ezifanelekileyo, kwaye yenze omnye umsebenzi wenkxaso ukuqinisekisa ukuba izicelo zomsebenzisi ezinezikhongozeli zinenkxaso eyaneleyo yeziseko ezingundoqo.
6. Umlawuli we-CRD ke udlulisela idatha efunyenweyo kwi-Kubernetes API ukwenzela ukuba iqhutywe ngumcwangcisi kwaye ifakwe kwimveliso.

Qaphela:: Oku kuhamba komsebenzi okhutshwe kwangaphambili kokusasazwa kwenzelwe abasebenzisi bokuqala beqonga elitsha le-k8s. Sithetha nje sikwinkqubo yokucokisa le nkqubo ukuze idityaniswe ngokupheleleyo neCI/CD yethu entsha. Oku kuthetha ukuba asinakukuxelela yonke into enxulumene neKubernetes. Sijonge phambili ekwabelaneni ngamava ethu kunye nenkqubela yeqela kweli cala kwibhlog yethu elandelayo, “Ukwakha iqonga le-CI/CD lePinterest.”

Iindidi zezibonelelo ezikhethekileyo

Ngokusekwe kwiimfuno ezithile zePinterest, siphuhlise ezi CRD zilandelayo ukuze zilungele ukuhamba komsebenzi okwahlukeneyo:

• I-PinterestService ziinkonzo ezingenammiselo eziye zaqhuba ixesha elide. Uninzi lweenkqubo zethu ezingundoqo zisekelwe kwiseti yeenkonzo ezinjalo.
• PinterestJobSet imifuziselo yomjikelo ogcweleyo webhetshi imisebenzi. Imeko eqhelekileyo kwiPinterest kukuba imisebenzi emininzi iqhuba izikhongozeli ezifanayo ngokuhambelanayo, kungakhathaliseki ukuba zeziphi ezinye iinkqubo ezifanayo.
• I-PinterestCronJob isetyenziswa ngokubanzi ngokubambisana nemithwalo emincinci yexesha. Lo ngumqulu womsebenzi we-cron wendabuko kunye neendlela zenkxaso zePinterest ezijongene nokhuseleko, i-traffic, iilogi kunye neemetrics.
• I-PinterestDaemon ibandakanya iiDaemons zeziseko zophuhliso. Olu sapho lusaqhubeka lukhula njengoko songeza inkxaso engakumbi kumaqela ethu.
• I-PinterestTrainingJob inabela kwi-Tensorflow kunye neenkqubo zePytorch, ibonelela ngenqanaba elifanayo lenkxaso yexesha lokubaleka njengazo zonke ezinye ii-CRD. Ekubeni i-Pinterest isebenzisa i-Tensorflow kunye nezinye iinkqubo zokufunda koomatshini, sinesizathu sokwakha i-CRD eyahlukileyo ngeenxa zonke kubo.

Sikwasebenza kwi-PinterestStatefulSet, eza kuthi kungekudala ilungelelaniswe kwindawo yokugcina idatha kunye nezinye iinkqubo ezisemthethweni.

Inkxaso yexesha lokusebenza

Xa i-pod yesicelo isebenza kwi-Kubernetes, ifumana ngokuzenzekelayo isatifikethi sokuzichaza. Esi satifikethi sisetyenziselwa ukufikelela kwindawo yokugcina eyimfihlo okanye ukunxibelelana nezinye iinkonzo nge-mTLS. Ngeli xesha, i-Container Init Configurator kunye ne-Daemon iyakukhuphela zonke izinto ezifunekayo ezixhomekeke phambi kokuqhuba isicelo esifakwe kwisingxobo. Xa yonke into sele ilungile, i-traffic sidecar kunye ne-Daemon ziya kubhalisa idilesi ye-IP yemodyuli kunye ne-Zookeeper yethu ukuze abathengi bayibhaqe. Konke oku kuya kusebenza kuba imodyuli yenethiwekhi yaqwalaselwe phambi kokuba isicelo siqaliswe.

Le ingentla yimizekelo eqhelekileyo yenkxaso yexesha lokuqhutywa komsebenzi. Ezinye iintlobo zemithwalo yomsebenzi zinokufuna inkxaso eyahlukileyo kancinane, kodwa zonke ziza ngendlela yeepod-level sidecars, i-node-level okanye i-virtual machine-level Daemons. Siqinisekisa ukuba konke oku kusetyenziswe ngaphakathi kweziseko zolawulo kwaye kuyahambelana kuzo zonke izicelo, ekugqibeleni kunciphisa kakhulu umthwalo ngokubhekiselele kumsebenzi wobugcisa kunye nenkxaso yabathengi.

Uvavanyo kunye ne-QA

Sakhe umbhobho wovavanyo wokuphela ukuya ekupheleni phezu kwesiseko esikhoyo sovavanyo lweKubernetes. Olu vavanyo lusebenza kuwo onke amaqela ethu. Umbhobho wethu udlule kuhlaziyo oluninzi ngaphambi kokuba ube yinxalenye yeqela lemveliso.

Ukongeza kwiinkqubo zokuvavanya, sineenkqubo zokubeka iliso kunye nezilumkiso ezihlala zibeka esweni imeko yamacandelo enkqubo, ukusetyenziswa kobutyebi kunye nezinye izibonakaliso ezibalulekileyo, ezisazisa kuphela xa kufuneka ungenelelo lomntu.

Iindlela ezizezinye

Siye sajonga ezinye iindlela kwizibonelelo zesiko, ezifana nabalawuli bokufikelela kwinguqu kunye neenkqubo zetemplate. Nangona kunjalo, zonke ziza nemingeni ebalulekileyo yokusebenza, ngoko ke sikhethe indlela yeCRD.

Umlawuli wokwamkelwa oguquguqukayo wasetyenziselwa ukwazisa ii-sidecars, ukuguquguquka kokusingqongileyo, kunye nenye inkxaso yexesha lokusebenza. Nangona kunjalo, ijongene neengxaki ezahlukeneyo, ezifana nokubophelela izibonelelo kunye nokulawulwa kwe-lifecycle, apho iingxaki ezinjalo zingaveli kwi-CRD.

Qaphela: Iinkqubo zetemplate ezifana neetshathi zeHelm zikwasetyenziswa ngokubanzi ukuqhuba izicelo ezinolungelelwaniso olufanayo. Nangona kunjalo, izicelo zethu zomsebenzi zahluke kakhulu ukuba zingalawulwa kusetyenziswa iitemplates. Kwakhona ngexesha lokuthunyelwa ngokuqhubekayo kuya kubakho iimpazamo ezininzi xa usebenzisa iitemplates.

Umsebenzi ozayo

Ngoku sijongene nomthwalo oxubeneyo kuwo onke amaqela ethu. Ukuxhasa ezo nkqubo zeentlobo ezahlukeneyo kunye nobukhulu, sisebenza kwezi ndawo zilandelayo:

• Ingqokelela yamaqela isasaza izicelo ezinkulu kuwo wonke amaqela ahlukeneyo ukwenzela ukulinganisa kunye nokuzinza.
• Ukuqinisekisa ukuzinza kweqela, i-scalability kunye nokubonakala ukudala uqhagamshelwano lwesicelo kunye nee-SLA.
• Ukulawula izixhobo kunye nezabelo ukwenzela ukuba izicelo zingangqubani omnye nomnye, kwaye isikali seqela silawulwa kwicala lethu.
• Iqonga elitsha le-CI/CD lokuxhasa kunye nokuthunyelwa kwezicelo kwi-Kubernetes.

umthombo: www.habr.com