Ngokuhamba kweminyaka, abasebenzisi be-Pinterest abayizigidi ezingama-300 badale izikhonkwane ezingaphezu kwezigidigidi ezingama-200 kumabhodi angaphezu kwezigidigidi ezi-4. Ukusebenzela leli butho labasebenzisi kanye nesisekelo sokuqukethwe esikhulu, ingosi ithuthukise izinkulungwane zezinsizakalo, kusukela kuma-microservices angaphathwa ama-CPU ambalwa, kuya kuma-monoliths amakhulu asebenza kuwo wonke uxhaxha lwemishini ebonakalayo. Kwabe sekufika isikhathi lapho amehlo enkampani ewela kuma-k8s. Kungani i-"cube" ibukeka kahle ku-Pinterest? Uzofunda ngalokhu ekuhumusheni kwethu isihloko sakamuva esivela ku .
ΠΡΠ°ΠΊ, ΡΠΎΡΠ½ΠΈ ΠΌΠΈΠ»Π»ΠΈΠΎΠ½ΠΎΠ² ΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°ΡΠ΅Π»Π΅ΠΉ ΠΈ ΡΠΎΡΠ½ΠΈ ΠΌΠΈΠ»Π»ΠΈΠ°ΡΠ΄ΠΎΠ² ΠΏΠΈΠ½ΠΎΠ². Π§ΡΠΎΠ±Ρ ΠΎΠ±ΡΠ»ΡΠΆΠΈΠ²Π°ΡΡ ΡΡΡ Π°ΡΠΌΠΈΡ ΠΏΠΎΠ»ΡΠ·ΠΎΠ²Π°ΡΠ΅Π»Π΅ΠΉ ΠΈ ΠΎΠ±ΡΠΈΡΠ½ΡΡ ΠΊΠΎΠ½ΡΠ΅Π½Ρ-Π±Π°Π·Ρ, ΠΌΡ ΡΠ°Π·ΡΠ°Π±ΠΎΡΠ°Π»ΠΈ ΡΡΡΡΡΠΈ ΡΠ΅ΡΠ²ΠΈΡΠΎΠ², Π½Π°ΡΠΈΠ½Π°Ρ ΠΎΡ ΠΌΠΈΠΊΡΠΎΡΠ΅ΡΠ²ΠΈΡΠΎΠ², Ρ ΠΊΠΎΡΠΎΡΡΠΌΠΈ ΠΌΠΎΠΆΠ΅Ρ ΡΠΏΡΠ°Π²ΠΈΡΡΡΡ Π½Π΅ΡΠΊΠΎΠ»ΡΠΊΠΎ CPU, ΠΈ Π·Π°ΠΊΠ°Π½ΡΠΈΠ²Π°Ρ Π³ΠΈΠ³Π°Π½ΡΡΠΊΠΈΠΌΠΈ ΠΌΠΎΠ½ΠΎΠ»ΠΈΡΠ°ΠΌΠΈ, ΠΊΠΎΡΠΎΡΡΠ΅ ΠΊΡΡΡΡΡΡΡ Π½Π° ΡΠ΅Π»ΠΎΠΌ ΠΏΠ°ΡΠΊΠ΅ Π²ΠΈΡΡΡΠ°Π»ΡΠ½ΡΡ
ΠΌΠ°ΡΠΈΠ½. ΠΡΠΎΠΌΠ΅ ΡΠΎΠ³ΠΎ, Ρ Π½Π°Ρ Π΅ΡΡΡ ΡΠ°Π·Π½ΠΎΠΎΠ±ΡΠ°Π·Π½ΡΠ΅ ΡΡΠ΅ΠΉΠΌΠ²ΠΎΡΠΊΠΈ, ΠΊΠΎΡΠΎΡΡΠ΅ ΡΠ°ΠΊΠΆΠ΅ ΠΌΠΎΠ³ΡΡ ΡΡΠ΅Π±ΠΎΠ²Π°ΡΡ ΡΠ΅ΡΡΡΡΠΎΠ² CPU, ΠΏΠ°ΠΌΡΡΠΈ ΠΈΠ»ΠΈ Π΄ΠΎΡΡΡΠΏΠ° ΠΊ ΠΎΠΏΠ΅ΡΠ°ΡΠΈΡΠΌ Π²Π²ΠΎΠ΄Π°-Π²ΡΠ²ΠΎΠ΄Π°.
Ekunakekeleni le zoo yamathuluzi, ithimba lokuthuthukisa libhekene nenani lezinselelo:
- Ayikho indlela efanayo yokuthi onjiniyela baqhube indawo yokukhiqiza. Izinsizakalo ezingenasisekelo, izinsizakalo ezisemthethweni kanye namaphrojekthi athuthukiswayo asekelwe ezitaki zobuchwepheshe ezihluke ngokuphelele. Lokhu kwaholela ekwakhiweni kwesifundo sonke sokuqeqesha onjiniyela, futhi futhi kwenza umsebenzi wethimba lethu lengqalasizinda ube nzima kakhulu.
- Onjiniyela abaneqoqo labo lemishini ebonakalayo badala umthwalo omkhulu kubaphathi bangaphakathi. Ngenxa yalokho, imisebenzi elula njengokuvuselela i-OS noma i-AMI kuthatha amasonto nezinyanga. Lokhu kuholela ekwandeni komthwalo womsebenzi ezimweni ezibonakala zingokoqobo nsuku zonke.
- Ubunzima bokudala amathuluzi okuphatha ingqalasizinda yomhlaba wonke phezu kwezixazululo ezikhona. Isimo siba nzima nakakhulu ukuthi ukuthola abanikazi bemishini ebonakalayo akulula. Okungukuthi, asazi ukuthi lo mthamo ungakhishwa ngokuphepha ukuze usebenze kwezinye izingxenye zengqalasizinda yethu.
Amasistimu wokucula ama-Container ayindlela yokuhlanganisa ukuphathwa komthwalo womsebenzi. Bavula ithuba lokwenyuka kwesivinini sentuthuko futhi benze lula ukuphathwa kwengqalasizinda, njengoba zonke izinsiza ezibandakanyekayo kulo msebenzi zilawulwa uhlelo olulodwa olulodwa.
Umfanekiso 1: Okubalulekile kwengqalasizinda (ukwethembeka, ukukhiqiza kukanjiniyela, nokusebenza kahle).
Ithimba le-Cloud Management Platform e-Pinterest lithole ama-K8 ngo-2017. Ngengxenye yokuqala ka-2017, sase sibhale amakhono ethu amaningi okukhiqiza, okuhlanganisa i-API nawo wonke amaseva ethu ewebhu. Ngemva kwalokho, senze ukuhlola okuphelele kwezinhlelo ezihlukahlukene zokuhlela izixazululo zeziqukathi, ukwakha amaqoqo nokusebenza nazo. Ngasekupheleni kuka-2017, sinqume ukusebenzisa i-Kubernetes. Bekuvumelana nezimo futhi kusekelwa kabanzi emphakathini wonjiniyela.
Kuze kube manje, sakhe amathuluzi ethu e-cluster boot asekelwe ku-Kops futhi sathutha izingxenye zengqalasizinda ezikhona ezifana nenethiwekhi, ukuphepha, amamethrikhi, ukuloga, ukuphathwa kobunikazi, kanye nethrafikhi eya e-Kubernetes. Siphinde sasebenzisa uhlelo lokumodela umthwalo womsebenzi wensiza yethu, inkimbinkimbi yakhona efihliwe konjiniyela. Manje sigxile ekuqinisekiseni ukuzinza kweqoqo, ukulilinganisa nokuxhuma amaklayenti amasha.
Kubernetes: Indlela ye-Pinterest
Ukuqalisa nge-Kubernetes esikalini se-Pinterest njengenkundla onjiniyela bethu abangayithanda kufike nezinselelo eziningi.
Njengenkampani enkulu, sitshale imali eningi kumathuluzi engqalasizinda. Izibonelo zifaka amathuluzi okuphepha aphatha ukucutshungulwa kwesitifiketi nokusatshalaliswa kokhiye, izingxenye zokulawula ithrafikhi, amasistimu okutholwa kwesevisi, izingxenye zokubonakala, kanye nezingxenye zokuthunyelwa kwelogi namamethrikhi. Konke lokhu kwaqoqwa ngesizathu: sidlule endleleni evamile yokuhlola namaphutha, ngakho-ke besifuna ukuhlanganisa zonke lezi zisetshenziswa nengqalasizinda entsha ku-Kubernetes esikhundleni sokuvuselela isondo elidala endaweni entsha. Sekukonke le ndlela yenza ukufuduka kwaba lula, njengoba konke ukusekelwa kohlelo lokusebenza sekukhona futhi akudingi ukudalwa kusukela ekuqaleni.
Ngakolunye uhlangothi, amamodeli wokubikezela umthwalo ku-Kubernetes ngokwayo (njengokuthunyelwa, imisebenzi, namasethi e-Daemon) awanele kuphrojekthi yethu. Lezi zinkinga zokusebenziseka ziyizithiyo ezinkulu zokuthuthela e-Kubernetes. Isibonelo, sizwe abathuthukisi besevisi bekhalaza mayelana nezilungiselelo zokungena ezilahlekile noma ezingalungile. Siphinde sabhekana nokusetshenziswa okungalungile kwezinjini zezifanekiso, lapho amakhulu amakhophi enziwa anencazelo efanayo nomsebenzi, okubangele izinkinga zokulungisa iphutha elibi.
Kwakunzima kakhulu futhi ukugcina izinguqulo ezihlukene kuqoqo elifanayo. Cabanga ngobunkimbinkimbi bokwesekwa kwamakhasimende uma udinga ukusebenza ngesikhathi esisodwa ezinguqulweni eziningi zendawo efanayo yesikhathi sokusebenza, nazo zonke izinkinga zazo, iziphazamisi nezibuyekezo.
Izakhiwo Zomsebenzisi we-Pinterest Nabalawuli
Ukwenza kube lula konjiniyela bethu ukusebenzisa i-Kubernetes, nokwenza lula nokusheshisa ingqalasizinda yethu, sizenzele ezethu izincazelo zensiza yangokwezifiso (ama-CRD).
Ama-CRD ahlinzeka ngokusebenza okulandelayo:
- Ukuhlanganisa izinsiza ezahlukene ze-Kubernetes zomdabu ukuze zisebenze njengomthwalo owodwa. Isibonelo, insiza ye-PinterestService ifaka phakathi ukusetshenziswa, isevisi yokungena ngemvume, nemephu yokulungiselela. Lokhu kuvumela abathuthukisi ukuthi bangakhathazeki ngokusetha i-DNS.
- Sebenzisa ukwesekwa kohlelo lokusebenza oludingekayo. Umsebenzisi udinga ukugxila kuphela ekucacisweni kwesiqukathi ngokuya ngengqondo yebhizinisi lakhe, kuyilapho isilawuli se-CRD sisebenzisa zonke iziqukathi ze-init ezidingekayo, okuguquguqukayo kwendawo kanye nokucaciswa kwe-pod. Lokhu kunikeza izinga lokunethezeka elihluke kakhulu konjiniyela.
- Izilawuli ze-CRD ziphinde zilawule umjikelezo wempilo wezisetshenziswa zomdabu futhi zithuthukise ukutholakala kokususa iphutha. Lokhu kuhlanganisa ukuvumelanisa izicaciso ezifunwayo nezingokoqobo, ukubuyekeza isimo se-CRD nokugcina amalogi omcimbi, nokuningi. Ngaphandle kwe-CRD, onjiniyela bazophoqeleka ukuthi baphathe izinsiza eziningi, okuzokwandisa kuphela amathuba ephutha.
Nasi isibonelo se-PinterestService nesisetshenziswa sangaphakathi esiphethwe isilawuli sethu:
Njengoba ungabona ngenhla, ukuze sisekele isiqukathi esingokwezifiso sidinga ukuhlanganisa isiqukathi se-init kanye nezengezo ezimbalwa ukuze sinikeze ukuvikeleka, ukubonakala, kanye nethrafikhi yenethiwekhi. Ngaphezu kwalokho, sidale izifanekiso zemephu yokumisa futhi safaka usekelo lwezifanekiso ze-PVC zemisebenzi yeqoqo, kanye nokulandelelwa kokuguquguquka kwemvelo okuningi ukuze kulandelelwe ubunikazi, ukusetshenziswa kwezinsiza, nokuqoqwa kukadoti.
Kunzima ukucabanga ukuthi abathuthukisi bangafuna ukubhala lawa mafayela okumisa ngesandla ngaphandle kokusekelwa kwe-CRD, ingasaphathwa eyokuqhubeka nokugcina nokulungisa amaphutha.
Ukugeleza komsebenzi wokuthunyelwa kwesicelo
Isithombe esingenhla sibonisa indlela yokusebenzisa insiza yangokwezifiso ye-Pinterest kuqoqo le-Kubernetes:
- Onjiniyela basebenzisana neqoqo lethu le-Kubernetes nge-CLI kanye nesixhumi esibonakalayo somsebenzisi.
- Amathuluzi e-CLI/UI athola ukucushwa kokugeleza komsebenzi amafayela e-YAML nezinye izakhiwo zokwakha (inguqulo ye-ID efanayo) ku-Artifactory abese ezithumela Kusevisi Yokuthumela Umsebenzi. Lesi sinyathelo siqinisekisa ukuthi izinguqulo zokukhiqiza kuphela ezilethwa kuqoqo.
- I-JSS iyisango lamapulatifomu ahlukahlukene, kufaka phakathi i-Kubernetes. Lapha umsebenzisi ugunyaziwe, ama-quota akhishwa futhi ukumiswa kwe-CRD yethu kuhlolwa kancane.
- Ngemva kokuhlola i-CRD ohlangothini lwe-JSS, ulwazi luthunyelwa ku-API yesikhulumi se-k8s.
- Isilawuli sethu se-CRD siqapha imicimbi kuzo zonke izinsiza zabasebenzisi. Iguqula ama-CRs iwenze izinsiza zomdabu ze-k8s, ingeze amamojula adingekayo, isethe okuguquguqukayo kwendawo efanele, futhi yenza omunye umsebenzi wokusekela ukuze kuqinisekiswe ukuthi izinhlelo zokusebenza zabasebenzisi ezifakwe esitsheni zinokusekelwa okwanele kwengqalasizinda.
- Isilawuli se-CRD sibe sesidlulisela idatha etholiwe ku-Kubernetes API ukuze ikwazi ukucutshungulwa umhleli futhi ifakwe ekukhiqizweni.
Ukubhala: Lokhu kugeleza komsebenzi okhishwe ngaphambilini kokuphakwa kwadalelwa abasebenzisi bokuqala benkundla entsha ye-k8s. Njengamanje sisemkhankasweni wokucwenga le nqubo ukuze ihlanganiswe ngokugcwele ne-CI/CD yethu entsha. Lokhu kusho ukuthi asikwazi ukukutshela yonke into ehlobene ne-Kubernetes. Sibheke ngabomvu ukwabelana ngolwazi lwethu kanye nenqubekelaphambili yeqembu kulokhu kubhulogi yethu elandelayo, βUkwakha inkundla ye-CI/CD ye-Pinterest.β
Izinhlobo zezinsiza ezikhethekile
Ngokusekelwe ezidingweni ezithile ze-Pinterest, sithuthukise ama-CRD alandelayo ukuze afanele ukuhamba komsebenzi okuhlukile:
- I-PinterestService yizinsizakalo ezingenasisekelo ezisebenze isikhathi eside. Iningi lamasistimu ethu abalulekile asekelwe kusethi yamasevisi anjalo.
- I-PinterestJobSet imodeli yemisebenzi yeqoqo yomjikelezo ogcwele. Isimo esivamile ku-Pinterest ukuthi imisebenzi eminingi isebenzisa iziqukathi ezifanayo ngokuhambisana, ngaphandle kokubheka ezinye izinqubo ezifanayo.
- I-PinterestCronJob isetshenziswa kabanzi ngokuhambisana nemithwalo emincane yezikhathi. Lesi isisonga somsebenzi we-cron womdabu onezindlela zokusekela ze-Pinterest ezinesibopho sokuphepha, ithrafikhi, amalogi namamethrikhi.
- I-PinterestDaemon ihlanganisa amaDaemons engqalasizinda. Lo mndeni uyaqhubeka nokukhula njengoba sengeza ukwesekwa okwengeziwe kumaqoqo ethu.
- I-PinterestTrainingJob inweba kuzinqubo ze-Tensorflow ne-Pytorch, ihlinzeka ngezinga elifanayo losekelo lwesikhathi sokusebenza njengawo wonke amanye ama-CRD. Njengoba i-Pinterest isebenzisa ngenkuthalo i-Tensorflow nezinye izinhlelo zokufunda zomshini, sibe nesizathu sokwakha i-CRD ehlukile kubo.
Siphinde sisebenze ku-PinterestStatefulSet, ezolungiselelwa maduze izindawo zokugcina idatha namanye amasistimu asezingeni eliphezulu.
Ukwesekwa kwesikhathi sokusebenza
Uma i-pod yohlelo lokusebenza isebenza ku-Kubernetes, ithola ngokuzenzakalelayo isitifiketi sokuzikhomba. Lesi sitifiketi sisetshenziselwa ukufinyelela isitoreji esiyimfihlo noma ukuxhumana namanye amasevisi nge-mTLS. Ngaleso sikhathi, i-Container Init Configurator kanye ne-Daemon bazolanda konke ukuncika okudingekayo ngaphambi kokusebenzisa uhlelo olufakwe esitsheni. Uma konke sekumi ngomumo, inqola yethrafikhi kanye ne-Daemon bazobhalisa ikheli le-IP lemojuli neZookeeper yethu ukuze amaklayenti akwazi ukuyithola. Konke lokhu kuzosebenza ngoba imojuli yenethiwekhi yalungiswa ngaphambi kokuthi uhlelo lokusebenza luqaliswe.
Okungenhla kuyizibonelo ezijwayelekile zosekelo lwesikhathi sokusebenza somthwalo womsebenzi. Ezinye izinhlobo zemithwalo yomsebenzi zingadinga ukwesekwa okuhluke kancane, kodwa zonke ziza ngendlela yama-sidecars anezinga le-pod, ama-Daemons asezingeni le-node noma ama-virtual level level. Siqinisekisa ukuthi konke lokhu kufakwa ngaphakathi kwengqalasizinda yokuphatha futhi kuyahambisana kuzo zonke izinhlelo zokusebenza, okugcina kwehlisa kakhulu umthwalo ngokomsebenzi wezobuchwepheshe kanye nokwesekwa kwamakhasimende.
Ukuhlola kanye ne-QA
Sakhe ipayipi lokuhlola eliya ekupheleni phezu kwengqalasizinda yokuhlola ekhona ye-Kubernetes. Lezi zivivinyo zisebenza kuwo wonke amaqoqo ethu. Ipayipi lethu lidlule ezibuyekezweni eziningi ngaphambi kokuthi libe yingxenye yeqoqo lomkhiqizo.
Ngokungeziwe kumasistimu okuhlola, sinezinhlelo zokuqapha nezokwazisa ezihlala ziqapha isimo sezingxenye zesistimu, ukusetshenziswa kwezinsiza nezinye izinkomba ezibalulekile, ezisazisa kuphela uma kudingeka ukungenelela komuntu.
Ezinye izindlela
Sibheke ezinye izindlela ezingasetshenziswa ezisetshenziswa ngokwezifiso, njengezilawuli zokufinyelela eziguquguqukayo namasistimu ezifanekiso. Nokho, zonke ziza nezinselelo ezinkulu zokusebenza, ngakho sikhethe umzila we-CRD.
Isilawuli sokwamukela esishintshayo sisetshenziswe ukwethula ama-sidecars, okuguquguqukayo kwendawo, nokunye usekelo lwesikhathi sokusebenza. Kodwa-ke, ibhekane nezinkinga ezahlukahlukene, njengokubophezela kwezinsiza kanye nokuphathwa komjikelezo wokuphila, lapho izinkinga ezinjalo zingaveli ku-CRD.
Qaphela: Amasistimu ezifanekiso afana namashadi e-Helm nawo asetshenziswa kabanzi ukuze kusetshenziswe izinhlelo zokusebenza ezinokucushwa okufanayo. Nokho, izinhlelo zethu zokusebenza zihluke kakhulu ukuthi zingaphathwa kusetshenziswa izifanekiso. Futhi ngesikhathi sokuthunyelwa okuqhubekayo kuzoba namaphutha amaningi kakhulu uma usebenzisa izifanekiso.
Umsebenzi ozayo
Njengamanje sibhekene nomthwalo oxubile kuwo wonke amaqoqo ethu. Ukuze sisekele izinqubo ezinjalo zezinhlobo nosayizi abahlukene, sisebenza ezindaweni ezilandelayo:
- Iqoqo lamaqoqo lisabalalisa izinhlelo zokusebenza ezinkulu kuwo wonke amaqoqo ahlukene ukuze kube nokuqina nokuzinza.
- Ukuqinisekisa ukuzinza kweqoqo, ukulinganisa nokubonakala ukuze udale uxhumano lohlelo lokusebenza kanye nama-SLA.
- Ukuphatha izinsiza nama-quota ukuze izinhlelo zokusebenza zingangqubuzani, futhi isikali seqoqo silawulwa thina.
- Inkundla entsha ye-CI/CD yokusekela nokuphakela izinhlelo zokusebenza ku-Kubernetes.
Source: www.habr.com