Kwazaka zambiri, ogwiritsa ntchito 300 miliyoni a Pinterest apanga mapini opitilira 200 biliyoni pama board opitilira 4 biliyoni. Kuti mutumikire gulu la ogwiritsira ntchito komanso malo ambiri okhutira, portal yapanga mautumiki masauzande ambiri, kuyambira ma microservices omwe amatha kuthandizidwa ndi ma CPU ochepa, mpaka ma monoliths akuluakulu omwe amayendetsa pagulu lonse la makina enieni. Ndiyeno nthawi inafika pamene maso a kampaniyo anagwera pa k8s. Chifukwa chiyani "cube" idawoneka bwino pa Pinterest? Muphunzira za izi kuchokera kumasulira kwathu nkhani yaposachedwa kuchokera .
Kotero, mazana a mamiliyoni a ogwiritsa ntchito ndi mazana a mabiliyoni a pini. Kuti titumikire gulu la anthu ogwiritsira ntchito komanso malo ambiri okhutira, tapanga mautumiki masauzande ambiri, kuyambira ma microservices omwe amatha kuthandizidwa ndi ma CPU ochepa, mpaka ma monoliths akuluakulu omwe amathamanga pamagulu onse a makina enieni. Kuphatikiza apo, tili ndi machitidwe osiyanasiyana omwe angafunikenso mwayi wa CPU, kukumbukira, kapena I/O.
Posamalira zoo ya zida izi, gulu lachitukuko likukumana ndi zovuta zingapo:
- Palibe njira yofananira yomwe mainjiniya amayendetsera malo opanga. Ntchito zopanda malire, ntchito zodziwika bwino komanso mapulojekiti omwe akutukuka kwambiri amachokera pamitundu yosiyanasiyana yaukadaulo. Izi zidapangitsa kuti pakhale maphunziro athunthu a mainjiniya, komanso kusokoneza kwambiri ntchito ya gulu lathu la zomangamanga.
- Madivelopa okhala ndi gulu lawo la makina pafupifupi amapanga cholemetsa chachikulu kwa oyang'anira mkati. Zotsatira zake, ntchito zosavuta monga kukonzanso OS kapena AMI zimatenga milungu ndi miyezi. Izi zimabweretsa kuchuluka kwa ntchito zomwe zimawoneka ngati zatsiku ndi tsiku.
- Zovuta pakupanga zida zoyendetsera ntchito padziko lonse lapansi pamwamba pa mayankho omwe alipo. Zinthu zimakhala zovuta kwambiri chifukwa kupeza eni makina enieni sikophweka. Ndiko kuti, sitikudziwa ngati mphamvuyi ingathe kuchotsedwa kuti igwire ntchito m'madera ena a zomangamanga.
Machitidwe a Container orchestration ndi njira yolumikizira kasamalidwe ka ntchito. Amatsegula chitseko chowonjezera liwiro lachitukuko ndikuchepetsa kasamalidwe ka zomangamanga, popeza zonse zomwe zikukhudzidwa ndi ntchitoyi zimayendetsedwa ndi dongosolo limodzi lapakati.
Chithunzi 1: Zofunikira pazachitukuko (kudalirika, zokolola zamapulogalamu, komanso kuchita bwino).
Gulu la Cloud Management Platform ku Pinterest linapeza ma K8 mu 2017. Pofika theka loyamba la 2017, tinali titalemba zambiri zomwe timapanga, kuphatikizapo API ndi ma seva athu onse. Pambuyo pake, tidawunikiranso bwino machitidwe osiyanasiyana opangira mayankho a chidebe, kumanga magulu ndikugwira nawo ntchito. Kumapeto kwa 2017, tinaganiza zogwiritsa ntchito Kubernetes. Zinali zosinthika komanso zothandizidwa kwambiri m'magulu opanga mapulogalamu.
Mpaka pano, tapanga zida zathu zoyambira ma cluster potengera Kops ndikusamutsa zida zomwe zidalipo kale monga ma network, chitetezo, ma metric, kudula mitengo, kasamalidwe ka identity, ndi traffic kupita ku Kubernetes. Tinakhazikitsanso dongosolo lachitsanzo cha ntchito yathu, zovuta zomwe zimabisika kwa opanga. Tsopano tikuyang'ana kwambiri kuonetsetsa kukhazikika kwa masango, kukulitsa ndi kulumikiza makasitomala atsopano.
Kubernetes: Njira ya Pinterest
Kuyamba ndi Kubernetes pamlingo wa Pinterest ngati nsanja yomwe mainjiniya athu angakonde idabwera ndi zovuta zambiri.
Monga kampani yayikulu, tayika ndalama zambiri pazida zogwirira ntchito. Zitsanzo zikuphatikizapo zida zachitetezo zomwe zimagwira ntchito yokonza satifiketi ndi kugawa makiyi, zida zowongolera magalimoto, makina odziwira ntchito, zowonekera, ndi zida zotumizira zolemba ndi ma metrics. Zonsezi zinasonkhanitsidwa pazifukwa: tinadutsa njira yachibadwa yoyesera ndi zolakwika, choncho tinkafuna kuphatikiza zipangizo zonsezi muzomangamanga zatsopano za Kubernetes m'malo mobwezeretsanso gudumu lakale pa nsanja yatsopano. Njirayi idapangitsa kusamukako kukhala kosavuta, chifukwa zonse zothandizira zilipo kale ndipo sizifunika kupangidwa kuyambira pachiyambi.
Kumbali ina, zolosera zamtundu wa Kubernetes palokha (monga kutumizidwa, ntchito, ndi ma seti a Daemon) sizokwanira pantchito yathu. Nkhani zogwiritsiridwa ntchito izi ndizolepheretsa zazikulu kusamukira ku Kubernetes. Mwachitsanzo, tamvapo okonza mautumiki akudandaula chifukwa chosowa kapena makonda olowera molakwika. Tidakumananso ndi kugwiritsa ntchito molakwika kwa injini zama template, pomwe mazana a makope adapangidwa ndi mawonekedwe ndi ntchito yofananira, zomwe zidadzetsa zovuta zowongolera zoopsa.
Zinalinso zovuta kwambiri kusunga Mabaibulo osiyanasiyana pagulu limodzi. Tangoganizirani zovuta za chithandizo chamakasitomala ngati mukufuna kugwira ntchito nthawi imodzi mumitundu ingapo yamalo othamanga omwewo, ndi mavuto awo onse, nsikidzi ndi zosintha.
Pinterest User Properties ndi Controllers
Kuti zikhale zosavuta kuti mainjiniya athu agwiritse ntchito Kubernetes, komanso kufewetsa ndikufulumizitsa zomangamanga zathu, tapanga matanthauzidwe athu azinthu (CRDs).
Ma CRD amapereka magwiridwe antchito awa:
- Kuphatikiza zida zosiyanasiyana zaku Kubernetes kuti azigwira ntchito ngati gawo limodzi. Mwachitsanzo, chida cha PinterestService chimaphatikizapo kutumiza, ntchito yolowera, ndi mapu okonzekera. Izi zimathandiza opanga kuti asadandaule za kukhazikitsa DNS.
- Gwiritsani ntchito chithandizo chofunikira. Wogwiritsa ntchito akuyenera kuyang'ana pazotengera zomwe ali nazo malinga ndi momwe amagwirira ntchito, pomwe woyang'anira CRD amagwiritsa ntchito zotengera zonse zofunika za init, zosintha zachilengedwe ndi ma pod. Izi zimapereka chitonthozo chosiyana kwenikweni kwa opanga.
- Owongolera a CRD amayang'aniranso moyo wazinthu zachilengedwe ndikuwongolera kupezeka kwa zovuta. Izi zikuphatikiza kuyanjanitsa zomwe mukufuna komanso zenizeni, kukonzanso mawonekedwe a CRD ndikusunga zipika za zochitika, ndi zina zambiri. Popanda CRD, opanga amakakamizidwa kuyang'anira zinthu zingapo, zomwe zingangowonjezera mwayi wolakwitsa.
Nachi chitsanzo cha PinterestService ndi chida chamkati chomwe chimayendetsedwa ndi woyang'anira wathu:
Monga mukuwonera pamwambapa, kuti tithandizire chidebe chokhazikika tiyenera kuphatikizira chidebe cha init ndi zowonjezera zingapo kuti tipereke chitetezo, mawonekedwe, ndi kuchuluka kwa maukonde. Kuphatikiza apo, tidapanga masanjidwe am'mapu ndikukhazikitsa chithandizo cha ma tempuleti a PVC a ntchito zamagulu, komanso kutsata mitundu ingapo ya chilengedwe kuti tiwunikire zomwe zimadziwika, kugwiritsa ntchito zinthu, komanso kutolera zinyalala.
Ndizovuta kulingalira kuti otukula angafune kulemba mafayilo osinthikawa ndi dzanja popanda thandizo la CRD, osasiyanso kusunga ndikuwongolera masanjidwewo.
Njira yoyendetsera ntchito
Chithunzi pamwambapa chikuwonetsa momwe mungatumizire gwero la Pinterest ku gulu la Kubernetes:
- Madivelopa amalumikizana ndi gulu lathu la Kubernetes kudzera pa CLI ndi mawonekedwe ogwiritsa ntchito.
- Zida za CLI/UI zimatenganso mafayilo a YAML osintha kachitidwe kantchito ndi zinthu zina zomanga (ID ya mtundu womwewo) kuchokera ku Artifactory ndikuzipereka ku Ntchito Yotumiza Ntchito. Gawoli limawonetsetsa kuti matembenuzidwe opangidwa okha amaperekedwa kumagulu.
- JSS ndi chipata cha nsanja zosiyanasiyana, kuphatikiza Kubernetes. Apa wogwiritsa ntchito amatsimikiziridwa, magawo amaperekedwa ndipo kasinthidwe ka CRD yathu kumawunikiridwa pang'ono.
- Pambuyo poyang'ana CRD kumbali ya JSS, chidziwitsocho chimatumizidwa ku k8s platform API.
- Wowongolera wathu wa CRD amayang'anira zochitika pazogwiritsa ntchito zonse. Imatembenuza ma CR kukhala zida zamtundu wa k8s, imawonjezera ma module ofunikira, imayika zosintha zoyenera, ndikuchita ntchito zina zothandizira kuwonetsetsa kuti mapulogalamu omwe ali ndi zida ali ndi chithandizo chokwanira cha zomangamanga.
- Woyang'anira CRD ndiye amapatsira zomwe adalandira ku Kubernetes API kotero kuti zitha kukonzedwa ndi wokonza mapulani ndikuyika kupanga.
ndemanga: Ntchito yotulutsidwa isanatulutsidweyi idapangidwa kwa ogwiritsa ntchito oyamba a nsanja yatsopano ya k8s. Panopa tikukonza ndondomekoyi kuti igwirizane ndi CI/CD yathu yatsopano. Izi zikutanthauza kuti sitingathe kukuuzani zonse zokhudzana ndi Kubernetes. Tikuyembekezera kugawana zomwe takumana nazo komanso momwe gulu likuyendera mbali iyi mu positi yathu yotsatira yabulogu, "Kumanga nsanja ya CI/CD ya Pinterest."
Mitundu yazinthu zapadera
Kutengera zosowa zenizeni za Pinterest, tapanga ma CRD otsatirawa kuti agwirizane ndi mayendedwe osiyanasiyana:
- PinterestService ndi ntchito zopanda malire zomwe zakhala zikuyenda kwa nthawi yayitali. Zambiri mwazinthu zathu zazikulu zimachokera kumagulu otere.
- PinterestJobSet mitundu yonse yozungulira batch jobs. Chochitika chodziwika bwino pa Pinterest ndikuti ntchito zingapo zimayendetsa zotengera zomwezo mofanana, mosasamala kanthu za njira zina zofananira.
- PinterestCronJob imagwiritsidwa ntchito kwambiri molumikizana ndi katundu wochepa wanthawi ndi nthawi. Ichi ndi chomangira chantchito yamba ya cron yokhala ndi njira zothandizira za Pinterest zomwe zimayang'anira chitetezo, magalimoto, zipika ndi ma metric.
- PinterestDaemon imaphatikizapo ma Daemons a zomangamanga. Banja ili likukulirakulirabe pamene tikuwonjezera thandizo kumagulu athu.
- PinterestTrainingJob imafikira ku njira za Tensorflow ndi Pytorch, zomwe zimapereka chithandizo chofanana cha nthawi yothamanga monga ma CRD ena onse. Popeza Pinterest imagwiritsa ntchito Tensorflow mwachangu ndi makina ena ophunzirira makina, tinali ndi chifukwa chopangira CRD yosiyana mozungulira iwo.
Tikugwiranso ntchito pa PinterestStatefulSet, yomwe posachedwapa idzasinthidwa kukhala malo osungiramo deta ndi machitidwe ena ovomerezeka.
Thandizo la nthawi yothamanga
Pulogalamu ya pulogalamu ikayamba Kubernetes, imangolandira satifiketi yodzizindikiritsa yokha. Satifiketiyi imagwiritsidwa ntchito kupeza malo osungira mwachinsinsi kapena kulumikizana ndi ntchito zina kudzera mTLS. Pakadali pano, Container Init Configurator ndi Daemon azitsitsa zonse zofunika asanagwiritse ntchito pulogalamuyo. Zonse zikakonzeka, galimoto yapamsewu ndi Daemon idzalembetsa adilesi ya IP ya module ndi Zookeeper yathu kuti makasitomala adziwe. Zonsezi zigwira ntchito chifukwa gawo la netiweki lidakhazikitsidwa pulogalamuyo isanayambike.
Zomwe zili pamwambazi ndi zitsanzo za chithandizo cha nthawi yothamanga pa ntchito. Mitundu ina yantchito ingafunike thandizo losiyana pang'ono, koma zonse zimabwera m'mawonekedwe a ma pod-level sidecars, node-level kapena ma Daemons amtundu wa makina. Timaonetsetsa kuti zonsezi zikugwiritsidwa ntchito mkati mwa kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe ka kayendetsedwe kabwino kake
Mayeso ndi QA
Tidapanga payipi yoyeserera yomaliza pamwamba pa zida zoyeserera za Kubernetes. Mayeserowa amagwira ntchito m'magulu athu onse. Mapaipi athu adasinthidwanso zambiri asanakhale gawo la gulu lazinthu.
Kuphatikiza pa machitidwe oyesera, tili ndi machitidwe owonetsetsa ndi ochenjeza omwe nthawi zonse amayang'anitsitsa momwe zigawo za dongosolo, kugwiritsa ntchito zipangizo ndi zizindikiro zina zofunika, zimatidziwitsa pokhapokha ngati kulowererapo kwa anthu kuli kofunikira.
Njira zina
Tidayang'ana njira zina zomwe zingasinthidwe ndi zinthu zomwe mwamakonda, monga ma mutation access controller ndi ma template system. Komabe, onse amabwera ndi zovuta zogwirira ntchito, chifukwa chake tidasankha njira ya CRD.
Wowongolera ovomerezeka adagwiritsidwa ntchito poyambitsa ma sidecars, zosintha zachilengedwe, ndi chithandizo china chanthawi yothamanga. Komabe, idakumana ndi zovuta zosiyanasiyana, monga kumangiriza zida ndi kasamalidwe ka moyo, pomwe zovuta zotere sizimatuluka mu CRD.
Taonani: Machitidwe a ma template monga ma chart a Helm amagwiritsidwanso ntchito kwambiri kuyendetsa mapulogalamu ndi masinthidwe ofanana. Komabe, ntchito zathu zogwirira ntchito ndizosiyana kwambiri kuti zisamayendetsedwe pogwiritsa ntchito ma template. Komanso pakutumiza mosalekeza padzakhala zolakwika zambiri mukamagwiritsa ntchito ma templates.
Ntchito yomwe ikubwera
Panopa tikulimbana ndi katundu wosakanikirana m'magulu athu onse. Kuthandizira njira zotere zamitundu ndi makulidwe osiyanasiyana, timagwira ntchito m'magawo awa:
- Gulu lamagulu limagawa ntchito zazikulu m'magulu osiyanasiyana kuti scalability ndi bata.
- Kuwonetsetsa kukhazikika kwamagulu, scalability ndi kuwoneka kuti apange kulumikizana kwa mapulogalamu ndi ma SLA.
- Kuwongolera zothandizira ndi ma quotas kuti mapulogalamu asasemphane, ndipo kukula kwa gulu kumayendetsedwa ndi ife.
- Pulatifomu yatsopano ya CI/CD yothandizira ndi kutumiza mapulogalamu pa Kubernetes.
Source: www.habr.com