Tupperware: wakupha Kubernetes wa Facebook?

Kuwongolera koyenera komanso kodalirika kwamagulu pamlingo uliwonse ndi Tupperware

Lero pa Systems @Scale msonkhano tidayambitsa Tupperware, kasamalidwe ka magulu athu omwe amawongolera zotengera mamiliyoni a maseva omwe amayendetsa pafupifupi ntchito zathu zonse. Tidatumiza Tupperware koyamba mu 2011, ndipo kuyambira pamenepo zomangamanga zathu zakula kuchokera 1 data center kwathunthu 15 geo-distributed data centers. Nthawi yonseyi, Tupperware sanayime ndikukula nafe. Tikuwonetsani momwe Tupperware imaperekera kasamalidwe kamagulu oyambira, kuphatikiza thandizo lothandizira mautumiki apamwamba, gulu limodzi loyang'anira malo onse a data, komanso kuthekera kogawa mphamvu pakati pa mautumiki munthawi yeniyeni. Tigawananso maphunziro omwe taphunzira pamene zomangamanga zathu zikukula.

Tupperware imagwira ntchito zosiyanasiyana. Opanga mapulogalamu amazigwiritsa ntchito popereka ndi kuyang'anira mapulogalamu. Imayika ma code ogwiritsira ntchito ndi zodalira mu chithunzi ndikuzipereka ku maseva ngati zotengera. Zotengera zimapereka kudzipatula pakati pa mapulogalamu omwe ali pa seva yomweyo kuti opanga athe kuthana ndi malingaliro ogwiritsira ntchito ndipo asamade nkhawa kupeza ma seva kapena kukonza zosintha. Tupperware imayang'aniranso magwiridwe antchito a seva, ndipo ikapeza kulephera, imasamutsa zotengera kuchokera pa seva yovuta.

Akatswiri opanga luso amagwiritsa ntchito Tupperware kugawa mphamvu za seva kumagulu kutengera bajeti ndi zovuta. Amagwiritsanso ntchito kupititsa patsogolo kugwiritsa ntchito seva. Ogwiritsa ntchito ma data center amatembenukira ku Tupperware kuti agawire zotengera bwino m'malo opangira data ndikuyimitsa kapena kusuntha zotengera pakukonza. Chifukwa cha izi, kusunga ma seva, maukonde ndi zida kumafuna kulowererapo kochepa kwa anthu.

Tupperware zomangamanga

Zomangamanga za Tupperware PRN ndi chimodzi mwa zigawo za malo athu a deta. Derali lili ndi nyumba zingapo za data center (PRN1 ndi PRN2) zomwe zili pafupi. Tikukonzekera kupanga gulu limodzi lowongolera lomwe lidzayang'anira ma seva onse m'chigawo chimodzi.

Opanga mapulogalamu amapereka ntchito ngati ntchito za Tupperware. Ntchito imakhala ndi zotengera zingapo, ndipo zonse zimakhala ndi code yofanana yogwiritsira ntchito.

Tupperware ili ndi udindo wopereka zotengera ndikuwongolera moyo wawo. Zili ndi zigawo zingapo:

• Tupperware frontend imapereka ma API a mawonekedwe ogwiritsira ntchito, CLI, ndi zida zina zodzipangira zomwe mungathe kuyanjana ndi Tupperware. Amabisa zonse zamkati kuchokera kwa eni ntchito a Tupperware.
• Tupperware Scheduler ndi gulu lowongolera lomwe limayang'anira chidebe ndi moyo wantchito. Imayikidwa kumadera ndi padziko lonse lapansi, pomwe wokonza chigawo amayang'anira ma seva m'chigawo chimodzi ndipo wokonza dziko lonse amayang'anira ma seva ochokera kumadera osiyanasiyana. Wopangayo amagawidwa kukhala shards, ndipo shard iliyonse imayang'anira ntchito zingapo.
• Tupperware's Scheduler Proxy amabisa sharding mkati ndikupereka galasi limodzi losavuta kwa ogwiritsa ntchito a Tupperware.
• Tupperware allocator amagawira zotengera ku maseva. Wokonza mapulani amayendetsa kuyimitsa, kuyambitsa, kukonzanso, ndi kulephera kwa zotengera. Pakadali pano, ogawa m'modzi amatha kuyang'anira dera lonse popanda kugawanika kukhala shards. (Dziwani kusiyana kwa mawu. Mwachitsanzo, ndandanda mu Tupperware imagwirizana ndi gulu lowongolera mu Kubernetes, ndipo Tupperware allocator amatchedwa scheduler ku Kubernetes.)
• Wothandizira zothandizira amasunga gwero la chowonadi cha seva ndi zochitika zautumiki. Timayendetsa gwero limodzi lazinthu pa data iliyonse, ndipo imasunga zonse zokhudzana ndi maseva omwe ali pa data center. Wothandizira zida ndi dongosolo loyang'anira mphamvu, kapena njira yoperekera zida, amasankha mwamphamvu kuti ndi ndani amene amayendetsa seva. Ntchito yoyang'anira zaumoyo imayang'anira ma seva ndikusunga zambiri zokhudzana ndi thanzi lawo muzothandizira. Ngati seva ili ndi zovuta kapena ikufunika kukonzedwa, wothandizirayo amauza woperekayo ndi wokonza mapulani kuti ayimitse zotengerazo kapena kuzisunthira ku maseva ena.
• Tupperware Agent ndi daemon yomwe ikuyenda pa seva iliyonse yomwe imayang'anira kupereka ndi kuchotsa zotengera. Mapulogalamu amayendera mkati mwa chidebe, zomwe zimawapangitsa kudzipatula komanso kuberekana. Yambani Msonkhano wa chaka chatha wa Systems @Scale Tafotokoza kale momwe zotengera za Tupperware zimapangidwira pogwiritsa ntchito zithunzi, btrfs, cgroupv2 ndi systemd.

Zodziwika bwino za Tupperware

Tupperware ndi yofanana m'njira zambiri ndi machitidwe ena owongolera magulu monga Kubernetes ndi Miyezi, koma palinso zosiyana:

• Thandizo lopangidwira la mautumiki apamwamba.
• Gulu limodzi lowongolera ma seva omwe ali m'malo osiyanasiyana a data kuti azitha kutumiza zotengera kutengera cholinga, kuchotsedwa kwamagulu ndi kukonza.
• Chotsani kugawikana kwa gulu lowongolera kuti muwonjezere.
• Elastic computing imakupatsani mwayi wogawa mphamvu pakati pa mautumiki munthawi yeniyeni.

Tapanga zinthu zabwinozi kuti zithandizire ntchito zosiyanasiyana zopanda malire komanso zodziwika bwino pagulu lalikulu lapadziko lonse lapansi logawana nawo.

Thandizo lopangidwira la mautumiki apamwamba.

Tupperware imagwira ntchito zingapo zofunika kwambiri zomwe zimasunga zomwe zimasungidwa pa Facebook, Instagram, Messenger ndi WhatsApp. Izi zitha kukhala masitolo akuluakulu amagulu amtengo wapatali (mwachitsanzo. ZippyDB) ndi kuyang'anira nkhokwe za deta (mwachitsanzo, ODS Gorilla и Scuba). Kusunga mautumiki ovomerezeka sikophweka, chifukwa dongosololi liyenera kuonetsetsa kuti zotengerazo zimatha kupirira kusokonezeka kwakukulu, kuphatikizapo kuzimitsa kwa ma netiweki kapena kuzimitsa kwa magetsi. Ndipo ngakhale njira wamba, monga kugawa zotengera m'malo olakwika, zimagwira bwino ntchito zopanda malire, mautumiki apamwamba amafunikira thandizo lina.

Mwachitsanzo, ngati kulephera kwa seva kumapangitsa kuti chofanana ndi database chisapezeke, kodi muzitha kukonza zokha zomwe zingasinthe ma cores pa ma seva 50 kuchokera pagulu la 10? Zimatengera momwe zinthu ziliri. Ngati imodzi mwa ma seva 50wa ili ndi chithunzi china cha database yomweyi, ndi bwino kudikirira osataya 2 replicas nthawi imodzi. Kuti tipange zisankho zokhuza kukonza ndi kagwiritsidwe ntchito ka makina, tifunika zambiri zokhuza kubwereza kwa data mkati ndi malingaliro a kakhazikitsidwe kantchito iliyonse yodziwika bwino.

Mawonekedwe a TaskControl amalola mautumiki apamwamba kuti akhudze zisankho zomwe zimakhudza kupezeka kwa data. Pogwiritsa ntchito mawonekedwewa, wokonza mapulani amadziwitsa zakunja za ntchito za chidebe (kuyambiranso, kusintha, kusamuka, kukonza). Ntchito yodziwika bwino imagwiritsa ntchito woyang'anira yemwe amauza Tupperware ngati kuli kotetezeka kuchita ntchito iliyonse, ndipo izi zitha kusinthidwa kapena kuchedwetsedwa kwakanthawi. Muchitsanzo chomwe chili pamwambapa, woyang'anira database akhoza kuuza Tupperware kuti asinthe 49 mwa ma seva 50, koma asiye seva yeniyeni (X) yokha pakalipano. Zotsatira zake, ngati nthawi yosinthira kernel ikadutsa ndipo nkhokwe ikulephera kubwezeretsanso zovuta, Tupperware idzasinthanso seva ya X.

Ntchito zambiri zodziwika bwino ku Tupperware zimagwiritsa ntchito TaskControl osati mwachindunji, koma kudzera pa ShardManager, nsanja wamba yopangira ntchito zabwino pa Facebook. Ndi Tupperware, Madivelopa atha kufotokoza zolinga zawo ndendende momwe zotengera ziyenera kugawidwira m'malo opangira ma data. Ndi ShardManager, Madivelopa amafotokoza cholinga chawo cha momwe ma data amayenera kugawidwira m'mitsuko. ShardManager akudziwa za kuyika kwa deta ndi kubwereza kwa mapulogalamu ake ndipo amalankhulana ndi Tupperware kudzera mu TaskControl mawonekedwe kuti akonze zogwirira ntchito popanda kugwiritsa ntchito mwachindunji. Kuphatikiza uku kumathandizira kasamalidwe ka ntchito zapamwamba, koma TaskControl imatha kuchita zambiri. Mwachitsanzo, gawo lathu lalikulu lapaintaneti ndi lopanda malire ndipo limagwiritsa ntchito TaskControl kuti lisinthire kuchuluka kwa zosintha pazida zonse. Potsirizira pake tsamba lawebusayiti limatha kumaliza mwachangu kutulutsa kwamapulogalamu angapo patsiku popanda kusokoneza kupezeka.

Kuwongolera ma seva mu malo opangira data

Tupperware itakhazikitsidwa koyamba mu 2011, gulu lililonse la seva limayang'aniridwa ndi ndandanda yosiyana. Kalelo, gulu la Facebook linali gulu la ma racks olumikizidwa ku netiweki imodzi, ndipo malo opangira data amakhala ndi masango angapo. Wokonza mapulani amatha kuyang'anira ma seva mugulu limodzi, kutanthauza kuti ntchitoyo sinafalikire m'magulu angapo. Zomangamanga zathu zidakula, tidasokoneza magulu. Popeza Tupperware sakanatha kusuntha ntchitoyo kuchokera kumagulu ochotsedwa kupita kumagulu ena popanda kusintha, zimafuna khama lalikulu ndi kugwirizana mosamala pakati pa opanga mapulogalamu ndi ogwira ntchito pa data center. Izi zidapangitsa kuti zinthu ziwonongeke pomwe ma seva anali osagwira ntchito kwa miyezi ingapo chifukwa chochotsa ntchito.

Tidapanga zida zothandizira kuthana ndi vuto la cluster decommissioning ndikugwirizanitsa ntchito zina zokonza. Wothandizira zothandizira amatsata zidziwitso zonse zolumikizidwa ndi seva ndipo amasankha mwachangu kuti ndi ndani amene amawongolera seva iliyonse. Kulumikiza mwamphamvu ma seva kwa okonza mapulani kumalola wokonza kuti azitha kuyang'anira ma seva m'malo osiyanasiyana a data. Popeza ntchito ya Tupperware sikhalanso ndi gulu limodzi, ogwiritsa ntchito a Tupperware atha kufotokoza momwe zotengera ziyenera kugawidwira m'malo olakwika. Mwachitsanzo, woyambitsa akhoza kulengeza cholinga chake (nenani: "yendetsani ntchito yanga pa madera olakwika a 2 m'dera la PRN") osatchula madera omwe alipo. Tupperware palokha ipeza ma seva oyenera kuti akwaniritse cholingachi, ngakhale gulu kapena ntchitoyo itachotsedwa.

Scalable kuthandizira dongosolo lonse lapadziko lonse lapansi

M'mbiri, zomangamanga zathu zagawidwa m'ma seva mazana odzipereka amagulu amodzi. Chifukwa cha kugawikana ndi kusowa kwa miyezo, tinali ndi ndalama zambiri zogwirira ntchito, ndipo ma seva opanda pake anali ovuta kugwiritsa ntchito kachiwiri. Pamsonkhano wa chaka chatha Systems @Scale tinapereka zomangamanga monga ntchito (IaaS), zomwe ziyenera kugwirizanitsa zomangamanga zathu kukhala paki yaikulu ya seva imodzi. Koma paki imodzi ya seva ili ndi zovuta zake. Iyenera kukwaniritsa zofunika zina:

• Scalability. Zomangamanga zathu zidakula pomwe tidawonjezera ma data mdera lililonse. Ma seva akhala ang'onoang'ono komanso amphamvu kwambiri, kotero pali ena ambiri m'dera lililonse. Zotsatira zake, wokonza mapulani m'dera lililonse sangathe kuthana ndi kuchuluka kwa zotengera zomwe zitha kuyendetsedwa ndi ma seva masauzande ambiri mdera lililonse.
• Kudalirika Ngakhale wokonza mapulaniwo atha kuchulukitsidwa chotere, kuchuluka kwa wokonzayo kumatanthauza kuti pali chiwopsezo chachikulu cha zolakwika ndipo gawo lonse la zotengera zitha kukhala zosasamalika.
• Kulekerera zolakwa. Pakachitika vuto lalikulu la zomangamanga (mwachitsanzo, ma seva omwe akuyendetsa pulogalamuyo amalephera chifukwa cha kulephera kwa ma netiweki kapena kutha kwa magetsi), zotsatirapo zoyipa ziyenera kukhudza gawo la ma seva m'derali.
• Kugwiritsa ntchito mosavuta. Zitha kuwoneka kuti mukufunika kuyendetsa angapo odziyimira pawokha adera limodzi. Koma kuchokera kumalingaliro osavuta, malo amodzi olowera mugawo logawana nawo amapangitsa kukhala kosavuta kuyang'anira mphamvu ndi ntchito.

Tinagawaniza scheduler kukhala shards kuti tithetse mavuto osunga dziwe lalikulu logawana. Shard iliyonse ya scheduler imayang'anira ntchito zake m'derali, ndipo izi zimachepetsa chiopsezo chokhudzana ndi wokonza. Pamene dziwe logawana likukulirakulira, titha kuwonjezera ma shards owonjezera. Kwa ogwiritsa ntchito a Tupperware, ma shards ndi ma proxies a scheduler amawoneka ngati gulu limodzi lowongolera. Iwo samasowa kugwira ntchito ndi gulu la shards lomwe limapanga ntchito. Scheduler shards ndi yosiyana kwambiri ndi ma klaster schedulers omwe tidagwiritsapo kale, pomwe gulu lowongolera lidagawika popanda kugawa magawo omwe adagawana nawo malinga ndi ma network topology.

Limbikitsani Kugwiritsa Ntchito Mwachangu ndi Elastic Computing

Kukula kwa zomangamanga zathu, ndikofunikira kwambiri kugwiritsa ntchito ma seva athu moyenera kuti tikwaniritse mtengo wa zomangamanga ndikuchepetsa katundu. Pali njira ziwiri zowonjezerera kugwiritsa ntchito seva:

• Kompyuta yokhazikika - chepetsani ntchito zapaintaneti nthawi yabata ndikugwiritsa ntchito maseva omasuka pantchito zapaintaneti, monga kuphunzira pamakina ndi ntchito za MapReduce.
• Kuchulukirachulukira - Ikani ntchito zapaintaneti ndikuchulukirachulukira pamaseva omwewo kuti ntchito zambiri ziziyenda mochepera.

Cholepheretsa m'malo athu a data ndi kugwiritsa ntchito mphamvu. Chifukwa chake, timakonda maseva ang'onoang'ono, osagwiritsa ntchito mphamvu zomwe pamodzi amapereka mphamvu zambiri zogwirira ntchito. Tsoka ilo, pa maseva ang'onoang'ono omwe ali ndi CPU yaying'ono ndi kukumbukira, kudzaza kwambiri sikuthandiza. Zachidziwikire, titha kuyika zotengera zingapo za mautumiki ang'onoang'ono pa seva imodzi yaying'ono yopanda mphamvu yomwe imawononga zinthu zochepa zama purosesa ndi kukumbukira, koma ntchito zazikuluzikulu zitha kukhala zotsika kwambiri pamtunduwu. Chifukwa chake, timalangiza omwe akupanga ntchito zathu zazikulu kuti akwaniritse bwino kuti agwiritse ntchito ma seva onse.


M'malo mwake, timagwiritsa ntchito ma elastic computing. Ntchito zathu zazikulu zambiri, monga News Feed, Mauthenga, ndi tsamba lakutsogolo, zimasiyana malinga ndi nthawi ya tsiku. Timachepetsa dala ntchito zapaintaneti nthawi yabata ndikugwiritsa ntchito maseva opanda ufulu pantchito zapaintaneti, monga kuphunzira pamakina ndi ntchito za MapReduce.

Tikudziwa kuchokera pazomwe takumana nazo kuti ndikwabwino kupereka ma seva athunthu ngati mayunitsi otanuka chifukwa ntchito zazikulu ndizopereka ndalama zazikulu komanso ogula akuluakulu amphamvu zotanuka, ndipo amakonzedwa kuti agwiritse ntchito ma seva onse. Seva ikatulutsidwa kuchokera pa intaneti nthawi yabata, wobwereketsa wothandizira amabwereketsa seva kwa wokonza kuti azitha kugwira ntchito popanda intaneti. Ngati ntchito yapaintaneti ikumana ndi chiwopsezo chachikulu, wobwereketsa amakumbukira mwachangu seva yobwerekedwa ndipo, pamodzi ndi wokonza mapulani, amabwezeretsanso ku ntchito yapaintaneti.

Maphunziro omwe aphunziridwa ndi mapulani amtsogolo

Pazaka 8 zapitazi, takhala tikupanga Tupperware kuti tigwirizane ndi kukula kwachangu kwa Facebook. Timagawana zomwe taphunzira ndipo tikukhulupirira kuti zithandiza ena kusamalira zomanga zomwe zikukula mwachangu:

• Khazikitsani kulumikizana kosinthika pakati pa gulu lowongolera ndi ma seva omwe amayang'anira. Kusinthasintha kumeneku kumapangitsa gulu lowongolera kuti lizitha kuyang'anira ma seva m'malo osiyanasiyana a data, kumathandizira kutsitsa ndikukonza masango, ndikupangitsa kugawa kwamphamvu kwamphamvu pogwiritsa ntchito makompyuta otanuka.
• Ndi gulu limodzi lowongolera m'derali, zimakhala zosavuta kugwira ntchito ndi ntchito komanso zosavuta kuyang'anira zombo zazikulu zogawana seva. Zindikirani kuti gulu lowongolera limasunga mfundo imodzi yolowera, ngakhale mawonekedwe ake amkati asiyanitsidwa chifukwa cha kukula kapena kulekerera zolakwika.
• Pogwiritsa ntchito pulogalamu yowonjezera, gulu lowongolera likhoza kudziwitsa zakunja zomwe zikubwera. Kuphatikiza apo, mautumiki apamwamba amatha kugwiritsa ntchito mawonekedwe a plugin kuti asinthe kasamalidwe ka chidebe. Ndi pulogalamu yowonjezera iyi, gulu lowongolera limapereka kuphweka pomwe likutumikira mogwira mtima mautumiki osiyanasiyana apamwamba.
• Timakhulupirira kuti computing yotanuka, komwe timachotsa ma seva onse kuchokera kwa opereka ntchito zamagulu, kuphunzira makina, ndi ntchito zina zosafunikira, ndiye njira yabwino kwambiri yopititsira patsogolo magwiridwe antchito a ma seva ang'onoang'ono, osagwiritsa ntchito mphamvu.

Tikuyamba kumene kukhazikitsa gulu limodzi lapadziko lonse lapansi logawana seva. Pakali pano pafupifupi 20% ya ma seva athu ali mu dziwe logawana. Kuti mukwaniritse 100%, nkhani zambiri ziyenera kuthetsedwa, kuphatikizapo kusunga dziwe losungiramo zinthu, kukonza makina, kuyang'anira zofunikira za anthu ogwira ntchito, kupititsa patsogolo kagwiritsidwe ntchito ka seva, komanso kupititsa patsogolo chithandizo cha ntchito yophunzirira makina. Sitingadikire kuti tithane ndi zovuta izi ndikugawana zomwe tapambana.

Source: www.habr.com