Tupperware: Kisan Kubernetes na Facebook?

Ingantacciyar kuma amintaccen gudanarwa na gungu a kowane ma'auni tare da Tupperware

Yau a Systems@Taron Sikeli mun gabatar da Tupperware, tsarin kula da gungu wanda ke tsara kwantena a cikin miliyoyin sabobin da ke tafiyar da kusan dukkanin ayyukanmu. Mun fara tura Tupperware a cikin 2011, kuma tun daga nan kayan aikin mu sun girma daga 1 data center ga duka Cibiyoyin bayanai 15 da aka rarraba ta geo. Duk wannan lokacin, Tupperware bai tsaya cik ba kuma ya haɓaka tare da mu. Za mu nuna muku yadda Tupperware ke ba da sarrafa gungu na aji na farko, gami da ingantaccen tallafi don sabis na jaha, rukunin sarrafawa guda ɗaya don duk cibiyoyin bayanai, da ikon rarraba iyawa tsakanin sabis a cikin ainihin lokaci. Za mu kuma raba darussan da muka koya yayin da kayan aikin mu ke haɓaka.

Tupperware yana yin ayyuka daban-daban. Masu haɓaka aikace-aikacen suna amfani da shi don sadar da sarrafa aikace-aikace. Yana tattara lambar aikace-aikacen da abin dogaro cikin hoto kuma yana isar da shi zuwa sabobin a matsayin kwantena. Kwantenala suna ba da keɓance tsakanin aikace-aikace akan sabar iri ɗaya don masu haɓakawa suyi hulɗa da dabarun aikace-aikacen kuma kada su damu game da nemo sabar ko sarrafa sabuntawa. Tupperware kuma yana lura da aikin uwar garken, kuma idan ya sami gazawa, yana canja wurin kwantena daga uwar garken matsala.

Injiniyoyin tsara ƙarfin aiki suna amfani da Tupperware don rarraba ƙarfin uwar garken ga ƙungiyoyi dangane da kasafin kuɗi da ƙuntatawa. Suna kuma amfani da shi don inganta amfani da uwar garke. Masu aikin cibiyar bayanai sun juya zuwa Tupperware don rarraba kwantena yadda yakamata a cikin cibiyoyin bayanai da tsayawa ko matsar da kwantena yayin kulawa. Godiya ga wannan, kiyaye sabobin, cibiyoyin sadarwa da kayan aiki na buƙatar ƙaramin sa hannun ɗan adam.

Tupperware gine-gine

Tupperware PRN gine yana ɗaya daga cikin yankuna na cibiyoyin bayanan mu. Yankin ya ƙunshi gine-ginen cibiyar bayanai da yawa (PRN1 da PRN2) da ke kusa. Muna shirin yin kwamiti mai sarrafawa guda ɗaya wanda zai sarrafa duk sabar a yanki ɗaya.

Masu haɓaka aikace-aikacen suna ba da sabis ta hanyar ayyukan Tupperware. Aiki ya ƙunshi kwantena da yawa, kuma dukkansu yawanci suna gudanar da lambar aikace-aikacen iri ɗaya.

Tupperware yana da alhakin samar da kwantena da sarrafa yanayin rayuwarsu. Ya ƙunshi abubuwa da yawa:

• Tupperware frontend yana samar da APIs don ƙirar mai amfani, CLI, da sauran kayan aikin sarrafa kansa ta inda zaku iya hulɗa tare da Tupperware. Suna ɓoye duk tsarin ciki daga masu aikin Tupperware.
• Mai tsara Jadawalin Tupperware shine kwamiti mai kulawa da ke da alhakin sarrafa kwantena da zagayen aiki. An tura shi a matakan yanki da na duniya, inda mai tsara tsarin yanki ke sarrafa sabobin a cikin yanki ɗaya kuma mai tsara tsarin duniya yana sarrafa sabobin daga yankuna daban-daban. An rarraba mai tsara jadawalin zuwa ɓangarorin, kuma kowane ɓangarorin yana sarrafa jerin ayyuka.
• Tupperware's Scheduler Proxy yana ɓoye ɓoyayyiyar ɓarna na ciki kuma yana ba da madaidaiciyar faren gilashi ɗaya ga masu amfani da Tupperware.
• Mai rarraba Tupperware yana ba da kwantena ga sabobin. Mai tsara jadawalin yana sarrafa tsayawa, farawa, sabuntawa, da gazawar kwantena. A halin yanzu, mai rarrabawa ɗaya zai iya sarrafa yankin gaba ɗaya ba tare da tsagewa cikin tarkace ba. (Ka lura da bambancin ƙamus. Misali, mai tsara jadawalin a cikin Tupperware yayi daidai da kwamitin sarrafawa a ciki Kubernetes, kuma Tupperware allocator ana kiransa mai tsarawa a cikin Kubernetes.)
• Dillalin albarkatun yana adana tushen gaskiya don uwar garken da abubuwan sabis. Muna gudanar da dillalin albarkatu guda ɗaya don kowace cibiyar bayanai, kuma tana adana duk bayanai game da sabar a waccan cibiyar bayanai. Dillalin albarkatu da tsarin sarrafa iya aiki, ko tsarin samar da albarkatu, a tsai da shawarar wane mai tsara isar da sabar ke sarrafa wace sabar. Sabis ɗin duba lafiyar yana lura da sabar da adana bayanai game da lafiyar su a cikin dillalin albarkatun. Idan uwar garken yana da matsala ko yana buƙatar kulawa, dillalin albarkatun yana gaya wa mai rarrabawa da mai tsarawa su dakatar da kwantena ko matsar da su zuwa wasu sabobin.
• Tupperware Agent shine daemon da ke gudana akan kowace uwar garken da ke kula da samarwa da cire kwantena. Aikace-aikace suna gudana a cikin akwati, wanda ke ba su ƙarin keɓewa da haɓakawa. Kunna Tsarin Tsarin @Scale taro na bara Mun riga mun bayyana yadda ake ƙirƙirar kwantena Tupperware guda ɗaya ta amfani da hotuna, btrfs, cgroupv2 da systemd.

Musamman fasali na Tupperware

Tupperware yayi kama da yawa ta hanyoyi da yawa zuwa sauran tsarin sarrafa tari kamar Kubernetes da Mesos, amma kuma akwai bambance-bambance:

• Ginin tallafi don ayyuka na jiha.
• Ƙungiyar sarrafawa guda ɗaya don sabobin a cikin cibiyoyin bayanai daban-daban don sarrafa isar da kwantena bisa niyya, ƙaddamar da gungu da kulawa.
• Share rabo na kula da panel don zuƙowa.
• Kwamfuta na roba yana ba ku damar rarraba wutar lantarki tsakanin ayyuka a cikin ainihin lokaci.

Mun ƙirƙira waɗannan fasalulluka masu kyau don tallafawa aikace-aikacen marasa ƙasa iri-iri da jaha a cikin babbar rundunar jiragen ruwa na sabar da aka raba.

Ginin tallafi don ayyuka na jiha.

Tupperware yana aiki da ayyuka masu mahimmanci iri-iri waɗanda ke adana bayanan samfur na dindindin don Facebook, Instagram, Messenger da WhatsApp. Waɗannan na iya zama manyan shaguna na nau'i-nau'i masu ƙima (misali. ZipDB) da kuma lura da ma'ajiyar bayanai (misali, ODS Gorilla и Scuba). Kula da ayyuka na gwamnati ba abu ne mai sauƙi ba, domin dole ne tsarin ya tabbatar da cewa samar da kwantena zai iya jure wa manyan tarzoma, gami da katsewar hanyar sadarwa ko katsewar wutar lantarki. Kuma yayin da dabaru na al'ada, kamar rarraba kwantena a cikin wuraren da ba su da kyau, suna aiki da kyau don ayyukan marasa jiha, ayyuka na jihohi suna buƙatar ƙarin tallafi.

Misali, idan gazawar uwar garken ya sa babu kwafin bayanai guda ɗaya, shin ya kamata ku ba da damar kiyayewa ta atomatik wanda zai sabunta maƙallan akan sabar 50 daga tafkin 10? Ya dogara da yanayin. Idan ɗayan waɗannan sabar guda 50 yana da wani kwafin bayanai iri ɗaya, yana da kyau a jira kar a rasa kwafi 2 lokaci guda. Domin yin tsai da shawara game da kiyaye tsarin da aiki, muna buƙatar bayani game da kwafin bayanai na ciki da dabarar jeri na kowane sabis na jaha.

Keɓancewar TaskControl yana ba da damar sabis na jaha don yin tasiri ga yanke shawara waɗanda ke shafar samuwar bayanai. Yin amfani da wannan ƙa'idar, mai tsara jadawalin yana sanar da aikace-aikacen waje game da ayyukan kwantena (sake farawa, sabuntawa, ƙaura, kiyayewa). Sabis mai ƙima yana aiwatar da mai sarrafawa wanda ke gaya wa Tupperware lokacin da ba shi da aminci don yin kowane aiki, kuma waɗannan ayyukan ana iya musanya su ko jinkirta su na ɗan lokaci. A cikin misalin da ke sama, mai sarrafa bayanai zai iya gaya wa Tupperware don sabunta 49 na sabobin 50, amma barin takamaiman sabar (X) ita kaɗai a yanzu. Sakamakon haka, idan lokacin sabuntawar kwaya ya wuce kuma bayanan har yanzu ba su iya dawo da kwafin matsala ba, Tupperware har yanzu zai sabunta sabar X.

Yawancin ayyuka na jihohi a cikin Tupperware suna amfani da TaskControl ba kai tsaye ba, amma ta hanyar ShardManager, dandamali na gama gari don ƙirƙirar ayyuka na gari akan Facebook. Tare da Tupperware, masu haɓakawa za su iya ƙayyadaddun niyyarsu don ainihin yadda ya kamata a rarraba kwantena a cikin cibiyoyin bayanai. Tare da ShardManager, masu haɓakawa suna ƙayyadaddun niyyarsu don yadda yakamata a rarraba ɓarna bayanai a cikin kwantena. ShardManager yana sane da sanya bayanan da kwafin aikace-aikacen sa kuma yana sadarwa tare da Tupperware ta hanyar TaskControl don tsara ayyukan kwantena ba tare da shigar da aikace-aikacen kai tsaye ba. Wannan haɗin kai yana sauƙaƙa sosai da sarrafa ayyuka na jihohi, amma TaskControl yana iya ƙarin. Misali, babban matakin gidan yanar gizon mu ba shi da ƙasa kuma yana amfani da TaskControl don daidaita ƙimar ɗaukakawa zuwa kwantena. A ƙarshe matakin gidan yanar gizon yana da ikon yin saurin kammala abubuwan fitar da software da yawa a kowace rana ba tare da lalata samuwa ba.

Sarrafa sabobin a cikin cibiyoyin bayanai

Lokacin da aka fara ƙaddamar da Tupperware a cikin 2011, kowane gungu na uwar garken ana sarrafa shi ta hanyar mai tsarawa daban. A wancan lokacin, gungu na Facebook rukuni ne na rakiyar uwar garken da aka haɗa zuwa hanyar sadarwa guda ɗaya, kuma cibiyar bayanai ta ƙunshi gungu da yawa. Mai tsara jadawalin zai iya sarrafa sabobin a cikin tari ɗaya kawai, ma'ana aikin ba zai iya yaɗuwa cikin gungu da yawa ba. Kayan aikin mu sun girma, mun ƙara rubuta gungu. Tun da Tupperware ba zai iya matsar da aikin daga gungu da aka soke zuwa wasu gungu ba tare da canje-canje ba, yana buƙatar ƙoƙari mai yawa da daidaitawa a hankali tsakanin masu haɓaka aikace-aikacen da masu sarrafa bayanan. Wannan tsari ya haifar da ɓarnatar albarkatu lokacin da sabobin ke aiki na tsawon watanni saboda ƙaddamar da hanyoyin.

Mun ƙirƙiri dillalin albarkatu don magance matsalar rushewar gungu da daidaita sauran nau'ikan ayyukan kulawa. Dillalin albarkatun yana kiyaye duk bayanan jiki da ke da alaƙa da uwar garken kuma yana yanke shawarar wanne mai tsarawa ke sarrafa kowace sabar. Haɗa sabobin a zahiri zuwa masu tsara tsarawa yana ba mai tsarawa damar sarrafa sabar a cibiyoyin bayanai daban-daban. Tun da aikin Tupperware baya iyakance ga gungu ɗaya, masu amfani da Tupperware za su iya tantance yadda ya kamata a rarraba kwantena a cikin wuraren da ba su da laifi. Misali, mai haɓakawa na iya bayyana aniyarsa (ce: "gudanar da aikina akan wuraren kuskure guda 2 a cikin yankin PRN") ba tare da fayyace takamaiman wuraren samuwa ba. Tupperware da kanta za ta sami sabar masu dacewa don aiwatar da wannan niyya, ko da gungu ko sabis ɗin ya ƙare.

Zazzagewa don tallafawa tsarin duniya gaba ɗaya

A tarihi, an raba kayan aikin mu zuwa ɗaruruwan wuraren tafkunan sabar uwar garken don ƙungiyoyi ɗaya. Saboda rarrabuwar kawuna da rashin ma'auni, muna da tsadar aiki mai yawa, kuma sabar marasa aiki sun fi wahalar sake amfani da su. A taron na bara Systems @Scale mun gabatar kayayyakin more rayuwa a matsayin sabis (IaaS), wanda yakamata ya haɗa kayan aikin mu zuwa babban wurin shakatawa guda ɗaya. Amma wurin shakatawa na uwar garken guda ɗaya yana da nasa matsalolin. Dole ne ya cika wasu buƙatu:

• Ƙimar ƙarfi. Ayyukanmu sun girma yayin da muka ƙara cibiyoyin bayanai a kowane yanki. Sabis ɗin sun zama ƙarami kuma sun fi ƙarfin kuzari, don haka akwai da yawa daga cikinsu a kowane yanki. Sakamakon haka, mai tsara jadawalin guda ɗaya a kowane yanki ba zai iya ɗaukar adadin kwantena waɗanda za a iya tafiyar da su akan dubban ɗaruruwan sabobin a kowane yanki.
• Dogara Ko da mai tsara jadawalin za a iya haɓaka da yawa haka, babban fa'idar mai tsarawa yana nufin akwai haɗarin kurakurai da yawa kuma duk yanki na kwantena na iya zama wanda ba a iya sarrafa shi.
• Haƙuri na kuskure. A yayin babban gazawar ababen more rayuwa (alal misali, sabar da ke tafiyar da mai tsara jadawalin sun gaza saboda gazawar hanyar sadarwa ko katsewar wutar lantarki), mummunan sakamakon ya kamata ya shafi wani yanki na sabobin a yankin.
• Ba da amfani. Yana iya zama kamar kuna buƙatar gudanar da jadawali masu zaman kansu da yawa don yanki ɗaya. Amma daga hangen nesa, wuri guda na shigarwa cikin tafkin da aka raba na yanki yana sauƙaƙa sarrafa iyawa da ayyuka.

Mun raba mai tsarawa zuwa shards don magance matsalolin kula da babban tafkin da aka raba. Kowane shard mai tsarawa yana gudanar da ayyukan kansa a yankin, kuma wannan yana rage haɗarin da ke tattare da mai tsarawa. Yayin da tafkin da aka raba ke girma, za mu iya ƙara ƙarin shards masu tsarawa. Ga masu amfani da Tupperware, shards da proxies masu tsarawa suna kama da kwamiti mai sarrafawa ɗaya. Ba dole ba ne su yi aiki tare da tarin shards waɗanda ke tsara ayyuka. Shards masu tsarawa sun bambanta sosai da masu tsara tsarin tari da muka yi amfani da su a baya, lokacin da aka raba kwamitin kulawa ba tare da an raba tafki na sabar ba bisa ga topology na cibiyar sadarwa.

Inganta Ingantacciyar Amfani tare da Kwamfuta na roba

Girman kayan aikin mu, mafi mahimmanci shine amfani da sabobin mu da kyau don inganta farashin kayan aiki da rage kaya. Akwai hanyoyi guda biyu don haɓaka ingantaccen amfani da uwar garken:

• Ƙwararren Ƙwararren Ƙwararren Ƙwararren Ƙwararren Ƙwararren Ƙwararren Ƙwararren Ƙwararren Ƙwararren Ƙwararren Ƙwararren Ƙwararren Ƙwararren Ƙwararren Ƙwararren Ƙwararren Ƙwararren Ƙwararren Ƙwararren Ƙwaƙwalwa na Ƙaƙwalwa na Ƙaƙwalwa na Ƙaƙwalwa na Ƙaƙwalwa na Ƙaƙwalwa na Ƙaƙwalwa na Ƙaƙwalwa na Ƙaƙwalwa ) - Ƙaddamar da ayyuka na kan layi a cikin sa'o'i na shiru da amfani da 'yantattun sabobin don ayyukan aiki na layi, kamar koyon inji da MapReduce jobs.
• Yin lodi - Sanya sabis na kan layi da kayan aikin batch akan sabar iri ɗaya don nauyin aikin batch yayi aiki da ƙarancin fifiko.

Kangin da ke cikin cibiyoyin bayanan mu shine Amfanin makamashi. Saboda haka, mun fi son ƙananan sabar masu amfani da makamashi waɗanda tare suke samar da ƙarin ƙarfin sarrafawa. Abin takaici, akan ƙananan sabobin da ke da ƙananan CPU da ƙwaƙwalwar ajiya, overloading ba shi da tasiri. Tabbas, zamu iya sanya kwantena da yawa na ƙananan ayyuka akan ɗayan ƙaramin sabar mai amfani da makamashi wanda ke cinye albarkatun sarrafa kayan aiki kaɗan da ƙwaƙwalwar ajiya, amma manyan ayyuka zasu sami ƙarancin aiki a cikin wannan yanayin. Don haka, muna ba da shawara ga masu haɓaka manyan ayyukanmu da su inganta su ta yadda za su yi amfani da dukkan sabobin.


Ainihin, muna inganta ingantaccen amfani ta amfani da na'ura mai kwakwalwa. Yawancin manyan ayyukanmu, kamar Ciyarwar Labarai, fasalin Saƙon, da matakin gidan yanar gizo na gaba-gaba, sun bambanta dangane da lokacin rana. Muna rage ayyukan kan layi da gangan a cikin sa'o'i masu natsuwa kuma muna amfani da sabar sabobin don ayyukan aiki na layi, kamar koyon injina da ayyukan MapReduce.

Mun sani daga gwaninta cewa yana da kyau a samar da duka sabobin a matsayin raka'a na iya aiki na roba saboda manyan ayyuka duka manyan masu ba da gudummawa ne da manyan masu amfani da ƙarfin roba, kuma an inganta su don amfani da duka sabobin. Lokacin da aka saki uwar garken daga sabis na kan layi a cikin sa'o'i na shiru, dillalin albarkatun yana ba da hayar uwar garken ga mai tsarawa don gudanar da ayyukan aiki na layi a kai. Idan sabis na kan layi ya sami babban nauyi, dillalin albarkatun yana da sauri tuno sabar da aka aro kuma, tare da mai tsarawa, ya mayar da shi zuwa sabis na kan layi.

Darussan da aka koya da tsare-tsare na gaba

A cikin shekaru 8 da suka gabata, muna haɓaka Tupperware don ci gaba da saurin haɓakar Facebook. Muna raba abin da muka koya kuma muna fatan zai taimaka wa wasu sarrafa abubuwan more rayuwa cikin sauri:

• Saita haɗin kai mai sassauƙa tsakanin sashin kulawa da sabar da yake gudanarwa. Wannan sassauci yana ba da damar kula da uwar garken don sarrafa sabobin a cikin cibiyoyin bayanai daban-daban, yana taimakawa ta atomatik ragewa da kuma kula da gungu, kuma yana ba da damar rarraba ƙarfin aiki ta amfani da na'ura mai kwakwalwa.
• Tare da kwamiti mai kulawa guda ɗaya a cikin yankin, ya zama mafi dacewa don yin aiki tare da ayyuka da sauƙi don sarrafa babban rukunin uwar garken da aka raba. Yi la'akari da cewa kwamitin kulawa yana kula da wurin shigarwa guda ɗaya, koda kuwa tsarinsa na ciki ya rabu saboda dalilai na ma'auni ko rashin haƙuri.
• Yin amfani da samfurin plugin, kwamitin kulawa zai iya sanar da aikace-aikacen waje na ayyukan kwantena masu zuwa. Haka kuma, ayyuka na jaha na iya amfani da kayan aikin plugin don keɓance sarrafa kwantena. Tare da wannan ƙirar plugin ɗin, kwamitin kulawa yana ba da sauƙi yayin da ya dace yana ba da sabis na jihohi daban-daban.
• Mun yi imanin cewa ƙididdiga na roba, inda muke cire duka sabobin daga sabis na masu ba da gudummawa don ayyukan batch, koyon inji, da sauran ayyukan da ba na gaggawa ba, ita ce hanya mafi kyau don inganta ingantaccen ƙananan sabar masu amfani da makamashi.

Mun fara aiwatarwa ƙungiyar sabar uwar garken guda ɗaya ta duniya. A halin yanzu kusan kashi 20% na sabobinmu suna cikin tafkin da aka raba. Don cimma 100%, ana buƙatar magance batutuwa da yawa, ciki har da kiyaye wurin ajiya mai raba, sarrafa sarrafa kansa, sarrafa buƙatun masu haya, haɓaka amfani da sabar, da haɓaka tallafi don aikin koyan injin. Ba za mu iya jira don ɗaukar waɗannan ƙalubalen da raba nasarorinmu ba.

source: www.habr.com