Miyezo yatsopano yosungira zinthu

Flying Fortress ndi Nele-Diel

S3 chinthu chosungira lamulo Mail.ru Cloud Storage anamasulira nkhani yokhudza zomwe zili zofunika posankha kusungirako chinthu. Zotsatirazi ndi zomwe wolemba amawonera.

Zikafika posungira zinthu, anthu amangoganiza za chinthu chimodzi: mtengo pa TB/GB. Zachidziwikire, metric iyi ndiyofunikira, koma imapangitsa njirayo kukhala mbali imodzi ndikufanizira kusungidwa kwa chinthu ndi chida chosungira zakale. Kuphatikiza apo, njira iyi imachepetsa kufunikira kwa kusungirako zinthu pagulu laukadaulo wamabizinesi.

Posankha kusungirako zinthu, muyenera kulabadira zinthu zisanu:

• ntchito;
• scalability;
• Zogwirizana ndi S3;
• kuyankha zolephera;
• umphumphu.

Makhalidwe asanuwa ndi njira zatsopano zosungira zinthu, pamodzi ndi mtengo. Tiyeni tiyang'ane pa iwo onse.

Kukonzekera

Malo ogulitsa zinthu zachikhalidwe alibe ntchito. Opereka chithandizo nthawi zonse amadzipereka kuti apeze mitengo yotsika. Komabe, ndi zinthu zamakono zosungira zinthu ndizosiyana.

Makina osiyanasiyana osungira amayandikira kapena kupitilira liwiro la Hadoop. Zofunikira zamakono zowerengera ndi kulemba liwiro: kuchokera ku 10 GB/s kwa hard drive, mpaka 35 GB/s kwa NVMe. 

Kutulutsa uku ndikokwanira Spark, Presto, Tensorflow, Teradata, Vertica, Splunk ndi machitidwe ena amakono apakompyuta mu stack ya analytics. Mfundo yakuti ma database a MPP akukonzedwa kuti asungidwe zinthu zikusonyeza kuti ikugwiritsidwa ntchito kwambiri ngati malo osungira.

Ngati makina anu osungira sapereka liwiro lomwe mukufuna, simungagwiritse ntchito deta ndikuchotsa mtengo kuchokera pamenepo. Ngakhale mutatenganso deta kuchokera kuzinthu zosungiramo zinthu ndikuyika mu-memory processing structure, mudzafunikabe bandwidth kuti musamutsire deta ndi kukumbukira. Malo ogulitsa zinthu zakale alibe zokwanira.

Iyi ndiye mfundo yofunika kwambiri: metric yatsopano yogwirira ntchito ndikudutsa, osati latency. Imafunika kuti pakhale deta pamlingo waukulu ndipo ndizomwe zimachitika muzinthu zamakono zamakono.

Ngakhale ma benchmark ndi njira yabwino yodziwira momwe ntchito ikuyendera, singayesedwe molondola musanagwiritse ntchito chilengedwe. Pokhapokha mutha kunena komwe kuli botolo: mu mapulogalamu, ma disks, network kapena pamlingo wamakompyuta.

Scalability

Scalability imatanthawuza kuchuluka kwa ma petabytes omwe amalowa mumalo amodzi. Zomwe ogulitsa amati ndizosavuta, zomwe samanena ndikuti akamakula, makina akuluakulu a monolithic amakhala osalimba, ovuta, osakhazikika, komanso okwera mtengo.

Metric yatsopano ya scalability ndi kuchuluka kwa mayina kapena makasitomala omwe mungagwiritse ntchito. Metric imatengedwa mwachindunji kuchokera ku hyperscalers, kumene zosungirako zosungirako zimakhala zazing'ono koma zimafika mabiliyoni a mayunitsi. Mwambiri, ichi ndi metric yamtambo.

Zomangamanga zikakhala zazing'ono, zimakhala zosavuta kukhathamiritsa chitetezo, kuwongolera mwayi, kasamalidwe ka malamulo, kasamalidwe ka moyo, komanso zosintha zosasokoneza. Ndipo pamapeto pake kuonetsetsa zokolola. Kukula kwa nyumba yomanga ndi ntchito ya kuwongolera kwa dera lolephera, momwe machitidwe okhazikika amamangidwira.

Multi-tenancy ali ndi makhalidwe ambiri. Ngakhale gawoli limalankhula za momwe mabungwe amaperekera mwayi wopeza deta ndi kugwiritsa ntchito, limatanthawuzanso mapulogalamu omwewo komanso malingaliro odzipatula kwa wina ndi mnzake.

Makhalidwe a njira yamakono yofikira makasitomala ambiri:

• M'kanthawi kochepa, chiwerengero cha makasitomala chikhoza kukula kuchokera mazana angapo kufika mamiliyoni angapo.
• Makasitomala ali olekanitsidwa kwathunthu kwa wina ndi mnzake. Izi zimawathandiza kuti azitha kuyendetsa mapulogalamu osiyanasiyana a mapulogalamu omwewo ndikusunga zinthu zosiyanasiyana, zilolezo, mawonekedwe, chitetezo ndi milingo yokonza. Izi ndizofunikira mukakulitsa ma seva atsopano, zosintha, ndi malo.
• Zosungirako ndi elastically scalable, zothandizira zimaperekedwa pakufunika.
• Ntchito iliyonse imayendetsedwa ndi API ndipo imapangidwa popanda kulowererapo kwa anthu.
• Mapulogalamu amatha kusungidwa m'mitsuko ndikugwiritsa ntchito makina oyimba wamba monga Kubernetes.

Zogwirizana ndi S3

Amazon S3 API ndiye mulingo wokhazikika pakusungira zinthu. Aliyense chinthu chosungira mapulogalamu ogulitsa amati n'zogwirizana ndi izo. Kugwirizana ndi S3 ndikosavuta: mwina kumakwaniritsidwa kapena ayi.

M'malo mwake, pali mazana kapena masauzande am'mphepete momwe china chake chimalakwika mukamagwiritsa ntchito kusungirako zinthu. Makamaka kuchokera kwa opereka mapulogalamu ndi mautumiki ake. Milandu yake yayikulu yogwiritsira ntchito ndikusungirako mwachindunji kapena kusungitsa, chifukwa chake pali zifukwa zochepa zotchulira API, zomwe zimagwiritsidwa ntchito ndizofanana.

Mapulogalamu otsegula ali ndi ubwino wambiri. Imakhudza zochitika zam'mphepete, kutengera kukula ndi mitundu yosiyanasiyana ya ntchito, makina ogwiritsira ntchito, ndi zomangamanga za Hardware.

Zonsezi ndizofunikira kwa opanga mapulogalamu, chifukwa chake ndikofunikira kuyesa pulogalamuyi ndi osungira. Open source imapangitsa kuti ntchitoyi ikhale yosavuta - ndikosavuta kumvetsetsa kuti ndi nsanja iti yomwe ili yoyenera pulogalamu yanu. Woperekayo angagwiritsidwe ntchito ngati malo amodzi osungira, kutanthauza kuti adzakwaniritsa zosowa zanu. 

Kutsegula kumatanthawuza: mapulogalamu samamangirizidwa ndi ogulitsa ndipo amawonekera bwino. Izi zimatsimikizira moyo wautali wogwiritsa ntchito.

Ndipo zolemba zina zingapo za gwero lotseguka ndi S3. 

Ngati mukugwiritsa ntchito pulogalamu yayikulu ya data, S3 SELECT imawongolera magwiridwe antchito ndikuchita bwino ndi dongosolo la ukulu. Imachita izi pogwiritsa ntchito SQL kuti itengenso zinthu zomwe mukufuna kuzisungira.

Mfundo yofunika kwambiri ndikuthandizira zidziwitso za ndowa. Zidziwitso za ndowa zimathandizira makompyuta opanda seva, gawo lofunikira pamapangidwe aliwonse a microservice omwe amaperekedwa ngati ntchito. Poganizira kuti kusungirako chinthu ndikusungirako mitambo, kuthekera uku kumakhala kofunikira pamene kusungirako zinthu kumagwiritsidwa ntchito ndi mapulogalamu amtambo.

Pomaliza, kukhazikitsidwa kwa S3 kuyenera kuthandizira ma API a Amazon S3 server-side encryption: SSE-C, SSE-S3, SSE-KMS. Ngakhale zili bwino, S3 imathandizira chitetezo chosokoneza chomwe chili chotetezeka. 

Kuyankha zolephera

Metric yomwe nthawi zambiri imanyalanyazidwa ndi momwe makina amagwirira ntchito zolephera. Zolephera zimachitika pazifukwa zosiyanasiyana, ndipo kusungirako zinthu kuyenera kuthana nazo zonse.

Mwachitsanzo, pali mfundo imodzi yolephera, metric ya izi ndi ziro.

Tsoka ilo, makina ambiri osungira zinthu amagwiritsa ntchito ma node apadera omwe amayenera kuthandizidwa kuti masango agwire bwino ntchito. Izi zikuphatikiza ma node kapena ma seva a metadata - izi zimapangitsa kulephera kumodzi.

Ngakhale pamene pali zolephera zambiri, kuthekera kopirira kulephera kowopsa ndikofunikira. Ma disks amalephera, ma seva amalephera. Chofunikira ndikupanga mapulogalamu opangidwa kuti athe kuthana ndi kulephera ngati mkhalidwe wabwinobwino. Ngati disk kapena node ikulephera, mapulogalamuwa adzapitiriza kugwira ntchito popanda kusintha.

Chitetezo chomangidwira pakufufutidwa kwa data ndikuwonongeka kwa data kumatsimikizira kuti mutha kutaya ma diski kapena ma node ambiri momwe muli ndi midadada yofananira - nthawi zambiri theka la ma disks. Pokhapokha pulogalamuyo idzalephera kubwezeretsa deta.

Kulephera sikumayesedwa kawirikawiri pansi pa katundu, koma kuyesa koteroko kumafunika. Kuyerekeza kulephera kwa katundu kudzawonetsa ndalama zonse zomwe zidachitika pambuyo polephera.

Kusasinthasintha

Chiwerengero chofanana cha 100% chimatchedwanso kusasinthasintha kokhazikika. Kugwirizana ndi gawo lofunikira la dongosolo lililonse losungirako, koma kusasinthasintha kwamphamvu ndikosowa. Mwachitsanzo, Amazon S3 ListObject siyokhazikika, imangokhala yokhazikika pamapeto.

Kodi kusasinthasintha kotheratu kumatanthauza chiyani? Pazochita zonse zotsatizana ndi PUT yotsimikizika, zotsatirazi ziyenera kuchitika:

• Mtengo wosinthidwa umawonekera powerenga kuchokera ku node iliyonse.
• Zosinthazo zimatetezedwa ku kulephera kwa node redundancy.

Izi zikutanthauza kuti ngati mukoka pulagi pakati pa kujambula, palibe chomwe chidzatayika. Dongosololi silibwezanso data yowonongeka kapena yachikale. Uwu ndi kapamwamba kwambiri komwe kumafunikira pazinthu zambiri, kuyambira pazosinthana mpaka zosunga zobwezeretsera ndikuchira.

Pomaliza

Awa ndi ma metrics atsopano osungira zinthu omwe amawonetsa momwe amagwiritsidwira ntchito m'mabungwe amasiku ano, momwe magwiridwe antchito, kusasinthika, kusasunthika, madomeni olakwika ndi kugwirizana kwa S3 ndizomwe zimamangira ntchito zamtambo komanso kusanthula kwakukulu kwa data. Ndikupangira kugwiritsa ntchito mndandandawu kuwonjezera pa mtengo pomanga ma data amakono. 

Zokhudza kusungidwa kwa chinthu cha Mail.ru Cloud Solutions: S3 zomangamanga. Zaka 3 zakusinthika kwa Mail.ru Cloud Storage.

Zomwe mungawerenge:

1. Chitsanzo cha pulogalamu yoyendetsedwa ndi zochitika potengera ma webhook mu S3 yosungirako zinthu Mail.ru Cloud Solutions.
2. Kuposa Ceph: Kusungirako kwamtambo kwa MCS 
3. Kugwira ntchito ndi Mail.ru Cloud Solutions S3 yosungirako zinthu ngati fayilo.
4. Njira yathu ya Telegraph yokhala ndi nkhani zosintha posungira S3 ndi zinthu zina. 

Source: www.habr.com