Zvitsva zvekuchengetedza zvinhu

Flying Fortress naNele-Diel

S3 chinhu chekuchengetedza murairo Mail.ru Cloud Storage yakashandura chinyorwa pamusoro pezvipi zvakakosha pakusarudza chinhu chekuchengetedza. Chinotevera ndicho chinyorwa kubva mumaonero emunyori.

Kana zvasvika pakuchengetedza chinhu, vanhu vanowanzo funga nezve chinhu chimwe chete: mutengo paTB/GB. Ehe, metric iyi yakakosha, asi inoita kuti nzira yacho ive nedivi uye inoenzanisa chengetedzo yechinhu nemudziyo wekuchengetedza dura. Uyezve, nzira iyi inoderedza kukosha kwekuchengetedza chinhu kune bhizinesi tekinoroji stack.

Paunosarudza kuchengetedza chinhu, iwe unofanirwa kuterera kune mashanu maitiro:

• performance;
• scalability;
• S3 inoenderana;
• mhinduro kune kukundikana;
• kuvimbika.

Aya maitiro mashanu mametric matsva ekuchengetedza chinhu, pamwe nemutengo. Ngativatarise vese.

Kubudirira

Zvitoro zvechivanhu zvinoshaya basa. Vanopa masevhisi vaigara vachizvibayira mukutsvaka mitengo yakaderera. Zvisinei, nezvinhu zvemazuva ano zvekuchengetedza zvinhu zvakasiyana.

Akasiyana siyana ekuchengetedza masisitimu anoswedera kana kutopfuura kumhanya kweHadoop. Zvinodiwa zvemazuva ano zvekuverenga nekunyora kumhanya: kubva ku10 GB/s kune hard drive, kusvika ku35 GB/s yeNVMe. 

Kubuda uku kwakaringana Spark, Presto, Tensorflow, Teradata, Vertica, Splunk uye mamwe mazuvano emakomputa masisitimu mune analytics stack. Icho chokwadi chekuti MPP dhatabhesi iri kugadzirirwa kuchengetedza chinhu inoratidza kuti iri kuwedzera kushandiswa seyekutanga kuchengetedza.

Kana yako yekuchengetedza system isingape kumhanya kwaunoda, haugone kushandisa iyo data uye kubvisa kukosha kubva mairi. Kunyangwe iwe ukadzoreredza data kubva kuchengetedzo yechinhu mu-mu-ndangariro yekugadzirisa chimiro, iwe uchazoda bandwidth kuendesa iyo data kuenda uye kubva mundangariro. Zvitoro zvenhaka hazvina kukwana nazvo.

Iyi ndiyo poindi yakakosha: iyo nyowani yekuita metric ndeye throughput, kwete latency. Inodiwa kune data pachiyero uye ndiyo yakajairwa mune yemazuva ano data infrastructure.

Nepo mabhenji ari nzira yakanaka yekuona mashandiro, haigone kuyerwa nemazvo usati wamhanyisa application munzvimbo. Chete mushure mekunge iwe unogona kutaura kuti ndepapi chaizvo iro bhodhoro riri: mune software, disks, network kana padanho rekombuta.

Kukwanisika

Scalability zvinoreva nhamba yemapetabytes anokwana munzvimbo imwe chete yezita. Zvinonzi nevatengesi zviri nyore scalability, zvavasingataure ndezvekuti sezvavanoyera, mahombe monolithic masisitimu anova asina kusimba, akaomarara, asina kugadzikana, uye anodhura.

Iyo metric nyowani ye scalability nhamba yemazita kana vatengi vaunogona kushandira. Iyo metric inotorwa zvakananga kubva ku hyperscalers, uko zvivharo zvekuvaka zvidiki asi zvakakura kusvika kumabhiriyoni emayuniti. Kazhinji, iyi igore metric.

Kana zvivharo zvekuvaka zvidiki, zviri nyore kukwidziridza kuchengetedzeka, kutonga kwekuwana, kutonga kwepolicy, kutenderera kwehupenyu, uye kusavhiringidza zvigadziriso. Uye pakupedzisira chengetedza kubereka. Kukura kwechivakwa chekuvaka ibasa rekudzora kwenzvimbo yekutadza, iyo ndiyo nzira inogadziriswa masisitimu anovakwa.

Multi-tenancy ine maitiro akawanda. Nepo iyo dimension ichitaura nezvekuti masangano anopa sei kuwana data uye maapplication, zvakare inoreva kune mashandisirwo acho uye pfungwa yekuzviparadzanisa kubva kune mumwe nemumwe.

Hunhu hwemazuva ano maitiro kune akawanda-vatengi:

• Munguva pfupi, nhamba yevatengi inogona kukura kubva pamazana akati wandei kusvika kumamiriyoni akati wandei.
• Vatengi vakaparadzaniswa zvachose kubva kune mumwe nemumwe. Izvi zvinovatendera kuti vamhanye akasiyana mavhezheni eiyo software uye kuchengetedza zvinhu zvine akasiyana masisitimu, mvumo, maficha, chengetedzo uye mazinga ekugadzirisa. Izvi zvinodikanwa kana uchiyera kumaseva matsva, zvigadziriso, uye geographies.
• Iyo yekuchengetera ndeye elastically scalable, zviwanikwa zvinopihwa pakuda.
• Kushanda kwega kwega kunodzorwa neAPI uye kunongoitwa pasina kupindira kwevanhu.
• Software inogona kugarwa mumidziyo uye kushandisa yakajairwa orchestration masisitimu akadai Kubernetes.

S3 inoenderana

Iyo Amazon S3 API ndiyo de facto chiyero chekuchengetedza chinhu. Yese chinhu chekuchengetedza software mutengesi anoti inoenderana nayo. Kuenderana neS3 ibhinari: ingave yakaitwa zvizere kana kuti haina.

Mukuita, kune mazana kana zviuru zvemapendero mamiriro apo chimwe chinhu chinokanganisa kana uchishandisa kuchengetedza chinhu. Kunyanya kubva kune vanopa proprietary software uye masevhisi. Makesi ayo makuru ekushandisa ndeaya archiving kana backup, saka pane zvikonzero zvishoma zvekufonera iyo API, makesi ekushandisa ane homogeneous.

Open source software ine zvakakosha zvakakosha. Iyo inovhara zvakanyanya kumucheto mamiriro, kupihwa saizi uye akasiyana ekushandisa, masisitimu anoshanda, uye hardware zvivakwa.

Zvese izvi zvakakosha kune vanogadzira maapplication, saka zvakafanira kuyedza application nevachengeti vekuchengetedza. Vhura sosi inoita kuti maitiro ave nyore-zviri nyore kunzwisisa kuti ndeipi puratifomu yakakodzera application yako. Mupi anogona kushandiswa senzvimbo imwe chete yekupinda mukuchengetedza, zvichireva kuti ichazadzisa zvaunoda. 

Open source zvinoreva: zvikumbiro hazvina kusungirirwa kune mutengesi uye zviri pachena. Izvi zvinovimbisa hupenyu hurefu hwekushandisa.

Uye mamwe mashoma manotsi nezve yakavhurika sosi uye S3. 

Kana iwe uchimhanyisa yakakura data application, S3 SARUDZA inovandudza mashandiro uye kugona nekurongeka kwehukuru. Inoita izvi nekushandisa SQL kutora chete zvinhu zvaunoda kubva mukuchengetedza.

Chinhu chakakosha kutsigira zviziviso zvebhaketi. Bucket zviziviso zvinofambisa serverless komputa, chinhu chakakosha chechero microservice architecture inounzwa sevhisi. Tichifunga kuti chengetedzo yechinhu inonyatso kuchengetedza gore, kugona uku kunova kwakakosha kana chengetedzo yechinhu ichishandiswa nemakore-based application.

Chekupedzisira, kuita kweS3 kunofanirwa kutsigira Amazon S3 server-side encryption APIs: SSE-C, SSE-S3, SSE-KMS. Kunyangwe zvirinani, S3 inotsigira tamper chengetedzo yakachengeteka zvechokwadi. 

Mhinduro kune kukundikana

Metric iyo inowanzo furatirwa ndeye mabatiro anoita system kukundikana. Kukundikana kunoitika nekuda kwezvikonzero zvakasiyana, uye kuchengetedza chinhu kunofanirwa kuzvibata zvese.

Semuenzaniso, pane imwe pfungwa yekutadza, metric yeiyi zero.

Nehurombo, masystem akawanda ekuchengetedza zvinhu anoshandisa node dzakakosha dzinofanirwa kugoneswa kuti cluster ishande nemazvo. Izvi zvinosanganisira mazita node kana metadata maseva - izvi zvinogadzira imwe pfungwa yekutadza.

Kunyangwe paine akawanda mapoinzi ekukundikana, kugona kutsungirira kutadza kunotyisa kwakakosha. Madhisiki anotadza, maseva anotadza. Chinokosha ndechekugadzira software yakagadzirirwa kubata kutadza seyakajairika mamiriro. Kana dhisiki kana node ikakundikana, software yakadaro icharamba ichishanda pasina shanduko.

Dziviriro yakavakirwa-mukati kubva mukudzimwa kwedata uye kushatiswa kwedata kunovimbisa kuti unogona kurasikirwa nemadhisiki akawanda kana node sezvaunenge uine parity blocks-kazhinji hafu yemadhisiki. Ipapo chete software ichatadza kudzosera data.

Iko kukundikana hakuwanzo kuedzwa pasi pemutoro, asi kuedza kwakadaro kunodiwa. Kutevedzera kutadza kwemutoro kucharatidza mutengo wakakwana wakaitwa mushure mekutadza.

Consistency

A consistency mamakisi e100% anonziwo kusimba kuenderana. Consistency chinhu chakakosha chechero system yekuchengetedza, asi kusimba kwakasimba hakuwanzo. Semuenzaniso, Amazon S3 ListObject haina kunyatsoenderana, inongoenderana pamagumo.

Chii chinorehwa nokunyatsoenderana? Kune ese mashandisirwo anotevera akasimbiswa PUT oparesheni, zvinotevera zvinofanirwa kuitika:

• Iko kukosha kwakagadziridzwa kunoonekwa kana uchiverenga kubva kune chero node.
• Iyo yekuvandudza inodzivirirwa kubva kune node kukundikana redundancy.

Izvi zvinoreva kuti kana iwe ukadhonza plug pakati pekurekodha, hapana chinorasika. Iyo sisitimu haidzoke data yakaora kana yechinyakare. Iyi ibhawa yakakwira iyo inokosha mune dzakawanda mamiriro, kubva kune transactional application kuenda kune backup uye kudzoreredza.

mhedziso

Aya ndiwo matsva ekuchengetedza zvinhu metrics anoratidza maitiro ekushandiswa mumasangano anhasi, uko kuita, kuenderana, scalability, kukanganisa madomasi uye kuenderana kweS3 ndizvo zvivharo zvekuvakisa zvekushandisa kwegore uye hombe data analytics. Ini ndinokurudzira kushandisa iyi runyorwa mukuwedzera kune mutengo pakuvaka yemazuva ano data stacks. 

Nezve Mail.ru Cloud Solutions kuchengetedza chinhu: S3 zvivakwa. 3 makore ekushanduka kweMail.ru Cloud Storage.

Zvimwe zvekuverenga:

1. Muenzaniso wechiitiko chinofambiswa nechiitiko chakavakirwa pawebhooks muS3 chinhu chekuchengetedza Mail.ru Cloud Solutions.
2. Kupfuura Ceph: MCS Cloud block kuchengetedza 
3. Kushanda neMail.ru Cloud Solutions S3 chinhu chekuchengetedza sefaira system.
4. Yedu yeTeregiramu chiteshi ine nhau dzekuvandudza kune S3 kuchengetedza uye zvimwe zvigadzirwa. 

Source: www.habr.com