TL; DR: Chikwata cheBloomberg Storage Injiniya chakagadzira kuchengetedza kwegore kuti zvishandiswe mukati izvo zvisingakanganise zvivakwa uye zvinokwanisa kumira kuremerwa kwekutengesa kusagadzikana panguva yedenda.
Mattew Leonard, kana achitaura nezvebasa rake semaneja wehunyanzvi pachikwata cheBloomberg Storage Engineering, anowanzoshandisa mazwi ekuti "kunetsa" uye "kunakidzwa." Matambudziko anomuka kubva pahupamhi hwekuchengetedza, kubva kune yazvino NVMe-yakavakirwa SAN arrays kuvhura sosi software-yakatsanangurwa chengetedzo muDevOps. Apa ndipo panotanga "kunakidzwa" (ona avatar yangu paHabrΓ©, approx. mushanduri).
Leonard nechikwata chake chevaanoshanda navo makumi maviri nevashanu vanotarisisa anopfuura zana petabytes ehukwanisi uye gore remukati remainjiniya zviuru zvitanhatu kugadzira zvikumbiro zveBloomberg Terminal, tekinoroji yakaita kuti Michael Bloomberg ave bhirioni. Chikwata chinogadzira, chinovaka uye chinochengetedza masisitimu ekuchengetera eBloomberg Engineering.
Kufanana nerimwe basa reIT, 2020 raive gore risina kujairika kunhengo dzeStorage Engineering timu sezvo COVID-19 yakavamanikidza kushanda kure. Leonard akati denda iri rakakanganisa "chikwata chake chakabatana" munharaunda sezvo kusangana kumeso kwakabviswa, asi vashandi vakajaira nekukurumidza kushanda kubva kumba pamalaptop uye musangano wevhidhiyo.
Zvinoshamisa, ndinoda kutaura kuti izvi hazvina kuita kuti zvinhu zvinyanye kuipa. Paive nechinguva chidiki chekugadzirisa - havasi vese vaive vakagadzirira kushanda kubva kumba. Pashure pevhiki kana maviri munhu wose akanzwisisa izvi. Takakwanisa kuwana nzira dzekuti tirambe takabatikana, kutenga nekusimudzira michina, uye kuwedzera mari yekutsigira kambani panguva idzi. Taifanira kugadzira, asi hatina kukuvara
Dambudziko guru rinogona kunge rakatangira pamusoro peCOVID-19. Izvi zvakakonzerwa nekusagadzikana kwekutengesa kwemusika nekuda kwekunetsekana nekukanganisa kwedenda pahupfumi hwenyika. Huwandu hwedhata inoyerera ichipinda muBloomberg terminals kubva kumisika mikuru yepasi rose yakapetwa kaviri, inosvika mazana maviri nemakumi mana ezviuru zvemashoko pane mamwe mazuva mukupera kwaKurume. Uyu muyedzo wakakomba wekuchengetedza masisitimu.
Paunongopeta kaviri zvaunoda kuchengetedza muzuva rimwe chete, zvinogadzira matambudziko anonakidza. Isu takakwanisa kukunda izvi uye nekuona kuti zvikwata zvekuvandudza maapplication zvakapihwa nzvimbo nekuita kwazvaida. Zvizhinji zveizvi zvine chekuita nemafungire atinoita nezvemagadzirirwo ekuchengetedza. Nhasi hatisi kugadzira chero chinhu. Hatiti, "Tinoshandisa ABC, saka tichavaka zvivakwa zveABC." Isu tinoita yatinodaidza kuti "dhata bhajeti" nezvikwata zvedu kufanotaura mashandisirwo, kuongorora mashandisirwo nemaitiro ekuita, uye isu tinotarisawo kuchengetedza. Iyi mhando yekuronga, kufunga, uye methodical kushingaira kwakafanira kunotibvumira kutora danho rakasimba pakuvhiya pasina kubuda dikita. Hongu, ndaitya, asi ndainzwa ndakasununguka kuva panzvimbo yangu.
Leonard nguva pfupi yadarika akataura neSearchStorage zvakadzama nezve manejimendi ekuchengetedza mabhizinesi anofambiswa nedata. Akakurukura kuti chii chaizotora kupa yakavanzika gore yekuchengetedza mhinduro, nekugona kupa maAWS maficha kune vashandisi vayo uchichengeta chero data muBloomberg data centers.
Kana kusisina denda, ndeapi matambudziko ane mainjiniya eBloomberg nekuchengetedza kuchengetedza?
Tine zvakawanda zvatinoda, tinongobvarurwa nenzira dzakasiyana. Saka isu tinofanirwa kupa akawanda akasiyana marudzi ezvigadzirwa pamatanho eSLA akasiyana kuti tibatsire vanogadzira maapplication kuti vatarise pamabasa avo pane kunetsekana nezve kuchengetera pachako.
Uye izano ripi raunotevera pane izvi?
Chikamu chezvatiri kuyedza kuita kugadzirisa mashandiro ekuchengetedza. Funga nezve modhi yeAWS apo injiniya wekusimudzira anofamba, odzvanya bhatani, uye ipapo "tinya" nemashiripiti anowana mhando yekuchengetera yakakodzera kugadzirisa dambudziko rake.
Mamiriro ako ekuchengetedza anotaridzika sei?
Nekuti isu tine akasiyana ecosystem uye akawanda akasiyana anovandudza, isu hatigone kupa chimwe chigadzirwa. Tine chinhu, faira uye block kuchengetedza. Izvi zvigadzirwa zvakasiyana uye isu tinopa akasiyana marudzi etekinoroji kuti azvibudise. Kune block tinoshandisa SAN. Isu tinewo SDS, iyo inopa imwe block kuchengetedza sarudzo ine akasiyana seti yekuita zvinodiwa. Kune mafaira tinoshandisa NFS. SDS inoshandiswawo kuchengetedza chinhu. Iyo block uye zvinhu zvikamu zvinoumba yemukati yakavanzika gore rekombuta uye kuchengetedza.
Saka haushandise ruzhinji kuchengetedza gore?
Izvo ndizvo. Zvimwe zvikwata zvebudiriro zvine mvumo yekushandisa makore eruzhinji. Asi nekuda kwechimiro chebhizinesi redu, isu tinosarudza kuve nekutonga kwakanyanya pamusoro pezvinhu zvinosiya madziro edu. Saka hongu, isu tine makore edu ari pasi pesimba redu. Iyi midziyo iri munzvimbo yedu yedata pasi pekutonga kwedu.
Munzvimbo dzedu dzedata, isu tinosarudza nzira yevakawanda-vatengesi. Ivo vatengesi vakakura, asi isu hatitaure kuti ndiani chaizvo (ndiyo mutemo weBloomberg wekusatsigira chero mutengesi, approx. mushanduri).
Iwe uri kushandisa hyperconverged zvivakwa kuvaka yako yakavanzika gore?
Aihwa. Isu paBloomberg tiri kusarudza gwara kwatisiri kuenda kune hyperconvergence. Tiri kuedza kudhirowa komputa kubva mudura kuti tikwanise kuayera takazvimiririra. Nzira yatiri kufamba mairi, kunyanya negore redu, ndeyekuti tikwanise kupatsanura zvinhu zviviri izvozvo. Uye zvese nekuti zvimwe zvinhu munyika yedu zvinoda kuverenga kwakadzama, nepo zvimwe zvinoda kuchengetedza. Kana ukaayera zvakaenzana, unorasikirwa nezviwanikwa, zvisinei nemari, kana nzvimbo munzvimbo dzedata, kana nekutenga huwandu hwausingade. Ndosaka isu tichida kuve neyakajairwa interface pakati pezviviri izvi, asi ita kuti ave akasiyana zvachose masisitimu uye anotungamirirwa nezvikwata zvakasiyana.
Ndezvipi zvipingamupinyi zvinofanirwa kukundwa kuvaka gore rakavanzika?
Dambudziko rechiyero. Sezvizhinji zvinhu, dhiabhori ari mune zvakadzama. Paunofunga nezve mashandisiro anoita zvinhu izvi, maitiro ekuita kuti zvigadzikane, mabatiro aunoita basa rekuita, matauriro aunoita nezvikwata zvemidziyo yenyama, zvinhu zvinowana kunakidza zvishoma. Dambudziko nderekutsvaga nzira yekuita kuti zvese zvive chigadzirwa chinokuvadza uye chinotsigirwa icho vagadziri vedu vekushandisa vangada kushandisa, kukwanisa kupfumisa chimiro chakaiswa uchiramba uri pamucheto wezviri kuitwa negore reruzhinji. Uye zvakare kuunza zvese pamwe chete kuti irambe ichishanda. Iri ndiro dambudziko redu guru - tinoshanda munzvimbo dzese dzebhizinesi, tichiedza kugutsa zvese zvinodiwa, asi tisingaregeredze zvimwe zvinodiwa.
Iwe unofunga iwe unoda izvo zvichangoburwa zvinowanikwa muAWS uye mamwe makore eruzhinji?
Chinhu chinonyanya kunakidza nezve S3 ndechekuti mwero wehupenyu unogara uchichinja, zvitsva zvinogara zvichiwedzerwa. Zvakafanana nechitoyi chitsva. Kana mumwe munhu akaona chinhu chitsva mukuburitswa kutsva, ivo vanochida. Haasi ese maAWS maficha anoshanda munharaunda yedu, saka zvakakosha uye zvinonakidza kuziva zvichabatsira vanogadzira uye kuti vangazviwana sei mumba.
Ndeipi midziyo yekuchengetedza yaunoshandisa?
Isu tinoshandisa midziyo yazvino. Gore redu remukati rakanyatsoenderana neNVMe Flash, inoita kuti masisitimu aya ave nesimba. Zvinoita kuti hupenyu hwedu huve nyore, uye zvakare chinhu chakanaka kune vagadziri vedu nekuti ivo havafanirwe kunetseka nezve kuchengetedza kuita.
Chii chaunoshandisa kuchengetedza chinhu?
Tine 6000 vagadziri vanoshanda pazvivakwa, ivo havana kubatana nechero imwe nyaya yekushandisa. Chero sarudzo yaungafunga nezvayo, isu tingangodaro tinayo mukuchengetedza chinhu. Zvimwe zvikwata zvinoishandisa kuchengetedza inotonhora, zvimwe zvekufambisa data, uye vamwe vanoishandisa kuita zvekutengesa. Ese aya makesi ekushandisa anoda mazinga akasiyana eSLA, saka sezvauri kuona, isu tine mhando dzakasiyana dzetraffic, marudzi ese ezvido zvevashandisi vakasiyana vezvivakwa zvedu. Iyi haisi homogeneous yekushandisa kesi inomhanya pamusoro pechero yedu yekuchengetedza, izvo zviri pachena zvinoita kuti zvinhu zviome.
Ibasa rakakura sei rinoitwa naKubernetes nemidziyo kwauri, uye izvo zvinokanganisa sei kuchengetedza?
Tiri kusundidzira kugadzirwa kwekuchengetedza kuti tigadzire pfungwa yegore, pfungwa yechimwe chinhu-se-sevhisi, uko kune bhatani revagadziri kuti vakurumidze hunyanzvi hwavo uye kubvisa zvivakwa munzira.
Mupepeti n.b.: Gumiguru 15, 2020 ichave yakagadzirira . Iwe unozodzidza Ceph network yekuchengetedza tekinoroji yekushandisa mumapurojekiti ako kuvandudza kukanganisa kushivirira.
Tine zvikwata zvitatu, yekutanga ndiyo yekuchengetedza API timu. Ivo vanoita chirongwa chekuwana, magumo, uye akafanotsanangurwa mafambiro evashandisi vekuvandudza app kuBloomberg. Ichi ichikwata chevakazara webhu vanogadzira, vanoshandisa node.js, python, yakavhurika sosi matekinoroji, akadai seApache Airflow, saka vanodzidza midziyo uye virtualization.
Isu tine zvikwata zviviri zvehunyanzvi zvinonyatso fambisa mabhiti nemabhayiti. Ivo vane hukama zvakananga nemidziyo. Tine michina yakawanda, uye zvikwata izvi hazvishandise virtualization uye midziyo.
Tiri kuyedza kufambirana nezviri kuitika muindastiri, tichidzidza madhiraivha eKubernetes CSI, uye tichishanda pamwe chete nechikwata chiri kuita Kubernetes kuBloomberg kuona kana tichigona kuita kuti Kubernetes chengetedzo ishande inoenderana nehunyanzvi hwatinahwo, uye tine. inoshanda . Isu tinoshandisa SDS kutsigira Kubernetes yakabatana kune inoenderera kuchengetedza. Takabudirira kugadzira tekinoroji iyi, uye nhaurirano dziri kuenderera mberi pakati pezvikwata zviviri izvi nezvekuti tingaite sei kuti izvi zviwanikwe kune wese munhu paBloomberg. Taratidza kuti izvi zvinogoneka.
Ndeipi imwe open source software yaunoshandisa, kunyanya yekuchengetedza?
Isu tinoshandisa Apache Airflow, HAProxy kudzikamisa application traffic. Isu tinoshandisawo Ceph, chikuva cheSDS. Nayo, iwe unogona kuve nehurongwa humwe hwemirairo, asi ipa akawanda mainterface kune vatengi. Imwe yemapuratifomu ekuona inomhanya paOpenStack - tinoshanda padhuze nechikwata ichi. Isu tine yakavhurika sosi virtualization chikuva inoshandisa yakavhurika sosi SDS chikuva chekuchengetedza. Zvinosekesa.
Ndeapi matekinoroji ekuchengetedza auri kufunga kwemakore maviri kusvika matatu anotevera?
Isu tinogara tichitarisa mune zvimwe zvinhu zvinotonhorera zviri kuitika muindasitiri yekuchengetedza. Ichi chikamu chebasa redu, harisi "heino SAN wako, tonga pano, uye heino NFS yako, tonga ipapo." Tinoedza kutaurirana nevatengi vedu, i.e. nevagadziri vedu vekushandisa. Isu tinoshanda pamwe chete kuti tinzwisise kuti ndeapi matambudziko avari kuyedza kugadzirisa uye kuti zvichakanganisa sei vatengi vedu veBloomberg vekunze - mabhangi nevamwe vanoshandisa software yedu. Uye tobva tadzokera munyika yekuchengetedza data kuti tiwane mikana yekuvabatsira kuzadzisa chinangwa chavo. Tingavabatsira sei kuwana tekinoroji yekuchengetera inoenderana neSLA yavo kana zvavari kuedza kuita? Nekuti isu tine akawanda mainjiniya anoita zvinhu zvinotonhorera, hazvifi zvinofinha.
Parizvino tiri kutsvaga nzira dzekuvandudza mashandiro eSDS ayo anogona kumhanyisa pane zvakajairwa chinangwa maseva. Saka tiri kushanda paNVMe pamusoro peTCP, iyi inonakidza uye inotonhorera danho, imwe yeakawanda. Isu tiri kushandawo nevanhu vakakosha muindasitiri uye vamwe vatengesi varipo kuti tizive zvavanopa uye kuti kuita chaiko kuchave chii, kana isu tichigona kutanga kuishandisa mukugadzira mukambani. Izvi zvinovhura mahorizons matsva akange asati awanikwa.
Rubatsiro rudiki muPS
PS Kana ndikagona, ndinoda kukuyeuchidza kuti September 28-30 ichaitwa , kune avo vasingazivi Kubernetes, asi vanoda kujairana nayo uye kutanga kushanda nayo.
Source: www.habr.com