TL; DR: Gulu la Bloomberg Storage Engineering lidapanga malo osungira mitambo kuti azigwiritsa ntchito mkati zomwe sizimasokoneza zomangamanga ndipo zimatha kupirira kusakhazikika kwa malonda panthawi ya mliri.
Matt Leonard, akamalankhula za ntchito yake monga manejala waukadaulo mu gulu la Bloomberg Storage Engineering, nthawi zambiri amagwiritsa ntchito mawu oti "zovuta" ndi "zosangalatsa." Zovuta zimabwera kuchokera kuzinthu zambiri zosungirako, kuchokera kumagulu aposachedwa kwambiri a NVMe-based SAN kuti atsegule gwero lokhazikitsidwa ndi mapulogalamu a DevOps. Apa ndipamene "zosangalatsa" zimayambira (onani avatar yanga pa HabrΓ©, pafupifupi. womasulira).
Leonard ndi gulu lake la anzake 25 amayang'anira mphamvu zopitirira ma petabytes 100 komanso mtambo wamkati wa mainjiniya 6000 omwe akupanga mapulogalamu a Bloomberg Terminal, ukadaulo womwe udapangitsa Michael Bloomberg kukhala mabiliyoni. Gululi limapanga, kumanga ndi kusunga makina osungira a Bloomberg Engineering.
Monga ntchito ina yonse ya IT, 2020 chinali chaka chachilendo kwa mamembala a gulu la Storage Engineering popeza COVID-19 idawakakamiza kugwira ntchito kutali. Leonard adati mliriwu udakhudzanso "timu yake yolumikizana kwambiri" pomwe kucheza kumaso ndi maso kudathetsedwa, koma ogwira ntchito adazolowera mwachangu kuti azigwira ntchito kunyumba pakompyuta ndi pamisonkhano yamakanema.
Modabwitsa, ndikufuna kunena kuti izi sizinapangitse zinthu kuipiraipira. Panali nthawi yochepa yosinthira - si onse omwe anali okonzeka kugwira ntchito kunyumba. Pambuyo pa sabata kapena ziwiri aliyense adamvetsetsa izi. Tinatha kupeza njira zokhalira otanganidwa, kugula ndi kukweza zida, ndikuwonjezera ndalama zothandizira kampani panthawiyi. Tinkayenera kuchita zinthu mwanzeru, koma sitinavulale
Vuto lalikulu litha kukhala lisanachitike pachimake cha COVID-19. Izi zidachitika chifukwa chakusokonekera kwa malonda amsika chifukwa cha nkhawa zomwe zakhudzidwa ndi mliri wapadziko lonse lapansi. Kuchuluka kwa zidziwitso zomwe zikuyenda m'malo a Bloomberg kuchokera kumisika yayikulu padziko lonse lapansi zidakwera pafupifupi kuwirikiza kawiri, kufika pazidziwitso 240 biliyoni masiku ena kumapeto kwa Marichi. Ichi ndi kuyesa kwakukulu kwa machitidwe osungira.
Mukangowonjezera zomwe mukufuna kusunga tsiku limodzi, zimadzetsa mavuto osangalatsa. Tinatha kuthana ndi izi ndikuwonetsetsa kuti magulu opititsa patsogolo ntchito amapatsidwa malo ndi magwiridwe antchito omwe amafunikira. Zambiri mwa izi zimakhudzana ndi momwe timaganizira zosungirako. Lero sitikupanga kalikonse. Sitikunena kuti, "Timagwiritsa ntchito ABC, kotero tidzamanga maziko a ABC." Timachita zomwe timatcha "data budgeting" ndi magulu athu kulosera za kagwiritsidwe ntchito, kusanthula kagwiritsidwe ntchito ndi kachitidwe, komanso timayang'ana chitetezo. Kukonzekera kotereku, kulingalira, ndi kulimbikira koyenera kumatilola kuchitapo kanthu pakuchita maopaleshoni popanda kutuluka thukuta. Inde, ndinali ndi mantha, koma ndinamva kukhala womasuka kukhala mβmalo mwanga.
Leonard posachedwapa adalankhula ndi SearchStorage mwatsatanetsatane za kuyang'anira kusungirako mabizinesi oyendetsedwa ndi data. Anakambirana zomwe zingatengere kuti apereke njira yosungiramo mitambo yachinsinsi, ndi kuthekera kopereka mawonekedwe a AWS kwa ogwiritsa ntchito pamene akusunga deta iliyonse mu Bloomberg data centers.
Ngati kulibenso mliri, kodi mainjiniya a Bloomberg ali ndi zovuta zotani pakuwongolera zosungirako?
Tili ndi zosowa zambiri, timangong'ambika mbali zosiyanasiyana. Chifukwa chake tikuyenera kupereka mitundu yambiri yazogulitsa pamagawo osiyanasiyana a SLA kuti tithandizire opanga mapulogalamu athu kuyang'ana kwambiri ntchito zawo m'malo modandaula ndi zosungirako zokha.
Nanga mumatsatira njira yanji pa izi?
Zina mwa zomwe tikuyesera kuchita ndikuwongolera magwiridwe antchito. Ganizirani za chitsanzo cha AWS kumene injiniya wachitukuko amalowamo, akusindikiza batani, ndiyeno "dinani" mwamatsenga amapeza mtundu woyenera wosungirako kuti athetse vuto lake.
Kodi malo anu osungira amawoneka bwanji?
Chifukwa tili ndi chilengedwe chosiyanasiyana komanso opanga osiyanasiyana, sitingathe kupereka chinthu chimodzi. Tili ndi chinthu, fayilo ndi block block. Izi ndizinthu zosiyanasiyana ndipo timapereka mitundu yosiyanasiyana yaukadaulo kuti tipereke. Pa block timagwiritsa ntchito SAN. Tilinso ndi SDS, yomwe imapereka njira ina yosungiramo chipika yokhala ndi zofunikira zosiyanasiyana zogwirira ntchito. Kwa mafayilo timagwiritsa ntchito NFS. SDS imagwiritsidwanso ntchito posungira zinthu. Ma block ndi zinthu zimapanga mtambo wachinsinsi wamkati wamakompyuta ndi kusungirako.
Ndiye simugwiritsa ntchito posungira mitambo pagulu?
Ndichoncho. Magulu ena achitukuko ali ndi chilolezo chogwiritsa ntchito mitambo yapagulu. Koma chifukwa cha chikhalidwe cha bizinesi yathu, timakonda kukhala ndi ulamuliro pa zinthu zomwe zimasiya makoma athu. Kotero inde, tili ndi mitambo yathu yomwe ili pansi pa ulamuliro wathu. Izi ndi zida zomwe zili mu data center yathu pansi pa kasamalidwe kathu.
M'malo athu a data, timakonda njira yopangira mavenda ambiri. Ndi ogulitsa akuluakulu, koma sitinena kuti ndani kwenikweni (ndi lamulo la Bloomberg kuti asavomereze ogulitsa aliyense, pafupifupi. womasulira).
Kodi mukugwiritsa ntchito hyperconverged zomangamanga kuti mupange mtambo wanu wachinsinsi?
Ayi. Ife ku Bloomberg tikusankha komwe sitikupita ku hyperconvergence. Tikuyesa kuchulutsa ma compute kuchokera ku zosungirako kuti tithe kuziwonjeza paokha. Momwe tikuyenda, makamaka ndi mtambo wathu, ndikuti tithe kusiyanitsa magulu awiriwa. Ndipo zonse chifukwa zinthu zina m'dziko lathu zimafuna kuwerengera kwambiri, pomwe zina zimafunikira kusungidwa. Mukawakulitsa mofanana, mudzataya chuma, ziribe kanthu ndalama, kapena malo m'malo opangira deta, kapena pogula mphamvu zomwe simukuzifuna. Ichi ndichifukwa chake timakonda kukhala ndi mawonekedwe ofanana pakati pa mabungwe awiriwa, koma akhale ndi machitidwe osiyana kotheratu ndikuyendetsedwa ndi magulu osiyanasiyana.
Ndi zopinga ziti zomwe ziyenera kugonjetsedwa kuti mupange mtambo wachinsinsi?
Vuto la sikelo. Mofanana ndi zinthu zambiri, mdierekezi ali mwatsatanetsatane. Mukamaganizira za momwe zinthuzi zimagwirira ntchito, momwe mungapangire kuti zikhale zolimba, momwe mungagwirire ntchito yogwira ntchito, momwe mumalankhulirana ndi magulu azinthu zakuthupi, zinthu zimakhala zosangalatsa pang'ono. Vuto ndikupeza njira yopangira chilichonse kuti chikhale chosavuta komanso chothandizira chomwe opanga mapulogalamu athu angafune kugwiritsa ntchito, ndikutha kulemeretsa zomwe zakhazikitsidwa pomwe mukukhalabe m'mphepete mwazomwe mitambo yapagulu ikuchita. Komanso kubweretsa zonse pamodzi kuti zipitirize kugwira ntchito. Ili ndiye vuto lathu lalikulu - timagwira ntchito kumadera onse abizinesi, kuyesera kukwaniritsa zosowa zonse, koma osanyalanyaza zosowa zina.
Kodi mukuganiza kuti mukufuna zaposachedwa kwambiri mu AWS ndi mitambo ina yapagulu?
Chosangalatsa kwambiri pa S3 ndikuti moyo umasintha nthawi zonse, zatsopano zimawonjezeredwa nthawi zonse. Zili ngati chidole chatsopano. Ngati wina awona chinthu chatsopano pakumasulidwa kwatsopano, akufuna. Sizinthu zonse za AWS zomwe zimagwira ntchito m'malo athu, kotero ndikofunikira komanso kosangalatsa kudziwa zomwe zingathandize omanga komanso momwe angazipezere mnyumba.
Mumagwiritsa ntchito zida zotani zosungiramo zinthu?
Timagwiritsa ntchito zipangizo zamakono. Mtambo wathu wamkati umachokera ku NVMe Flash, zomwe zimapangitsa kuti machitidwewa akhale amphamvu kwambiri. Zimapangitsa moyo wathu kukhala wosavuta pang'ono, komanso ndi chinthu chabwino kwa opanga athu chifukwa sayenera kuda nkhawa ndi momwe kusungirako kumagwirira ntchito.
Kodi mumagwiritsa ntchito chiyani posungira zinthu?
Tili ndi opanga 6000 omwe akugwira ntchito pazomangamanga, samalumikizana ndi vuto lililonse. Njira iliyonse yomwe mungaganizire, mwina tili nayo posungira zinthu. Magulu ena amawagwiritsa ntchito posungira zakale, ena potumiza deta, ndipo ena amawagwiritsa ntchito pochita malonda. Zonsezi zimagwiritsidwa ntchito zimafuna magawo osiyanasiyana a SLA, monga momwe mukuonera, tili ndi mitundu yosiyanasiyana ya magalimoto, mitundu yonse ya zosowa za ogwiritsa ntchito osiyanasiyana a zomangamanga zathu. Iyi si njira yogwiritsira ntchito yofanana yomwe ikuyenda pamwamba pa zosungira zathu zilizonse, zomwe mwachiwonekere zimapangitsa kuti zinthu zikhale zovuta kwambiri.
Kodi Kubernetes ndi makontena ali ndi gawo lalikulu bwanji kwa inu, ndipo izi zimakhudza bwanji kusunga?
Tikukankhira zokolola zosungirako kuti tipange mtambo, kumverera kwa chinachake-monga-ntchito, pomwe pali batani la omanga kuti afulumizitse luso lawo ndikuchotsa zomangamanga panjira.
Mkonzi wa NB: October 15, 2020 adzakhala okonzeka . Muphunzira ukadaulo wa Ceph network yosungirako kuti mugwiritse ntchito muma projekiti anu kuti muwongolere zolakwika.
Tili ndi magulu atatu, yoyamba ndi gulu la API yosungirako. Amapanga mwayi wofikira mwadongosolo, malekezero, ndi mayendedwe ofotokozedweratu amakasitomala opititsa patsogolo mapulogalamu ku Bloomberg. Ili ndi gulu la omanga mawebusayiti ambiri, amagwiritsa ntchito node.js, python, matekinoloje otseguka, monga Apache Airflow, kotero amaphunzira zotengera ndi virtualization.
Tilinso ndi magulu awiri aukadaulo omwe amasuntha ma bits ndi ma byte. Zimagwirizana kwambiri ndi zida. Tili ndi zida zambiri, ndipo maguluwa sagwiritsa ntchito virtualization ndi zotengera.
Tikuyesera kuti titsatire zomwe zikuchitika m'makampani, kuphunzira madalaivala a Kubernetes CSI, ndikugwiranso ntchito limodzi ndi gulu lomwe likugwiritsa ntchito Kubernetes ku Bloomberg kuti tiwone ngati titha kupanga Kubernetes kuti agwire ntchito mosasinthasintha ndi matekinoloje omwe tili nawo, ndipo tili nawo. zikugwira ntchito. Timagwiritsa ntchito SDS kuthandizira Kubernetes yolumikizidwa ndi kusungidwa kosalekeza. Tapanga bwino lusoli, ndipo zokambirana zikupitilira pakati pa magulu awiriwa za momwe tingapangire kuti izi zitheke kwa wina aliyense ku Bloomberg. Tawonetsa kuti izi ndizotheka.
Ndi mapulogalamu ena ati otseguka omwe mumagwiritsa ntchito, makamaka posungira?
Timagwiritsa ntchito Apache Airflow, HAProxy kuti tichepetse kuchuluka kwa magalimoto. Timagwiritsanso ntchito Ceph, nsanja ya SDS. Ndi iyo, mutha kukhala ndi dongosolo limodzi lamalamulo, koma perekani mawonekedwe angapo kwa makasitomala. Imodzi mwamapulatifomu owoneka bwino imayenda pa OpenStack - timagwira ntchito limodzi ndi gululi. Tili ndi nsanja yotsegulira gwero lotseguka lomwe limagwiritsa ntchito nsanja yotseguka ya SDS posungira. Ndizoseketsa.
Ndi njira ziti zosungira zomwe mukuganizira zaka ziwiri kapena zitatu zikubwerazi?
Nthawi zonse timayang'ana zinthu zina zatsopano zomwe zikuchitika m'makampani osungira zinthu. Ili ndi gawo la ntchito yathu, si "pano pali SAN wanu, samalirani apa, ndipo nayi NFS yanu, samalirani pamenepo." Timayesa kulankhulana ndi makasitomala athu, i.e. ndi opanga mapulogalamu athu. Timagwira ntchito limodzi kuti timvetsetse mavuto omwe akuyesera kuthetsa komanso momwe angakhudzire makasitomala athu akunja a Bloomberg - mabanki ndi ena omwe amagwiritsa ntchito mapulogalamu athu. Kenako timabwerera kudziko losungirako deta kuti tipeze mipata yowathandiza kukwaniritsa cholinga chawo. Kodi tingawathandize bwanji kupeza ukadaulo woyenera wosungira zinthu womwe umagwirizana ndi SLA yawo kapena zomwe akuyesera kuchita? Chifukwa tili ndi mainjiniya ambiri omwe akuchita zinthu zabwino, sizikhala zotopetsa.
Pakali pano tikuyang'ana njira zosinthira magwiridwe antchito a SDS omwe atha kuyendetsedwa ndi maseva a zolinga wamba. Chifukwa chake tikugwira ntchito pa NVMe pa TCP, iyi ndi njira yosangalatsa komanso yabwino, imodzi mwambiri. Tikugwiranso ntchito ndi anthu ofunikira m'makampani ndi ena mwa ogulitsa omwe alipo kuti adziwe zomwe amapereka komanso zomwe zidzachitike, ngati tingayambe kuzigwiritsa ntchito popanga kampani. Izi zimatsegula mawonekedwe atsopano omwe sanapezekepo kale.
Thandizo pang'ono mu PS
PS Ngati ndingathe, ndikufuna ndikukumbutseni kuti September 28-30 idzachitika , kwa iwo omwe sakudziwa Kubernetes, koma akufuna kudziwana nawo ndikuyamba kugwira nawo ntchito.
Source: www.habr.com