Mazano nemisangano nezve mamwe maitiro anogona kuve otomatiki anomuka mumabhizinesi ehukuru hwakasiyana mazuva ese. Asi nekuwedzera kune chokwadi chekuti nguva yakawanda inogona kushandiswa pakugadzira modhi, unofanirwa kuipedza uchiiongorora uye kutarisa kuti mhedzisiro yacho haina kurongeka. Mushure mekuita, chero modhi inofanirwa kuongororwa uye nguva nenguva kuongororwa.
Uye aya ndiwo ese matanho aunofanirwa kupfuura mune chero kambani, zvisinei nehukuru hwayo. Kana isu tichitaura pamusoro pechiyero uye nhaka yeSberbank, nhamba yezvakanaka-tunings inowedzera kuwedzera. Pakupera kwa2019, Sberbank yakanga yatoshandisa anopfuura zviuru zviviri zvemhando. Izvo hazvina kukwana kungogadzira modhi, zvinodikanwa kubatanidza nemaindasitiri masisitimu, kugadzira data mats emhando dzekuvaka, uye kuve nechokwadi chekutonga kwekushanda kwayo pasumbu.
Chikwata chedu chiri kugadzira Sber.DS chikuva. Iyo inokutendera kuti ugadzirise matambudziko ekudzidza muchina, inomhanyisa maitiro ekuyedza hypotheses, musimboti inorerutsa maitiro ekugadzira uye kusimbisa modhi, uye zvakare inodzora mhedzisiro yemuenzaniso muPROM.
Kuti urege kunyengedza zvaunotarisira, ndinoda kutaurira mberi kuti iyi chinyorwa chiziviso, uye pasi pekucheka, kwekutanga, inoudzwa pamusoro pezvinonyanya pasi pehodhi yeSber.DS platform. Isu tichataura nyaya nezve kutenderera kwehupenyu hwemuenzaniso kubva pakusikwa kusvika pakuitwa zvakasiyana.
Sber.DS ine zvikamu zvakati wandei, zvakakosha kuve raibhurari, hurongwa hwekuvandudza, uye maitiro ekuita maitiro.
Raibhurari inodzora kutenderera kwehupenyu hwemuenzaniso kubva panguva iyo pfungwa yekuvandudza inoratidzika kusvika pakuitwa kwayo muPROM, kutarisa uye kubvisa. Mazhinji maficha eraibhurari anotemerwa nemitemo yemutongi, semuenzaniso, kushuma uye kuchengetwa kwekudzidziswa uye masampula ekusimbisa. Muchokwadi, iyi irejista yemamodheru edu ese.
Iyo yekuvandudza sisitimu inoitirwa kusimudzira kwekuona kwemamodhi uye nzira dzekusimbisa. Iwo mamodheru akagadziridzwa anoenda kusimbiswa kwekutanga uye anounzwa kune ekuuraya sisitimu kuti vaite bhizinesi ravo mabasa. Zvakare, muhurongwa hwekuuraya, modhi inogona kuiswa pachitarisiko kuitira kuti nguva nenguva itange nzira dzekusimbisa kudzora kushanda kwayo.
Kune akati wandei marudzi emanodhi muhurongwa. Mamwe akagadzirirwa kubatanidza kune akasiyana data masosi, mamwe - kushandura iyo sosi data uye kuipfumisa (markup). Kune akawanda ma node ekuvaka akasiyana mamodheru uye node yekusimbisa kwavo. Iyo inovandudza inogona kurodha data kubva kune chero masosi, shandura, sefa, kuona yepakati data, kupatsanura kuita zvikamu.
Ipuratifomu zvakare ine akagadzirira-akagadzirwa mamodule anogona kudhonzerwa munzvimbo yeprojekiti. Zvese zviito zvinoitwa uchishandisa inooneka interface. Muchokwadi, iwe unogona kugadzirisa dambudziko pasina mutsara wekodhi.
Kana iyo yakavakirwa-mukati kugona isina kukwana, saka sisitimu inopa kugona kukurumidza kugadzira yako mamodule. Isu takagadzira yakasanganiswa yekuvandudza modhi yakavakirwa pa kune avo vanogadzira mamodule matsva kubva kutanga.
Iyo Sber.DS architecture yakavakirwa pamicroservices. Kune akawanda maonero ekuti chii chinonzi microservices. Vamwe vanhu vanofunga kuti zvakakwana kupatsanura iyo monolithic kodhi kuita zvikamu, asi ivo vachiri kuenda kune imwechete database. Yedu microservice inofanirwa kutaurirana neimwe microservice chete kuburikidza neREST API. Hapana maworkaround ekuwana iyo database zvakananga.
Isu tinoedza kuita kuti masevhisi asave akakura uye ane usimbe: chiitiko chimwe chete hachifanire kushandisa anopfuura 4-8 gigabytes ye RAM uye inofanirwa kukwanisa kuyera yakachinjika zvikumbiro nekutanga zviitiko zvitsva. Sevhisi yega yega inotaurirana nevamwe chete kuburikidza neREST API () Chikwata chinotarisira sevhisi chinodiwa kuchengetedza API kumashure inoenderana kusvika mutengi wekupedzisira anoishandisa.
Iyo yakakosha yekushandisa yakanyorwa muJava uchishandisa iyo Spring Framework. Mhinduro yacho yakatanga kugadzirwa kuti ikurumidze kuendeswa muhurongwa hwegore, saka chishandiso chinovakwa pachishandiswa mudziyo wekutakura. () Iyi puratifomu inogara ichishanduka, zvese maererano nekuwedzera bhizinesi mashandiro (zvitsva zvinongedzo, AutoML inowedzerwa), uye maererano nehunyanzvi hwekuita tekinoroji.
Imwe ye "chips" yepuratifomu yedu ndeyekuti isu tinokwanisa kumhanya kodhi yakagadziridzwa mune inotaridzika interface pane chero Sberbank modhi execution system. Ikozvino kwatove nevaviri vavo: imwe paHadoop, imwe paOpenShift (Docker). Isu hatimire ipapo uye tinogadzira ekubatanidza mamodule ekumhanyisa kodhi pane chero zvivakwa, kusanganisira pa-nzvimbo uye mugore. Panyaya yemikana yekubatanidzwa kunobudirira muSberbank ecosystem, isu tinorongawo kutsigira basa nenzvimbo dziripo dzekumhanya. Mune ramangwana, mhinduro inogona kuchinjika yakabatanidzwa "kunze kwebhokisi" mune chero mamiriro echero sangano.
Avo vakamboedza kuchengetedza mhinduro inomhanya Python paHadoop muPROM vanoziva kuti hazvina kukwana kugadzirira uye kuendesa tsika yepython nharaunda kune imwe neimwe datanode. Huwandu hukuru hweC / C ++ maraibhurari ekudzidza muchina anoshandisa maPython modules haakusiye iwe kuzorora murunyararo. Hatifanire kukanganwa kugadzirisa mapakeji kana tichiwedzera maraibhurari matsva kana maseva, tichichengetedza kuenderana nekudzokera kumashure nekodhi yemhando yakatoitwa.
Pane nzira dzakawanda dzekuita izvi. Semuenzaniso, gadzirira maraibhurari akati wandei anowanzo shandiswa pachine nguva uye woashandisa muPROM. Cloudera's Hadoop kugovera kunowanzo shandisa . Zvakare ikozvino muHadoop pane mukana wekumhanya - midziyo. Mune zvimwe zviitiko zviri nyore zvinokwanisika kuendesa iyo kodhi pamwe chete nepakeji .
Bhengi inotora chengetedzo yekumhanyisa yechitatu-bato kodhi zvakanyanya, saka isu tinoshandisa zvakanyanya zvezvinhu zvitsva zveLinux kernel, uko maitiro ari kushanda munzvimbo iri kure. , unogona kurambidza, somuenzaniso, kupinda kune network uye disk yemunharaunda, iyo inoderedza zvikuru mukana wekodhi yakaipa. Dhipatimendi rega rega nzvimbo dzedata dzakachengetedzwa uye dzinongowanikwa kune varidzi veiyo data. Iyi puratifomu inovimbisa kuti data kubva kune imwe dura inogona chete kupinda mune imwe dura kuburikidza nedhisheni yekushambadzira maitiro nekutonga pamatanho ese kubva pakuwana masosi kusvika kune yekumhara data mune inotangwa mart.
Gore rino isu tinoronga kupedzisa iyo MVP yeanomhanya mamodheru akanyorwa muPython/R/Java paHadoop. Isu takazvimisikidza iro rinoshuvira basa rekudzidza maitiro ekumhanyisa chero mushandisi nharaunda paHadoop, kuti tirege kudzikamisa vashandisi vepuratifomu yedu chero nzira.
Uye zvakare, sezvazvakazoitika, nyanzvi zhinji dzeDS dzakanakisisa pamasvomhu nenhamba, dzinogadzira modhi dzinotonhorera, asi hadzina ruzivo rwakanyanya mukushandurwa kukuru kwedata, uye vanoda rubatsiro rweinjiniya yedu yedata kugadzirira masampuli ekudzidziswa. Isu takasarudza kubatsira vatinoshanda navo uye kugadzira ari nyore mamodule eshanduko yakajairwa uye gadziriro yezvimiro zvemhando pane Spark injini. Izvi zvinobvumira nguva yakawanda yekupihwa mukugadzira mamodheru uye kwete kumirira mainjiniya edata kuti vagadzirire dataset nyowani.
Tine vanhu vane ruzivo munzvimbo dzakasiyana: Linux uye DevOps, Hadoop uye Spark, Java uye Spring, Scala neAkka, OpenShift uye Kubernetes. Nguva inotevera tichazotaura nezve raibhurari yemamodheru, kuti modhi inofamba sei nehupenyu mukati mekambani, kuti kusimbiswa uye kuita kunoitika sei.
Source: www.habr.com