Molweni nonke, igama lam nguAlexander, kwaye ndiyinjineli yoMgangatho weDatha ojonga idatha yomgangatho wayo. Eli nqaku liza kuthetha malunga nokuba ndeza njani kule nto kwaye kutheni ngo-2020 le ndawo yovavanyo yayikwindawo yovavanyo.
Intsingiselo yehlabathi
Ihlabathi lanamhlanje lijongene nenye inguqu kwezobuchwepheshe, enye inkalo kukusetyenziswa kwedatha eqokelelweyo zizo zonke iintlobo zeenkampani ukukhuthaza i-flywheel yazo yokuthengisa, ingeniso kunye ne-PR. Kubonakala ngathi ubukho beenkcukacha ezilungileyo (ezisemgangathweni), kunye nobuchopho obunobuchule obunokwenza imali kuyo (inkqubo echanekileyo, imbono, ukwakha imodeli yokufunda koomatshini, njl.), iye yaba ngundoqo kwimpumelelo yabaninzi namhlanje. Ukuba i-15-20 kwiminyaka edlulileyo iinkampani ezinkulu bezibandakanyeka kakhulu kumsebenzi onzima kunye nokuqokelela idatha kunye nokwenza imali, namhlanje oku kuninzi phantse bonke abantu abanengqondo.
Kule nkalo, kwiminyaka eliqela eyadlulayo, zonke iiphothali ezinikezelwe kukhangelo lwemisebenzi kwihlabathi liphela zaqala ukuzaliswa zizithuba zeeNzululwazi zeDatha, kuba wonke umntu wayeqinisekile ukuba, xa eqeshe ingcali enjalo, kuya kwenzeka ukwakha i-supermodel yokufunda koomatshini. , qikelela ikamva kwaye wenze "i-quantum leap" yenkampani. Ngokuhamba kwexesha, abantu baqaphela ukuba le ndlela phantse ayisebenzi naphi na, kuba ayizizo zonke iinkcukacha eziwela ezandleni zeengcali ezinjalo ezifanelekileyo kwiimodeli zoqeqesho.
Kwaye izicelo ezivela kwiiNzululwazi zeDatha zaqala: "Masithenge idatha engakumbi kwezi kunye nezo...", "Asinayo idatha yaneleyo...", "Sifuna enye idatha, ngokukodwa ekumgangatho ophezulu..." . Ngokusekwe kwezi zicelo, unxibelelwano oluninzi luqale ukwakhiwa phakathi kweenkampani eziphethe enye okanye iseti yedatha. Ngokwemvelo, oku kufuna umbutho wobugcisa wale nkqubo - ukudibanisa kumthombo wedatha, ukuyikhuphela, ukujonga ukuba ilayishwe ngokupheleleyo, njl. Inani leenkqubo ezinjalo zaqala ukukhula, kwaye namhlanje sinesidingo esikhulu solunye uhlobo. iingcali - Iinjineli zoMgangatho weDatha - abo baya kubeka iliso ekuhambeni kwedatha kwinkqubo (imibhobho yedatha), umgangatho wedatha kwigalelo kunye nesiphumo, kwaye bafikelele kwizigqibo malunga nokwanela kwabo, ingqibelelo kunye nezinye iimpawu.
Umkhwa weenjineli zoMgangatho weDatha weza kuthi zivela e-USA, apho, phakathi kwexesha elivuthayo lobungxowankulu, akukho mntu ulungele ukuphulukana nedabi ledatha. Apha ngezantsi ndibonelele ngezikrini ukusuka kwiindawo ezimbini ezidumileyo zokukhangela umsebenzi e-US: ΠΈ - ebonisa idatha ukususela ngo-Matshi 17, 2020 kwinani lezithuba ezithunyelwe ezifunyenwe kusetyenziswa amagama angundoqo: Umgangatho weDatha kunye neNzululwazi yeDatha.
IiNzululwazi zeDatha - izithuba ze-21416
Umgangatho wedatha - 41104 izithuba
IiNzululwazi zeDatha - izithuba ezingama-404
Umgangatho weDatha - izithuba ze-2020
Ngokucacileyo, ezi misebenzi azikhuphisani. Ngemifanekiso-skrini, bendifuna nje ukubonisa imeko ekhoyo ngoku kwimarike yabasebenzi ngokwezicelo zeenjineli zoMgangatho weDatha, ekufuneka uninzi lwazo ngoku kuneeNzululwazi zeDatha.
NgoJuni ka-2019, i-EPAM, iphendula kwiimfuno zemakethi ye-IT yanamhlanje, yahlula uMgangatho weDatha kwisenzo esahlukileyo. Iinjineli zoMgangatho weDatha, ekuhambeni komsebenzi wazo wemihla ngemihla, zilawula idatha, zijonga indlela yokuziphatha kwayo kwiimeko ezintsha kunye neenkqubo, zibeke iliso kukubaluleka kwedatha, ukwanela kwayo kunye nokufaneleka. Ngayo yonke le nto, ngengqiqo esebenzayo, iinjineli zoMgangatho weDatha zichitha ixesha elincinci kuvavanyo olusebenzayo lwakudala, KUNYE oku kuxhomekeke kakhulu kwiprojekthi (ndiya kunika umzekelo ongezantsi).
Uxanduva lweNjineli yoMgangatho weDatha ayipheleliselwanga kuphela kwiitshekhi zesiqhelo/ezizenzekelayo βezingeyomfuneko, zokubala kunye nezibaloβ kwiitheyibhile zedatha, kodwa zifuna ukuqonda okunzulu kwiimfuno zeshishini lomthengi kwaye, ngokufanelekileyo, ukukwazi ukuguqula idatha ekhoyo ibe ulwazi lweshishini oluluncedo.
Ithiyori yoMgangatho weDatha
Ukuze ucinge ngakumbi ngendima yenjineli enjalo, makhe sifumanise ukuba yintoni uMgangatho weDatha kwithiyori.
Umgangatho weDatha - esinye sezigaba zoLawulo lweDatha (ihlabathi liphela esiya kukushiya ukuba uzifundele ngokwakho) kwaye unoxanduva lokuhlalutya idatha ngokwezi khrayitheriya zilandelayo:
Ndicinga ukuba akukho mfuneko yokucacisa inqaku ngalinye (kwithiyori zibizwa ngokuba "yimilinganiselo yedatha"), zichazwe kakuhle emfanekisweni. Kodwa inkqubo yovavanyo ngokwayo ayithethi ukukopa ngokungqongqo ezi mpawu kwiimeko zovavanyo kunye nokuzijonga. KuMgangatho weDatha, njengakuluphi na uhlobo lovavanyo, kuyimfuneko, okokuqala, ukwakha kwiimfuno zomgangatho wedatha ekuvunyelwene ngazo kunye nabathathi-nxaxheba beprojekthi abenza izigqibo zoshishino.
Ngokuxhomekeke kwiprojekthi yoMgangatho weDatha, injineli inokwenza imisebenzi eyahlukeneyo: ukusuka kumvavanyi oqhelekileyo ozenzekelayo kunye novavanyo oluphezulu lomgangatho wedatha, kumntu oqhuba iphrofayili enzulu yedatha ngokwemiqathango engentla.
Inkcazo eneenkcukacha kakhulu yoLawulo lweDatha, uMgangatho weDatha kunye neenkqubo ezinxulumene nazo zichazwe kakuhle kwincwadi ebizwa "I-DAMA-DMBOK: IZiko loLawulo lweDatha yoLwazi: uHlelo lwe-2". Ndiyincoma kakhulu le ncwadi njengentshayelelo yesi sihloko (uya kufumana ikhonkco kuyo ekupheleni kwenqaku).
Ibali lam
Kwishishini le-IT, ndasebenza indlela yam ukusuka kumvavanyi weJunior kwiinkampani zemveliso ukuya kwiNjineli yoMgangatho weDatha eKhokelo kwi-EPAM. Emva kweminyaka emalunga nemibini ndisebenza njengomvavanyi, ndiqinisekile ukuba ndizenzile zonke iintlobo zovavanyo: ukuhlehla, ukusebenza, uxinzelelo, uzinzo, ukhuseleko, i-UI, njl.njl. - kwaye ndazama inani elikhulu lezixhobo zokuvavanya, wasebenza ngexesha elinye kwiilwimi ezintathu zokucwangcisa: iJava, iScala, iPython.
Ukujonga emva, ndiyasiqonda isizathu sokuba isethi yam yobuchule yayihluke kakhulu-ndandibandakanyeka kwiiprojekthi eziqhutywe ngedatha, ezinkulu nezincinci. Oku koko kundizise kwihlabathi lezixhobo ezininzi kunye namathuba okukhula.
Ukuxabisa izixhobo ezahlukeneyo kunye namathuba okufumana ulwazi olutsha kunye nezakhono, jonga nje umfanekiso ongezantsi, obonisa ezona zidumileyo kwihlabathi "Idatha & ne-AI".
Olu hlobo lomzekeliso luqulunqwa minyaka le ngomnye wongxowankulu abadumileyo uMat Turck, osuka kuphuhliso lwesoftware. Apha kwiblogi yakhe kwaye , apho asebenza khona njengeqabane.
Ndakhula ngokomsebenzi ngakumbi ngokukhawuleza xa ndinguye kuphela umvavanyi kwiprojekthi, okanye ubuncinane ekuqaleni kweprojekthi. Kungelo xesha ke apho kufuneka ube noxanduva kuyo yonke inkqubo yovavanyo, kwaye awunalo ithuba lokuhlehla, ukuya phambili kuphela. Ekuqaleni kwakusoyikeka, kodwa ngoku zonke izibonelelo zovavanyo olunjalo zicacile kum:
- Uqala ukunxibelelana neqela lonke ngendlela ongazange wakwazi ngayo ngaphambili, kuba akukho proxy yonxibelelwano: ingenguye nomphathi wovavanyo okanye abanye abavavanyi.
- Ukuntywiliselwa kwiprojekthi kuba nzulu kakhulu, kwaye unolwazi malunga nawo onke amacandelo, ngokubanzi kunye neenkcukacha.
- Abaphuhlisi abakujongi "njengomfo ovela kuvavanyo ongayazi into ayenzayo," kodwa njengomntu olinganayo ovelisa izibonelelo ezimangalisayo zeqela kunye novavanyo oluzenzekelayo kunye nolindelo lwezinambuzane ezivela kwicandelo elithile leqela. imveliso.
- Ngenxa yoko, usebenza ngakumbi, ufanelekile ngakumbi, kwaye ufuna ngakumbi.
Njengoko iprojekthi ikhula, kwi-100% yamatyala ndiye ndaba ngumcebisi wabavavanyi abatsha, ndibafundisa kwaye ndidlulisa ulwazi endilufunde ngokwam. Kwangaxeshanye, kuxhomekeke kwiprojekthi, andisoloko ndifumana inqanaba eliphezulu leengcali zovavanyo lwemoto kulawulo kwaye bekukho isidingo sokubaqeqesha kwi-automation (kwabo banomdla) okanye benze izixhobo zokusetyenziswa kwimisebenzi yemihla ngemihla (izixhobo). ukuvelisa idatha kunye nokulayisha kwisistim, isixhobo sokwenza uvavanyo lomthwalo / uvavanyo lokuzinza "ngokukhawuleza", njl.).
Umzekelo weprojekthi ethile
Ngelishwa, ngenxa yezibophelelo ezingabonakaliyo, andinakukwazi ukuthetha ngokubanzi malunga neeprojekthi endisebenze kuzo, kodwa ndiya kunika imizekelo yemisebenzi eqhelekileyo yeNjineli yoMgangatho weDatha kwenye yeeprojekthi.
Undoqo weprojekthi kukuphumeza iqonga lokulungiselela idatha yoqeqesho lweemodeli zokufunda zoomatshini ezisekelwe kuyo. Umthengi wayeyinkampani enkulu yamayeza evela e-USA. Ngokobuchwephesha yayiliqela , ukunyuka ukuya iimeko, ezineenkonzo ezincinci ezininzi kunye neprojekthi yoMthombo oVulekileyo we-EPAM - , iguqulelwe kwiimfuno zomthengi othile (ngoku iprojekthi iye yazalwa ngokutsha ). Iinkqubo ze-ETL zacwangciswa kusetyenziswa kunye nedatha esuka iinkqubo zabathengi kwi Iibhakethi. Emva koko, umfanekiso we-Docker womzekelo wokufunda umatshini wafakwa kwiqonga, elaqeqeshwa kwidatha entsha kwaye, usebenzisa i-REST API interface, yavelisa izibikezelo ezinomdla kwishishini kwaye yasombulula iingxaki ezithile.
Ngokubonakalayo, yonke into yayijongeka ngolu hlobo:
Kwakukho ubuninzi bovavanyo olusebenzayo kule projekthi, kwaye kunikwe isantya sophuhliso lweempawu kunye nesidingo sokugcina isantya somjikelo wokukhulula (iiveki ezimbini zee-sprints), bekuyimfuneko ukuba ucinge ngokukhawuleza malunga novavanyo oluzenzekelayo lwawona macandelo abaluleke kakhulu. inkqubo. Uninzi lweqonga elisekelwe ku-Kubernetes ngokwalo ligutyungelwe zi-autotests eziphunyezwe kuyo + I-Python, kodwa kwakuyimfuneko ukuyixhasa kunye nokwandisa. Ukongezelela, ukwenzela ukuba kube lula umthengi, i-GUI yadalwa ukulawula iimodeli zokufunda zoomatshini ezifakwe kwiqela, kunye nokukwazi ukucacisa ukuba kuphi na apho idatha kufuneka idluliselwe ukuqeqesha iimodeli. Olu longezo lubanzi lubandakanya ukwandiswa kovavanyo olusebenzayo oluzenzekelayo, olwenziwa kakhulu ngeefowuni ze-REST API kunye nenani elincinci leemvavanyo ze-UI zokuphela kwe-2. Malunga ne-ikhweyitha yayo yonke le ntshukumo, siye sadityaniswa ngumvavanyi wezandla owenze umsebenzi obalaseleyo ngovavanyo lokwamkelwa kweenguqulelo zemveliso kunye nokunxibelelana nomthengi malunga nokwamkelwa kokukhutshwa okulandelayo. Ukongeza, ngenxa yokufika kwengcali entsha, siye sakwazi ukubhala umsebenzi wethu kwaye songeza iitshekhi ezininzi ezibaluleke kakhulu ekunzima ukuzenzela ngoko nangoko.
Kwaye ekugqibeleni, emva kokuba sizuze uzinzo kwiqonga kunye ne-GUI yokongeza phezu kwayo, saqala ukwakha imibhobho ye-ETL usebenzisa i-Apache Airflow DAGs. Ukutshekisha okuzenzekelayo komgangatho wedatha kwenziwa ngokubhala i-Airflow DAG ezikhethekileyo ezitshekisha idatha esekelwe kwiziphumo zenkqubo ye-ETL. Njengenxalenye yale projekthi, sibe nethamsanqa kwaye umthengi usinike ukufikelela kwiiseti zedatha ezingachazwanga apho sivavanye khona. Sihlolisise umgca wedatha ngomgca wokuthotyelwa kweentlobo, ubukho bedatha ephukileyo, inani elipheleleyo leerekhodi ngaphambi nangemva, ukuthelekiswa kweenguqu ezenziwe yinkqubo ye-ETL yokudibanisa, ukuguqula amagama ekholomu, kunye nezinye izinto. Ukongeza, ezi zitshekisho zilinganiswe kwimithombo eyahlukeneyo yedatha, umzekelo, ukongeza kwi-SalesForce, nakwi-MySQL.
Ukuhlolwa komgangatho wedatha wokugqibela sele kuqhutywe kwinqanaba le-S3, apho zigcinwe khona kwaye zilungele ukusetyenziswa kwiimodeli zokufunda zoomatshini. Ukufumana idatha kwifayile ye-CSV yokugqibela ebekwe kwi-Bucket ye-S3 kwaye uyiqinisekise, ikhowudi yabhalwa kusetyenziswa .
Kwakhona kwakukho imfuneko evela kumthengi ukugcina inxalenye yedatha kwi-Bucket ye-S3 enye kwaye inxalenye kwenye. Oku kukwafuna ukuba kubhalwe iitshekhi ezongezelelweyo ukujonga ukuthembeka kolo kuhlelwa.
Amava ngokubanzi avela kwezinye iiprojekthi
Umzekelo wolona luhlu luqhelekileyo lwemisebenzi yenjineli yoMgangatho weDatha:
- Lungisa idatha yovavanyo (engasebenziyo enkulu encinci encinci) ngesixhobo esizenzekelayo.
- Layisha isethi yedatha elungisiweyo kumthombo wokuqala kwaye ukhangele ukuba ilungele ukusetyenziswa.
- Qalisa iinkqubo ze-ETL zokucubungula iseti yedatha ukusuka kwindawo yokugcina umthombo ukuya kugcino lokugqibela okanye oluphakathi usebenzisa isethi ethile yezicwangciso (ukuba kunokwenzeka, misela iiparamitha ezilungelelanisiweyo zomsebenzi we-ETL).
- Qinisekisa idatha eqhutywe yinkqubo ye-ETL yomgangatho wayo kunye nokuthotyelwa kweemfuno zoshishino.
Kwangaxeshanye, ujoliso oluphambili lokutshekishwa alufanele lube kuphela kwinto yokuba ukuhamba kwedatha kwinkqubo, ngokwemigaqo, yasebenza kwaye yafikelela esiphelweni (eyinxalenye yovavanyo olusebenzayo), kodwa ikakhulu ekujongeni nasekuqinisekiseni idatha. ukuthobela iimfuno ezilindelekileyo, ukuchonga izinto ezingaqhelekanga kunye nezinye izinto.
Zixhobo
Enye yeendlela zokulawulwa kwedatha enjalo ingaba yintlangano yokutshekisha ikhonkco kwinqanaba ngalinye lokucutshungulwa kwedatha, into ebizwa ngokuba "yi-data chain" kuncwadi - ukulawulwa kwedatha ukusuka kumthombo ukuya kwindawo yokusetyenziswa kokugqibela. Ezi ntlobo zetshekhi zidla ngokuphunyezwa ngokubhala kuqwalaselwe imibuzo yeSQL. Kucacile ukuba imibuzo enjalo kufuneka ibe yinto elula ngokusemandleni kwaye ukhangele iingcezu zomntu ngamnye womgangatho wedatha (imethadatha yeetheyibhile, imigca engenanto, i-NULLs, Iimpazamo kwi-syntax - ezinye iimpawu ezifunekayo ukujonga).
Kwimeko yovavanyo lokubuyisela umva, osebenzisa iiseti zedatha esele zenziwe (ezingenakuguquguquka, ezitshintshiweyo kancinane), ikhowudi ye-autotest inokugcina itemplates esele zilungele ukujonga idatha yokuthotyelwa komgangatho (iinkcazo zemethadatha yetafile elindelekileyo; izinto zesampulu zomqolo ezinokubakho. ekhethwe ngokungakhethiyo ngexesha lovavanyo, njl. njl.).
Kwakhona, ngexesha lovavanyo, kufuneka ubhale iinkqubo zovavanyo lwe-ETL usebenzisa izikhokelo ezifana ne-Apache Airflow, okanye isixhobo sohlobo lwebhokisi elimnyama , Kwaye nangokunjalo. Le meko inyanzelisa injineli yokuvavanya ukuba igxininise kwimigaqo yokusebenza kwezi zixhobo zingentla kwaye ngokufanelekileyo zombini ukuqhuba uvavanyo olusebenzayo (umzekelo, iinkqubo ze-ETL ezikhoyo kwiprojekthi) kwaye uzisebenzise ukujonga idatha. Ngokukodwa, i-Apache Airflow inabaqhubi abasele belungile ukuze basebenze neenkcukacha zohlalutyo ezidumileyo, umzekelo . Owona mzekelo usisiseko wokusetyenziswa kwawo sele uchaziwe , ngoko andiyi kuphinda.
Ngaphandle kwezisombululo esele zenziwe, akukho mntu ukwalela ukuba uphumeze ezakho iindlela kunye nezixhobo. Oku akuyi kuba luncedo kuphela kwiprojekthi, kodwa nakwiNjineli yoMgangatho weDatha ngokwakhe, oya kuthi ngaloo ndlela aphucule i-horizons yakhe yobugcisa kunye nezakhono zokubhala iikhowudi.
Isebenza njani kwiprojekthi yokwenyani
Umzobo omhle wemihlathi yokugqibela malunga "nekhonkco ledatha", i-ETL kunye nokutshekisha kuyo yonke indawo yinkqubo elandelayo evela kwenye yeeprojekthi zangempela:
Apha, idatha eyahlukeneyo (ngokwemvelo, elungiselelwe sithi) faka igalelo "ifenitshala" yenkqubo yethu: esebenzayo, engasebenziyo, exutywe, njl. kwaye zibekwe kwindawo yokugcina yokugcina , apho, emva koko, i-analytics, ukwakha i-data data kunye nokukhangela ulwazi lwezoshishino luya kwenziwa. Kwinkqubo enjalo, ngaphandle kokujonga ngokusebenzayo ukusebenza kweenkqubo ze-ETL, sigxininise kumgangatho wedatha ngaphambi nangemva kokuguqulwa, kunye nokuphuma kwi-analytics.
Ukushwankathela okungasentla, nokuba zeziphi iindawo endisebenza kuzo, kuyo yonke indawo bendibandakanyeka kwiiprojekthi zeDatha ezabelane ngezi mpawu zilandelayo:
- Kuphela ngokuzenzekela unokuvavanya iimeko ezithile kwaye ufezekise umjikelo wokukhululwa owamkelekileyo kwishishini.
- Umvavanyi kwiprojekthi enjalo ungomnye wamalungu ahloniphekileyo eqela, njengoko izisa inzuzo enkulu kubathathi-nxaxheba ngamnye (ukukhawuleza kokuvavanya, idatha efanelekileyo evela kwiNzululwazi yeNzululwazi, ukuchongwa kweziphene kwizigaba zokuqala).
- Ayinamsebenzi nokuba usebenza kweyakho ihardware okanye emafini - zonke izixhobo zitsalwa kwiqela elinjengeHortonworks, Cloudera, Mesos, Kubernetes, njl.
- Iiprojekthi zakhelwe kwindlela yenkonzo encinci, isasazwa kwaye icomputing ehambelanayo iphambili.
Ndingathanda ukuqaphela ukuba xa usenza uvavanyo kwintsimi yoMgangatho weDatha, ingcali yokuvavanya iguqula ugxininiso lwakhe lobuchwephesha kwikhowudi yemveliso kunye nezixhobo ezisetyenzisiweyo.
Iimpawu ezahlukileyo zovavanyo loMgangatho weDatha
Ukongeza, kum, ndichonge oku kulandelayo (ndiya kwenza ngoko nangoko ugcino ukuba ZIBALULEKILE KAKHULU kwaye zixhomekeke kuphela) iimpawu ezahlukileyo zovavanyo kwiiprojekthi zeDatha (iDatha enkulu) (iinkqubo) kunye nezinye iindawo:
amakhonkco aluncedo
- Ithiyori: .
- I-EPAM
- Imathiriyeli ecetyiswayo kwinjineli yomgangatho wedatha yokuqala:
- Ikhosi yasimahla kwiStepik: .
- Ikhosi kwiLinkedIn yokuFunda: .
- Amanqaku:
- ;
- ;
- ;
- Ividiyo:
- ;
- ;
isiphelo
Umgangatho weDatha licala elithembisayo eliselula kakhulu, ukuba yinxalenye yalo nto ithetha ukuba yinxalenye yesiqalo. Nje ukuba ube kuMgangatho weDatha, uya kuntywiliselwa kwinani elikhulu letekhnoloji yale mihla, efunwayo, kodwa okona kubaluleke kakhulu, amathuba amakhulu aya kukuvulelwa ukuba uvelise kwaye uphumeze izimvo zakho. Uya kuba nakho ukusebenzisa indlela yokuphucula eqhubekayo kungekuphela nje kwiprojekthi, kodwa nawe ngokwakho, ngokuqhubekayo uphuhlisa njengengcali.
umthombo: www.habr.com