Sanibonani nonke, igama lami ngingu-Alexander, futhi ngingunjiniyela Wekhwalithi Yedatha ohlola idatha ngekhwalithi yayo. Lesi sihloko sizokhuluma ngokuthi ngifike kanjani kulokhu nokuthi kungani ngo-2020 le ndawo yokuhlola yayiphezu kwegagasi.
Umkhuba womhlaba wonke
Izwe lanamuhla libhekene nolunye uguquko kwezobuchwepheshe, isici esisodwa esiwukusetshenziswa kwedatha eqoqwe yizo zonke izinhlobo zezinkampani ukuze zithuthukise ezazo amasondo okuthengisa, inzuzo kanye ne-PR. Kubonakala sengathi ukuba khona kwedatha enhle (yekhwalithi), kanye nobuchopho obunekhono obungakwazi ukwenza imali kuyo (ukucubungula kahle, ukubona ngeso lengqondo, ukwakha amamodeli wokufunda ngomshini, njll.), kube yisihluthulelo sempumelelo kwabaningi namuhla. Uma eminyakeni engu-15-20 edlule izinkampani ezinkulu bezihileleke kakhulu emsebenzini onzima ngokuqoqwa kwedatha nokwenza imali, namuhla lokhu kuningi cishe bonke abantu abanengqondo.
Mayelana nalokhu, eminyakeni embalwa edlule, zonke izingosi ezinikezelwe ekufuneni umsebenzi emhlabeni wonke zaqala ukugcwaliswa ngezikhala zeDatha Scientists, ngoba wonke umuntu wayenesiqiniseko sokuthi, njengoba eqashe uchwepheshe onjalo, kwakungenzeka ukwakha i-supermodel yokufunda umshini. , bikezela ikusasa futhi wenze "i-quantum leap" yenkampani. Ngokuhamba kwesikhathi, abantu baqaphela ukuthi le ndlela cishe ayisebenzi noma kuphi, ngoba akuyona yonke idatha ewela ezandleni zochwepheshe abanjalo efanelekile kumamodeli okuqeqesha.
Futhi izicelo ezivela ku-Data Scientists zaqala: “Masithenge idatha eyengeziwe kulawa nalawo...”, “Asinayo idatha eyanele...”, “Sidinga enye idatha, okungcono kakhulu esezingeni eliphezulu...” . Ngokusekelwe kulezi zicelo, ukusebenzisana okuningi kwaqala ukwakhiwa phakathi kwezinkampani eziphethe isethi eyodwa noma enye yedatha. Ngokwemvelo, lokhu kwakudinga inhlangano yezobuchwepheshe yale nqubo - ukuxhuma emthonjeni wedatha, ukuyilanda, ukuhlola ukuthi ilayishwe ngokugcwele, njll. Inani lezinqubo ezinjalo laqala ukukhula, futhi namuhla sinesidingo esikhulu solunye uhlobo ochwepheshe - Onjiniyela Bekhwalithi Yedatha - labo abangaqapha ukuhamba kwedatha ohlelweni (amapayipi edatha), ikhwalithi yedatha kokokufakayo nokuphumayo, futhi bafinyelele iziphetho mayelana nokwanela kwabo, ubuqotho nezinye izici.
Umkhuba wonjiniyela Wekhwalithi Yedatha weza kithi uvela e-USA, lapho, phakathi nesikhathi esishubile sobunxiwankulu, akekho olungele ukulahlekelwa impi yedatha. Ngezansi nginikeze izithombe-skrini ezivela kumasayithi amabili okusesha aziwa kakhulu e-US: и - okubonisa idatha kusukela ngoMashi 17, 2020 enanini lezikhala ezithunyelwe ezitholwe kusetshenziswa amagama angukhiye: Ikhwalithi Yedatha kanye Nososayensi Wedatha.
Ososayensi Bedatha - 21416 izikhala
Ikhwalithi Yedatha - 41104 izikhala
Ososayensi Bedatha - 404 izikhala
Ikhwalithi Yedatha - izikhala ze-2020
Ngokusobala, le misebenzi ayiqhudelani nangayiphi indlela. Ngezithombe-skrini, bengifuna ukukhombisa isimo samanje emakethe yezemisebenzi ngokwezicelo zonjiniyela Bekhwalithi Yedatha, okudingeka okuningi kakhulu kubo manje kunoSosayensi Bedatha.
NgoJuni 2019, i-EPAM, isabela ezidingweni zemakethe yesimanje ye-IT, yahlukanisa Ikhwalithi Yedatha yaba umkhuba ohlukile. Onjiniyela Bekhwalithi Yedatha, phakathi nomsebenzi wabo wansuku zonke, baphatha idatha, bahlole ukuziphatha kwayo ezimeni ezintsha nezinhlelo, baqaphe ukuhlobana kwedatha, ukwanela kwayo kanye nokuhambisana kwayo. Ngakho konke lokhu, ngomqondo ongokoqobo, onjiniyela Bekhwalithi Yedatha bachitha isikhathi esincane ekuhlolweni kokusebenza kwakudala, BUT kuncike kakhulu kuphrojekthi (ngizonikeza isibonelo ngezansi).
Izibopho zikanjiniyela Wekhwalithi Yedatha azikhawulelwe kuphela ekuhloleni okwenziwa ngesandla/okuzenzakalelayo “okungenalutho, ukubala nezibalo” kumathebula esizindalwazi, kodwa kudinga ukuqonda okujulile kwezidingo zebhizinisi lekhasimende, ngokufanele, nekhono lokuguqula idatha etholakalayo ibe yisizindalwazi. ulwazi lwebhizinisi oluwusizo.
Ithiyori Yekhwalithi Yedatha
Ukuze ucabange ngokugcwele indima kanjiniyela onjalo, ake sithole ukuthi iyini Ikhwalithi Yedatha emcabangweni.
Ikhwalithi yedatha - esinye sezigaba Zokuphathwa Kwedatha (umhlaba wonke esizowushiya ukuze uzifundele wedwa) futhi unesibopho sokuhlaziya idatha ngokuya ngalezi zindlela ezilandelayo:
Ngicabanga ukuthi asikho isidingo sokucacisa iphuzu ngalinye (ngokombono abizwa ngokuthi "ubukhulu bedatha"), achazwe kahle esithombeni. Kodwa inqubo yokuhlola ngokwayo ayisho ukukopisha ngokuqinile lezi zici emacaleni okuhlola nokuzihlola. Kukhwalithi Yedatha, njengakunoma yiluphi olunye uhlobo lokuhlola, kuyadingeka, okokuqala, ukwakha phezu kwezimfuneko zekhwalithi yedatha okuvunyelwane ngazo nabahlanganyeli bephrojekthi abenza izinqumo zebhizinisi.
Kuye ngephrojekthi Yekhwalithi Yedatha, unjiniyela angenza imisebenzi ehlukene: kusukela kumhloli ozenzakalelayo ojwayelekile onokuhlola okukha phezulu kwekhwalithi yedatha, kuye kumuntu owenza iphrofayela ejulile yedatha ngokuya ngemibandela engenhla.
Incazelo enemininingwane eminingi yokuphathwa Kwedatha, Ikhwalithi Yedatha kanye nezinqubo ezihlobene nayo ichazwe kahle encwadini ebizwa "I-DAMA-DMBOK: I-Data Management Body of Knowledge: Uhlelo Lwesibili". Ngincoma kakhulu le ncwadi njengesingeniso salesi sihloko (uzothola isixhumanisi kuso ekupheleni kwesihloko).
Indaba yami
Embonini ye-IT, ngisebenze indlela yami esuka kumhloli weJunior ezinkampanini zomkhiqizo ukuya kuNjiniyela Wekhwalithi Yedatha Eholayo kwa-EPAM. Cishe ngemva kweminyaka emibili ngisebenza njengomhloli, ngaba nokuqiniseka okuqinile ukuthi ngangenze zonke izinhlobo zokuhlola: ukwehla, ukusebenza, ukucindezeleka, ukuzinza, ukuphepha, i-UI, njll. - futhi ngazama inqwaba yamathuluzi okuhlola, wasebenza ngesikhathi esifanayo ezilimini zokuhlela ezintathu: Java, Scala, Python.
Uma ngibheka emuva, ngiyaqonda ukuthi kungani isethi yami yamakhono yayihluke kakhulu—ngangihileleka kumaphrojekthi aqhutshwa yidatha, amakhulu namancane. Yilokhu okungilethe emhlabeni wamathuluzi amaningi namathuba okukhula.
Ukuze ujabulele izinhlobonhlobo zamathuluzi namathuba okuthola ulwazi namakhono amasha, vele ubheke isithombe esingezansi, esibonisa adume kakhulu emhlabeni "wedatha ne-AI".
Lolu hlobo lomfanekiso luhlanganiswa minyaka yonke ngomunye wonxiwankulu abadumile uMat Turck, ovela ekuthuthukisweni kwesoftware. Lapha kubhulogi yakhe futhi , lapho esebenza khona njengozakwethu.
Ngakhula ngokomsebenzi ikakhulukazi ngokushesha lapho ngiwukuphela komhloli kuphrojekthi, noma okungenani ekuqaleni kwephrojekthi. Kungaleso sikhathi lapho kufanele ube nesibopho sayo yonke inqubo yokuhlola, futhi awunalo ithuba lokuhlehla, ukuya phambili kuphela. Ekuqaleni kwakwethusa, kodwa manje zonke izinzuzo zokuhlolwa okunjalo zisobala kimi:
- Uqala ukuxhumana nethimba lonke kunanini ngaphambili, njengoba ungekho ummeleli wokuxhumana: nomphathi wokuhlola noma ozakwethu abahloli.
- Ukucwiliswa kuphrojekthi kuba kujule ngendlela emangalisayo, futhi unolwazi mayelana nazo zonke izingxenye, kokubili ngokujwayelekile nangemininingwane.
- Onjiniyela abakubheki “njengomuntu ophuma ekuhlolweni ongazi ukuthi wenzani,” kodwa njengokulingana okhiqiza izinzuzo ezimangalisayo zeqembu ngokuhlolwa kwakhe okuzenzakalelayo nokulindela iziphazamisi ezivela engxenyeni ethile ye- umkhiqizo.
- Ngenxa yalokho, uphumelela kakhulu, ufanelekile kakhulu, futhi udingeka kakhulu.
Njengoba iphrojekthi ikhula, ngamaphesenti angu-100 ngaphenduka umeluleki wabahloli abasha, ngibafundisa futhi ngidlulisela ulwazi engangizifundele mina. Ngaso leso sikhathi, kuye ngephrojekthi, angizange ngihlale ngithola izinga eliphezulu lochwepheshe bokuhlola izimoto kubaphathi futhi bekunesidingo sokuthi ngibaqeqeshe ngokuzenzakalelayo (kulabo abanentshisekelo) noma ngidale amathuluzi azosetshenziswa emisebenzini yansuku zonke (amathuluzi). ukukhiqiza idatha nokuyilayisha kusistimu , ithuluzi lokwenza ukuhlola komthwalo/ukuhlola ukuzinza “ngokushesha”, njll.).
Isibonelo sephrojekthi ethile
Ngeshwa, ngenxa yezibopho zokungadaluli, angikwazi ukukhuluma ngokuningiliziwe ngamaphrojekthi engisebenze kuwo, kodwa ngizonikeza izibonelo zemisebenzi evamile kaNjiniyela Wekhwalithi Yedatha kwenye yamaphrojekthi.
Umongo wephrojekthi ukusebenzisa inkundla yokulungiselela idatha yamamodeli okufunda emishini esuselwe kuyo. Ikhasimende kwakuyinkampani enkulu eyenza imithi evela e-USA. Ngobuchwepheshe bekuyiqoqo ,esukuma ku izimo, ngamasevisi amancane ambalwa kanye nephrojekthi yomthombo ovulekile eyisisekelo ye-EPAM - , evumelaniswe nezidingo zekhasimende elithile (manje iphrojekthi isiphinde yazalwa ). Izinqubo ze-ETL zahlelwa kusetshenziswa futhi yahambisa idatha kusuka amasistimu wamakhasimende ku Amabhakede. Okulandelayo, isithombe se-Docker semodeli yokufunda yomshini safakwa endaweni yesikhulumi, esaqeqeshwa ngedatha entsha futhi, kusetshenziswa isixhumi esibonakalayo se-REST API, sakhiqiza izibikezelo ezazinentshisekelo ebhizinisini futhi zaxazulula izinkinga ezithile.
Ngokubukeka, yonke into yayibukeka kanjena:
Bekunokuhlolwa okuningi okusebenzayo kule phrojekthi, futhi kunikezwe isivinini sokuthuthukiswa kwesici kanye nesidingo sokugcina ijubane lomjikelezo wokukhululwa (ama-sprints amasonto amabili), bekudingeka ukuthi kucatshangwe ngokushesha mayelana nokuhlolwa okuzenzakalelayo kwezingxenye ezibaluleke kakhulu uhlelo. Iningi leplatifomu esekwe ku-Kubernetes uqobo yambozwa ama-autotests asetshenziswe kuwo + I-Python, kodwa futhi kwakudingeka ukuyisekela futhi yandise. Ngaphezu kwalokho, ukuze kube lula ikhasimende, i-GUI yadalwa ukuze ilawule amamodeli okufunda omshini athunyelwe kuqoqo, kanye nekhono lokucacisa ukuthi idatha idinga ukudluliselwa kuphi futhi kuphi ukuze kuqeqeshwe amamodeli. Lokhu kungezwa okubanzi kuhlanganisa ukunwetshwa kokuhlolwa kokusebenza okuzenzakalelayo, obekwenziwa kakhulukazi ngamakholi we-REST API kanye nenani elincane lokuhlolwa kwe-UI yokuphela kwezi-2. Ngaphakathi kwenkabazwe yakho konke lokhu kunyakaza, sajoyinwa umhloli owenzelwa mathupha owenza umsebenzi omuhle kakhulu ngokuhlolwa kokwamukelwa kwezinguqulo zomkhiqizo kanye nokuxhumana nekhasimende mayelana nokwamukela ukukhishwa okulandelayo. Ngaphezu kwalokho, ngenxa yokufika kukachwepheshe omusha, sakwazi ukubhala umsebenzi wethu futhi sengeza ukuhlola okwenziwa ngesandla okumbalwa okubaluleke kakhulu obekunzima ukukuzenzela khona manjalo.
Futhi ekugcineni, ngemva kokuba sesizuze ukuzinza kusuka endaweni yesikhulumi kanye nesengezo se-GUI phezu kwayo, saqala ukwakha amapayipi e-ETL sisebenzisa ama-Apache Airflow DAG. Ukuhlolwa kwekhwalithi yedatha okuzenzakalelayo kwenziwa ngokubhala ama-Airflow DAG akhethekile ahlola idatha ngokusekelwe emiphumeleni yenqubo ye-ETL. Njengengxenye yale phrojekthi, sibe nenhlanhla futhi ikhasimende lisinikeze ukufinyelela kumasethi edatha angaziwa esiwahlole. Sihlole umugqa wedatha ngomugqa ukuthi uyahambisana yini nezinhlobo, ukuba khona kwedatha ephukile, ingqikithi yenani lamarekhodi ngaphambi nangemuva, ukuqhathaniswa koshintsho olwenziwa inqubo ye-ETL yokuhlanganisa, ukushintsha amagama ekholomu, nezinye izinto. Ngaphezu kwalokho, lokhu kuhlola kukalwe emithonjeni yedatha ehlukene, isibonelo, ngaphezu kwe-SalesForce, naku-MySQL.
Ukuhlolwa kwekhwalithi yokugcina yedatha kwenziwa kakade ezingeni le-S3, lapho yayigcinwe khona futhi isilungele ukusetshenziswa kumamodeli okufunda emishini yokuqeqesha. Ukuze uthole idatha efayelini lokugcina le-CSV elitholakala Kubhakede le-S3 futhi uliqinisekise, ikhodi yabhalwa kusetshenziswa .
Bekukhona nemfuneko ephuma kukhasimende ukuthi igcine ingxenye yedatha kuBhakede elilodwa le-S3 bese iyingxenye kwelinye. Lokhu futhi kwakudinga ukubhala amasheke engeziwe ukuze kubhekwe ukwethembeka kokuhlunga okunjalo.
Umuzwa ojwayelekile ovela kwamanye amaphrojekthi
Isibonelo sohlu olujwayelekile kakhulu lwemisebenzi kanjiniyela Wekhwalithi Yedatha:
- Lungiselela idatha yokuhlola (encane engavumelekile enkulu engavumelekile) ngethuluzi elizenzakalelayo.
- Layisha isethi yedatha elungisiwe emthonjeni wangempela futhi uhlole ukuthi isilungele ukusetshenziswa.
- Qalisa izinqubo ze-ETL zokucubungula isethi yedatha isuka kumthombo wokulondoloza uye kwisitoreji sokugcina noma esimaphakathi usebenzisa isethi ethile yezilungiselelo (uma kungenzeka, setha imingcele elungisekayo yomsebenzi we-ETL).
- Qinisekisa idatha ecutshungulwe yinqubo ye-ETL ngekhwalithi yayo nokuhambisana nezidingo zebhizinisi.
Ngesikhathi esifanayo, ukugxila okuyinhloko kokuhlola akufanele kube kuphela eqinisweni lokuthi ukugeleza kwedatha ohlelweni, empeleni, kusebenzile futhi kwafinyelela ekupheleni (okuyingxenye yokuhlolwa kokusebenza), kodwa ikakhulukazi ekuhloleni nasekuqinisekiseni idatha ye ukuhambisana nezidingo ezilindelekile, ukuhlonza okudidayo nezinye izinto.
Amathuluzi
Enye yezindlela zokulawulwa kwedatha enjalo kungaba inhlangano yokuhlola amaketanga esigabeni ngasinye sokucubungula idatha, lokho okubizwa ngokuthi "uchungechunge lwedatha" ezincwadini - ukulawulwa kwedatha kusuka emthonjeni kuya endaweni yokusetshenziswa kokugcina. Lezi zinhlobo zokuhlola zivame ukusetshenziswa ngokubhala ukuhlola imibuzo ye-SQL. Kuyacaca ukuthi imibuzo enjalo kufanele ibe lula ngangokunokwenzeka futhi ihlole izingcezu ngazinye zekhwalithi yedatha (imethadatha yamathebula, imigqa engenalutho, NULLs, Amaphutha ku-syntax - ezinye izibaluli ezidingekayo ukuze zihlolwe).
Esimeni sokuhlolwa kokuhlehla, okusebenzisa amasethi edatha esenziwe ngomumo (angenakuguquleka, ashintsheka kancane), ikhodi yokuhlola ezenzakalelayo ingagcina izifanekiso esezilungile ukuze kuhlolwe idatha ukuze ihambisane nekhwalithi (izincazelo zemethadatha yethebula elilindelekile; izinto zesampula zomugqa ezingaba kukhethwe ngokungahleliwe ngesikhathi sokuhlolwa, njll.).
Futhi, ngesikhathi sokuhlolwa, kufanele ubhale izinqubo zokuhlola ze-ETL usebenzisa izinhlaka ezifana ne-Apache Airflow, noma ithuluzi lohlobo lwamafu elimnyama , Njalo njalo. Lesi simo siphoqa unjiniyela wokuhlola ukuthi agxile ezimisweni zokusebenza kwamathuluzi angenhla futhi ngempumelelo nakakhulu kokubili enze ukuhlola okusebenzayo (isibonelo, izinqubo ezikhona ze-ETL kuphrojekthi) futhi azisebenzisele ukuhlola idatha. Ikakhulukazi, i-Apache Airflow inama-opharetha enziwe ngomumo ukuze asebenze nesizindalwazi sokuhlaziya esidumile, ngokwesibonelo . Isibonelo esiyisisekelo sokusetshenziswa kwayo sesivele sibekiwe , ngakho ngeke ngiphinde.
Ngaphandle kwezixazululo ezenziwe ngomumo, akekho okwenqabela ukusebenzisa amasu akho namathuluzi. Lokhu ngeke kube yinzuzo kuphrojekthi kuphela, kodwa futhi nakuNjiniyela Wekhwalithi Yedatha ngokwakhe, ozokwenza kanjalo athuthukise ama-horizons akhe obuchwepheshe namakhono okubhala amakhodi.
Isebenza kanjani kuphrojekthi yangempela
Umfanekiso omuhle wezigaba zokugcina mayelana "nochungechunge lwedatha", i-ETL kanye nokuhlolwa okutholakala yonke indawo inqubo elandelayo evela kwelinye lamaphrojekthi angempela:
Lapha, idatha ehlukahlukene (ngokwemvelo, elungiselelwe yithi) ifaka "ifaneli" yokufaka yesistimu yethu: evumelekile, engavumelekile, exutshwe, njll., bese iyahlungwa futhi igcine isitoreji esiphakathi, bese iphinda idlule uchungechunge lwezinguquko. futhi abekwe endaweni yokugcina yokugcina , lapho, kamuva, ukuhlaziya, ukwakha amamaki edatha kanye nokusesha imininingwane yebhizinisi kuzokwenziwa. Kuhlelo olunjalo, ngaphandle kokuhlola ukusebenza kwezinqubo ze-ETL, sigxila kwikhwalithi yedatha ngaphambi nangemva koshintsho, kanye nasemikhiqizweni yokuhlaziya.
Ukufingqa lokhu okungenhla, kungakhathaliseki ukuthi yiziphi izindawo engangisebenza kuzo, yonke indawo ngangihileleke kumaphrojekthi wedatha abelane ngezici ezilandelayo:
- Ngokuzenzakalela kuphela lapho ungakwazi ukuhlola ezinye izimo futhi uzuze umjikelezo wokukhishwa owamukelekayo ebhizinisini.
- Umhloli kuphrojekthi enjalo ungomunye wamalungu eqembu ahlonishwa kakhulu, njengoba kuletha izinzuzo ezinkulu kubahlanganyeli ngamunye (ukusheshisa ukuhlolwa, idatha enhle evela ku-Data Scientist, ukuhlonza amaphutha ezigabeni zokuqala).
- Akukhathalekile ukuthi usebenza ku-hardware yakho noma emafwini - zonke izinsiza zikhishwa zibe yiqoqo elifana neHortonworks, Cloudera, Mesos, Kubernetes, njll.
- Amaphrojekthi akhiwe ngendlela ye-microservice, esabalalisiwe futhi i-parallel computing ihamba phambili.
Ngingathanda ukuqaphela ukuthi lapho kuhlolwa emkhakheni Wekhwalithi Yedatha, uchwepheshe wokuhlola ushintsha agxile emsebenzini wakhe agxile kukhodi yomkhiqizo namathuluzi asetshenzisiwe.
Izici ezihlukile zokuhlola Ikhwalithi Yedatha
Ngaphezu kwalokho, ngokwami, ngizihlonze lokhu okulandelayo (ngizokwenza ngokushesha ukubhuka ukuthi ZIFUMANE KAKHULU futhi zizimele kuphela) izici ezihlukile zokuhlola kumaphrojekthi weDatha (Big Data) (amasistimu) nakwezinye izindawo:
Izixhumanisi eziwusizo
- Ithiyori: .
- I-EPAM
- Izinto ezinconyiwe zonjiniyela wokuqala Wekhwalithi Yedatha:
- Isifundo samahhala ku-Stepik: .
- Isifundo sokufunda nge-LinkedIn: .
- Izindatshana:
- ;
- ;
- ;
- Ividiyo:
- ;
- ;
isiphetho
Ikhwalithi yedatha iyisiqondiso esithembisayo esincane kakhulu, ukuba yingxenye okusho ukuba yingxenye yesiqalo. Uma usufinyelele Ikhwalithi Yedatha, uzocwiliswa enanini elikhulu lobuchwepheshe besimanje, obufunwa kakhulu, kodwa okubaluleke kakhulu, amathuba amakhulu azovulelwa wena ukuthi ukhiqize futhi usebenzise imibono yakho. Uzokwazi ukusebenzisa indlela yokuthuthukisa eqhubekayo hhayi kuphrojekthi kuphela, kodwa futhi nakuwe, ngokuqhubekayo uthuthuka njengochwepheshe.
Source: www.habr.com