Lawa magama amabili e-Data Science adida abantu abaningi. Ukumbiwa Kwedatha kuvame ukuqondwa kabi njengokukhipha kanye nokubuyisa idatha, kodwa iqiniso liyinkimbinkimbi kakhulu. Kulokhu okuthunyelwe, ake senze amachashazi Imayini futhi sithole umehluko phakathi Kwemayini Yedatha kanye Nokukhishwa Kwedatha.
Iyini i-Data Mining?
Idatha yezimayini, ebizwa nangokuthi I-Database Knowledge Discovery (KDD), kuyindlela evame ukusetshenziswa ukuhlaziya amasethi edatha amakhulu kusetshenziswa izindlela zezibalo nezibalo ukuze kutholwe amaphethini afihliwe noma amathrendi futhi kukhishwe inani kuwo.
Yini engenziwa nge-Data Mining?
Ngokuzenzakalela inqubo, ingaphequlula imininingo egciniwe futhi yembule ngempumelelo amaphethini afihliwe. Kumabhizinisi, ukumbiwa kwedatha kuvame ukusetshenziselwa ukuthola amaphethini nobudlelwano kudatha ukusiza ukwenza izinqumo ezingcono zebhizinisi.
Izibonelo zohlelo lokusebenza
Ngemva kokuba ukumbiwa kwedatha kusakazeke kabanzi ngeminyaka yawo-1990, izinkampani ezimbonini eziningi ezihlukene, ezihlanganisa ezokuthengisa, ezezimali, ezokunakekelwa kwempilo, ezokuthutha, ezokuxhumana, ezohwebo nge-inthanethi, njll., zaqala ukusebenzisa izindlela zokumba idatha ukuze zithole ulwazi ngesisekelo sedatha. Ukumbiwa kwedatha kungasiza ukuhlukanisa amakhasimende, ukuhlonza ukukhwabanisa, ukubikezela ukuthengiswa, nokuningi.
- Ukuhlukaniswa kwekhasimende
Ngokuhlaziya idatha yekhasimende nokuhlonza izici zamakhasimende okuqondiwe, izinkampani zingawahlanganisa abe yiqembu elihlukile futhi zinikeze izipesheli ezihlangabezana nezidingo zazo. - Market Basket Analysis
Le nqubo isekelwe embonweni wokuthi uma uthenga iqembu elithile lemikhiqizo, kungenzeka ukuthi uthenge iqembu elihlukile lemikhiqizo. Isibonelo esisodwa esidumile: lapho obaba bethengela izingane zabo amanabukeni, bavame ukuthenga ubhiya kanye namanabukeni. - Isibikezelo sokuthengisa
Kungase kubonakale kufana nokuhlaziya ubhasikidi wemakethe, kodwa kulokhu ukuhlaziywa kwedatha kusetshenziselwa ukubikezela ukuthi ikhasimende lizophinde lithenge nini umkhiqizo esikhathini esizayo. Isibonelo, umqeqeshi uthenga ithini lamaprotheni okufanele lihlale izinyanga eziyi-9. Isitolo esidayisa le protein sihlela ukukhipha entsha ezinyangeni eziyisi-9 ukuze umqeqeshi azophinde ayithenge. - Ukutholwa kokukhwabanisa
Ukumbiwa kwedatha kusiza ekwakheni amamodeli okutholwa kokukhwabanisa. Ngokuqoqa amasampula emibiko yokukhwabanisa neyiqiniso, amabhizinisi anikwa amandla okunquma ukuthi yikuphi okwenziwayo okusolisayo. - Ukutholwa kwephethini ekukhiqizeni
Embonini yokukhiqiza, ukumbiwa kwedatha kusetshenziselwa ukusiza ukuklama izinhlelo ngokuhlonza ubudlelwano phakathi kwezakhiwo zomkhiqizo, iphrofayili, kanye nezidingo zamakhasimende. Ukumbiwa kwedatha kungase futhi kubikezele izikhathi zokuthuthukiswa komkhiqizo nezindleko.
Futhi lezi yizimo ezimbalwa zokusebenzisa ukumbiwa kwedatha.
Izigaba zokumbiwa kwedatha
Ukumbiwa kwedatha kuyinqubo ephelele yokuqoqa, ukukhetha, ukuhlanza, ukuguqula, kanye nokukhipha idatha ukuze kuhlolwe amaphethini futhi, ekugcineni, kukhishwe inani.
Ngokuvamile, yonke inqubo yokumbiwa kwedatha ingafingqwa ngezinyathelo eziyi-7:
- Ukuhlanza idatha
Emhlabeni wangempela, idatha ayihlali ihlanzwa futhi yakhiwe. Zivame ukuba nomsindo, aziphelele, futhi zingaqukatha amaphutha. Ukuqinisekisa ukuthi umphumela wokumbiwa kwedatha unembile, udinga kuqala ukuhlanza idatha. Ezinye izindlela zokuhlanza zihlanganisa ukugcwalisa amanani angekho, izilawuli ezizenzakalelayo nezandla, njalonjalo. - Ukuhlanganiswa kwedatha
Lesi yisigaba lapho idatha evela emithonjeni ehlukene ikhishwa, ihlanganiswe futhi ihlanganiswe. Imithombo ingaba isizindalwazi, amafayela ombhalo, amaspredishithi, amadokhumenti, amasethi edatha ahlukahlukene, i-inthanethi, njalo njalo. - Ukusampula kwedatha
Ngokuvamile, akuyona yonke idatha ehlanganisiwe edingekayo ekumbiweni kwedatha. Ukusampula kwedatha yisiteji lapho kukhethwa idatha ewusizo kuphela futhi ikhishwe kusizindalwazi esikhulu. - Ukuguqulwa kwedatha
Uma idatha isikhethiwe, iguqulwa ibe amafomu afanele ukumbiwa. Le nqubo ihlanganisa ukujwayela, ukuhlanganisa, ukwenziwa ngokujwayelekile, njll. - Ukumbiwa kwedatha
Nakhu kufika ingxenye ebaluleke kakhulu yokumbiwa kwedatha - kusetshenziswa izindlela ezihlakaniphile ukuthola amaphethini kuzo. Inqubo ihlanganisa ukuhlehla, ukuhlukanisa, ukubikezela, ukuhlanganisa, ukufunda ngokuhlanganisa, nokuningi. - Ukuhlolwa kwemodeli
Lesi sinyathelo sihlose ukuhlonza amaphethini angaba usizo, okulula ukuwaqonda, kanye namaphethini asekela imibono eqanjiwe. - Ukumelwa kolwazi
Esigabeni sokugcina, ulwazi olutholiwe lwethulwa ngendlela ekhangayo kusetshenziswa ukumelwa kolwazi nezindlela zokubona ngeso lengqondo.
Ububi beMayini Yedatha
- Ukutshalwa kwezimali okukhulu kwesikhathi nomsebenzi
Njengoba ukumbiwa kwedatha kuyinqubo ende futhi eyinkimbinkimbi, kudinga umsebenzi omningi ovela kubantu abakhiqizayo nabanekhono. Ososayensi bedatha bangasebenzisa amathuluzi anamandla okumba idatha, kodwa badinga ochwepheshe ukuze balungiselele idatha futhi baqonde imiphumela. Ngenxa yalokho, kungase kuthathe isikhathi ukucubungula lonke ulwazi. - Ubumfihlo bedatha nokuvikeleka
Ngoba ukumbiwa kwedatha kuqoqa ulwazi lwekhasimende ngezindlela zemakethe, kungephula ubumfihlo bomsebenzisi. Ngaphezu kwalokho, abaduni bangathola idatha egcinwe ezinhlelweni zokumbiwa kwedatha. Lokhu kubangela usongo ekuvikelekeni kwedatha yekhasimende. Uma idatha eyebiwe isetshenziswa kabi, ingalimaza abanye kalula.
Okungenhla yisingeniso esifushane sokumbiwa kwedatha. Njengoba ngike ngasho, ukumbiwa kwedatha kuqukethe inqubo yokuqoqa nokuhlanganisa idatha, okuhlanganisa inqubo yokukhipha idatha (ukukhishwa kwedatha). Kulokhu, kuphephile ukusho ukuthi ukukhishwa kwedatha kungaba yingxenye yenqubo ende yokumbiwa kwedatha.
Kuyini Ukukhishwa Kwedatha?
Eyaziwa nangokuthi "izimayini zedatha yewebhu" kanye "ne-web scraping", le nqubo iyisenzo sokukhipha idatha emithonjeni yedatha (imvamisa engahlelekile noma engahlelwanga kahle) iyiswe ezindaweni ezimaphakathi kanye nokuhlanganisa indawo eyodwa ukuze igcinwe noma iqhutshwe phambili. Ngokukhethekile, imithombo yedatha engahlelekile ihlanganisa amakhasi ewebhu, i-imeyili, amadokhumenti, amafayela e-PDF, umbhalo oskeniwe, imibiko yozimele oyinhloko, amafayela we-reel, izimemezelo, njalonjalo. Isitoreji esimaphakathi kungaba sendawo, ifu noma ingxubevange. Kubalulekile ukukhumbula ukuthi ukukhishwa kwedatha akubandakanyi ukucubungula noma okunye ukuhlaziya okungenzeka kwenzeke kamuva.
Yini engenziwa nge-Data Extraction?
Ngokuyisisekelo, izinjongo zokukhipha idatha ziwela ezigabeni ezi-3.
- Igcina kungobo yomlando
Ukukhishwa kwedatha kungaguqula idatha kusuka kumafomethi aphathekayo njengezincwadi, amaphephandaba, ama-invoyisi kuya kumafomethi edijithali afana nesizindalwazi sokulondoloza noma ikhophi yasenqolobaneni. - Ukushintsha ifomethi yedatha
Uma ufuna ukuthutha idatha isuka kusayithi lakho yamanje iye kwentsha esakhiwayo, ungakwazi ukuqoqa idatha kusayithi lakho ngokuyikhipha. - Ukuhlaziywa kwedatha
Kuvamile ukuqhubeka nokuhlaziya idatha ekhishiwe ukuze uthole ukuqonda kuyo. Lokhu kungase kuzwakale kufana nokumbiwa kwedatha, kodwa khumbula ukuthi ukumbiwa kwedatha kuwumgomo wokumbiwa kwedatha, hhayi ingxenye yakho. Ngaphezu kwalokho, idatha ihlaziywa ngokuhlukile. Isibonelo esisodwa ukuthi abanikazi bezitolo eziku-inthanethi badonsa imininingwane yomkhiqizo kumasayithi e-e-commerce afana ne-Amazon ukuze baqaphe amasu esincintisana nawo ngesikhathi sangempela. Njengokumbiwa kwedatha, ukukhipha idatha kuyinqubo ezenzakalelayo enezinzuzo eziningi. Esikhathini esedlule, abantu babekopisha futhi banamathisele idatha mathupha besuka endaweni eyodwa baye kwenye, okwakudla isikhathi kakhulu. Ukukhishwa kwedatha kusheshisa ukuqoqwa futhi kuthuthukisa kakhulu ukunemba kwedatha ekhishiwe.
Ezinye izibonelo zokusebenzisa i-Data Extraction
Ngokufana nokumbiwa kwedatha, ukumbiwa kwedatha kusetshenziswa kabanzi ezimbonini ezahlukahlukene. Ngokungeziwe ekuqaphelweni kwentengo ye-e-commerce, ukumbiwa kwedatha kungasiza ngocwaningo lwakho, ukuhlanganisa izindaba, ukumaketha, izindlu ezithengiswayo, ezokuvakasha nezokuvakasha, ukubonisana, ezezimali, nokuningi.
- Isizukulwane sokuhola
Izinkampani zingakhipha idatha kunkhombandlela: Yelp, Crunchbase, Yellowpages futhi zikhiqize imikhondo yokuthuthukiswa kwebhizinisi. Ungabuka ividiyo engezansi ukuze ufunde ukuthi ungayikhipha kanjani idatha ku-Yellowpages nge . - Ukuhlanganiswa kokuqukethwe nezindaba
Amawebhusayithi ahlanganisa okuqukethwe angathola ukuphakelwa kwedatha okuvamile okuvela emithonjeni eminingi futhi agcine amasayithi abo enolwazi lwakamuva. - Ukuhlaziya Imizwa
Ngemva kokukhipha ukubuyekezwa, ukuphawula, nobufakazi ezinkundleni zokuxhumana ezifana ne-Instagram ne-Twitter, ochwepheshe bangahlaziya izimo zengqondo ezicashile futhi bathole imininingwane yokuthi umkhiqizo, umkhiqizo, noma isenzakalo sibonwa kanjani.
Izinyathelo Zokukhipha Idatha
Ukukhishwa kwedatha yisigaba sokuqala se-ETL (Khipha, Ukuguqula, Ukulayisha: Khipha, Ukuguqula, Ukulayisha) kanye ne-ELT (I-Extract, Layisha, ne-Transform). I-ETL ne-ELT ngokwazo ziyingxenye yesu eliphelele lokuhlanganisa idatha. Ngamanye amazwi, ukukhipha idatha kungaba yingxenye yokukhishwa kwabo.
Khipha, guqula, layisha
Nakuba ukumbiwa kwedatha kumayelana nokukhipha ulwazi enanini elikhulu ledatha, ukukhishwa kwedatha kuyinqubo emfushane kakhulu futhi elula. Kungancishiswa kube izigaba ezintathu:
- Ukukhetha umthombo wedatha
Khetha umthombo ofuna ukukhipha idatha kuwo, njengewebhusayithi. - Ukuqoqwa kwedatha
Thumela isicelo esithi "GET" kusayithi futhi uhlukanise idokhumenti ye-HTML ewumphumela usebenzisa izilimi zokuhlela ezifana nePython, PHP, R, Ruby, njll. - Isitoreji sedatha
Londoloza idatha kusizindalwazi sakho sendawo noma isitoreji samafu ukuze uyisebenzise esikhathini esizayo. Uma ungumfundi onolwazi ofuna ukukhipha idatha, izinyathelo ezingenhla zingabonakala zilula kuwe. Kodwa-ke, uma ungeyena umhleli, kunesinqamuleli - sebenzisa amathuluzi okumba idatha afana . Amathuluzi okukhipha idatha, njengamathuluzi okumba idatha, aklanyelwe ukonga amandla futhi enze ukucubungula idatha kube lula kuwo wonke umuntu. Lawa mathuluzi awawona nje ukonga kuphela, kodwa futhi awusizo kwabaqalayo. Ivumela abasebenzisi ukuthi baqoqe idatha phakathi namaminithi, bayigcine emafini, futhi bayithumele kumafomethi amaningi: i-Excel, i-CSV, i-HTML, i-JSON, noma kusizindalwazi esikusayithi nge-API.
Ububi bokukhishwa kwedatha
- Ukuphahlazeka kweseva
Lapho ukhipha idatha ngesilinganiso esikhulu, iseva yewebhu yesayithi okuhlosiwe ingase ilayishwe ngokweqile, okungaholela ekuphahlazekeni kweseva. Lokhu kuzolimaza izintshisekelo zomnikazi wesayithi. - Vimbela nge-IP
Uma umuntu eqoqa idatha kaningi, amawebhusayithi angavimba ikheli lakhe le-IP. Insiza ingavimba ngokuphelele ikheli le-IP noma ikhawulele ukufinyelela ngokwenza idatha ingaphelele. Ukubuyisa idatha futhi ugweme ukuvinjwa, udinga ukukwenza ngesivinini esimaphakathi futhi usebenzise amasu athile okuvimbela ukuvimba. - Izinkinga nomthetho
Ukukhipha idatha kuwebhu kuwela endaweni empunga uma kuziwa emthethweni. Amasayithi amakhulu afana ne-Linkedin ne-Facebook asho ngokucacile emigomeni yawo yokusebenzisa ukuthi noma yikuphi ukukhipha okuzenzakalelayo kwedatha akuvunyelwe. Kube khona amacala amaningi phakathi kwezinkampani ngenxa yemisebenzi ye-bot.
Umehluko Obalulekile Phakathi Kwezimayini Zedatha Nokukhishwa Kwedatha
- Ukumbiwa kwedatha kubizwa nangokuthi ukutholwa kolwazi kusizindalwazi, ukukhishwa kolwazi, ukuhlaziya idatha/iphethini, ukuqoqwa kolwazi. Ukukhishwa kwedatha kusetshenziswa ngokushintshana nokukhipha idatha yewebhu, ukuskena kwekhasi lewebhu, ukuqoqwa kwedatha, nokunye.
- Ucwaningo lokumbiwa kwedatha lusekelwe kakhulu kudatha ehlelekile kuyilapho ukumbiwa kwedatha ngokuvamile kuthola emithonjeni engahlelekile noma engakhiwe kahle.
- Umgomo wokumbiwa kwedatha ukwenza idatha isebenziseke kakhulu ekuhlaziyeni. Ukukhishwa kwedatha ukuqoqwa kwedatha endaweni eyodwa lapho ingagcinwa khona noma isetshenzwe.
- Ukuhlaziywa kokumbiwa kwedatha kusekelwe ezindleleni zezibalo zokuhlonza amaphethini noma amathrendi. Ukukhishwa kwedatha kusekelwe ezilimini zokuhlela noma amathuluzi okukhipha idatha ukuze udlule imithombo.
- Inhloso yokumbiwa kwedatha ukuthola amaqiniso abengaziwa noma ashaywe indiva ngaphambilini, kuyilapho ukukhishwa kwedatha kuphathelene nolwazi olukhona.
- Ukumbiwa kwedatha kuyinkimbinkimbi kakhulu futhi kudinga ukutshalwa kwezimali okukhulu ekuqeqesheni abantu. Ukukhishwa kwedatha ngethuluzi elifanele kungaba lula kakhulu futhi kungabizi kakhulu.
Sisiza abasaqalayo ukuthi bangadideki kudatha. Ikakhulukazi kuma-habravchans, senze ikhodi yokuphromotha I-HABR, enikeza isaphulelo esingeziwe esingu-10% kwisaphulelo esiboniswe kusibhengezo.
Izifundo ezengeziwe
Izihloko Ezifakiwe
Source: www.habr.com