Ezi buzzwords ezimbini zenzululwazi yedatha zibhidanisa abantu abaninzi. I-Data Mining isoloko iqondwa phosakeleyo njengokukhupha kunye nokubuyisela idatha, kodwa eneneni inzima kakhulu. Kule post, makhe sibeke ukugqiba kwiMigodi kwaye sifumane umahluko phakathi kweDatha yeMigodi kunye nokuThathwa kweDatha.
Yintoni i-Data Mining?
Ukuchithwa kwedatha, ebizwa ngokuba UkuFunyaniswa koLwazi kwiDatabase (KDD), bubuchule obuhlala busetyenziswa ukuhlalutya amanani amakhulu edatha kusetyenziswa ubuchule bamanani kunye nezibalo ukufumana iipateni ezifihliweyo okanye iintsingiselo kunye nokukhupha ixabiso kuzo.
Yintoni onokuyenza nge-Data Mining?
Ngokuzenzekela inkqubo, inokuskena oovimba beenkcukacha kwaye ichonge ngokusebenzayo iipateni ezifihliweyo. Kumashishini, ukumbiwa kwedatha kudla ngokusetyenziswa ukuchonga iipateni kunye nobudlelwane kwidatha ukunceda ukwenza izigqibo ezingcono zoshishino.
Imizekelo yesicelo
Emva kokuba i-data mining ixhaphake kakhulu kwi-1990s, iinkampani ezikuluhlu olubanzi lwamashishini, kubandakanywa ukuthengisa, imali, ukhathalelo lwempilo, ukuthutha, unxibelelwano, i-e-commerce, njl. Ukumbiwa kwedatha kunokunceda abathengi bamacandelo, babone ubuqhetseba, ukuthengisa kwangaphambili, nokunye okuninzi.
- Ukwahlulwa kwabathengi
Ngokuhlalutya idatha yabathengi kunye nokuchonga iimpawu zabathengi ekujoliswe kubo, iinkampani zinokujolisa kwiqela elihlukeneyo kwaye zibonelele ngezibonelelo ezikhethekileyo ezihlangabezana neemfuno zabo. - Uhlalutyo lweBhasikiti yeMarike
Obu buchule busekelwe kwingcamango yokuba xa uthenga iqela elithile leemveliso, kunokwenzeka ukuba uthenge elinye iqela leemveliso. Omnye umzekelo odumileyo: xa ootata bethengela iintsana zabo iidiapers, badla ngokuthenga ibhiya kunye namadiapers. - Uqikelelo lwentengiso
Oku kunokubonakala kufana nokuhlalutya ibhasikidi yeemarike, kodwa ngeli xesha uhlalutyo lwedatha lusetyenziselwa ukuqikelela ukuba umthengi uza kuthenga nini umkhiqizo kwakhona kwixesha elizayo. Umzekelo, umqeqeshi uthenga itoti yeprotheyini, ekufuneka ihlale iinyanga ezili-9. Ivenkile ethengisa le proteni iceba ukukhulula entsha kwiinyanga ze-9 ukuze umqeqeshi athenge kwakhona. - Ukufunyanwa kobuqhophololo
Ukumbiwa kwedatha kunceda kwiimodeli zokwakha ukufumanisa ubuqhophololo. Ngokuqokelela iisampulu zeengxelo zobuqhophololo nezisemthethweni, amashishini axhotyisiwe ukuze amisele ukuba zeziphi iintengiselwano ezikrokrelekayo. - Ukufunyanwa kweepateni kwimveliso
Kwishishini lemveliso, ukumbiwa kwedatha kusetyenziselwa ukuncedisa kuyilo lwenkqubo ngokuchonga ubudlelwane phakathi koyilo lwemveliso, iprofayili kunye neemfuno zabathengi. Ukumbiwa kwedatha kunokuqikelela amaxesha ophuhliso lwemveliso kunye neendleko.
Kwaye ezi ziimeko nje ezimbalwa zokusebenzisa umgodi wedatha.
Amanqanaba oMgodi weDatha
Ukumbiwa kwedatha yinkqubo epheleleyo yokuqokelela, ukukhetha, ukucoca, ukuguqula nokukhupha idatha ukuvavanya iipateni kwaye ekugqibeleni kutsalwe ixabiso.
Njengomthetho, yonke inkqubo yokumbiwa kwedatha inokushwankathelwa ngokwezigaba ezisi-7:
- Ukucoca idatha
Kwihlabathi lokwenyani, idatha ayisoloko icocwa kwaye icwangcisiwe. Zihlala zinengxolo, aziphelelanga, kwaye zinokuqulatha iimpazamo. Ukuqinisekisa ukuba isiphumo sokumbiwa kwedatha sichanekile, kufuneka uqale ucoce idatha. Ezinye iindlela zokucoca ziquka ukuzaliswa kwamaxabiso alahlekileyo, ukukhangela ngokuzenzekelayo kunye nokukhangela ngesandla, njl. - Ukudityaniswa kwedatha
Eli nqanaba apho idatha evela kwimithombo eyahlukeneyo ikhutshwe, idibaniswe kwaye idibaniswe. Imithombo inokuba nguvimba weenkcukacha, iifayile ezibhaliweyo, i-spreadsheets, amaxwebhu, iiseti zedatha ezininzi, i-Intanethi, njalo njalo. - Isampula yedatha
Ngokuqhelekileyo, akuyiyo yonke idatha edibeneyo efunekayo kwimigodi yedatha. Isampula yedatha yinqanaba apho idatha eluncedo kuphela ikhethiweyo kwaye ikhutshwe kwi-database enkulu. - Uguqulo lwedatha
Emva kokuba idatha ikhethiwe, iguqulwa ibe ziifom ezifanelekileyo zokumbiwa kwemigodi. Le nkqubo ibandakanya ukuqheleka, ukuhlanganiswa, ukulungelelanisa, njl. - Imayini yedatha
Apha kuza eyona nxalenye ibalulekileyo yokumbiwa kwedatha - usebenzisa iindlela ezikrelekrele ukufumana iipatheni kuyo. Inkqubo ibandakanya ukuhlehla, ukuhlelwa, ukuqikelela, ukuhlanganisana, ukufunda ngokunxulumana, nokunye. - Uvavanyo lomzekelo
Eli nyathelo lijolise ekuchongeni iipateni ezinokuba luncedo, ekulula ukuziqonda, kunye ne-hypothesis-support. - Ukumelwa kolwazi
Kwinqanaba lokugqibela, ulwazi olufunyenweyo lunikezelwa ngendlela ekhangayo ngokusebenzisa ukubonakaliswa kolwazi kunye neendlela zokubonisa.
Ukungalungi koMgodi weDatha
- Utyalo-mali olukhulu lwexesha kunye nomsebenzi
Ekubeni ukuchithwa kwedatha yinkqubo ende kunye neyinkimbinkimbi, idinga umsebenzi omningi ovela kubantu abanemveliso kunye nabanobuchule. Abavukuzi bedatha banokusebenzisa izixhobo ezinamandla zokumbiwa kwedatha, kodwa zifuna iingcali ukuba zilungiselele idatha kwaye ziqonde iziphumo. Ngenxa yoko, kungathatha ixesha ukusetyenzwa lonke ulwazi. - Ubumfihlo kunye nokhuseleko lwedatha
Ekubeni i-data mining iqokelela ulwazi lwabathengi ngeendlela zemarike, inokwaphula ubumfihlo bomsebenzisi. Ukongezelela, abahlaseli banokufumana idatha egcinwe kwiinkqubo zokumbiwa kwedatha. Oku kubangela ingozi kukhuseleko lwedatha yabathengi. Ukuba idatha ebiweyo isetyenziswa kakubi, inokubenzakalisa ngokulula abanye.
Ngentla yintshayelelo emfutshane yokwembiwa kwedatha. Njengoko sele ndikhankanyile, ukuchithwa kwedatha kubandakanya inkqubo yokuqokelela kunye nokudibanisa idatha, equka inkqubo yokukhutshwa kwedatha. Kule meko, kukhuselekile ukuthetha ukuba ukukhutshwa kwedatha kunokuba yinxalenye yenkqubo yexesha elide lokumba idatha.
Yintoni ukutsalwa kwedatha?
Eyaziwa nangokuthi "i-web data mining" kunye "ne-web scraping," le nkqubo sisenzo sokukhupha idatha kwimithombo yedatha (ehlala ingalungiswanga okanye ingacwangciswanga kakuhle) kwiindawo eziphakathi kwaye ibekwe kwindawo enye kwindawo yokugcina okanye ukuqhubeka. Ngokukodwa, imithombo yedatha engacwangciswanga ibandakanya amaphepha ewebhu, i-imeyile, amaxwebhu, iifayile zePDF, isicatshulwa esiskeniweyo, iingxelo ze-mainframe, iifayile ze-reel-to-reel, iintengiso, njl. Ugcino olusembindini lunokuba lwendawo, ilifu, okanye ingxubevange. Kubalulekile ukukhumbula ukuba ukukhutshwa kwedatha akubandakanyi ukuqhutyelwa phambili okanye olunye uhlalutyo olunokuthi lwenzeke kamva.
Unokwenza ntoni ngokutsalwa kweDatha?
Ngokwenene, iinjongo zokukhutshwa kwedatha ziwela kwiindidi ezi-3.
- Ugcino lwamaxwebhu
Ukukhutshwa kwedatha kunokuguqula idatha kwiifomathi ezibonakalayo: iincwadi, amaphephandaba, ii-invoyisi kwiifomathi zedijithali, ezifana nedatha yokugcina okanye yokugcina. - Ukutshintsha ifomati yedatha
Xa ufuna ukufuduka idatha ukusuka kwindawo yakho yangoku ukuya kwentsha phantsi kophuhliso, unokuqokelela idatha kwindawo yakho ngokuyikhupha. - Uhlalutyo lwedatha
Uhlalutyo olongezelelweyo lwedatha ekhutshiweyo ukufumana ingqiqo luqhelekileyo. Oku kunokubonakala kufana nokuchithwa kwedatha, kodwa khumbula ukuba ukuchithwa kwedatha yinjongo yokunciphisa idatha, kungekhona inxalenye yayo. Ngaphezu koko, idatha ihlalutywa ngokwahlukileyo. Umzekelo omnye: Abanini beevenkile ze-intanethi bakhupha ulwazi lwemveliso kwiindawo ze-e-commerce ezifana ne-Amazon ukubeka iliso kwizicwangciso zabakhuphisana ngexesha langempela. Njengokumbiwa kwedatha, ukukhutshwa kwedatha yinkqubo ezenzekelayo enenzuzo ezininzi. Ngaphambili, abantu babesebenzisa ukukopa kunye nokuncamathisela idatha ngesandla ukusuka kwenye indawo ukuya kwenye, eyayidla ixesha kakhulu. Ukutsalwa kwedatha kukhawulezisa ukuqokelela kwaye kuphucula kakhulu ukuchaneka kwedatha ekhutshiweyo.
Eminye imizekelo yokusebenzisa ukutsalwa kwedatha
Ngokufana nokuchithwa kwedatha, ukuchithwa kwedatha kusetyenziswa ngokubanzi kumashishini ahlukeneyo. Ukongeza ekubekeni iliso kumaxabiso kwi-e-commerce, imigodi yedatha inokunceda kuphando lwakho, ukuhlanganiswa kweendaba, ukuthengisa, indawo yokuhlala, uhambo kunye nokhenketho, ukubonisana, imali kunye nokunye okuninzi.
- Isizukulwana esikhokelayo
Iinkampani zinokukhupha idatha kwiinkcukacha: i-Yelp, i-Crunchbase, i-Yellowpages kwaye ivelise izikhokelo zophuhliso lwezoshishino. Ungayibukela le vidiyo ingezantsi ukuze ufunde indlela yokukhupha idatha kwiYellowpages usebenzisa . - Ukudityaniswa komxholo kunye neendaba
Iiwebhusayithi zokudibanisa umxholo zinokufumana imilambo rhoqo yedatha evela kwimithombo emininzi kwaye igcine iisayithi zabo zihlaziyiwe. - Uhlalutyo lweemvakalelo
Ngokukhupha uphononongo, amagqabantshintshi, kunye nengxelo kwiisayithi zemidiya yoluntu njenge-Instagram kunye ne-Twitter, iingcali zinokuhlalutya iimvakalelo ezisisiseko kwaye zifumane ukuqonda malunga nendlela uphawu, imveliso, okanye into ebonwa ngayo.
Amanyathelo okutsalwa kwedatha
Ukukhutshwa kwedatha yinqanaba lokuqala le-ETL (isifinyezo se-Extract, Transform, Load) kunye ne-ELT (i-extract, load and transform). I-ETL kunye ne-ELT ngokwazo ziyinxalenye yesicwangciso esipheleleyo sokudibanisa idatha. Ngamanye amazwi, ukutsalwa kwedatha kunokuba yinxalenye ye-data mining.
Khupha, guqula, layisha
Nangona i-data mining imalunga nokukhupha ulwazi kwiimali ezininzi zedatha, ukukhutshwa kwedatha yinkqubo emfutshane kakhulu kwaye ilula. Inokuncitshiswa ibe ngamanqanaba amathathu:
- Ukukhetha umthombo wedatha
Khetha umthombo ofuna ukukhupha idatha kuwo, njengewebhusayithi. - Ukuqokelelwa kwedatha
Thumela isicelo "GET" kwindawo kwaye ucazulule uxwebhu lweHTML olunesiphumo usebenzisa iilwimi zeprogram ezifana nePython, PHP, R, Ruby, njl. - Ukugcinwa kwedatha
Gcina idatha kwisiseko sedatha yakho yasekhaya okanye kwindawo yokugcina ilifu ukuze uyisebenzise kwixesha elizayo. Ukuba ungumdwebi wenkqubo onamava ofuna ukukhupha idatha, la manyathelo angasentla anokubonakala elula kuwe. Nangona kunjalo, ukuba awuyi khowudi, indlela emfutshane kukusebenzisa izixhobo zokutsalwa kwedatha, umz. . Izixhobo zokutsalwa kwedatha, njengezixhobo zokumbiwa kwedatha, ziyilelwe ukonga amandla kunye nokwenza inkqubo yedata ibe lula kumntu wonke. Ezi zixhobo azikho nje kuphela uqoqosho kodwa zikwanomdla kwabaqalayo. Bavumela abasebenzisi ukuba baqokelele idatha ngaphakathi kwemizuzu, bayigcine efini kwaye bayithumele kwiifomathi ezininzi: i-Excel, i-CSV, i-HTML, i-JSON okanye i-database yewebhusayithi nge-API.
Izinto ezingalunganga zokutsalwa kwedatha
- Ukuwa kwiseva
Xa kufunyanwa idatha kumlinganiselo omkhulu, iseva yewebhu yendawo ekujoliswe kuyo inokugcwala kakhulu, nto leyo enokubangela ukuba umncedisi aphuke. Oku kuya kwenzakalisa umdla womnini sayithi. - Ukuvalwa kwe-IP
Xa umntu eqokelela idatha rhoqo, iiwebhusayithi zinokuvala idilesi yazo ye-IP. Isixhobo sinokukhanyela ngokupheleleyo idilesi ye-IP okanye ukufikelela umda, okwenza idatha ingagqibekanga. Ukubuyisela idatha kwaye ugweme ukuvimba, kufuneka uyenze ngesantya esiphakathi kwaye usebenzise ezinye iindlela zokulwa nokuthintela. - Iingxaki nomthetho
Ukukhupha idatha kwiwebhu kuwela kwindawo engwevu xa kuziwa kumthetho. Iziza ezinkulu ezifana ne-Linkedin kunye ne-Facebook zichaza ngokucacileyo kwimimiselo yazo yokusetyenziswa ukuba nayiphi na i-extraction data ezenzekelayo inqatshelwe. Kukho amatyala amaninzi phakathi kweenkampani ngenxa yomsebenzi we-bot.
Iiyantlukwano eziphambili phakathi kweDatha yeMigodi kunye nokuThwalwa kweDatha
- Ukumbiwa kwedatha kukwabizwa ngokuba kukufunyanwa kolwazi koovimba beenkcukacha, ukutsalwa kolwazi, uhlalutyo lwedatha/iphethini, ukuqokelela ulwazi. Ukutsalwa kwedatha kusetyenziswa ngokungafaniyo kunye nokukhutshwa kwedatha yewebhu, ukukhasa kwewebhu, ukuchithwa kwedatha, njalo njalo.
- Uphando lokumbiwa kwedatha lusekwe ikakhulu kwidatha ecwangcisiweyo, ngelixa ukumbiwa kwedatha idla ngokukhutshwa kwimithombo engacwangciswanga okanye engacwangciswanga kakuhle.
- Injongo yokumbiwa kwedatha kukwenza idatha ibe luncedo ngakumbi kuhlalutyo. Ukutsalwa kwedatha kukuqokelela kwedatha kwindawo enye apho inokugcinwa okanye iqhutywe khona.
- Uhlalutyo kwimigodi yedatha lusekwe kwiindlela zemathematika zokuchonga iipateni okanye iintsingiselo. Ukutsalwa kwedatha kusekwe kwiilwimi zokucwangcisa okanye izixhobo zokutsalwa kwedatha ukukhasa kwimithombo.
- Injongo yokumbiwa kwedatha kukufumana iinyani ebezingaziwa okanye ebezingahoywa ngaphambili, ngelixa ukutsalwa kwedatha kujongana nolwazi olukhoyo.
- Ukumbiwa kwedatha kuntsonkothile kakhulu kwaye kufuna utyalo-mali olukhulu ekuqeqesheni abantu. Ukutsalwa kwedatha, xa kusetyenziswe ngesixhobo esifanelekileyo, kunokuba lula kakhulu kwaye kungabizi kakhulu.
Sinceda abaqalayo ukuba bangabhideki kwiDatha. Senze ikhowudi yokuthengisa ngakumbi kubahlali baseKhabra IHABR, ukunika isaphulelo esongezelelweyo se-10% kwisaphulelo esiboniswe kwibhena.
Izifundo ezingakumbi
Amanqaku akhoyo
umthombo: www.habr.com