Ob Cov Ntaub Ntawv Kev Tshawb Fawb buzzwords ua rau ntau tus neeg tsis meej pem. Cov ntaub ntawv Mining feem ntau nkag siab yuam kev raws li kev rho tawm thiab khaws cov ntaub ntawv, tab sis qhov tseeb yog qhov nyuaj dua. Hauv tsab xov xwm no, cia peb teev Mining thiab nrhiav qhov sib txawv ntawm Data Mining thiab Data Extraction.
Data Mining yog dab tsi?
Cov ntaub ntawv mining, kuj hu ua Database Knowledge Discovery (KDD), yog cov txheej txheem feem ntau siv los txheeb xyuas cov ntaub ntawv loj uas siv cov kev txheeb cais thiab lej los nrhiav cov qauv zais lossis cov qauv thiab rho tawm tus nqi ntawm lawv.
Yuav ua li cas nrog Data Mining?
Los ntawm automating txheej txheem, tuaj yeem xauj databases thiab zoo tshaj tawm cov qauv zais. Rau cov lag luam, cov ntaub ntawv mining feem ntau yog siv los nrhiav cov qauv thiab kev sib raug zoo hauv cov ntaub ntawv los pab txiav txim siab ua lag luam zoo dua.
Daim ntawv thov piv txwv
Tom qab cov ntaub ntawv mining tau nthuav dav hauv xyoo 1990, cov tuam txhab hauv ntau qhov kev lag luam, suav nrog cov khw muag khoom, nyiaj txiag, kev noj qab haus huv, kev thauj mus los, kev sib txuas lus, kev lag luam e-lag luam, thiab lwm yam, pib siv cov ntaub ntawv mining txoj kev kom tau txais cov ntaub ntawv ntawm cov ntaub ntawv hauv paus. Kev tshawb nrhiav cov ntaub ntawv tuaj yeem pab faib cov neeg siv khoom, txheeb xyuas kev dag ntxias, kwv yees kev muag khoom, thiab lwm yam.
- Cov neeg siv khoom segmentation
Los ntawm kev txheeb xyuas cov neeg siv khoom cov ntaub ntawv thiab txheeb xyuas qhov zoo ntawm cov neeg siv khoom lub hom phiaj, cov tuam txhab tuaj yeem muab lawv tso rau hauv ib pab pawg thiab muab cov kev tshwj xeeb uas ua tau raws li lawv cov kev xav tau. - Market Basket Analysis
Cov txheej txheem no yog ua raws li txoj kev xav tias yog tias koj yuav ib pawg ntawm cov khoom, koj muaj peev xwm yuav ntau pawg ntawm cov khoom. Ib qho piv txwv nto moo: thaum leej txiv yuav daim pawm rau lawv cov menyuam mos, lawv nyiam yuav npias nrog rau cov pawm. - Kev muag khoom kwv yees
Nws yuav zoo li zoo ib yam li kev lag luam pob tawb tsom xam, tab sis lub sij hawm no cov ntaub ntawv tsom xam yog siv los kwv yees thaum twg tus neeg yuav khoom yuav khoom dua yav tom ntej. Piv txwv li, tus kws qhia yuav ib lub kaus poom ntawm cov protein uas yuav tsum kav ntev li 9 lub hlis. Lub khw uas muag cov protein no npaj yuav tso ib qho tshiab hauv 9 lub hlis kom tus kws qhia yuav rov yuav nws dua. - Nrhiav kev dag ntxias
Cov ntaub ntawv mining pab hauv kev tsim qauv rau kev tshawb nrhiav kev dag ntxias. Los ntawm kev sau cov qauv ntawm kev dag thiab cov ntaub ntawv tseeb, cov lag luam muaj zog los txiav txim siab qhov kev lag luam twg yog qhov tsis txaus ntseeg. - Qauv nrhiav tau hauv kev tsim khoom
Hauv kev tsim khoom lag luam, cov ntaub ntawv tsuas yog siv los pab tsim cov txheej txheem los ntawm kev txheeb xyuas kev sib raug zoo ntawm cov khoom tsim, profile, thiab cov neeg siv khoom xav tau. Cov ntaub ntawv tsuas tuaj yeem kwv yees lub sijhawm tsim khoom thiab cov nqi.
Thiab cov no tsuas yog qee qhov kev siv rau cov ntaub ntawv mining.
Cov theem ntawm cov ntaub ntawv mining
Cov ntaub ntawv tsuas yog cov txheej txheem ntawm kev sau, xaiv, tu, hloov pauv, thiab rho tawm cov ntaub ntawv txhawm rau ntsuas cov qauv thiab, thaum kawg, rho tawm tus nqi.
Feem ntau, tag nrho cov ntaub ntawv mining txheej txheem tuaj yeem muab sau ua 7 kauj ruam:
- Kev ntxuav cov ntaub ntawv
Hauv lub ntiaj teb tiag tiag, cov ntaub ntawv tsis yog ib txwm ntxuav thiab tsim qauv. Lawv feem ntau nrov, tsis tiav, thiab tej zaum yuav muaj qhov yuam kev. Yuav kom paub tseeb tias cov ntaub ntawv mining tshwm sim yog qhov tseeb, koj thawj zaug yuav tsum tau ntxuav cov ntaub ntawv. Qee txoj hauv kev tu muaj xws li sau cov txiaj ntsig uas ploj lawm, kev tswj tsis siv neeg thiab phau ntawv, thiab lwm yam. - Kev koom ua ke cov ntaub ntawv
Qhov no yog theem uas cov ntaub ntawv los ntawm ntau qhov chaw raug muab rho tawm, ua ke thiab sib xyaw ua ke. Cov peev txheej tuaj yeem yog cov ntaub ntawv, cov ntawv sau, cov ntawv nthuav qhia, cov ntaub ntawv, cov ntaub ntawv ntau yam, Internet, thiab lwm yam. - Cov ntaub ntawv sampling
Feem ntau, tsis yog tag nrho cov ntaub ntawv sib xyaw xav tau hauv cov ntaub ntawv mining. Cov ntaub ntawv sampling yog theem uas tsuas yog cov ntaub ntawv tseem ceeb raug xaiv thiab muab rho tawm los ntawm cov ntaub ntawv loj. - Kev hloov ntaub ntawv
Thaum cov ntaub ntawv raug xaiv, nws yog hloov dua siab tshiab rau hauv cov ntaub ntawv tsim nyog rau mining. Cov txheej txheem no suav nrog normalization, aggregation, generalization, thiab lwm yam. - Cov ntaub ntawv mining
Ntawm no los txog qhov tseem ceeb tshaj plaws ntawm cov ntaub ntawv mining - siv cov kev txawj ntse los nrhiav cov qauv hauv lawv. Cov txheej txheem suav nrog kev thim rov qab, kev faib tawm, kev kwv yees, kev sib koom ua ke, kev kawm koom nrog, thiab lwm yam. - Kev ntsuas tus qauv
Cov kauj ruam no yog txhawm rau txheeb xyuas cov qauv uas muaj txiaj ntsig zoo, nkag siab yooj yim, nrog rau cov qauv uas txhawb kev xav. - Kev Paub Txog Kev Paub
Nyob rau theem kawg, cov ntaub ntawv tau txais yog nthuav tawm hauv txoj kev txaus nyiam siv kev paub sawv cev thiab kev pom pom.
Disadvantages ntawm Data Mining
- Kev nqis peev loj ntawm lub sijhawm thiab kev ua haujlwm
Txij li cov ntaub ntawv tsuas yog ib txoj haujlwm ntev thiab nyuaj, nws yuav tsum tau ua haujlwm ntau los ntawm cov neeg tsim khoom thiab cov neeg txawj ntse. Cov kws tshawb fawb cov ntaub ntawv tuaj yeem siv cov ntaub ntawv mining muaj zog, tab sis lawv xav tau cov kws tshaj lij los npaj cov ntaub ntawv thiab nkag siab cov txiaj ntsig. Yog li ntawd, nws yuav siv sij hawm qee lub sij hawm los ua txhua yam ntaub ntawv. - Cov ntaub ntawv ntiag tug thiab kev ruaj ntseg
Vim cov ntaub ntawv mining sau cov neeg siv khoom cov ntaub ntawv los ntawm kev ua lag luam, nws tuaj yeem ua txhaum tus neeg siv khoom ntiag tug. Tsis tas li ntawd, hackers tuaj yeem tau txais cov ntaub ntawv khaws cia hauv cov ntaub ntawv mining systems. Qhov no ua rau muaj kev hem thawj rau kev ruaj ntseg ntawm cov neeg siv khoom cov ntaub ntawv. Yog tias cov ntaub ntawv raug nyiag siv tsis raug, nws tuaj yeem ua mob rau lwm tus.
Cov saum toj no yog cov lus qhia luv luv rau cov ntaub ntawv mining. Raws li kuv twb tau hais lawm, cov ntaub ntawv mining muaj cov txheej txheem ntawm kev sau thiab sib sau cov ntaub ntawv, uas suav nrog cov txheej txheem ntawm kev rho tawm cov ntaub ntawv (cov ntaub ntawv rho tawm). Hauv qhov no, nws muaj kev nyab xeeb los hais tias kev rho tawm cov ntaub ntawv tuaj yeem yog ib feem ntawm cov ntaub ntawv mining ntev.
Data Extraction yog dab tsi?
Kuj tseem hu ua "web data mining" thiab "web scraping", cov txheej txheem no yog ua los ntawm kev rho tawm cov ntaub ntawv los ntawm (feem ntau tsis muaj teeb meem lossis cov txheej txheem tsis zoo) cov ntaub ntawv mus rau hauv qhov chaw nruab nrab thiab nruab nrab hauv ib qho chaw rau kev khaws cia lossis ua haujlwm ntxiv. Tshwj xeeb, cov ntaub ntawv tsis muaj teeb meem suav nrog nplooj ntawv web, email, cov ntaub ntawv, cov ntaub ntawv PDF, cov ntawv luam theej duab, cov ntawv ceeb toom tseem ceeb, cov ntaub ntawv reel, tshaj tawm, thiab lwm yam. Centralized cia tuaj yeem yog hauv zos, huab lossis hybrid. Nws yog ib qho tseem ceeb kom nco ntsoov tias kev rho tawm cov ntaub ntawv tsis suav nrog kev ua haujlwm lossis lwm yam kev tshuaj xyuas uas yuav tshwm sim tom qab.
Yuav ua li cas nrog Data Extraction?
Yeej, cov ntaub ntawv rho tawm lub hom phiaj poob rau hauv 3 pawg.
- Khawv koob
Cov ntaub ntawv rho tawm tuaj yeem hloov cov ntaub ntawv los ntawm lub cev tawm tswv yim xws li cov phau ntawv, ntawv xov xwm, invoice rau cov hom ntawv xws li cov ntaub ntawv khaws cia lossis thaub qab. - Hloov cov ntaub ntawv hom
Thaum koj xav hloov cov ntaub ntawv los ntawm koj qhov chaw tam sim no mus rau ib qho tshiab hauv kev txhim kho, koj tuaj yeem sau cov ntaub ntawv los ntawm koj tus kheej qhov chaw los ntawm kev rho tawm. - Cov ntaub ntawv tsom xam
Nws yog ib qho uas yuav tau txheeb xyuas ntxiv cov ntaub ntawv rho tawm kom tau txais kev pom rau nws. Qhov no yuav zoo ib yam li cov ntaub ntawv mining, tab sis nco ntsoov tias cov ntaub ntawv mining yog lub hom phiaj ntawm cov ntaub ntawv mining, tsis yog ib feem ntawm nws. Ntxiv mus, cov ntaub ntawv raug txheeb xyuas txawv. Ib qho piv txwv yog tias cov tswv khw hauv online rub cov khoom lag luam los ntawm cov chaw lag luam e-lag luam xws li Amazon los saib xyuas cov tswv yim sib tw hauv lub sijhawm. Zoo li cov ntaub ntawv mining, cov ntaub ntawv rho tawm yog cov txheej txheem automated nrog ntau cov txiaj ntsig. Yav dhau los, tib neeg tau theej thiab muab cov ntaub ntawv los ntawm ib qho chaw mus rau lwm qhov, uas siv sijhawm ntau heev. Cov ntaub ntawv rho tawm nrawm nrawm thiab txhim kho qhov tseeb ntawm cov ntaub ntawv rho tawm.
Qee qhov piv txwv ntawm kev siv Data Extraction
Zoo ib yam li cov ntaub ntawv mining, cov ntaub ntawv tsuas yog siv dav hauv ntau yam lag luam. Ntxiv rau kev saib xyuas tus nqi lag luam e-lag luam, kev tshawb nrhiav cov ntaub ntawv tuaj yeem pab nrog koj tus kheej kev tshawb fawb, kev sib sau xov xwm, kev lag luam, vaj tse, kev mus ncig thiab kev ncig xyuas, kev sab laj, nyiaj txiag, thiab lwm yam.
- Lead tiam
Cov tuam txhab lag luam tuaj yeem rho tawm cov ntaub ntawv los ntawm cov npe: Yelp, Crunchbase, Yellowpages thiab tsim cov thawj coj rau kev txhim kho kev lag luam. Koj tuaj yeem saib cov yeeb yaj kiab hauv qab no kom paub yuav ua li cas rho cov ntaub ntawv los ntawm Yellowpages nrog . - Aggregation ntawm cov ntsiab lus thiab xov xwm
Cov ntsiab lus sib sau cov vev xaib tuaj yeem tau txais cov ntaub ntawv tsis tu ncua los ntawm ntau qhov chaw thiab khaws lawv cov vev xaib mus txog hnub tim. - Kev ntsuas siab
Tom qab tshem tawm cov kev tshuaj xyuas, cov lus pom, thiab cov lus pov thawj los ntawm kev sib raug zoo xws li Instagram thiab Twitter, cov kws tshaj lij tuaj yeem txheeb xyuas tus cwj pwm hauv qab thiab tau txais kev nkag siab txog yuav ua li cas lub hom phiaj, cov khoom, lossis qhov tshwm sim pom.
Cov kauj ruam Extraction
Cov ntaub ntawv rho tawm yog thawj theem ntawm ETL (Extract, Transform, Load: Extract, Transform, Load) thiab ELT (Extract, Load, and Transform). ETL thiab ELT yog lawv tus kheej ib feem ntawm kev ua tiav cov ntaub ntawv kev sib koom ua ke. Hauv lwm lo lus, rho tawm cov ntaub ntawv tuaj yeem yog ib feem ntawm lawv cov extraction.
Extract, hloov, load
Thaum cov ntaub ntawv mining yog txhua yam hais txog kev rho tawm cov ntaub ntawv los ntawm cov ntaub ntawv loj, cov ntaub ntawv rho tawm yog cov txheej txheem luv dua thiab yooj yim dua. Nws tuaj yeem raug txo mus rau peb theem:
- Xaiv cov ntaub ntawv qhov chaw
Xaiv qhov chaw koj xav rho tawm cov ntaub ntawv los ntawm, xws li lub vev xaib. - Kev sau cov ntaub ntawv
Xa ib daim ntawv thov "GET" rau lub xaib thiab txheeb xyuas cov ntaub ntawv HTML uas siv cov lus programming xws li Python, PHP, R, Ruby, thiab lwm yam. - Cov ntaub ntawv cia
Txuag cov ntaub ntawv rau koj lub zos database lossis huab cia rau yav tom ntej. Yog tias koj yog tus paub txog programmer uas xav rho tawm cov ntaub ntawv, cov kauj ruam saum toj no yuav zoo li yooj yim rau koj. Txawm li cas los xij, yog tias koj tsis yog programmer, muaj ib qho shortcut - siv cov ntaub ntawv mining cov cuab yeej zoo li . Cov cuab yeej rho tawm cov ntaub ntawv, ib yam li cov ntaub ntawv mining, yog tsim los txuag lub zog thiab ua kom cov ntaub ntawv ua tau yooj yim rau txhua tus. Cov cuab yeej no tsis yog tsuas yog kev lag luam xwb, tab sis kuj yog cov neeg pib tshiab. Lawv tso cai rau cov neeg siv sau cov ntaub ntawv hauv feeb, khaws cia hauv huab, thiab xa tawm mus rau ntau hom: Excel, CSV, HTML, JSON, lossis rau cov ntaub ntawv hauv qhov chaw ntawm API.
Disadvantages ntawm Data Extraction
- Server crash
Thaum rho tawm cov ntaub ntawv ntawm cov nplai loj, lub vev xaib server ntawm lub hom phiaj lub vev xaib yuav dhau mus, uas tuaj yeem ua rau lub server poob. Qhov no yuav ua phem rau qhov kev txaus siab ntawm tus tswv ntawm lub xaib. - txwv los ntawm IP
Thaum ib tug neeg sau cov ntaub ntawv ntau dhau lawm, cov vev xaib tuaj yeem thaiv lawv tus IP chaw nyob. Cov peev txheej tuaj yeem txwv tus IP chaw nyob lossis txwv tsis pub nkag los ntawm kev ua cov ntaub ntawv tsis tiav. Txhawm rau khaws cov ntaub ntawv thiab tsis txhob thaiv kev thaiv, koj yuav tsum ua nws ntawm qhov nruab nrab ceev thiab siv qee cov txheej txheem los tiv thaiv kev thaiv. - Teeb meem nrog txoj cai
Kev rho tawm cov ntaub ntawv los ntawm lub vev xaib poob rau hauv cheeb tsam grey thaum nws los txog rau kev raug cai. Cov chaw loj xws li Linkedin thiab Facebook qhia meej meej hauv lawv cov ntsiab lus ntawm kev siv uas txwv tsis pub rho tawm cov ntaub ntawv tsis siv neeg. Muaj ntau qhov kev foob ntawm cov tuam txhab vim yog cov haujlwm bot.
Qhov tseem ceeb sib txawv ntawm cov ntaub ntawv mining thiab cov ntaub ntawv rho tawm
- Cov ntaub ntawv tsuas yog hu ua kev paub tshawb pom hauv databases, kev paub txog kev rho tawm, cov ntaub ntawv / qauv tsom xam, kev sib sau cov ntaub ntawv. Kev rho tawm cov ntaub ntawv yog siv sib hloov nrog cov ntaub ntawv hauv web extraction, web page scanning, sau cov ntaub ntawv, thiab lwm yam.
- Kev tshawb nrhiav cov ntaub ntawv tsuas yog feem ntau raws li cov ntaub ntawv tsim kho thaum cov ntaub ntawv mining feem ntau rub los ntawm cov khoom siv tsis zoo lossis cov txheej txheem tsis zoo.
- Lub hom phiaj ntawm cov ntaub ntawv tsuas yog ua kom cov ntaub ntawv muaj txiaj ntsig zoo rau kev txheeb xyuas. Cov ntaub ntawv rho tawm yog kev sau cov ntaub ntawv rau hauv ib qho chaw uas nws tuaj yeem khaws cia lossis ua tiav.
- Kev tshuaj xyuas hauv cov ntaub ntawv tsuas yog ua raws li kev ua lej los txheeb xyuas cov qauv lossis cov qauv. Cov ntaub ntawv rho tawm yog ua raws li cov lus programming lossis cov cuab yeej rho tawm cov ntaub ntawv los hla cov peev txheej.
- Lub hom phiaj ntawm cov ntaub ntawv tsuas yog nrhiav qhov tseeb uas tsis tau paub dhau los lossis tsis quav ntsej, thaum cov ntaub ntawv rho tawm cuam tshuam nrog cov ntaub ntawv uas twb muaj lawm.
- Cov ntaub ntawv tsuas yog qhov nyuaj dua thiab xav tau kev nqis peev loj hauv kev cob qhia tib neeg. Kev rho tawm cov ntaub ntawv nrog cov cuab yeej zoo tuaj yeem yooj yim heev thiab siv tau zoo.
Peb pab cov pib tshiab kom tsis txhob tsis meej pem hauv Cov Ntaub Ntawv. Tshwj xeeb tshaj yog rau habravchans, peb tau ua ib tug promotional code HABR, muab 10% luv nqi ntxiv rau qhov luv nqi qhia rau ntawm daim ntawv lo.
Cov chav kawm ntxiv
Cov ntsiab lus tshwj xeeb
Tau qhov twg los: www.hab.com