Kuv tab tom qhia koj los ntawm kev paub txog tus kheej yam twg muaj txiaj ntsig nyob qhov twg thiab thaum twg. Nws yog lub ntsiab lus thiab thesis, kom nws paub meej tias dab tsi thiab qhov twg koj tuaj yeem khawb ntxiv - tab sis ntawm no kuv muaj kev paub txog tus kheej tshwj xeeb, tej zaum txhua yam txawv kiag li rau koj.
Vim li cas thiaj tseem ceeb kom paub thiab siv tau cov lus nug? Ntawm nws qhov tseem ceeb, Cov Ntaub Ntawv Kev Tshawb Fawb muaj ntau theem tseem ceeb ntawm kev ua haujlwm, thiab qhov tseem ceeb tshaj plaws (tsis muaj nws, yeej tsis muaj dab tsi yuav ua haujlwm!) yog tau txais lossis rho tawm cov ntaub ntawv. Feem ntau, cov ntaub ntawv yog zaum qhov chaw hauv qee daim ntawv thiab yuav tsum tau "rov qab" los ntawm qhov ntawd.
Cov lus nug tso cai rau koj rho tawm cov ntaub ntawv no heev! Thiab hnub no kuv yuav qhia koj txog cov lus nug uas muaj txiaj ntsig zoo rau kuv thiab kuv yuav qhia koj thiab qhia koj qhov twg thiab qhov tseeb - vim li cas nws thiaj li yuav tsum tau kawm.
Yuav muaj peb qhov tseem ceeb ntawm cov ntaub ntawv nug hom, uas peb yuav tham hauv kab lus no:
- "Standard" query languages yog qhov feem ntau nkag siab thaum tham txog cov lus nug, xws li kev sib raug zoo algebra lossis SQL.
- Scripting query languages: Piv txwv li, Python yam pandas, numpy lossis plhaub scripting.
- Cov lus nug rau kev paub graphs thiab graph databases.
Txhua yam uas sau ntawm no tsuas yog ib qho kev paub ntawm tus kheej xwb, dab tsi yog qhov tseem ceeb, nrog rau cov lus piav qhia ntawm cov xwm txheej thiab "vim li cas nws thiaj li xav tau" - txhua tus tuaj yeem sim seb qhov xwm txheej zoo li cas tuaj yeem tuaj rau koj thiab sim npaj rau lawv ua ntej los ntawm kev nkag siab cov lus no ua ntej koj yuav tsum tau thov (kho) ntawm ib qhov project lossis txawm mus rau qhov project uas lawv xav tau.
"Standard" lus nug
Standard query languages yog precisely nyob rau hauv lub siab hais tias peb feem ntau xav txog lawv thaum peb tham txog queries.
Relational algebra
Vim li cas kev sib raug zoo algebra niaj hnub no? Txhawm rau kom muaj kev nkag siab zoo txog vim li cas cov lus nug tau teeb tsa hauv ib txoj hauv kev thiab siv lawv lub siab, koj yuav tsum nkag siab cov ntsiab lus hauv qab lawv.
Relational algebra yog dab tsi?
Cov ntsiab lus tseem ceeb yog raws li hauv qab no: kev sib raug zoo algebra yog qhov kaw kaw ntawm kev ua haujlwm ntawm kev sib raug zoo hauv cov ntaub ntawv sib raug zoo. Txhawm rau muab nws me ntsis ntxiv rau tib neeg, qhov no yog qhov kev ua haujlwm ntawm cov ntxhuav xws li qhov tshwm sim yog ib lub rooj.
Saib tag nrho cov haujlwm kev sib raug zoo hauv tsab xov xwm los ntawm Habr - ntawm no peb piav qhia vim li cas koj yuav tsum paub thiab qhov twg nws tuaj yeem siv tau.
Vim li cas?
Pib nkag siab txog cov lus nug dab tsi yog txhua yam hais txog thiab kev ua haujlwm dab tsi tom qab cov lus qhia hauv cov lus nug tshwj xeeb feem ntau muab kev nkag siab tob txog qhov ua haujlwm hauv cov lus nug thiab yuav ua li cas.
Sau los ntawm cov ntawv. Ib qho piv txwv ntawm kev ua haujlwm: koom, uas koom nrog cov rooj.
Cov ntaub ntawv rau kev kawm:
. Feem ntau, muaj ntau cov ntaub ntawv ntawm kev sib raug zoo algebra thiab txoj kev xav - Coursera, Udacity. Kuj tseem muaj ntau yam khoom siv hauv online, suav nrog qhov zoo . Kuv tus kheej cov lus qhia: koj yuav tsum nkag siab txog kev sib raug zoo algebra zoo heev - qhov no yog lub hauv paus ntawm cov hauv paus.
SQL
Sau los ntawm lus.
SQL yog qhov tseem ceeb ntawm kev siv ntawm kev sib raug zoo algebra - nrog rau qhov tseem ceeb caveat, SQL yog tshaj tawm! Ntawd yog, thaum sau cov lus nug hauv cov lus ntawm kev sib raug zoo algebra, koj yeej hais tias yuav suav li cas - tab sis nrog SQL koj qhia qhov koj xav rho tawm, thiab tom qab ntawd DBMS twb tsim tawm (zoo) kab lus hauv cov lus ntawm kev sib raug zoo algebra (lawv. kev sib npaug yog paub rau peb ).
Sau los ntawm lus.
Vim li cas?
Relational DBMSs: Oracle, Postgres, SQL Server, thiab lwm yam tseem muaj nyob txhua qhov chaw thiab muaj lub caij nyoog siab kawg uas koj yuav tau cuam tshuam nrog lawv, uas txhais tau hais tias koj yuav tau nyeem SQL (uas yog qhov yuav muaj) lossis sau nws ( tsis zoo li ib yam).
Nyeem thiab kawm dab tsi
Raws li tib cov kev sib txuas saum toj no (hais txog kev sib raug zoo algebra), muaj cov khoom siv tsis txaus ntseeg, piv txwv li, .
Los ntawm txoj kev, NoSQL yog dab tsi?
"Nws tsim nyog hais dua ib zaug ntxiv tias lo lus "NoSQL" muaj lub hauv paus chiv keeb kiag li thiab tsis muaj cov ntsiab lus lees paub lossis lub koom haum tshawb fawb tom qab nws." Sib raug ntawm Habr.
Qhov tseeb, tib neeg pom tau hais tias tus qauv kev sib raug zoo tsis tas yuav tsum tau daws ntau yam teeb meem, tshwj xeeb tshaj yog rau cov uas, piv txwv li, kev ua tau zoo yog qhov tseem ceeb thiab qee cov lus nug yooj yim nrog kev sib sau ua ke - qhov tseem ceeb yog qhov tseem ceeb rau kev ntsuas kev ntsuas sai sai thiab sau rau hauv database, thiab feem ntau cov yam ntxwv yog kev sib raug zoo muab tawm tsis tau tsuas yog tsis tsim nyog, tab sis kuj muaj teeb meem - yog vim li cas normalize ib yam dab tsi yog hais tias nws yuav lwj qhov tseem ceeb tshaj plaws rau peb (rau tej hauj lwm tshwj xeeb) - productivity?
Tsis tas li ntawd, cov kev hloov pauv hloov tau feem ntau xav tau hloov pauv cov lej ruaj khov ntawm cov qauv kev sib raug zoo ntawm cov qauv qub - thiab qhov no ua kom yooj yim rau kev txhim kho daim ntawv thov thaum nws tseem ceeb heev rau kev xa cov kab ke thiab pib ua haujlwm sai, ua cov txiaj ntsig - lossis schema thiab hom khaws cia cov ntaub ntawv. tsis tseem ceeb li.
Piv txwv li, peb tab tom tsim cov txheej txheem kws tshaj lij thiab xav khaws cov ntaub ntawv ntawm cov npe tshwj xeeb nrog rau qee cov ntaub ntawv meta - peb yuav tsis paub tag nrho cov teb thiab tsuas yog khaws JSON rau txhua cov ntaub ntawv - qhov no muab rau peb ib puag ncig yooj yim rau kev nthuav tawm cov ntaub ntawv. qauv thiab sai iterating - yog li nyob rau hauv no qhov teeb meem no, NoSQL yuav txawm preferable thiab nyeem tau ntau dua. Piv txwv nkag (los ntawm ib qho ntawm kuv cov haujlwm uas NoSQL yog qhov uas nws xav tau).
{"en_wikipedia_url":"https://en.wikipedia.org/wiki/Johnny_Cash",
"ru_wikipedia_url":"https://ru.wikipedia.org/wiki/?curid=301643",
"ru_wiki_pagecount":149616,
"entity":[42775,"Джонни Кэш","ru"],
"en_wiki_pagecount":2338861}
Koj tuaj yeem nyeem ntxiv hais txog NoSQL.
Yuav kawm dab tsi?
Ntawm no, theej, koj tsuas yog yuav tsum tau ua tib zoo txheeb xyuas koj txoj haujlwm, nws muaj dab tsi thiab cov NoSQL systems muaj dab tsi uas yuav haum rau cov lus piav qhia - thiab tom qab ntawd pib kawm cov kab ke no.
Scripting Query Languages
Thaum xub thawj, nws zoo li, dab tsi Python yuav tsum ua nrog nws feem ntau - nws yog lus programming, thiab tsis yog hais txog cov lus nug.
- Pandas yog ib tug Swiss Army riam ntawm Data Science; ib tug loj npaum li cas ntawm cov ntaub ntawv transformation, aggregation, thiab lwm yam tshwm sim nyob rau hauv nws.
- Numpy - vector xam, matrices thiab linear algebra muaj.
- Scipy - muaj ntau cov lej hauv cov pob no, tshwj xeeb tshaj yog cov stats.
- Jupyter lab - ntau qhov kev tshawb nrhiav cov ntaub ntawv txheeb xyuas haum zoo rau hauv lub laptops - pab tau kom paub.
- Kev thov - ua haujlwm nrog lub network.
- Pyspark yog nrov heev ntawm cov ntaub ntawv engineers, feem ntau koj yuav tau cuam tshuam nrog qhov no los yog Spark, tsuas yog vim lawv qhov chaw.
- * Selenium - muaj txiaj ntsig zoo rau kev sau cov ntaub ntawv los ntawm cov chaw thiab cov peev txheej, qee zaum tsis muaj lwm txoj hauv kev kom tau txais cov ntaub ntawv.
Kuv cov lus qhia tseem ceeb: kawm Python!
pandas
Cia peb ua tus lej hauv qab no ua piv txwv:
import pandas as pd
df = pd.read_csv(“data/dataset.csv”)
# Calculate and rename aggregations
all_together = (df[df[‘trip_type’] == “return”]
.groupby(['start_station_name','end_station_name'])
.agg({'trip_duration_seconds': [np.size, np.mean, np.min, np.max]})
.rename(columns={'size': 'num_trips',
'mean': 'avg_duration_seconds',
'amin': min_duration_seconds',
‘amax': 'max_duration_seconds'}))Qhov tseem ceeb, peb pom tias cov cai haum rau hauv cov qauv SQL classic.
SELECT start_station_name, end_station_name, count(trip_duration_seconds) as size, …..
FROM dataset
WHERE trip_type = ‘return’
GROUPBY start_station_name, end_station_nameTab sis qhov tseem ceeb yog qhov chaws no yog ib feem ntawm tsab ntawv thiab cov raj xa dej; qhov tseeb, peb tab tom embedding cov lus nug rau hauv Python pipeline. Hauv qhov xwm txheej no, cov lus nug tuaj rau peb los ntawm cov tsev qiv ntawv xws li Pandas lossis pySpark.
Feem ntau, hauv pySpark peb pom ib yam zoo sib xws ntawm cov ntaub ntawv hloov pauv los ntawm cov lus nug hauv lub siab ntawm:
df.filter(df.trip_type = “return”)
.groupby(“day”)
.agg({duration: 'mean'})
.sort()Qhov twg thiab nyeem dab tsi
Ntawm Python nws tus kheej feem ntau nrhiav cov ntaub ntawv los kawm. Muaj ntau qhov kev qhia hauv online , thiab cov chav kawm rau (thiab los ntawm nws tus kheej ). Zuag qhia tag nrho, cov ntsiab lus ntawm no yog qhov zoo rau googling, thiab yog tias kuv yuav tsum xaiv ib pob los tsom rau, nws yuav yog pandas, tau kawg. Hais txog kev sib xyaw ntawm DS + Python cov ntaub ntawv ib yam nkaus .
Plhaub ua lus nug
Heev ob peb cov ntaub ntawv ua thiab tsom xam tej yaam num uas kuv tau ua hauj lwm nrog yog, qhov tseeb, plhaub scripts uas hu code nyob rau hauv Python, Java, thiab lub plhaub commands lawv tus kheej. Yog li, feem ntau, koj tuaj yeem xav txog cov kav dej hauv bash / zsh / thiab lwm yam raws li qee yam ntawm cov lus nug siab (koj tuaj yeem, ntawm chav kawm, cov khoom loops nyob rau hauv, tab sis qhov no tsis raug rau DS code hauv plhaub hom lus), cia peb muab ib qho piv txwv yooj yim - Kuv xav tau ua QID daim ntawv qhia ntawm wikidata thiab tag nrho cov txuas mus rau Lavxias teb sab thiab Askiv wikis, rau qhov no kuv tau sau ib daim ntawv thov yooj yim los ntawm cov lus txib hauv bash thiab rau cov zis kuv tau sau ib tsab ntawv yooj yim hauv Python, uas kuv ua ke zoo li no:
pv “data/latest-all.json.gz” |
unpigz -c |
jq --stream $JQ_QUERY |
python3 scripts/post_process.py "output.csv"
qhov twg
JQ_QUERY = 'select((.[0][1] == "sitelinks" and (.[0][2]=="enwiki" or .[0][2] =="ruwiki") and .[0][3] =="title") or .[0][1] == "id")' Qhov no yog, qhov tseeb, tag nrho cov raj xa dej uas tsim cov phiaj xwm xav tau; raws li peb pom, txhua yam ua haujlwm hauv kwj hom:
- pv filepath - muab qhov kev nce qib raws li cov ntaub ntawv loj thiab hla nws cov ntsiab lus txuas ntxiv mus
- unpigz -c nyeem ib feem ntawm archive thiab muab rau jq
- jq nrog tus yuam sij - kwj tam sim ua qhov tshwm sim thiab dhau mus rau tus txheej txheem postprocessor (ib yam li piv txwv thawj) hauv Python
- internally, lub postprocessor yog lub xeev yooj yim tshuab uas formatted cov zis
Nyob rau hauv tag nrho, ib tug complex pipeline ua hauj lwm nyob rau hauv ndlwg hom ntawm cov ntaub ntawv loj (0.5TB), tsis muaj peev xwm tseem ceeb thiab ua los ntawm ib tug yooj yim pipeline thiab ob peb cov cuab yeej.
Lwm cov lus qhia tseem ceeb: tuaj yeem ua haujlwm tau zoo thiab ua haujlwm tau zoo hauv lub davhlau ya nyob twg thiab sau bash / zsh / thiab lwm yam.
Nws yuav pab tau qhov twg? Yog, yuav luag txhua qhov chaw - dua, muaj ntau yam ntaub ntawv los kawm hauv Is Taws Nem. Tshwj xeeb, ntawm no kuv tsab xov xwm dhau los.
R sau ntawv
Ib zaug ntxiv, tus nyeem ntawv yuav qw - zoo, qhov no yog tag nrho cov lus programming! Thiab ntawm chav kawm, nws yuav yog. Txawm li cas los xij, kuv feem ntau ntsib R hauv cov ntsiab lus uas, qhov tseeb, nws zoo ib yam li cov lus nug.
R yog ib puag ncig kev suav txheeb thiab cov lus rau kev suav thiab kev pom zoo (raws li ).
coj . Los ntawm txoj kev, kuv xav kom nws, cov khoom zoo.
Vim li cas tus kws tshawb fawb cov ntaub ntawv yuav tsum paub R? Tsawg kawg, vim tias muaj txheej txheej loj ntawm cov neeg tsis yog IT uas txheeb xyuas cov ntaub ntawv hauv R. Kuv tuaj hla nws hauv qhov chaw hauv qab no:
- Pharmaceutical sector.
- Cov kws tshawb fawb txog biologist.
- Nyiaj txiag sector.
- Cov neeg uas muaj kev kawm lej nkaus xwb uas cuam tshuam nrog stats.
- Cov qauv kev txheeb cais tshwj xeeb thiab cov qauv kev kawm tshuab (uas feem ntau tuaj yeem pom nyob rau hauv tus sau version raws li pob R).
Vim li cas nws yeej yog lus nug? Nyob rau hauv daim ntawv nyob rau hauv uas nws yog feem ntau pom, nws yog tiag tiag ib tug thov los tsim ib tug qauv, nrog rau kev nyeem cov ntaub ntawv thiab kho cov lus nug (qauv) tsis, raws li zoo raws li pom cov ntaub ntawv nyob rau hauv tej pob khoom xws li ggplot2 - qhov no kuj yog ib daim ntawv ntawm kev sau queries. .
Piv txwv queries rau visualization
ggplot(data = beav,
aes(x = id, y = temp,
group = activ, color = activ)) +
geom_line() +
geom_point() +
scale_color_manual(values = c("red", "blue"))Feem ntau, ntau lub tswv yim los ntawm R tau tsiv mus rau hauv cov pob python xws li pandas, numpy lossis scipy, zoo li dataframes thiab cov ntaub ntawv vectorization - yog li feem ntau ntau yam hauv R yuav zoo li paub thiab yooj yim rau koj.
Muaj ntau qhov chaw los kawm, piv txwv li, .
Kev paub graphs
Ntawm no kuv muaj kev paub txawv me ntsis, vim tias kuv feem ntau yuav tsum tau ua haujlwm nrog kev paub graphs thiab cov lus nug rau graphs. Yog li ntawd, cia peb luv luv mus dhau lub hauv paus, vim qhov no yog qhov txawv me ntsis ntxiv.
Nyob rau hauv classical relational databases peb muaj ib tug ruaj schema, tab sis ntawm no lub schema yog hloov tau, txhua tus predicate yog ib tug "kem" thiab txawm ntau.
Xav txog tias koj tab tom ua qauv rau ib tug neeg thiab xav piav txog tej yam tseem ceeb, piv txwv li, cia peb coj ib tus neeg tshwj xeeb, Douglas Adams, thiab siv cov lus piav qhia no ua lub hauv paus.
Yog tias peb siv cov ntaub ntawv sib raug zoo, peb yuav tau tsim ib lub rooj lossis cov rooj uas muaj cov kab loj loj, feem ntau yuav yog NULL lossis sau nrog qee qhov tsis raug nqi, piv txwv li, tsis zoo li ntau ntawm peb muaj ib qho. nkag mus rau hauv lub tsev qiv ntawv Kauslim lub teb chaws - tau kawg, peb tuaj yeem muab lawv tso rau hauv cov rooj sib cais, tab sis qhov no thaum kawg yuav yog qhov kev sim ua qauv hloov pauv qhov kev xav tau yooj yim nrog cov lus qhia uas siv cov kev sib raug zoo.
Yog li xav txog tias tag nrho cov ntaub ntawv khaws cia ua ib daim duab lossis ua binary thiab unary boolean kab lus.
Koj txawm yuav ntsib qhov twg? Ua ntej, ua haujlwm nrog , thiab nrog txhua daim duab databases lossis cov ntaub ntawv txuas nrog.
Cov hauv qab no yog cov lus nug tseem ceeb uas kuv tau siv thiab ua haujlwm nrog.
SPARQL
Wiki:
SPARQL ( los ntawm SPARQL raws tu qauv thiab RDF Query Language) - - , sawv cev los ntawm tus qauv Thiab xa cov lus thov no thiab teb rau lawv. SPARQL yog qhov kev pom zoo thiab ib qho ntawm cov technologies .
Tab sis nyob rau hauv kev muaj tiag nws yog ib lo lus nug lus rau cov logical unary thiab binary predicates. Koj tsuas yog ua raws li kev qhia meej txog dab tsi yog kho hauv Boolean qhia thiab dab tsi tsis yog (qhov yooj yim heev).
RDF (Resource Description Framework) pib nws tus kheej, dhau qhov SPARQL cov lus nug raug tua, yog triple object, predicate, subject - thiab cov lus nug xaiv qhov xav tau triples raws li cov kev txwv tshwj xeeb hauv tus ntsuj plig: nrhiav X xws li p_55(X, q_33) yog qhov tseeb - qhov twg, tau kawg, p_55 yog qee yam kev sib raug zoo nrog ID 55, thiab q_33 yog ib qho yam khoom nrog ID 33 (ntawm no thiab tag nrho zaj dab neeg, dua tshem tawm txhua yam ntsiab lus).
Piv txwv ntawm kev nthuav qhia cov ntaub ntawv:
Cov duab thiab piv txwv nrog cov teb chaws ntawm no .
Basic Query Piv txwv
Qhov tseeb, peb xav nrhiav tus nqi ntawm ?lub teb chaws sib txawv xws li rau qhov predicate
member_of, nws muaj tseeb tias member_of(?lub teb chaws, q458) thiab q458 yog tus ID ntawm European Union.
Ib qho piv txwv ntawm cov lus nug SPARQL tiag hauv lub cav python:
Feem ntau, kuv yuav tsum tau nyeem SPARQL es tsis sau nws - nyob rau hauv qhov xwm txheej ntawd, nws yuav yog ib qho txuj ci tseem ceeb kom nkag siab cov lus yam tsawg kawg ntawm theem pib kom nkag siab raws nraim li cas cov ntaub ntawv rov qab tau.
Muaj ntau yam ntaub ntawv los kawm online: piv txwv li, ntawm no и . Kuv feem ntau google tshwj xeeb tsim thiab piv txwv thiab qhov txaus rau tam sim no.
Lus nug lus nug
Koj tuaj yeem nyeem ntxiv txog lub ntsiab lus hauv kuv tsab xov xwm . Thiab ntawm no, peb tsuas yog tshawb xyuas luv luv vim li cas cov lus muaj tswv yim zoo tsim nyog rau kev sau cov lus nug. Qhov tseem ceeb, RDF tsuas yog ib txheej ntawm cov ntsiab lus ntawm daim ntawv p(X) thiab h(X,Y), thiab cov lus nug txog kev xav muaj hauv qab no:
output(X) :- country(X), member_of(X,“EU”).
Ntawm no peb tab tom tham txog kev tsim cov khoom lag luam tawm tshiab / 1 (/ 1 txhais tau tias tsis muaj npe), muab tias rau X nws yog qhov tseeb tias lub teb chaws (X) - piv txwv li, X yog lub teb chaws thiab kuj member_of(X,"EU").
Uas yog, nyob rau hauv cov ntaub ntawv no, ob qho tib si cov ntaub ntawv thiab cov kev cai yog nthuav nyob rau hauv tib txoj kev, uas ua rau nws muaj peev xwm ua qauv teeb meem yooj yim heev thiab zoo.
Koj ntsib qhov twg hauv kev lag luam?: tag nrho cov phiaj xwm loj nrog lub tuam txhab uas sau cov lus nug hauv cov lus zoo li no, nrog rau cov phiaj xwm tam sim no hauv lub hauv paus ntawm lub kaw lus - nws yuav zoo li qhov no yog qhov txawv txawv, tab sis qee zaum nws tshwm sim.
Ib qho piv txwv ntawm cov kab lus tawg nyob rau hauv cov lus qhia txog kev ua haujlwm wikidata:
Cov ntaub ntawv: Kuv mam li muab ob peb qhov txuas mus rau cov lus niaj hnub logical programming Lus Teb Teeb Programming - Kuv pom zoo kom kawm nws:
Tau qhov twg los: www.hab.com