Ina gaya muku daga gwaninta abin da ke da amfani a ina da kuma lokacin. Yana da bayyani da kasida, domin ya bayyana abin da kuma inda za ku iya kara tonowa - amma a nan ina da ƙwarewa ta musamman na sirri, watakila komai ya bambanta a gare ku.
Me yasa yake da mahimmanci sani da samun damar amfani da yarukan tambaya? A ainihinsa, Kimiyyar Bayanai tana da matakai masu mahimmanci na aiki, kuma na farko kuma mafi mahimmanci (ba tare da shi ba, hakika babu abin da zai yi aiki!) shine samun ko cire bayanai. Mafi sau da yawa, bayanan suna zaune a wani wuri a wani nau'i kuma yana buƙatar "dawo" daga can.
Harsunan tambaya suna ba ku damar fitar da wannan ainihin bayanan! Kuma a yau zan gaya muku game da waɗannan harsunan tambaya waɗanda suka kasance masu amfani a gare ni kuma zan gaya muku kuma in nuna muku inda kuma yadda daidai - dalilin da yasa ake buƙatar yin karatu.
Za a sami manyan tubalan nau'ikan neman bayanai guda uku, waɗanda za mu tattauna a wannan labarin:
- Harsunan tambaya "Standard" sune abin da aka fi fahimta yayin magana game da yaren tambaya, kamar algebra na dangantaka ko SQL.
- Rubutun tambayoyin harsuna: misali, Python things pandas, numpy ko rubutun harsashi.
- Harsunan tambaya don zane-zane na ilimi da bayanan bayanai.
Duk abin da aka rubuta a nan shi ne kawai na sirri gwaninta, abin da yake da amfani, tare da bayanin yanayi da kuma "me yasa ake bukata" - kowa da kowa zai iya gwada yadda irin wannan yanayi zai iya zo muku da kuma kokarin shirya musu a gaba ta hanyar fahimtar wadannan harsuna kafin ka nemi (gaggawa) a kan wani aiki ko ma zuwa aikin da ake buƙata.
Harsunan tambaya "Standard".
Madaidaitan harsunan tambaya daidai suke a ma'anar cewa yawanci muna tunanin su lokacin da muke magana game da tambayoyi.
Algebra na dangantaka
Me yasa ake buƙatar algebra mai alaƙa a yau? Don samun kyakkyawar fahimtar dalilin da yasa aka tsara harsunan tambaya ta wata hanya da amfani da su da hankali, kuna buƙatar fahimtar ainihin tushen su.
Menene alaƙar algebra?
Ma'anar ma'anar ita ce kamar haka: algebra na dangantaka rufaffiyar tsarin ayyuka ne akan alaƙa a cikin ƙirar bayanan alaƙa. Don sanya shi ɗan ƙaramin ɗan adam, wannan shine tsarin aiki akan tebur wanda sakamakon shine koyaushe tebur.
Dubi duk ayyukan da ke da alaƙa a ciki labarin daga Habr - a nan mun bayyana dalilin da ya sa kuke buƙatar sani da kuma inda ya zo da amfani.
Me ya sa?
Fara fahimtar abin da harsunan tambaya suke game da kuma menene ayyuka ke bayan maganganu a cikin takamaiman yarukan tambaya sau da yawa yana ba da zurfin fahimtar abin da ke aiki a cikin yarukan tambaya da ta yaya.
An ɗauko daga labarai. Misalin aiki: haɗawa, wanda ke haɗa tebur.
Kayayyakin karatu:
. Gabaɗaya, akwai abubuwa da yawa akan algebra na alaƙa da ka'idar - Coursera, Udacity. Hakanan akwai adadi mai yawa na kayan kan layi, gami da mai kyau . Shawarata ta sirri: kuna buƙatar fahimtar algebra mai alaƙa da kyau - wannan shine tushen tushe.
SQL
An ɗauko daga labarai.
SQL shine ainihin aiwatar da algebra mai alaƙa - tare da muhimmin fa'ida, SQL shela ce! Wato, lokacin rubuta tambaya a cikin harshen algebra mai alaƙa, a zahiri za ku faɗi yadda ake ƙididdigewa - amma tare da SQL za ku ƙididdige abin da kuke son cirewa, sannan DBMS ya riga ya haifar da maganganu (masu tasiri) a cikin harshen algebra mai alaƙa (su). daidai da aka sani a gare mu kamar yadda ).
An ɗauko daga labarai.
Me ya sa?
DBMSs na dangantaka: Oracle, Postgres, SQL Server, da dai sauransu har yanzu suna kusan ko'ina kuma akwai babban damar da za ku yi hulɗa da su, wanda ke nufin ko dai dole ne ku karanta SQL (wanda ke da yuwuwar) ko rubuta shi ( ba zai yuwu ba kuma).
Abin da za a karanta da karatu
Dangane da mahaɗin da ke sama (game da algebra na dangantaka), akwai adadin abu mai ban mamaki, misali, .
Af, menene NoSQL?
"Yana da kyau a sake jaddada cewa kalmar "NoSQL" tana da cikakkiyar asali kuma ba ta da ma'anar da aka yarda da ita ko kuma cibiyar kimiyya a bayansa." Daidaitawa da Habr.
A gaskiya ma, mutane sun fahimci cewa ba a buƙatar cikakken samfurin haɗin gwiwa don magance matsaloli da yawa, musamman ga waɗanda, alal misali, aikin aiki yana da mahimmanci kuma wasu ƙananan tambayoyin tare da tarawa sun mamaye - inda yake da mahimmanci don ƙididdige ma'auni da sauri kuma rubuta su zuwa ga database, kuma mafi yawan siffofin suna da alaƙa sun juya ba kawai ba dole ba ne, amma kuma cutarwa - me yasa normalize wani abu idan zai ɓata abu mafi mahimmanci a gare mu (don wani takamaiman aiki) - yawan aiki?
Har ila yau, ana buƙatar gyare-gyare masu sassauƙa sau da yawa maimakon ƙayyadaddun tsarin ƙididdiga na ƙirar alaƙa na gargajiya - kuma wannan yana sauƙaƙe haɓaka aikace-aikacen yayin da yake da mahimmanci don tura tsarin kuma fara aiki da sauri, sarrafa sakamakon - ko tsari da nau'ikan bayanan da aka adana. ba su da mahimmanci.
Misali, muna ƙirƙirar tsarin ƙwararru kuma muna son adana bayanai akan takamaiman yanki tare da wasu bayanan meta - ƙila ba za mu san duk filayen ba kuma kawai adana JSON don kowane rikodin - wannan yana ba mu yanayi mai sassauƙa don faɗaɗa bayanan. samfuri da saurin maimaitawa - don haka a cikin wannan A wannan yanayin, NoSQL zai kasance ma fi so kuma mafi karantawa. Shigar da misali (daga ɗaya daga cikin ayyukana inda NoSQL yake daidai inda ake buƙata).
{"en_wikipedia_url":"https://en.wikipedia.org/wiki/Johnny_Cash",
"ru_wikipedia_url":"https://ru.wikipedia.org/wiki/?curid=301643",
"ru_wiki_pagecount":149616,
"entity":[42775,"Джонни Кэш","ru"],
"en_wiki_pagecount":2338861}
Kara karantawa da NoSQL.
Me karatu?
Anan, maimakon haka, kawai kuna buƙatar bincika aikinku sosai, menene kaddarorin da yake da su da kuma menene tsarin NoSQL waɗanda zasu dace da wannan bayanin - sannan ku fara nazarin wannan tsarin.
Harsunan Tambayar Rubutu
Da farko, da alama, menene Python ya yi da shi gabaɗaya - yaren shirye-shirye ne, kuma ba game da tambayoyi ba kwata-kwata.
- Pandas a zahiri wuka ce ta Sojan Swiss na Kimiyyar Bayanai; babban adadin canjin bayanai, tarawa, da sauransu yana faruwa a ciki.
- Numpy - lissafin vector, matrices da algebra na layi a can.
- Scipy - akwai lissafin lissafi da yawa a cikin wannan kunshin, musamman ƙididdiga.
- Lab Jupyter - yawancin bincike na bayanan bincike sun dace da kwamfyutocin kwamfyutoci - masu amfani don sani.
- Buƙatun - aiki tare da hanyar sadarwa.
- Pyspark ya shahara sosai a tsakanin injiniyoyin bayanai, da alama za ku yi hulɗa da wannan ko Spark, kawai saboda shaharar su.
- *Selenium - yana da matukar amfani don tattara bayanai daga shafuka da albarkatu, wani lokacin kawai babu wata hanyar samun bayanan.
Babban shawarata: koyi Python!
Panda
Mu dauki wannan lambar a matsayin misali:
import pandas as pd
df = pd.read_csv(“data/dataset.csv”)
# Calculate and rename aggregations
all_together = (df[df[‘trip_type’] == “return”]
.groupby(['start_station_name','end_station_name'])
.agg({'trip_duration_seconds': [np.size, np.mean, np.min, np.max]})
.rename(columns={'size': 'num_trips',
'mean': 'avg_duration_seconds',
'amin': min_duration_seconds',
‘amax': 'max_duration_seconds'}))Ainihin, mun ga cewa lambar ta dace da tsarin SQL na gargajiya.
SELECT start_station_name, end_station_name, count(trip_duration_seconds) as size, …..
FROM dataset
WHERE trip_type = ‘return’
GROUPBY start_station_name, end_station_nameAmma abin da ke da mahimmanci shi ne cewa wannan lambar wani ɓangare ne na rubutun da bututun; a gaskiya, muna shigar da tambayoyi a cikin bututun Python. A cikin wannan yanayin, harshen tambaya yana zuwa mana daga dakunan karatu kamar Pandas ko pySpark.
Gabaɗaya, a cikin pySpark muna ganin irin wannan nau'in canjin bayanai ta hanyar harshen tambaya a cikin ruhun:
df.filter(df.trip_type = “return”)
.groupby(“day”)
.agg({duration: 'mean'})
.sort()Inda da abin da za a karanta
Akan Python kanta gabaɗaya nemo kayan karatu. Akwai babbar adadin koyawa akan layi , da kuma darussa a kan (kuma da kanta ). Gabaɗaya, abubuwan da ke cikin nan suna da kyau don yin wasa, kuma idan na ɗauki fakiti ɗaya don mayar da hankali a kai, zai zama pandas, ba shakka. Game da haɗin DS+Python kayan kuma .
Shell azaman harshen tambaya
Yawancin ayyukan sarrafa bayanai da bincike da na yi aiki da su sune, a zahiri, rubutun harsashi waɗanda ke kiran lamba a Python, Java, kuma harsashi yana ba da umarnin kansu. Sabili da haka, gabaɗaya, zaku iya yin la'akari da bututun bututu a cikin bash / zsh / da sauransu azaman wani nau'in tambaya mai girma (zaku iya, ba shakka, madaukai na kaya a can, amma wannan ba al'ada bane ga lambar DS a cikin harsunan harsashi), bari mu ba. misali mai sauƙi - Ina buƙatar yin taswirar QID na wikidata da cikakkun hanyoyin haɗi zuwa wikis na Rasha da Ingilishi, don wannan na rubuta buƙatu mai sauƙi daga umarni a cikin bash kuma don fitarwa na rubuta rubutu mai sauƙi a cikin Python, wanda na rubuta. a hada su kamar haka:
pv “data/latest-all.json.gz” |
unpigz -c |
jq --stream $JQ_QUERY |
python3 scripts/post_process.py "output.csv"
inda
JQ_QUERY = 'select((.[0][1] == "sitelinks" and (.[0][2]=="enwiki" or .[0][2] =="ruwiki") and .[0][3] =="title") or .[0][1] == "id")' Wannan shi ne, a zahiri, duk bututun da ya haifar da taswirar da ake buƙata; kamar yadda muke gani, komai yayi aiki a yanayin rafi:
- pv filepath - yana ba da sandar ci gaba dangane da girman fayil kuma yana wuce abubuwan da ke ciki gaba
- unpigz -c karanta sashin tarihin kuma ya ba jq
- jq tare da maɓalli - rafi nan da nan ya samar da sakamakon kuma ya mika shi ga mai sarrafawa (daidai da misalin farko) a Python.
- a ciki, na'ura mai sarrafawa ta kasance na'ura ce mai sauƙi wacce ta tsara kayan aiki
Gabaɗaya, ƙayyadaddun bututun da ke aiki a cikin yanayin kwarara akan manyan bayanai (0.5TB), ba tare da mahimman albarkatu ba kuma an yi shi daga bututu mai sauƙi da kayan aiki biyu.
Wani muhimmin bayani: iya yin aiki da kyau da inganci a cikin tashar kuma rubuta bash / zsh / sauransu.
A ina zai zama da amfani? Ee, kusan ko'ina - kuma, akwai abubuwa da yawa don yin nazari akan Intanet. Musamman a nan labarina na baya.
Rubutun R
Har ila yau, mai karatu na iya cewa - da kyau, wannan yaren shirye-shirye ne duka! Kuma ba shakka, zai yi gaskiya. Koyaya, yawanci nakan ci karo da R a cikin irin wannan mahallin wanda, a zahiri, yayi kama da harshen tambaya.
R yanayi ne na lissafin ƙididdiga da harshe don ƙididdige ƙididdiga da gani (bisa ga ).
dauka . Af, Ina ba da shawarar shi, abu mai kyau.
Me yasa masanin kimiyyar bayanai ke buƙatar sanin R? Akalla, saboda akwai ɗimbin mutane waɗanda ba IT ba waɗanda ke nazarin bayanai a cikin R. Na ci karo da shi a wurare masu zuwa:
- Bangaran magunguna.
- Masana ilmin halitta.
- Sashin hada-hadar kudi.
- Mutanen da ke da ilimin lissafi zalla waɗanda ke ma'amala da ƙididdiga.
- Samfuran ƙididdiga na musamman da ƙirar injuna (waɗanda galibi ana iya samun su a cikin sigar marubucin azaman fakitin R).
Me yasa a zahiri yaren tambaya ne? A cikin sigar da ake yawan samunsa, a zahiri buƙatar ƙirƙira samfuri ne, gami da karanta bayanai da daidaita sigogin tambaya (samfurin), da kuma ganin bayanai a cikin fakiti irin su ggplot2 - wannan kuma wani nau'i ne na tambayoyin rubutawa. .
Misalin tambayoyin don gani
ggplot(data = beav,
aes(x = id, y = temp,
group = activ, color = activ)) +
geom_line() +
geom_point() +
scale_color_manual(values = c("red", "blue"))Gabaɗaya, ra'ayoyi da yawa daga R sun yi ƙaura zuwa fakitin Python kamar pandas, numpy ko scipy, kamar tsarin bayanai da vectorization na bayanai - don haka gabaɗaya abubuwa da yawa a cikin R za su zama kamar saba da dacewa a gare ku.
Akwai hanyoyi da yawa don yin nazari, misali, .
Hotunan ilimi
Anan ina da ɗan gogewa mai ban mamaki, saboda sau da yawa dole in yi aiki tare da zane-zane na ilimi da harsunan tambaya don jadawali. Don haka, bari mu ɗan ɗanɗana abubuwan yau da kullun, saboda wannan ɓangaren yana da ɗanɗano kaɗan.
A cikin ma'ajin bayanai na al'ada muna da ƙayyadaddun tsari, amma a nan tsarin yana da sassauƙa, kowane tsinkaya a zahiri “ginshiƙi” ne har ma da ƙari.
Ka yi tunanin cewa kana yin samfurin mutum kuma kana son bayyana mahimman abubuwa, alal misali, bari mu ɗauki takamaiman mutum, Douglas Adams, mu yi amfani da wannan bayanin a matsayin tushe.
Idan muka yi amfani da bayanan da ke da alaƙa, dole ne mu ƙirƙiri babban tebur ko teburi tare da ginshiƙan adadi mai yawa, waɗanda yawancinsu za su zama NULL ko cike da wasu ƙima na ƙarya, alal misali, da wuya yawancinmu suna da shigarwa a cikin ɗakin karatu na ƙasar Koriya - ba shakka, za mu iya sanya su a cikin tebur daban-daban, amma wannan zai zama yunƙuri na ƙirar da'irar ma'ana mai sauƙi tare da tsinkaya ta amfani da ƙayyadaddun alaƙa.
Don haka yi tunanin cewa an adana duk bayanan azaman jadawali ko azaman binary da maganganun boolean unary.
A ina ma za ku iya haduwa da wannan? Da farko, aiki tare da , kuma tare da kowane bayanan bayanan hoto ko bayanan da aka haɗa.
Waɗannan su ne manyan yarukan tambaya waɗanda na yi amfani da su kuma na yi aiki da su.
SPARQL
Wiki:
SPARQL ( daga SPARQL Protocol da RDF Query Language) - , wanda samfurin ya wakilta Kuma don isar da waɗannan buƙatun da amsa su. SPARQL shawara ce kuma daya daga cikin fasahohin .
Amma a zahiri yaren tambaya ne don ma'ana unary da binary predicates. Kuna kawai ƙayyadaddun sharadi kawai abin da aka gyara a cikin furcin Boolean da abin da ba (a sauƙaƙe).
Tushen RDF (Tsarin Bayanin Albarkatun) da kansa, wanda aka aiwatar da tambayoyin SPARQL, sau uku ne. object, predicate, subject - kuma tambayar ta zaɓi sau uku da ake buƙata bisa ga ƙayyadaddun ƙuntatawa a cikin ruhun: nemo X irin wannan p_55 (X, q_33) gaskiya ne - inda, ba shakka, p_55 wani nau'in dangantaka ne da ID 55, kuma q_33 shine. abu mai ID 33 (a nan da dukan labarin, sake barin kowane nau'in bayanai).
Misalin gabatarwar bayanai:
Hotuna da misali tare da kasashe a nan .
Misalin Tambaya na asali
A zahiri, muna so mu nemo ƙimar ?ƙasa mai canzawa irin wannan don predicate
memba_na, gaskiya ne memba_na(?kasa,q458) da q458 shine ID na Tarayyar Turai.
Misali na ainihin tambayar SPARQL a cikin injin Python:
Yawanci, dole ne in karanta SPARQL maimakon rubuta shi - a wannan yanayin, zai iya zama fasaha mai amfani don fahimtar harshen aƙalla a matakin asali don fahimtar ainihin yadda ake dawo da bayanai.
Akwai abubuwa da yawa don nazarin kan layi: misali, a nan и . Yawancin lokaci ina google takamaiman ƙira da misalai kuma hakan ya isa a yanzu.
Harsunan tambaya na ma'ana
Kuna iya karanta ƙarin akan batun a cikin labarina . Kuma a nan, za mu ɗan bincika dalilin da yasa harsunan ma'ana suka dace da rubuta tambayoyin. Ainihin, RDF saitin bayanai ne na hankali kawai na nau'in p(X) da h(X,Y), kuma tambaya ta hankali tana da nau'i mai zuwa:
output(X) :- country(X), member_of(X,“EU”).
Anan muna magana ne game da ƙirƙirar sabon fitarwa / 1 (/ 1 yana nufin unary), muddin ga X gaskiya ne cewa ƙasa (X) - watau X ƙasa ce kuma memba ce ta (X, EU).
Wato, a wannan yanayin, duka bayanai da ka'idoji suna gabatar da su ta hanya ɗaya, wanda ke ba mu damar yin samfura cikin sauƙi da kyau.
A ina kuka hadu a masana'antar?: wani babban babban aiki tare da kamfani wanda ke rubuta tambayoyi a cikin irin wannan harshe, da kuma kan aikin da ake yi a yanzu a cikin tsarin tsarin - yana da alama cewa wannan abu ne mai ban mamaki, amma wani lokacin yana faruwa.
Misalin guntun lamba a cikin sarrafa harshe mai ma'ana wikidata:
Kayayyaki: Zan ba a nan wasu hanyoyin haɗin kai zuwa harshen shirye-shirye na zamani na zamani Amsa Saitin Shirye-shiryen - Ina ba da shawarar karanta shi:
source: www.habr.com