Ke haʻi aku nei au iā ʻoe mai ka ʻike pilikino i ka mea i pono i hea a i hea. ʻO ia ka ʻike a me ka thesis, no laila e maopopo i ka mea a me kahi e hiki ai iā ʻoe ke eli hou aʻe - akā eia kaʻu ʻike pilikino pili wale nō, ʻokoʻa paha nā mea āpau iā ʻoe.
No ke aha he mea nui ka ʻike a hiki ke hoʻohana i nā ʻōlelo nīnau? Ma kāna kumu, ʻo ka ʻikepili ʻepekema he mau hana koʻikoʻi o ka hana, a ʻo ka mea mua loa a me ka mea nui loa (me ka ʻole, ʻoiaʻiʻo, ʻaʻohe mea e hana!) ʻO ka pinepine, e noho ana ka ʻikepili ma kahi ʻano a pono e "hoʻihoʻi" mai laila.
ʻAe nā ʻōlelo nīnau iā ʻoe e unuhi i kēia ʻikepili! A i kēia lā e haʻi aku wau iā ʻoe e pili ana i kēlā mau nīnau nīnau i pono iaʻu a e haʻi aku wau iā ʻoe a hōʻike iā ʻoe i hea a pehea pololei - no ke aha e pono ai ke aʻo.
ʻEkolu mau poloka nui o nā ʻano nīnau ʻikepili, a mākou e kūkākūkā ai ma kēia ʻatikala:
- ʻO nā ʻōlelo nīnau "maʻamau" ka mea i hoʻomaopopo pinepine ʻia ke kamaʻilio e pili ana i kahi ʻōlelo nīnau, e like me ka algebra relational a i ʻole SQL.
- Nā ʻōlelo noiʻi palapala: no ka laʻana, Python mea pandas, numpy a shell scripting.
- Nīnau ʻōlelo no nā kiʻi ʻike a me nā ʻikepili kiʻi.
ʻO nā mea a pau i kākau ʻia ma ʻaneʻi he ʻike pilikino wale nō, he aha ka mea e pono ai, me ka wehewehe ʻana i nā kūlana a me "no ke aha e pono ai" - hiki i nā mea a pau ke hoʻāʻo pehea e hiki mai ai nā kūlana like i kou ala a hoʻāʻo e hoʻomākaukau no lākou ma mua o ka hoʻomaopopo ʻana i kēia mau ʻōlelo ma mua o kou noi ʻana (wikiwiki) i kahi papahana a i ʻole i kahi papahana kahi e pono ai lākou.
Nā ʻōlelo nīnau "maʻamau".
ʻO nā ʻōlelo noiʻi maʻamau ma ke ʻano he mea maʻamau mākou e noʻonoʻo ai iā lākou ke kamaʻilio mākou e pili ana i nā nīnau.
Algebra pili
No ke aha e pono ai ka algebra pili i kēia lā? I mea e ʻike maikaʻi ai i ke kumu i hoʻonohonoho ʻia ai nā ʻōlelo noiʻi ma kekahi ʻano a hoʻohana pono iā lākou, pono ʻoe e hoʻomaopopo i ke kumu kumu o lākou.
He aha ka pilina pili?
Penei ka wehewehe ʻana: ʻo ka algebra pili he ʻōnaehana pani o nā hana ma nā pilina i loko o kahi kumu hoʻohālike pili. No ka hoʻonui iki ʻana i ke kanaka, he ʻōnaehana kēia o nā hana ma nā papa e like me ka hopena he papaʻaina mau.
E ʻike i nā hana pili a pau ma ʻatikala mai Habr - eia mākou e wehewehe nei i ke kumu e pono ai ʻoe e ʻike a me kahi e hiki mai ai.
No ke aha?
ʻO ka hoʻomaka ʻana e hoʻomaopopo i ke ʻano o nā ʻōlelo nīnau a me nā hana ma hope o nā hōʻike i nā ʻōlelo nīnau kikoʻī e hāʻawi pinepine i ka ʻike hohonu o ka hana ma nā ʻōlelo nīnau a pehea.
Lawe ʻia mai ʻatikala. ʻO kahi laʻana o kahi hana: hui, e hoʻohui i nā papa.
Nā mea no ke aʻo ʻana:
. Ma keʻano laulā, nui nā mea e pili ana i ka algebra relational a me ke kumumanaʻo - Coursera, Udacity. Loaʻa ka nui o nā mea ma ka pūnaewele, me ka maikaʻi . ʻO kaʻu ʻōlelo aʻo pilikino: pono ʻoe e hoʻomaopopo pono i ka algebra relational - ʻo ia ke kumu o nā kumu.
SQL
Lawe ʻia mai ʻatikala.
He hoʻokō pono ʻo SQL i ka algebra relational - me kahi hōʻailona koʻikoʻi, ʻōlelo ʻo SQL! ʻO ia hoʻi, i ka wā e kākau ai i kahi nīnau ma ka ʻōlelo o ka algebra relational, ʻōlelo maoli ʻoe pehea e helu ai - akā me SQL ʻoe e kuhikuhi i ka mea āu e makemake ai e unuhi, a laila ua hoʻopuka ka DBMS i nā ʻōlelo (kūpono) i ka ʻōlelo o ka relational algebra (ko lākou. ʻike ʻia ke kaulike iā mākou ).
Lawe ʻia mai ʻatikala.
No ke aha?
ʻO nā DBMS pili: Oracle, Postgres, SQL Server, a me nā mea ʻē aʻe ma nā wahi āpau a aia kahi manawa kiʻekiʻe loa e pono ai ʻoe e launa pū me lākou, ʻo ia hoʻi, pono ʻoe e heluhelu i ka SQL (ʻo ia paha) a kākau paha ( ʻaʻole hiki ʻole paha).
He aha e heluhelu ai a e aʻo ai
E like me nā loulou i luna (e pili ana i ka algebra relational), he nui ka nui o nā mea, no ka laʻana, .
Ma ke ala, he aha ka NoSQL?
"He mea pono ke hoʻokūpaʻa hou i ka huaʻōlelo "NoSQL" he kumu maoli maoli nō ia a ʻaʻohe manaʻo i ʻae ʻia a i ʻole ʻepekema ʻepekema ma hope o ia. Pili ana ma Habr.
ʻO kaʻoiaʻiʻo, ua ʻike ka poʻe ʻaʻole pono kahi kumu hoʻohālike piha e hoʻoponopono ai i nā pilikia he nui, ʻoi aku hoʻi no nā wahi, no ka laʻana, he koʻikoʻi ka hana a ʻo kekahi mau nīnau maʻalahi me ka hoʻohuihui ʻana - kahi mea koʻikoʻi e helu wikiwiki i nā metric a kākau iā lākou i ka waihona, a me ka hapa nui o na hiʻona i relational huli mai e ole wale pono ole, akā, no ka mea, no ka mea, normalize mea ina e hao i ka mea nui loa no mākou (no kekahi hana kiko'ī) - productivity?
Eia kekahi, makemake pinepine ʻia nā ʻōkuhi maʻalahi ma mua o nā ʻōkuhi makemakika paʻa o ke kumu hoʻohālike kuʻuna maʻamau - a maʻalahi kēia i ka hoʻomohala noiʻi i ka wā e koʻikoʻi ai ke kau ʻana i ka ʻōnaehana a hoʻomaka e hana wikiwiki, e hoʻoponopono i nā hopena - a i ʻole ka schema a me nā ʻano o ka ʻikepili i mālama ʻia. ʻaʻole nui loa.
No ka laʻana, ke hana nei mākou i kahi ʻōnaehana loea a makemake mākou e mālama i ka ʻike ma kahi kikowaena kikoʻī me kekahi ʻike meta - ʻaʻole paha mākou i ʻike i nā kahua āpau a mālama wale iā JSON no kēlā me kēia moʻolelo - hāʻawi kēia iā mākou i kahi ʻano maʻalahi no ka hoʻonui ʻana i ka ʻikepili. ka hoʻohālike a me ka wikiwiki wikiwiki - no laila ma kēia hihia, ʻoi aku ka maikaʻi o NoSQL a hiki ke heluhelu ʻia. Hoʻohālike hoʻohālike (mai kekahi o kaʻu mau papahana kahi i kūpono ai ʻo NoSQL i kahi e pono ai).
{"en_wikipedia_url":"https://en.wikipedia.org/wiki/Johnny_Cash",
"ru_wikipedia_url":"https://ru.wikipedia.org/wiki/?curid=301643",
"ru_wiki_pagecount":149616,
"entity":[42775,"Джонни Кэш","ru"],
"en_wiki_pagecount":2338861}
Hiki iā ʻoe ke heluhelu hou aku e pili ana iā NoSQL.
He aha ka mea e aʻo ai?
Eia naʻe, pono ʻoe e nānā pono i kāu hana, he aha nā waiwai a me nā ʻōnaehana NoSQL i loaʻa e kūpono i kēia wehewehe - a laila hoʻomaka e aʻo i kēia ʻōnaehana.
Nā ʻŌlelo Nīnau Palapala
I ka wā mua, ʻike ʻia, he aha ka pili o Python me ia ma ke ʻano maʻamau - he ʻōlelo hoʻonohonoho ia, ʻaʻole e pili ana i nā nīnau.
- ʻO Pandas maoli ka pahi Swiss Army o Data Science; he nui ka nui o ka hoʻololi ʻana i ka ʻikepili, ka hōʻuluʻulu ʻana, a pēlā aku.
- Numpy - helu helu vector, matrices a me ka algebra linear ma laila.
- Scipy - he nui ka makemakika i loko o kēia pūʻolo, ʻoi aku ka helu.
- ʻO Jupyter lab - ka nui o ka ʻikepili ʻimi noiʻi i kūpono i nā kamepiula - pono e ʻike.
- Nā noi - hana pū me ka pūnaewele.
- He mea kaulana loa ʻo Pyspark i waena o nā ʻenekinia data, ʻoi aku paha ʻoe e launa pū me kēia a i ʻole Spark, ma muli o ko lākou kaulana.
- *Selenium - maikaʻi loa no ka ʻohi ʻana i ka ʻikepili mai nā pūnaewele a me nā kumuwaiwai, i kekahi manawa ʻaʻohe ala ʻē aʻe e kiʻi ai i ka ʻikepili.
ʻO kaʻu ʻōlelo aʻo nui: e aʻo iā Python!
Nā Pandas
E lawe kākou i kēia code ma ke ʻano he laʻana:
import pandas as pd
df = pd.read_csv(“data/dataset.csv”)
# Calculate and rename aggregations
all_together = (df[df[‘trip_type’] == “return”]
.groupby(['start_station_name','end_station_name'])
.agg({'trip_duration_seconds': [np.size, np.mean, np.min, np.max]})
.rename(columns={'size': 'num_trips',
'mean': 'avg_duration_seconds',
'amin': min_duration_seconds',
‘amax': 'max_duration_seconds'}))ʻO ka mea nui, ʻike mākou ua kūpono ke code i ke ʻano SQL maʻamau.
SELECT start_station_name, end_station_name, count(trip_duration_seconds) as size, …..
FROM dataset
WHERE trip_type = ‘return’
GROUPBY start_station_name, end_station_nameAkā ʻo ka mea nui, ʻo kēia code he ʻāpana o ka script a me ka pipeline; ʻoiaʻiʻo, ke hoʻokomo nei mākou i nā nīnau i ka pipeline Python. Ma kēia kūlana, hiki mai ka ʻōlelo nīnau iā mākou mai nā hale waihona puke e like me Pandas a i ʻole pySpark.
Ma keʻano laulā, ma pySpark mākou e ʻike ai i kahi ʻano like o ka hoʻololi ʻana i ka ʻikepili ma o ka ʻōlelo nīnau ma ka ʻuhane o:
df.filter(df.trip_type = “return”)
.groupby(“day”)
.agg({duration: 'mean'})
.sort()Ma hea a me ka mea e heluhelu ai
Ma ka Python pono'ī ma ka laulā e imi i na mea e ao ai. Nui ka nui o nā haʻawina ma ka pūnaewele , a me nā papa ma luna (a me ia iho ). Ma ka holoʻokoʻa, maikaʻi ka ʻike ma ʻaneʻi no ka googling, a inā pono wau e koho i hoʻokahi pūʻolo e nānā aku ai, ʻo ia nō nā pandas, ʻoiaʻiʻo. E pili ana i ka hui pū ʻana o nā mea DS+Python .
ʻO Shell ma ke ʻano he ʻōlelo nīnau
ʻO kekahi mau papahana hoʻoili ʻikepili a me ka nānā ʻana aʻu i hana pū ai me ka ʻoiaʻiʻo, ʻo nā script shell e kāhea ana i ke code ma Python, Java, a me ka shell i kauoha iā lākou iho. No laila, ma ka laulā, hiki iā ʻoe ke noʻonoʻo i nā pipelines ma bash/zsh/etc e like me ke ʻano o ka nīnau kiʻekiʻe (hiki iā ʻoe, ʻoiaʻiʻo, nā puka lou i laila, akā ʻaʻole maʻamau kēia no ka code DS ma nā ʻōlelo shell), e hāʻawi mākou. he laʻana maʻalahi - pono wau e hana i kahi palapala QID o wikidata a me nā loulou piha i nā wiki Lūkini a me English, no kēia mea ua kākau wau i kahi noi maʻalahi mai nā kauoha i ka bash a no ka hopena ua kākau wau i kahi palapala maʻalahi ma Python, aʻu i kākau ai. hui pū me kēia:
pv “data/latest-all.json.gz” |
unpigz -c |
jq --stream $JQ_QUERY |
python3 scripts/post_process.py "output.csv"
kahi
JQ_QUERY = 'select((.[0][1] == "sitelinks" and (.[0][2]=="enwiki" or .[0][2] =="ruwiki") and .[0][3] =="title") or .[0][1] == "id")' ʻO kēia, ʻo ka pipeline holoʻokoʻa i hana i ka palapala palapala i makemake ʻia; e like me kā mākou e ʻike ai, ua hana nā mea āpau i ke ʻano kahe:
- pv filepath - hāʻawi i kahi pae holomua e pili ana i ka nui o ka faila a hāʻawi i kāna mau ʻike ma mua
- unpigz -c heluhelu i kekahi hapa o ka waihona a haawi ia jq
- jq me ke kī - hoʻopuka koke ke kahawai i ka hopena a hāʻawi iā ia i ka postprocessor (e like me ka hiʻohiʻona mua loa) ma Python
- i loko, ʻo ka postprocessor he mīkini mokuʻāina maʻalahi i hoʻohālikelike i ka hoʻopuka
Ma ka huina, he pipeline paʻakikī e hana ana i ke kahe kahe ma ka ʻikepili nui (0.5TB), me ka ʻole o nā kumu waiwai nui a hana ʻia mai kahi pipeline maʻalahi a me nā mea hana ʻelua.
ʻO kekahi manaʻo koʻikoʻi: hiki ke hana maikaʻi a maikaʻi hoʻi i ka pahu a kākau i ka bash/zsh/etc.
Ma hea e pono ai? ʻAe, kokoke i nā wahi āpau - hou, he nui nā mea e aʻo ai ma ka Pūnaewele. Eia kekahi, maanei kaʻu ʻatikala mua.
R kākau ʻana
Eia hou, hiki i ka mea heluhelu ke hoʻōho - pono, he ʻōlelo papahana holoʻokoʻa kēia! A ʻoiaʻiʻo, e pololei ʻo ia. Eia nō naʻe, ʻike pinepine au iā R i loko o kahi ʻano pōʻaiapili, ʻoiaʻiʻo, ua like loa ia me kahi ʻōlelo nīnau.
ʻO R kahi kaiapuni helu helu helu a me ka ʻōlelo no ka helu helu static a me ka nānā ʻana (e like me ).
lawe ʻia . Ma ke ala, paipai wau iā ia, mea maikaʻi.
No ke aha e ʻike ai ka ʻepekema ʻikepili iā R? Ma ka liʻiliʻi loa, no ka mea, aia kahi papa nui o ka poʻe ʻaʻole IT nāna e kālailai i ka ʻikepili ma R. Ua loaʻa iaʻu ma kēia mau wahi:
- ʻĀpana lāʻau lapaʻau.
- ʻO nā mea ʻike olaola.
- ʻĀpana kālā.
- ʻO ka poʻe me ka hoʻonaʻauao makemakika maʻemaʻe e pili ana i nā stats.
- Nā hiʻohiʻona helu helu kūikawā a me nā hiʻohiʻona aʻo mīkini (hiki ke ʻike pinepine ʻia ma ka mana o ka mea kākau ma ke ʻano he pūʻulu R).
No ke aha he ʻōlelo nīnau maoli? Ma ke ʻano i ʻike pinepine ʻia, he noi maoli ia e hana i kahi hoʻohālike, me ka heluhelu ʻana i ka ʻikepili a me ka hoʻoponopono ʻana i nā ʻāpana noiʻi (model), a me ka nānā ʻana i nā ʻikepili i loko o nā pūʻolo e like me ggplot2 - he ʻano hoʻi kēia o nā nīnau kākau. .
Nā nīnau laʻana no ka nānā ʻana
ggplot(data = beav,
aes(x = id, y = temp,
group = activ, color = activ)) +
geom_line() +
geom_point() +
scale_color_manual(values = c("red", "blue"))Ma keʻano laulā, nui nā manaʻo mai R i neʻe i loko o nā pūʻulu python e like me pandas, numpy a scipy paha, e like me ka dataframes a me ka vectorization data - no laila ma ke ʻano he nui nā mea ma R e ʻike a maʻalahi iā ʻoe.
Nui nā kumu e aʻo ai, no ka laʻana, .
Kiʻi ʻike
Ma ʻaneʻi, loaʻa iaʻu kahi ʻike maʻamau, no ka mea, pono pinepine wau e hana me nā kiʻi ʻike a me nā ʻōlelo nīnau no nā kiʻi. No laila, e hele pōkole kākou i nā kumu, no ka mea, ʻoi aku ka liʻiliʻi o kēia ʻāpana.
Ma nā ʻikepili pili pili kahiko loaʻa iā mākou kahi schema paʻa, akā ma ʻaneʻi ua maʻalahi ka schema, ʻo kēlā me kēia predicate he "column" a ʻoi aku.
E noʻonoʻo ʻoe e hoʻohālike ana ʻoe i kahi kanaka a makemake ʻoe e wehewehe i nā mea nui, no ka laʻana, e lawe kāua i kahi kanaka kikoʻī, ʻo Douglas Adams, a hoʻohana i kēia wehewehe i kumu.
Inā mākou i hoʻohana i kahi waihona pili, pono mākou e hana i kahi papaʻaina nui a i ʻole nā papakaukau me ka nui o nā kolamu, ʻo ka hapa nui o ia mau mea he NULL a i hoʻopiha ʻia me kahi waiwai False paʻamau, no ka laʻana, ʻaʻole paha he nui o mākou komo i loko o ka hale waihona puke ʻāina ʻo Korea - ʻoiaʻiʻo, hiki iā mākou ke hoʻokomo iā lākou i nā papa ʻokoʻa, akā ʻo ia ka mea i hoʻāʻo e hoʻohālike i kahi kaapuni loiloi maʻalahi me nā predicates me ka hoʻohana ʻana i kahi pilina paʻa.
No laila e noʻonoʻo e mālama ʻia nā ʻikepili āpau ma ke ʻano he pakuhi a i ʻole ma ke ʻano he binary a me unary boolean expression.
Ma hea ʻoe e hālāwai ai me kēia? ʻO ka mea mua, e hana pū me , a me nā ʻikepili kiʻi a i ʻole nā ʻikepili pili.
Eia nā ʻōlelo nīnau nui aʻu i hoʻohana ai a hana pū me.
SPARQL
Wiki:
SPARQL ( от SPARQL Protocol a me RDF Nīnau Language) - , hōʻike ʻia e ke kumu hoʻohālike , a me e hoʻouna i kēia mau noi a pane aku iā lākou. He ʻōlelo paipai ʻo SPARQL a ʻo kekahi o nā ʻenehana .
Akā ʻo ka ʻoiaʻiʻo he ʻōlelo nīnau ia no nā predicates unary a me binary. Ke hōʻike wale nei ʻoe i ka mea i hoʻopaʻa ʻia i kahi ʻōlelo Boolean a me ka mea ʻaʻole (maʻalahi loa).
ʻO ka waihona RDF (Resource Description Framework) ponoʻī, kahi e hoʻokō ʻia ai nā nīnau SPARQL, he ʻekolu. object, predicate, subject - a koho ka hulina i nā ʻekolu i koi ʻia e like me nā kapu i ʻōlelo ʻia ma ka ʻuhane: e ʻimi i kahi X e like me ka p_55(X, q_33) ʻoiaʻiʻo - kahi, ʻoiaʻiʻo, p_55 kekahi ʻano pili me ID 55, a ʻo q_33 he mea me ka ID 33 (eia a me ka moʻolelo holoʻokoʻa, waiho hou i nā ʻano kikoʻī āpau).
Ka laʻana o ka hōʻike ʻikepili:
Nā kiʻi a me nā laʻana me nā ʻāina ma aneʻi .
Laʻana Nīnau Kumu
ʻoiaʻiʻo, makemake mākou e ʻimi i ka waiwai o ka ʻano hoʻololi ʻāina e like me ka predicate
member_of, he ʻoiaʻiʻo ʻo member_of(?country,q458) a me q458 ka ID o ka European Union.
ʻO kahi hiʻohiʻona o kahi nīnau SPARQL maoli i loko o ka mīkini python:
ʻO ka maʻamau, pono wau e heluhelu i ka SPARQL ma mua o ke kākau ʻana - ma ia kūlana, he mākaukau kūpono paha ia e hoʻomaopopo i ka ʻōlelo ma ka liʻiliʻi ma kahi pae kumu e maopopo pono ai ke kiʻi ʻia ʻana o ka ʻikepili.
Nui nā mea e aʻo ai ma ka pūnaewele: no ka laʻana, ma aneʻi и . ʻIke pinepine au i nā hoʻolālā kikoʻī a me nā hiʻohiʻona a ua lawa ia i kēia manawa.
Nā ʻōlelo nīnau lokahi
Hiki iā ʻoe ke heluhelu hou aku i ke kumuhana ma kaʻu ʻatikala . A ma ʻaneʻi, e noʻonoʻo pōkole wale mākou i ke kumu e kūpono ai nā ʻōlelo loiloi no ke kākau ʻana i nā nīnau. ʻO ka mea nui, ʻo RDF he pūʻulu o nā ʻōlelo kūpono o ke ʻano p(X) a me h(X,Y), a ʻo kahi nīnau noiʻi e like me kēia:
output(X) :- country(X), member_of(X,“EU”).
Eia mākou e kamaʻilio nei e pili ana i ka hana ʻana i kahi huaʻōlelo predicate hou / 1 (/1 ʻo ia ka unary), inā no ka X he ʻoiaʻiʻo kēlā ʻāina (X) - ʻo ia hoʻi, he ʻāina ʻo X a he lālā nō hoʻi o (X,"EU ").
ʻO ia hoʻi, i kēia hihia, hōʻike ʻia nā ʻikepili a me nā lula ma ke ʻano like, e hiki ai iā mākou ke hoʻohālike i nā pilikia me ka maʻalahi a maikaʻi.
Ma hea ʻoe i hui ai ma ka ʻoihana?: he papahana nui holoʻokoʻa me kahi hui e kākau i nā nīnau ma ia ʻōlelo, a me ka papahana o kēia manawa i ke kumu o ka ʻōnaehana - me he mea lā he mea ʻē aʻe kēia, akā i kekahi manawa e hiki mai ana.
ʻO kahi laʻana o kahi ʻāpana code i ka wikidata e hoʻoponopono ana i ka ʻōlelo.
Mea Hana: E hāʻawi wau i ʻelua mau loulou i ka ʻōlelo hoʻonohonoho loiloi o kēia manawa.
Source: www.habr.com