Ẹ kí, Khabrovsk olugbe. Ni ifojusona ti awọn ibere ti awọn dajudaju A ti pese itumọ awọn ohun elo ti o nifẹ si.
Ni ọpọlọpọ awọn igba lilo ti a fẹ Gẹgẹbi a ti rii lori awọn oju opo wẹẹbu awọn alabara wa, alaye ti o ni ibatan ti wa ni pamọ ni awọn asopọ laarin awọn nkan, fun apẹẹrẹ nigba itupalẹ awọn ibatan laarin awọn olumulo, awọn igbẹkẹle laarin awọn eroja, tabi awọn asopọ laarin awọn sensọ. Iru awọn ọran lilo bẹẹ ni a maa n ṣe apẹrẹ lori aworan kan. Ni ibẹrẹ ọdun yii, Amazon ṣe ifilọlẹ data data ayaworan tuntun rẹ, Neptune. Ninu ifiweranṣẹ yii a fẹ lati pin awọn imọran akọkọ wa, awọn iṣe ti o dara ati ohun ti o le ni ilọsiwaju ni akoko pupọ.
Kini idi ti a nilo Amazon Neptune
Awọn apoti isura infomesonu ayaworan ṣe ileri lati mu awọn eto data ti o ni asopọ pọ si dara julọ ju awọn ibaramu ibatan wọn lọ. Ninu iru awọn ipilẹ data, alaye ti o yẹ ni a fipamọ nigbagbogbo ni awọn ibatan laarin awọn nkan. A lo iṣẹ akanṣe ṣiṣi data iyalẹnu lati ṣe idanwo Neptune . MusicBrainz n gba gbogbo iru metadata orin ti a ro, gẹgẹbi alaye nipa awọn oṣere, awọn orin, awọn idasilẹ awo-orin tabi awọn ere orin, bakanna bi ẹniti olorin lẹhin orin ṣe ifowosowopo pẹlu tabi nigbati awo-orin naa ti tu silẹ ni orilẹ-ede wo. MusicBrainz le rii bi nẹtiwọọki nla ti awọn nkan ti o ni ọna kan ti o sopọ si ile-iṣẹ orin.
Ipilẹ data MusicBrainz ti pese bi idalẹnu CSV ti ibi ipamọ data ibatan kan. Ni apapọ, idalenu naa ni awọn ori ila 93 milionu ni awọn tabili 157. Lakoko ti diẹ ninu awọn tabili wọnyi ni awọn data ipilẹ gẹgẹbi awọn oṣere, awọn iṣẹlẹ, awọn gbigbasilẹ, awọn idasilẹ tabi awọn orin, awọn miiran awọn tabili asopọ - tọju awọn ibatan laarin awọn oṣere ati awọn gbigbasilẹ, awọn oṣere miiran tabi awọn idasilẹ, ati bẹbẹ lọ… Wọn ṣe afihan eto ayaworan ti ṣeto data kan. Nigbati o ba n yi dataset pada si awọn ilọpo mẹta RDF, a gba to awọn iṣẹlẹ 500 milionu.
Da lori iriri ati awọn iwunilori ti awọn alabaṣiṣẹpọ ise agbese pẹlu ẹniti a ṣiṣẹ, a ṣafihan eto kan ninu eyiti a lo ipilẹ imọ yii lati gba alaye tuntun. Ni afikun, a nireti pe yoo ni imudojuiwọn nigbagbogbo, fun apẹẹrẹ nipa fifi awọn idasilẹ titun kun tabi mimudojuiwọn awọn ọmọ ẹgbẹ ẹgbẹ.
Ṣe akanṣe
Gẹgẹbi a ti ṣe yẹ, fifi Amazon Neptune sori ẹrọ rọrun. O jẹ alaye pupọ . O le ṣe ifilọlẹ data data aworan ni awọn jinna diẹ. Sibẹsibẹ, nigbati o ba de si iṣeto ni alaye diẹ sii, soro lati ri. Nitorinaa, a fẹ tọka si paramita iṣeto kan.
Sikirinifoto iṣeto ni fun awọn ẹgbẹ paramita
Amazon sọ pe Neptune dojukọ awọn iṣẹ ṣiṣe iṣowo lairi kekere, eyiti o jẹ idi ti akoko ibeere aiyipada jẹ awọn aaya 120. A ti, sibẹsibẹ, ṣe idanwo ọpọlọpọ awọn ọran lilo itupalẹ ninu eyiti a de opin nigbagbogbo. Aago akoko yi le yipada nipasẹ ṣiṣẹda ẹgbẹ paramita tuntun fun Neptune ati eto neptune_query_timeout ti o baamu ihamọ.
Gbigba Data
Ni isalẹ a yoo jiroro ni apejuwe bi a ṣe kojọpọ data MusicBrainz sinu Neptune.
Ibasepo ni meta
Ni akọkọ, a yi data MusicBrainz pada si RDF meteta. Nitorinaa, fun tabili kọọkan, a ṣe asọye awoṣe kan ti o ṣalaye bi o ṣe jẹ aṣoju iwe kọọkan ninu ẹẹmẹta. Ni apẹẹrẹ yii, ori ila kọọkan lati tabili oṣere ti ya aworan si awọn mẹtala RDF mejila.
<http://musicbrainz.foo/artist/${id}> <http://musicbrainz.foo/gid> "${gid}"^^<http://www.w3.org/2001/XMLSchema#string> .
<http://musicbrainz.foo/artist/${id}> <http://musicbrainz.foo/name> "${name}"^^<http://www.w3.org/2001/XMLSchema#string> .
<http://musicbrainz.foo/artist/${id}> <http://musicbrainz.foo/sort-name> "${sort_name}"^^<http://www.w3.org/2001/XMLSchema#string> .
<http://musicbrainz.foo/artist/${id}> <http://musicbrainz.foo/begin-date> "${begin_date_year}-${begin_date_month}-${begin_date_day}"^^xsd:<http://www.w3.org/2001/XMLSchema#date> .
<http://musicbrainz.foo/artist/${id}> <http://musicbrainz.foo/end-date> "${end_date_year}-${end_date_month}-${end_date_day}"^^xsd:<http://www.w3.org/2001/XMLSchema#date> .
<http://musicbrainz.foo/artist/${id}> <http://musicbrainz.foo/type> <http://musicbrainz.foo/artist-type/${type}> .
<http://musicbrainz.foo/artist/${id}> <http://musicbrainz.foo/area> <http://musicbrainz.foo/area/${area}> .
<http://musicbrainz.foo/artist/${id}> <http://musicbrainz.foo/gender> <http://musicbrainz.foo/gender/${gender}> .
<http://musicbrainz.foo/artist/${id}> <http://musicbrainz.foo/comment> "${comment}"^^<http://www.w3.org/2001/XMLSchema#string> .
<http://musicbrainz.foo/artist/${id}> <http://musicbrainz.foo/edits-pending> "${edits_pending}"^^<http://www.w3.org/2001/XMLSchema#int> .
<http://musicbrainz.foo/artist/${id}> <http://musicbrainz.foo/last-updated> "${last_updated}"^^<http://www.w3.org/2001/XMLSchema#dateTime> .
<http://musicbrainz.foo/artist/${id}> <http://musicbrainz.foo/ended> "${ended}"^^<http://www.w3.org/2001/XMLSchema#boolean> .
olopobobo ikojọpọ
Ọna ti a daba lati ṣajọpọ awọn oye nla ti data sinu Neptune jẹ nipasẹ ilana ikojọpọ olopobobo nipasẹ S3. Lẹhin ikojọpọ awọn faili meteta rẹ si S3, o bẹrẹ ikojọpọ pẹlu ibeere POST kan. Ninu ọran tiwa, o gba to wakati 24 fun 500 milionu mẹta. A nireti pe yoo yarayara.
curl -X POST -H 'Content-Type: application/json' http://your-neptune-cluster:8182/loader -d '{
"source" : "s3://your-s3-bucket",
"format" : "ntriples",
"iamRoleArn" : "arn:aws:iam::your-iam-user:role/NeptuneLoadFromS3",
"region" : "eu-west-1",
"failOnError" : "FALSE"
}'Lati yago fun ilana gigun yii ni gbogbo igba ti a ṣe ifilọlẹ Neptune, a pinnu lati mu pada apẹẹrẹ pada lati inu aworan kan ninu eyiti o ti kojọpọ awọn meteta wọnyi tẹlẹ. Nṣiṣẹ lati aworan aworan jẹ yiyara ni pataki, ṣugbọn tun gba to wakati kan titi Neptune yoo wa fun awọn ibeere.
Nigbati o ba n ṣajọ awọn mẹta mẹta si Neptune, a pade awọn aṣiṣe oriṣiriṣi.
{
"errorCode" : "PARSING_ERROR",
"errorMessage" : "Content after '.' is not allowed",
"fileName" : [...],
"recordNum" : 25
}Diẹ ninu wọn n ṣe itupalẹ awọn aṣiṣe, bi a ṣe han loke. Titi di oni, a ko tii pinnu kini gangan ti ko tọ ni aaye yii. Awọn alaye diẹ diẹ sii yoo dajudaju ṣe iranlọwọ nibi. Aṣiṣe yii waye fun isunmọ 1% ti awọn ilọpo mẹta ti a fi sii. Ṣugbọn bi o ṣe jẹ idanwo Neptune, a gba otitọ pe a nikan ṣiṣẹ pẹlu 99% ti alaye lati MusicBrainz.
Paapaa botilẹjẹpe eyi rọrun fun awọn eniyan ti o faramọ pẹlu SPARQL, ṣe akiyesi pe awọn ẹẹmẹta RDF gbọdọ jẹ asọye pẹlu awọn iru data ti o fojuhan, eyiti o tun le fa awọn aṣiṣe.
Gbigbawọle ṣiṣanwọle
Gẹgẹbi a ti sọ loke, a ko fẹ lati lo Neptune gẹgẹbi ibi-itaja data aimi, ṣugbọn dipo bi ipilẹ imo ti o rọ ati idagbasoke. Nitorinaa a nilo lati wa awọn ọna lati ṣafihan awọn meteta tuntun nigbati ipilẹ imọ ba yipada, fun apẹẹrẹ nigbati awo-orin tuntun kan ti tẹjade tabi nigba ti a fẹ lati ni imọ ti ari.
Neptune ṣe atilẹyin awọn oniṣẹ igbewọle nipasẹ awọn ibeere SPARQL, mejeeji aise ati ipilẹ-apẹẹrẹ. A yoo jiroro awọn ọna mejeeji ni isalẹ.
Ọkan ninu awọn ibi-afẹde wa ni lati tẹ data sii ni ọna ṣiṣanwọle. Gbero jijade awo-orin kan ni orilẹ-ede titun kan. Lati irisi MusicBrainz, eyi tumọ si pe fun itusilẹ ti o pẹlu awọn awo-orin, awọn ẹyọkan, EPs, ati bẹbẹ lọ, a ṣafikun titẹsi tuntun si tabili. tu-orilẹ-ede. Ni RDF, a baramu alaye yi pẹlu meji titun triples.
INSERT DATA { <http://musicbrainz.foo/release-country/737041> <http://musicbrainz.foo/release> <http://musicbrainz.foo/release/435759> };INSERT DATA { <http://musicbrainz.foo/release-country/737041> <http://musicbrainz.foo/date-year> "2018"^^<http://www.w3.org/2001/XMLSchema#int> };Ipinnu miiran ni lati gba imọ tuntun lati ori aworan naa. Jẹ ki a sọ pe a fẹ lati gba nọmba awọn idasilẹ ti oṣere kọọkan ti gbejade ni iṣẹ wọn. Iru ibeere bẹẹ jẹ eka pupọ ati pe o gba to iṣẹju 20 ni Neptune, nitorinaa a nilo lati lo abajade naa lati le tun lo imọ tuntun yii ni ibeere miiran. Nitorinaa a ṣafikun awọn meteta pẹlu alaye yii pada si awọnyaya, ti nwọle abajade ti abẹlẹ naa.
INSERT {
?artist_credit <http://musicbrainz.foo/number-of-releases> ?number_of_releases
} WHERE {
SELECT ?artist_credit (COUNT(*) as ?number_of_releases)
WHERE {
?artist_credit <http://musicbrainz.foo/rdftype> <http://musicbrainz.foo/artist-credit> .
?release_group <http://musicbrainz.foo/artist-credit> ?artist_credit .
?release_group <http://musicbrainz.foo/rdftype> <http://musicbrainz.foo/release-group> .
?release_group <http://musicbrainz.foo/name> ?release_group_name .
}
GROUP BY ?artist_credit
}Ṣafikun awọn meteta ẹyọkan si awọnyaya gba to milliseconds diẹ, lakoko ti akoko ipaniyan fun fifi abajade ti abẹlẹ kan da lori akoko ipaniyan ti ipilẹ-ara funrararẹ.
Botilẹjẹpe a ko lo nigbagbogbo, Neptune tun fun ọ laaye lati yọ awọn mẹta mẹta ti o da lori awọn ayẹwo tabi data ti o fojuhan, eyiti o le ṣee lo lati ṣe imudojuiwọn alaye.
SPARQL ibeere
Nipa iṣafihan iṣapẹẹrẹ iṣaaju, eyiti o da nọmba awọn idasilẹ pada fun oṣere kọọkan, a ti ṣafihan tẹlẹ iru ibeere akọkọ ti a fẹ dahun ni lilo Neptune. Ṣiṣe ibeere kan ni Neptune rọrun - fi ibeere POST ranṣẹ si aaye ipari SPARQL, bi a ṣe han ni isalẹ:
curl -X POST --data-binary 'query=SELECT ?artist ?p ?o where {?artist <http://musicbrainz.foo/name> "Elton John" . ?artist ?p ?o . }' http://your-neptune-cluster:8182/sparqlNi afikun, a ti ṣe imuse ibeere kan ti o da profaili olorin pada ti o ni alaye ninu nipa orukọ wọn, ọjọ ori, tabi orilẹ-ede abinibi. Fiyesi pe awọn oṣere le jẹ ẹni kọọkan, awọn ẹgbẹ, tabi awọn akọrin. Ni afikun, a ṣe afikun data yii pẹlu alaye nipa nọmba awọn idasilẹ ti a tu silẹ nipasẹ awọn oṣere lakoko ọdun. Fun awọn oṣere adashe, a tun ṣafikun alaye nipa awọn ẹgbẹ ẹgbẹ ti oṣere kopa ninu ọdun kọọkan.
SELECT
?artist_name ?year
?releases_in_year ?releases_up_year
?artist_type_name ?releases
?artist_gender ?artist_country_name
?artist_begin_date ?bands
?bands_in_year
WHERE {
# Bands for each artist
{
SELECT
?year
?first_artist
(group_concat(DISTINCT ?second_artist_name;separator=",") as ?bands)
(COUNT(DISTINCT ?second_artist_name) AS ?bands_in_year)
WHERE {
VALUES ?year {
1960 1961 1962 1963 1964 1965 1966 1967 1968 1969
1970 1971 1972 1973 1974 1975 1976 1977 1978 1979
1980 1981 1982 1983 1984 1985 1986 1987 1988 1989
1990 1991 1992 1993 1994 1995 1996 1997 1998 1999
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
2010 2011 2012 2013 2014 2015 2016 2017 2018
}
?first_artist <http://musicbrainz.foo/name> "Elton John" .
?first_artist <http://musicbrainz.foo/rdftype> <http://musicbrainz.foo/artist> .
?first_artist <http://musicbrainz.foo/type> ?first_artist_type .
?first_artist <http://musicbrainz.foo/name> ?first_artist_name .
?second_artist <http://musicbrainz.foo/rdftype> <http://musicbrainz.foo/artist> .
?second_artist <http://musicbrainz.foo/type> ?second_artist_type .
?second_artist <http://musicbrainz.foo/name> ?second_artist_name .
optional { ?second_artist <http://musicbrainz.foo/begin-date-year> ?second_artist_begin_date_year . }
optional { ?second_artist <http://musicbrainz.foo/end-date-year> ?second_artist_end_date_year . }
?l_artist_artist <http://musicbrainz.foo/entity0> ?first_artist .
?l_artist_artist <http://musicbrainz.foo/entity1> ?second_artist .
?l_artist_artist <http://musicbrainz.foo/link> ?link .
optional { ?link <http://musicbrainz.foo/begin-date-year> ?link_begin_date_year . }
optional { ?link <http://musicbrainz.foo/end-date-year> ?link_end_date_year . }
FILTER (!bound(?link_begin_date_year) || ?link_begin_date_year <= ?year)
FILTER (!bound(?link_end_date_year) || ?link_end_date_year >= ?year)
FILTER (!bound(?second_artist_begin_date_year) || ?second_artist_begin_date_year <= ?year)
FILTER (!bound(?second_artist_end_date_year) || ?second_artist_end_date_year >= ?year)
FILTER (?first_artist_type NOT IN (<http://musicbrainz.foo/artist-type/2>, <http://musicbrainz.foo/artist-type/5>, <http://musicbrainz.foo/artist-type/6>))
FILTER (?second_artist_type IN (<http://musicbrainz.foo/artist-type/2>, <http://musicbrainz.foo/artist-type/5>, <http://musicbrainz.foo/artist-type/6>))
}
GROUP BY ?first_artist ?year
}
# Releases up to a year
{
SELECT
?artist
?year
(group_concat(DISTINCT ?release_name;separator=",") as ?releases)
(COUNT(*) as ?releases_up_year)
WHERE {
VALUES ?year {
1960 1961 1962 1963 1964 1965 1966 1967 1968 1969
1970 1971 1972 1973 1974 1975 1976 1977 1978 1979
1980 1981 1982 1983 1984 1985 1986 1987 1988 1989
1990 1991 1992 1993 1994 1995 1996 1997 1998 1999
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
2010 2011 2012 2013 2014 2015 2016 2017 2018
}
?artist <http://musicbrainz.foo/name> "Elton John" .
?artist_credit_name <http://musicbrainz.foo/artist-credit> ?artist_credit .
?artist_credit_name <http://musicbrainz.foo/rdftype> <http://musicbrainz.foo/artist-credit-name> .
?artist_credit_name <http://musicbrainz.foo/artist> ?artist .
?artist_credit <http://musicbrainz.foo/rdftype> <http://musicbrainz.foo/artist-credit> .
?release_group <http://musicbrainz.foo/artist-credit> ?artist_credit .
?release_group <http://musicbrainz.foo/rdftype> <http://musicbrainz.foo/release-group> .
?release_group <http://musicbrainz.foo/name> ?release_group_name .
?release <http://musicbrainz.foo/release-group> ?release_group .
?release <http://musicbrainz.foo/name> ?release_name .
?release_country <http://musicbrainz.foo/release> ?release .
?release_country <http://musicbrainz.foo/date-year> ?release_country_year .
FILTER (?release_country_year <= ?year)
}
GROUP BY ?artist ?year
}
# Releases in a year
{
SELECT ?artist ?year (COUNT(*) as ?releases_in_year)
WHERE {
VALUES ?year {
1960 1961 1962 1963 1964 1965 1966 1967 1968 1969
1970 1971 1972 1973 1974 1975 1976 1977 1978 1979
1980 1981 1982 1983 1984 1985 1986 1987 1988 1989
1990 1991 1992 1993 1994 1995 1996 1997 1998 1999
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009
2010 2011 2012 2013 2014 2015 2016 2017 2018
}
?artist <http://musicbrainz.foo/name> "Elton John" .
?artist_credit_name <http://musicbrainz.foo/artist-credit> ?artist_credit .
?artist_credit_name <http://musicbrainz.foo/rdftype> <http://musicbrainz.foo/artist-credit-name> .
?artist_credit_name <http://musicbrainz.foo/artist> ?artist .
?artist_credit <http://musicbrainz.foo/rdftype> <http://musicbrainz.foo/artist-credit> .
?release_group <http://musicbrainz.foo/artist-credit> ?artist_credit .
?release_group <http://musicbrainz.foo/rdftype> <http://musicbrainz.foo/release-group> .
?release_group <http://musicbrainz.foo/name> ?release_group_name .
?release <http://musicbrainz.foo/release-group> ?release_group .
?release_country <http://musicbrainz.foo/release> ?release .
?release_country <http://musicbrainz.foo/date-year> ?release_country_year .
FILTER (?release_country_year = ?year)
}
GROUP BY ?artist ?year
}
# Master data
{
SELECT DISTINCT ?artist ?artist_name ?artist_gender ?artist_begin_date ?artist_country_name
WHERE {
?artist <http://musicbrainz.foo/name> ?artist_name .
?artist <http://musicbrainz.foo/name> "Elton John" .
?artist <http://musicbrainz.foo/gender> ?artist_gender_id .
?artist_gender_id <http://musicbrainz.foo/name> ?artist_gender .
?artist <http://musicbrainz.foo/area> ?birth_area .
?artist <http://musicbrainz.foo/begin-date-year> ?artist_begin_date.
?birth_area <http://musicbrainz.foo/name> ?artist_country_name .
FILTER(datatype(?artist_begin_date) = xsd:int)
}Nitori idiju iru ibeere bẹẹ, a le ṣe awọn ibeere aaye nikan fun oṣere kan pato, gẹgẹbi Elton John, ṣugbọn kii ṣe fun gbogbo awọn oṣere. Neptune ko dabi pe o mu iru ibeere bẹẹ pọ si nipa sisọ awọn asẹ sinu awọn yiyan. Nitorinaa, yiyan kọọkan gbọdọ jẹ titọ pẹlu ọwọ nipasẹ orukọ olorin.
Neptune ni awọn idiyele wakati mejeeji ati fun-I/O. Fun idanwo wa, a lo apẹẹrẹ Neptune ti o kere ju, eyiti o jẹ $ 0,384 fun wakati kan. Ninu ọran ibeere ti o wa loke, eyiti o ṣe iṣiro profaili fun oṣiṣẹ kan, Amazon n gba wa lọwọ ẹgbẹẹgbẹrun awọn iṣẹ I / O, ti o tumọ idiyele ti $ 0.02.
ipari
Ni akọkọ, Amazon Neptune ntọju ọpọlọpọ awọn ileri rẹ. Gẹgẹbi iṣẹ iṣakoso, o jẹ aaye data iyaya kan ti o rọrun pupọ lati fi sori ẹrọ ati pe o le wa ni oke ati ṣiṣiṣẹ laisi iṣeto pupọ. Eyi ni awọn awari bọtini wa marun:
- Gbigbe olopobobo rọrun ṣugbọn o lọra. Ṣugbọn o le ni idiju pẹlu awọn ifiranṣẹ aṣiṣe ti ko ṣe iranlọwọ pupọ.
- Gbigbawọle ṣiṣanwọle ṣe atilẹyin ohun gbogbo ti a nireti ati pe o yara pupọ
- Awọn ibeere rọrun, ṣugbọn kii ṣe ibaraenisepo to lati ṣiṣe awọn ibeere itupalẹ
- Awọn ibeere SPARQL gbọdọ jẹ iṣapeye pẹlu ọwọ
- Awọn sisanwo Amazon nira lati ṣe iṣiro nitori pe o ṣoro lati ṣe iṣiro iye data ti a ṣayẹwo nipasẹ ibeere SPARQL kan.
Gbogbo ẹ niyẹn. Forukọsilẹ fun .
orisun: www.habr.com