Bonjou tout moun. Nou ap devlope yon pwodwi pou analiz trafik offline. Pwojè a gen yon travay ki gen rapò ak analiz estatistik sou wout vizitè atravè rejyon yo.
Kòm yon pati nan travay sa a, itilizatè yo ka mande kesyon sistèm yo nan kalite sa a:
- konbyen vizitè ki te pase soti nan zòn "A" nan zòn "B";
- konbyen vizitè ki te pase soti nan zòn "A" nan zòn "B" nan zòn "C" epi answit nan zòn "D";
- konbyen tan li te pran pou yon sèten kalite vizitè vwayaje soti nan zòn "A" nan zòn "B".
ak yon kantite demann analyse menm jan an.
Mouvman vizitè a atravè zòn yo se yon graf dirije. Apre mwen fin li Entènèt la, mwen dekouvri ke graf DBMS yo itilize tou pou rapò analyse. Mwen te gen yon anvi wè ki jan DBMSs graf ta fè fas ak demann sa yo (TL; DR; mal).
Mwen te chwazi sèvi ak DBMS la , kòm yon reprezantan eksepsyonèl nan graf DBMS sous louvri, ki depann sou yon pil teknoloji ki gen matirite, ki (nan opinyon mwen) ta dwe bay li ak karakteristik operasyon desan:
- Backend depo BerkeleyDB, Apache Cassandra, Scylla;
- endis konplèks yo ka estoke nan Lucene, Elasticsearch, Solr.
Otè yo nan JanusGraph ekri ke li se apwopriye pou tou de OLTP ak OLAP.
Mwen te travay ak BerkeleyDB, Apache Cassandra, Scylla ak ES, ak pwodwi sa yo yo souvan itilize nan sistèm nou yo, kidonk mwen te optimis sou tès sa a DBMS graf. Mwen te jwenn li enpè yo chwazi BerkeleyDB sou RocksDB, men sa a pwobableman akòz kondisyon tranzaksyon yo. Nan nenpòt ka, pou évolutive, itilizasyon pwodwi, li sijere yo sèvi ak yon backend sou Cassandra oswa Scylla.
Mwen pa t 'konsidere Neo4j paske clustering mande pou yon vèsyon komèsyal, se sa ki, pwodwi a se pa sous louvri.
Grafik DBMS yo di: "Si li sanble ak yon graf, trete li tankou yon graf!" - bote!
Premyèman, mwen te trase yon graf, ki te fè egzakteman dapre kanon yo nan graf DBMS:
Gen yon sans Zone, responsab zòn nan. Si ZoneStep fè pati sa a Zone, Lè sa a, li refere a li. Sou sans Area, ZoneTrack, Person Pa peye atansyon, yo fè pati domèn nan epi yo pa konsidere kòm yon pati nan tès la. An total, yon rechèch chèn pou yon estrikti graf sa a ta sanble:
g.V().hasLabel('Zone').has('id',0).in_()
.repeat(__.out()).until(__.out().hasLabel('Zone').has('id',19)).count().next()Ki sa ki nan Larisi se yon bagay tankou sa a: jwenn yon Zòn ki gen ID = 0, pran tout somè ki soti nan ki yon kwen ale nan li (ZoneStep), stomp san yo pa tounen jiskaske ou jwenn ZoneSteps sa yo ki soti nan ki gen yon kwen nan Zòn la ak ID = 19, konte kantite chenn sa yo.
Mwen pa pretann konnen tout sibtilite ki genyen nan rechèch sou graf, men rechèch sa a te pwodwi ki baze sou liv sa a ().
Mwen chaje 50 mil tras ki soti nan 3 a 20 pwen nan longè nan yon baz done graf JanusGraph lè l sèvi avèk backend BerkeleyDB, te kreye endèks dapre .
Script telechaje Python:
from random import random
from time import time
from init import g, graph
if __name__ == '__main__':
points = []
max_zones = 19
zcache = dict()
for i in range(0, max_zones + 1):
zcache[i] = g.addV('Zone').property('id', i).next()
startZ = zcache[0]
endZ = zcache[max_zones]
for i in range(0, 10000):
if not i % 100:
print(i)
start = g.addV('ZoneStep').property('time', int(time())).next()
g.V(start).addE('belongs').to(startZ).iterate()
while True:
pt = g.addV('ZoneStep').property('time', int(time())).next()
end_chain = random()
if end_chain < 0.3:
g.V(pt).addE('belongs').to(endZ).iterate()
g.V(start).addE('goes').to(pt).iterate()
break
else:
zone_id = int(random() * max_zones)
g.V(pt).addE('belongs').to(zcache[zone_id]).iterate()
g.V(start).addE('goes').to(pt).iterate()
start = pt
count = g.V().count().next()
print(count)Nou itilize yon VM ak 4 am ak 16 GB RAM sou yon SSD. JanusGraph te deplwaye lè l sèvi avèk kòmandman sa a:
docker run --name janusgraph -p8182:8182 janusgraph/janusgraph:latestNan ka sa a, done ak endèks yo itilize pou rechèch matche ak egzak yo estoke nan BerkeleyDB. Lè m fin egzekite demann yo te bay pi bonè a, mwen te resevwa yon tan ki egal a plizyè dizèn segonn.
Lè mwen kouri 4 scripts ki anwo yo an paralèl, mwen te jere yo vire DBMS a nan yon joumou ak yon kouran kè kontan nan Java stacktraces (e nou tout renmen li Java stacktraces) nan mòso Docker yo.
Apre kèk reflechi, mwen deside senplifye dyagram nan graf nan sa ki annapre yo:
Deside ke rechèch pa atribi antite ta pi vit pase rechèch pa kwen. Kòm rezilta, demann mwen an te tounen bagay sa yo:
g.V().hasLabel('ZoneStep').has('id',0).repeat(__.out().simplePath()).until(__.hasLabel('ZoneStep').has('id',19)).count().next()Ki sa ki nan Ris se yon bagay tankou sa a: jwenn ZoneStep ak ID = 0, stomp san yo pa tounen jiskaske ou jwenn ZoneStep ak ID = 19, konte kantite chenn sa yo.
Mwen tou senplifye script loading yo bay pi wo a yo nan lòd yo pa kreye koneksyon nesesè, limite tèt mwen nan atribi.
Demann lan te toujou pran plizyè segonn pou konplete, ki te konplètman inakseptab pou travay nou an, paske li pa t 'ditou apwopriye pou rezon ki fè demann AdHoc nan nenpòt kalite.
Mwen te eseye deplwaye JanusGraph lè l sèvi avèk Scylla kòm aplikasyon Cassandra ki pi rapid, men sa a tou pa t mennen nan okenn chanjman pèfòmans enpòtan.
Se konsa, malgre lefèt ke "li sanble ak yon graf", mwen pa t 'kapab jwenn DBMS graf la trete li byen vit. Mwen konplètman sipoze ke gen yon bagay mwen pa konnen e ke JanusGraph ka fè rechèch sa a nan yon fraksyon nan yon segonn, sepandan, mwen pa t 'kapab fè li.
Depi pwoblèm nan toujou bezwen rezoud, mwen te kòmanse panse sou JOIN ak Pivot nan tab, ki pa t 'enspire optimis an tèm de distenksyon, men li ta ka yon opsyon konplètman travayabl nan pratik.
Pwojè nou an deja itilize Apache ClickHouse, kidonk mwen deside teste rechèch mwen an sou DBMS analyse sa a.
Deplwaye ClickHouse lè l sèvi avèk yon resèt senp:
sudo docker run -d --name clickhouse_1
--ulimit nofile=262144:262144
-v /opt/clickhouse/log:/var/log/clickhouse-server
-v /opt/clickhouse/data:/var/lib/clickhouse
yandex/clickhouse-serverMwen te kreye yon baz done ak yon tab ladan l tankou sa a:
CREATE TABLE
db.steps (`area` Int64, `when` DateTime64(1, 'Europe/Moscow') DEFAULT now64(), `zone` Int64, `person` Int64)
ENGINE = MergeTree() ORDER BY (area, zone, person) SETTINGS index_granularity = 8192Mwen ranpli li ak done lè l sèvi avèk script sa a:
from time import time
from clickhouse_driver import Client
from random import random
client = Client('vm-12c2c34c-df68-4a98-b1e5-a4d1cef1acff.domain',
database='db',
password='secret')
max = 20
for r in range(0, 100000):
if r % 1000 == 0:
print("CNT: {}, TS: {}".format(r, time()))
data = [{
'area': 0,
'zone': 0,
'person': r
}]
while True:
if random() < 0.3:
break
data.append({
'area': 0,
'zone': int(random() * (max - 2)) + 1,
'person': r
})
data.append({
'area': 0,
'zone': max - 1,
'person': r
})
client.execute(
'INSERT INTO steps (area, zone, person) VALUES',
data
)Depi foure yo vini an pakèt, ranpli te pi vit pase pou JanusGraph.
Konstwi de demann lè l sèvi avèk JOIN. Pou deplase soti nan pwen A a nan pwen B:
SELECT s1.person AS person,
s1.zone,
s1.when,
s2.zone,
s2.when
FROM
(SELECT *
FROM steps
WHERE (area = 0)
AND (zone = 0)) AS s1 ANY INNER JOIN
(SELECT *
FROM steps AS s2
WHERE (area = 0)
AND (zone = 19)) AS s2 USING person
WHERE s1.when <= s2.whenPou ale nan 3 pwen:
SELECT s3.person,
s1z,
s1w,
s2z,
s2w,
s3.zone,
s3.when
FROM
(SELECT s1.person AS person,
s1.zone AS s1z,
s1.when AS s1w,
s2.zone AS s2z,
s2.when AS s2w
FROM
(SELECT *
FROM steps
WHERE (area = 0)
AND (zone = 0)) AS s1 ANY INNER JOIN
(SELECT *
FROM steps AS s2
WHERE (area = 0)
AND (zone = 3)) AS s2 USING person
WHERE s1.when <= s2.when) p ANY INNER JOIN
(SELECT *
FROM steps
WHERE (area = 0)
AND (zone = 19)) AS s3 USING person
WHERE p.s2w <= s3.whenDemann yo, nan kou, gade byen pè; pou itilizasyon reyèl, ou bezwen kreye yon ekipay dèlko lojisyèl. Sepandan, yo travay epi yo travay byen vit. Tou de premye ak dezyèm demann yo ranpli nan mwens pase 0.1 segonn. Men yon egzanp tan egzekisyon demann pou konte (*) pase nan 3 pwen:
SELECT count(*)
FROM
(
SELECT
s1.person AS person,
s1.zone AS s1z,
s1.when AS s1w,
s2.zone AS s2z,
s2.when AS s2w
FROM
(
SELECT *
FROM steps
WHERE (area = 0) AND (zone = 0)
) AS s1
ANY INNER JOIN
(
SELECT *
FROM steps AS s2
WHERE (area = 0) AND (zone = 3)
) AS s2 USING (person)
WHERE s1.when <= s2.when
) AS p
ANY INNER JOIN
(
SELECT *
FROM steps
WHERE (area = 0) AND (zone = 19)
) AS s3 USING (person)
WHERE p.s2w <= s3.when
┌─count()─┐
│ 11592 │
└─────────┘1 rows in set. Elapsed: 0.068 sec. Processed 250.03 thousand rows, 8.00 MB (3.69 million rows/s., 117.98 MB/s.)Yon nòt sou IOPS. Lè peple done, JanusGraph te pwodwi yon kantite IOPS ase (1000-1300 pou kat fil popilasyon done) ak IOWAIT te byen wo. An menm tan an, ClickHouse te pwodwi chaj minim sou subsistèm ki gen kapasite a.
Konklizyon
Nou deside sèvi ak ClickHouse pou sèvis kalite demann sa a. Nou ka toujou plis optimize demann lè l sèvi avèk opinyon materyalize ak paralelizasyon pa pre-pwosesan kouran evènman an lè l sèvi avèk Apache Flink anvan yo chaje yo nan ClickHouse.
Pèfòmans la tèlman bon ke nou pwobableman pa pral menm gen panse sou pivote tab pwogramasyon. Précédemment, nou te oblije fè pivot nan done rekipere nan Vertica atravè Upload nan Apache Parquet.
Malerezman, yon lòt tantativ pou itilize yon graf DBMS pa t reyisi. Mwen pa t 'jwenn JanusGraph gen yon ekosistèm zanmitay ki te rann li fasil jwenn jiska vitès ak pwodwi a. An menm tan an, pou konfigirasyon sèvè a, yo itilize fason tradisyonèl Java a, ki pral fè moun ki pa abitye ak Java kriye dlo nan je san:
host: 0.0.0.0
port: 8182
threadPoolWorker: 1
gremlinPool: 8
scriptEvaluationTimeout: 30000
channelizer: org.janusgraph.channelizers.JanusGraphWsAndHttpChannelizer
graphManager: org.janusgraph.graphdb.management.JanusGraphManager
graphs: {
ConfigurationManagementGraph: conf/janusgraph-cql-configurationgraph.properties,
airlines: conf/airlines.properties
}
scriptEngines: {
gremlin-groovy: {
plugins: { org.janusgraph.graphdb.tinkerpop.plugin.JanusGraphGremlinPlugin: {},
org.apache.tinkerpop.gremlin.server.jsr223.GremlinServerGremlinPlugin: {},
org.apache.tinkerpop.gremlin.tinkergraph.jsr223.TinkerGraphGremlinPlugin: {},
org.apache.tinkerpop.gremlin.jsr223.ImportGremlinPlugin: {classImports: [java.lang.Math], methodImports: [java.lang.Math#*]},
org.apache.tinkerpop.gremlin.jsr223.ScriptFileGremlinPlugin: {files: [scripts/airline-sample.groovy]}}}}
serializers:
# GraphBinary is here to replace Gryo and Graphson
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphBinaryMessageSerializerV1, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphBinaryMessageSerializerV1, config: { serializeResultToString: true }}
# Gryo and Graphson, latest versions
- { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV3d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}
- { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV3d0, config: { serializeResultToString: true }}
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV3d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}
# Older serialization versions for backwards compatibility:
- { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}
- { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0, config: { serializeResultToString: true }}
- { className: org.apache.tinkerpop.gremlin.driver.ser.GryoLiteMessageSerializerV1d0, config: {ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV2d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV1d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistryV1d0] }}
- { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistryV1d0] }}
processors:
- { className: org.apache.tinkerpop.gremlin.server.op.session.SessionOpProcessor, config: { sessionTimeout: 28800000 }}
- { className: org.apache.tinkerpop.gremlin.server.op.traversal.TraversalOpProcessor, config: { cacheExpirationTime: 600000, cacheMaxSize: 1000 }}
metrics: {
consoleReporter: {enabled: false, interval: 180000},
csvReporter: {enabled: false, interval: 180000, fileName: /tmp/gremlin-server-metrics.csv},
jmxReporter: {enabled: false},
slf4jReporter: {enabled: true, interval: 180000},
gangliaReporter: {enabled: false, interval: 180000, addressingMode: MULTICAST},
graphiteReporter: {enabled: false, interval: 180000}}
threadPoolBoss: 1
maxInitialLineLength: 4096
maxHeaderSize: 8192
maxChunkSize: 8192
maxContentLength: 65536
maxAccumulationBufferComponents: 1024
resultIterationBatchSize: 64
writeBufferHighWaterMark: 32768
writeBufferHighWaterMark: 65536
ssl: {
enabled: false}Mwen te jere aksidantèlman "mete" vèsyon BerkeleyDB JanusGraph la.
Dokimantasyon an byen kwochi an tèm de endèks, depi jere endèks mande pou ou fè kèk shamanism olye etranj nan Groovy. Pou egzanp, kreye yon endèks dwe fè pa ekri kòd nan konsole Gremlin la (ki, nan chemen an, pa travay soti nan bwat la). Soti nan dokiman ofisyèl JanusGraph la:
graph.tx().rollback() //Never create new indexes while a transaction is active
mgmt = graph.openManagement()
name = mgmt.getPropertyKey('name')
age = mgmt.getPropertyKey('age')
mgmt.buildIndex('byNameComposite', Vertex.class).addKey(name).buildCompositeIndex()
mgmt.buildIndex('byNameAndAgeComposite', Vertex.class).addKey(name).addKey(age).buildCompositeIndex()
mgmt.commit()
//Wait for the index to become available
ManagementSystem.awaitGraphIndexStatus(graph, 'byNameComposite').call()
ManagementSystem.awaitGraphIndexStatus(graph, 'byNameAndAgeComposite').call()
//Reindex the existing data
mgmt = graph.openManagement()
mgmt.updateIndex(mgmt.getGraphIndex("byNameComposite"), SchemaAction.REINDEX).get()
mgmt.updateIndex(mgmt.getGraphIndex("byNameAndAgeComposite"), SchemaAction.REINDEX).get()
mgmt.commit()Apreword
Nan yon sans, eksperyans ki anwo a se yon konparezon ant cho ak mou. Si w panse sou sa, yon graf DBMS fè lòt operasyon pou jwenn menm rezilta yo. Sepandan, kòm yon pati nan tès yo, mwen te fè tou yon eksperyans ak yon demann tankou:
g.V().hasLabel('ZoneStep').has('id',0)
.repeat(__.out().simplePath()).until(__.hasLabel('ZoneStep').has('id',1)).count().next()ki reflete distans mache. Sepandan, menm sou done sa yo, graf DBMS la te montre rezilta ki te ale pi lwen pase kèk segonn... Sa a, nan kou, se akòz lefèt ke te gen chemen tankou 0 -> X -> Y ... -> 1, ki motè graf la tcheke tou.
Menm pou yon demann tankou:
g.V().hasLabel('ZoneStep').has('id',0).out().has('id',1)).count().next()Mwen pa t 'kapab jwenn yon repons pwodiktif ak yon tan pwosesis nan mwens pase yon segonn.
Moral la nan istwa a se ke yon lide bèl ak modèl paradigmatik pa mennen nan rezilta a vle, ki se demontre ak efikasite pi wo lè l sèvi avèk egzanp lan nan ClickHouse. Ka itilizasyon ki prezante nan atik sa a se yon anti-modèl klè pou DBMSs graf, byenke li sanble apwopriye pou modèl nan paradigm yo.
Sous: www.habr.com