Lumelang bohle! Ho tsa hae Ke ngotse ka ho hlophisa mokhoa oa ho shebella oa modular bakeng sa meralo ea microservice. Ha ho letho le emeng, morero oa rona o lula o hola, 'me ho joalo le ka palo ea metrics e bolokiloeng. Re ile ra hlophisa phetoho joang ho tloha Graphite+Whisper ho ea Graphite+ClickHouse tlas'a maemo a phahameng a boima, bala ka litebello tse tsoang ho eona le liphello tsa ho falla tlas'a sehiloeng.
Pele ke le bolella hore na re hlophisitse joang phetoho ea ho tloha ho boloka metrics ho Graphite+Whisper to Graphite+ClickHouse, ke kopa ho fana ka tlhahisoleseding mabapi le mabaka a ho etsa qeto e joalo le ka bofokoli ba Whisper eo re phetseng le eona nako e telele.
Mathata a Graphite+Whisper
1. Moroalo o phahameng ho subsystem ea disk
Nakong ea phetoho, li-metric tse ka bang limilione tse 1.5 li ne li fihla ho rona motsotso ka mong. Ka phallo e joalo, ts'ebeliso ea disk ho li-server e ne e le ~ 30%. Ka kakaretso, sena se ne se amoheleha haholo - ntho e 'ngoe le e' ngoe e ne e sebetsa ka mokhoa o tsitsitseng, e ngotsoe ka potlako, e baloa ka potlako ... Ho fihlela sehlopha se seng sa nts'etsopele se hlahisa karolo e ncha 'me se qala ho re romella li-metric tse limilione tse 10 ka motsotso. Ke ha tsamaiso ea disk e tiisitsoe, 'me re bone ts'ebeliso ea 100%. Bothata bo ile ba rarolloa ka potlako, empa ho ile ha sala masala.
2. Ho hloka ho pheta-pheta le ho tsitsa
Ho ka etsahala hore ebe, joalo ka motho e mong le e mong ea sebelisang/a sebelisang Graphite+Whisper, re ile ra tšela metrics e tšoanang ho li-server tse' maloa tsa Graphite ka nako e le 'ngoe ho theha mamello ea liphoso. 'Me ho ne ho se na mathata a khethehileng ka sena - ho fihlela motsotso oo e' ngoe ea li-server e ileng ea senyeha ka lebaka le itseng. Ka linako tse ling re ne re khona ho nka seva e oeleng kapele, 'me carbon-c-relay e khonne ho kenya metrics ho tsoa ho cache ea eona ho ea ho eona, empa ka linako tse ling ho se joalo. 'Me ho ne ho e-na le lesoba ho metrics, eo re ileng ra e tlatsa ka rsync. Mokhoa ona o ne o le telele haholo. Mohau o le mong o pholosang e ne e le hore sena se etsahetse ka seoelo. Hape nako le nako re ne re nka li-metrics tse sa reroang ebe re li bapisa le tse ling tsa mofuta o tšoanang ho li-node tsa boahelani tsa sehlopha. Hoo e ka bang 5% ea linyeoe, litekanyetso tse 'maloa li ne li fapane, tseo re neng re sa li thabele haholo.
3. Tsela e kholo ea maoto
Kaha ha re ngole ka Graphite feela lisebelisoa tsa motheo, empa hape le metrics ea khoebo ('me hona joale le metrics e tsoang Kubernetes), hangata re fumana boemo boo metric e nang le litekanyetso tse seng kae feela,' me faele ea .wsp e bōptjoa ho nahanoa ka ho boloka tsohle. nako, 'me e nka sebaka se fanoeng esale pele, seo ho rona e neng e le ~2MB. Bothata bo mpefatsoa le ho feta ke taba ea hore lifaele tse ngata tse tšoanang li hlaha ha nako e ntse e ea, 'me ha ho etsoa litlaleho ho tsona, ho bala lintlha tse se nang letho ho nka nako le lisebelisoa tse ngata.
Ke kopa ho hlokomela hang-hang hore mathata a hlalositsoeng ka holimo a ka sebetsanoa le mekhoa e sa tšoaneng le ka mekhoa e sa tšoaneng ea katleho, empa ha u qala ho fumana lintlha tse ngata, li mpefala le ho feta.
Ho ba le tsohle tse ka holimo (ho nahanne ka tse fetileng ), hammoho le keketseho e sa khaotseng ea palo ea metrics e amoheloang, takatso ea ho fetisetsa metrics eohle ho nako ea polokelo ea metsotsoana e 30. (ho fihla ho metsotsoana e 10 ha ho hlokahala), re nkile qeto ea ho leka Graphite+ClickHouse e le mokhoa o ts'episang ho Whisper.
Graphite+ClickHouse. Litebello
Ka mor'a ho etela likopano tse 'maloa tsa bashanyana ba tsoang Yandex, ha ba bala , ha re se re fetile litokomaneng 'me re fumane likarolo tse bohlale bakeng sa ho tlama ClickHouse tlas'a Graphite, re ile ra etsa qeto ea ho nka khato!
Ke kopa ho fumana tse latelang:
- fokotsa tšebeliso ea tsamaiso ea disk ho tloha ho 30% ho ea ho 5%;
- fokotsa sebaka sa sebaka se nkiloeng ho tloha ho 1TB ho isa ho 100GB;
- khona ho amohela li-metric tse limilione tse 100 ka motsotso ka har'a seva;
- ho pheta-pheta ha data le mamello ea liphoso ka ntle ho lebokose;
- u se ke ua lula morerong ona selemo le ho etsa phetoho ka nako e loketseng;
- switjha ntle le ho theoha.
O na le takatso e matla, ho joalo?
Graphite+ClickHouse. Likaroloana
Ho fumana data ka protocol ea Graphite ebe o e rekota ho ClickHouse, ke e khethile (golang).
Tokollo ea morao-rao ea ClickHouse, mofuta o tsitsitseng 1.1.54253, e khethiloe e le database bakeng sa ho boloka letoto la nako. Ho ne ho e-na le mathata ha ho sebetsa le eona: thaba ea liphoso e ile ea tšolleloa ka har'a lifate, 'me ho ne ho sa hlaka ka ho feletseng hore na u etse'ng ka tsona. Puisanong le (mongoli oa carbon-clickhouse, graphite-clickhouse le tse ling tse ngata, tse ngata) ho ile ha khethoa e moholo . Liphoso li ile tsa nyamela - ntho e 'ngoe le e' ngoe e ile ea qala ho sebetsa ka ho phatloha.
E khethiloe ho bala lintlha tse tsoang ho ClickHouse (golang). Joalo ka API ea Graphite − (golang). ClickHouse e ne e sebelisetsoa ho hlophisa phetisetso lipakeng tsa litafole . Bakeng sa metrics ea ho tsamaisa, re siile baratuoa ba rona (HO TSOANG) .
Graphite+ClickHouse. Sebopeho sa tafole
"graphite" ke polokelo ea litaba eo re e entseng bakeng sa litafole tsa ho beha leihlo.
"graphite.metrics" - tafole e nang le enjene ea ReplicatedReplacingMergeTree (e qotsitsoeng ). Tafole ena e boloka mabitso a metrics le litsela tse eang ho tsona.
CREATE TABLE graphite.metrics ( Date Date, Level UInt32, Path String, Deleted UInt8, Version UInt32 ) ENGINE = ReplicatedReplacingMergeTree('/clickhouse/tables/replicator/graphite.metrics', ‘r1’, Date, (Level, Path), 8192, Version);"graphite.data" - tafole e nang le enjene ea ReplicatedGraphiteMergeTree (e qotsitsoeng ). Tafole ena e boloka litekanyetso tsa metric.
CREATE TABLE graphite.data ( Path String, Value Float64, Time UInt32, Date Date, Timestamp UInt32 ) ENGINE = ReplicatedGraphiteMergeTree('/clickhouse/tables/replicator/graphite.data', 'r1', Date, (Path, Time), 8192, 'graphite_rollup')"graphite.date_metrics" ke tafole e tlatsitsoeng maemong a itseng e nang le enjene ea ReplicatedReplacingMergeTree. Tafole ena e tlaleha mabitso a li-metrics tsohle tse fumanoeng mots'ehare. Mabaka a ho bōptjoa ha eona a hlalositsoe karolong qetellong ea sehlooho sena.
CREATE MATERIALIZED VIEW graphite.date_metrics ( Path String, Level UInt32, Date Date) ENGINE = ReplicatedReplacingMergeTree('/clickhouse/tables/replicator/graphite.date_metrics', 'r1', Date, (Level, Path, Date), 8192) AS SELECT toUInt32(length(splitByChar('.', Path))) AS Level, Date, Path FROM graphite.data"graphite.data_stat" - tafole e tlatsitsoeng ho latela maemo, e nang le enjene ea ReplicatedAggregatingMergeTree (e qotsitsoeng ). Tafole ena e tlaleha palo ea metrics e tlang, e arotsoe ho maemo a 4 a ho haha.
CREATE MATERIALIZED VIEW graphite.data_stat ( Date Date, Prefix String, Timestamp UInt32, Count AggregateFunction(count)) ENGINE = ReplicatedAggregatingMergeTree('/clickhouse/tables/replicator/graphite.data_stat', 'r1', Date, (Timestamp, Prefix), 8192) AS SELECT toStartOfMonth(now()) AS Date, replaceRegexpOne(Path, '^([^.]+.[^.]+.[^.]+).*$', '1') AS Prefix, toUInt32(toStartOfMinute(toDateTime(Timestamp))) AS Timestamp, countState() AS Count FROM graphite.data GROUP BY Timestamp, PrefixGraphite+ClickHouse. Setšoantšo sa tšebelisano ea likarolo
Graphite+ClickHouse. Phalliso ea data
Joalo ka ha re hopola ho tsoa litebellong tsa projeke ena, phetoho ea ho ClickHouse e lokela ho ba ntle le linako tse theohileng; ka hona, re ile ra tlameha ho fetola sistimi eohle ea rona ea ho beha leihlo polokelong e ncha ka mokhoa o pepeneneng kamoo ho ka khonehang bakeng sa basebelisi ba rona.
Ke kamoo re entseng kateng.
-
Ho kentsoe molao ho carbon-c-relay ho romela molapo o mong oa metrics ho carbon-clickhouse ea e 'ngoe ea li-server tse nkang karolo ho phetoang litafole tsa ClickHouse.
-
Re ngotse script e nyenyane ka python, eo, ka ho sebelisa laebrari ea ho lahla, ho bala lifaele tsohle tsa .wsp ho tloha polokelong ea rona mme ra romela lintlha tsena ho carbon-clickhouse e hlalositsoeng ka holimo ka likhoele tse 24. Palo ea litekanyetso tsa metric tse amoheloang ka har'a carbon-clickhouse e fihlile ho limilione tse 125 / min, mme ClickHouse ha ea ka ea fufuleloa.
-
Re thehile DataSource e arohaneng Grafana ho lokisa lits'ebetso tse sebelisoang ho li-dashboard tse seng li ntse li le teng. Re khethile lethathamo la mesebetsi eo re e sebelisitseng, empa ha ea ka ea kengoa tšebetsong ka carbonapi. Re kentse mesebetsi ena mme ra romela li-PR ho bangoli ba carbonapi (liteboho tse khethehileng ho bona).
- Ho fetola mojaro oa ho bala ho li-setting tsa balancer, re fetotse li-endpoints ho tloha graphite-api (API interface bakeng sa Graphite + Whisper) ho carbonapi.
Graphite+ClickHouse. liphetho
-
ho fokotsa tšebeliso ea tsamaiso ea disk ho tloha ho 30% ho ea ho 1%;
- fokotsa sebaka sa sebaka se nkiloeng ho tloha ho 1 TB ho ea ho 300 GB;
- re na le bokhoni ba ho amohela metrics ea limilione tse 125 ka motsotso ho seva (litlhōrō ka nako ea ho falla);
- e fetiselitse metrics eohle ho nako ea polokelo ea metsotsoana e mashome a mararo;
- e amohetse phetisetso ea data le mamello ea liphoso;
- e fetotsoe ntle le ho fokotsa nako;
- Ho nkile libeke tse ka bang 7 ho phethela tsohle.
Graphite+ClickHouse. Mathata
Tabeng ea rona, ho ne ho e-na le maraba. Sena ke seo re ileng ra kopana le sona ka mor'a phetoho.
- ClickHouse ha se kamehla e balang litlhophiso ho fofa; ka linako tse ling e hloka ho qala bocha. Ka mohlala, tabeng ea tlhaloso ea sehlopha sa zookeeper ho ClickHouse config, ha ea ka ea sebelisoa ho fihlela clickhouse-server e tsosolosoa.
- Likopo tse kholo tsa ClickHouse ha lia ka tsa feta, kahoo ho graphite-clickhouse khoele ea rona ea khokahano ea ClickHouse e shebahala tjena:
url = "http://localhost:8123/?max_query_size=268435456&max_ast_elements=1000000" - ClickHouse hangata e lokolla mefuta e mecha ea litokollo tse tsitsitseng; e kanna ea ba le lintho tse makatsang: ela hloko.
- Lijana tse entsoeng ka matla ho kubernetes li romella palo e kholo ea metriki ka nako e khuts'oane le e sa reroang. Ha ho na lintlha tse ngata bakeng sa metrics e joalo, 'me ha ho na mathata a sebaka. Empa ha ho etsoa lipotso, ClickHouse e nka palo e kholo ea metrics e tšoanang ho tsoa tafoleng ea 'metrics'. Maemong a 90%, ha ho na data ho bona ntle le fensetere (lihora tse 24). Empa nako e sebelisoa ho batla lintlha tsena tafoleng ea 'data', 'me qetellong e fella ka nako. E le ho rarolla bothata bona, re ile ra qala ho boloka pono e fapaneng ka lintlha tse mabapi le metrics tse ileng tsa kopana le tsona motšehare. Kahoo, ha re haha litlaleho (li-graph) bakeng sa lijana tse entsoeng ka matla, re botsa feela metrics e ileng ea kopana le fensetere e fanoeng, eseng ka nako eohle, e leng se ileng sa potlakisa kaho ea litlaleho ho tsona. Bakeng sa tharollo e hlalositsoeng ka holimo, ke ile ka bokella , e kenyeletsang ts'ebetsong ea ho sebetsa le tafole ea date_metrics.
Graphite+ClickHouse. Li-tag
Ka mofuta oa 1.1.0 Graphite e ile ea fetoha semmuso . 'Me re ntse re nahana ka matla hore na re ka etsa eng le hore na re ka etsa eng ho ts'ehetsa morero ona ho graphite+clickhouse stack.
Graphite+ClickHouse. Anomaly detector
Ho ipapisitsoe le lits'ebetso tse hlalositsoeng kaholimo, re kentse ts'ebetso ea sesupo sa anomaly, 'me sea sebetsa! Empa ho eketsehileng ka eena sehloohong se latelang.
Ngolisa, tobetsa motsu o holimo 'me u thabe!
Source: www.habr.com