Nyob zoo sawv daws! Hauv nws kab lus kawg Kuv tau sau txog kev teeb tsa ib qho kev saib xyuas qhov system rau microservice architecture. Tsis muaj dab tsi nyob twj ywm, peb qhov project yog pheej loj zuj zus, thiab yog li ntawd yog tus naj npawb ntawm cov metrics khaws cia. Yuav ua li cas peb txhim kho qhov kev hloov pauv ntawm Graphite + Whisper rau Graphite + ClickHouse nyob rau hauv cov xwm txheej hnyav, nyeem txog kev cia siab los ntawm nws thiab cov txiaj ntsig ntawm kev tsiv teb tsaws chaw nyob rau hauv kev txiav.

Ua ntej kuv qhia koj tias peb tau npaj qhov kev hloov pauv ntawm kev khaws cov ntsuas hauv Graphite + Ntxhi rau Graphite + ClickHouse, Kuv xav muab cov ntaub ntawv hais txog cov laj thawj ntawm kev txiav txim siab zoo li no thiab txog qhov tsis zoo ntawm Whisper uas peb nyob nrog ntev.

Graphite+Ntxhi cov teeb meem

1. Siab load ntawm lub disk subsystem

Thaum lub sijhawm hloov pauv, kwv yees li 1.5 lab metrics tau tuaj txog ntawm peb ib feeb. Nrog xws li qhov ntws, kev siv disk ntawm servers yog ~ 30%. Feem ntau, qhov no tau txais txiaj ntsig zoo - txhua yam ua haujlwm ruaj khov, tau sau sai, nyeem sai sai ... Txog thaum ib pab pawg txhim kho tau nthuav tawm qhov tshiab thiab pib xa peb 10 lab metrics ib feeb. Qhov ntawd yog thaum lub disk subsystem nruj, thiab peb pom 100% kev siv. Qhov teeb meem tau daws sai, tab sis ib qho residue tseem nyob.

2. Tsis muaj replication thiab sib xws

Feem ntau, zoo li txhua tus neeg siv / siv Graphite + ntxhi, peb tau nchuav tib cov kwj ntawm cov ntsuas mus rau ntau lub Graphite servers ib zaug txhawm rau tsim kev ua txhaum cai. Thiab tsis muaj teeb meem tshwj xeeb nrog qhov no - txog thaum lub sijhawm thaum ib tus servers poob rau qee yam. Qee lub sij hawm peb tau tswj xyuas cov neeg rau zaub mov poob sai txaus, thiab carbon-c-relay tau tswj kom thauj cov ntsuas los ntawm nws cov cache rau hauv nws, tab sis qee zaum tsis yog. Thiab ces muaj ib lub qhov nyob rau hauv cov metrics, uas peb sau nrog rsync. Cov txheej txheem tau ntev heev. Qhov kev cawm dim tsuas yog qhov no tshwm sim tsis tshua muaj. Peb kuj tseem niaj hnub coj ib qho kev ntsuas qhov ntsuas thiab muab piv rau lwm tus ntawm tib hom ntawm cov nyob sib ze ntawm pawg. Hauv kwv yees li 5% ntawm cov xwm txheej, ntau qhov txiaj ntsig sib txawv, uas peb tsis zoo siab heev.

3. Loj hneev taw

Txij li thaum peb sau hauv Graphite tsis yog tsuas yog infrastructure, tab sis kuj ua lag luam metrics (thiab tam sim no kuj metrics los ntawm Kubernetes), peb feem ntau tau txais ib qho teeb meem nyob rau hauv uas cov metric muaj tsuas yog ob peb qhov tseem ceeb, thiab cov ntaub ntawv .wsp yog tsim los coj mus rau hauv tus account tag nrho cov tuav. lub sij hawm, thiab yuav siv sij hawm li qhov chaw npaj ua ntej, uas rau peb yog ~ 2MB. Qhov teeb meem yog qhov hnyav ntxiv los ntawm qhov tseeb tias ntau cov ntaub ntawv zoo sib xws tshwm sim dhau sijhawm, thiab thaum tsim cov ntawv ceeb toom rau lawv, nyeem cov ntsiab lus khoob yuav siv sij hawm ntau thiab cov peev txheej.

Kuv xav tam sim ntawd nco ntsoov tias cov teeb meem tau piav qhia saum toj no tuaj yeem daws tau los ntawm kev siv ntau txoj hauv kev thiab nrog ntau qib ntawm kev ua tau zoo, tab sis qhov ntau cov ntaub ntawv koj pib tau txais, qhov ntau lawv ua phem.

Muaj tag nrho cov saum toj no (siv rau hauv tus account yav dhau los lus), nrog rau qhov nce ntxiv ntawm tus naj npawb ntawm cov metrics tau txais, lub siab xav hloov tag nrho cov metrics mus rau lub sijhawm cia ntawm 30 vib nas this. (txog 10 vib nas this yog tias tsim nyog), peb tau txiav txim siab sim Graphite + ClickHouse ua lwm txoj hauv kev rau Whisper.

Graphite + ClickHouse. Kev cia siab

Tau mus xyuas ntau qhov kev sib ntsib ntawm cov txiv neej los ntawm Yandex, tau nyeem ob peb tsab xov xwm ntawm Habre, tau dhau los ntawm cov ntaub ntawv thiab pom cov khoom zoo rau kev khi ClickHouse hauv Graphite, peb txiav txim siab los nqis tes ua!

Kuv xav tau cov hauv qab no:

txo kev siv disk subsystem los ntawm 30% mus rau 5%;
txo qhov chaw nyob ntawm 1TB rau 100GB;
tuaj yeem tau txais 100 lab metrics ib feeb rau hauv lub server;
cov ntaub ntawv replication thiab ua txhaum cai tawm ntawm lub thawv;
tsis txhob zaum ntawm qhov project no rau ib xyoos thiab hloov pauv hauv lub sijhawm tsim nyog;
hloov tsis muaj downtime.

Heev ambition, txoj cai?

Graphite + ClickHouse. Cheebtsam

Txhawm rau tau txais cov ntaub ntawv ntawm Graphite raws tu qauv thiab tom qab ntawd sau nws hauv ClickHouse, kuv xaiv carbon-clickhouse (kawg).

Qhov tseeb tso tawm ntawm ClickHouse, ruaj khov version 1.1.54253, tau raug xaiv los ua cov ntaub ntawv khaws cia lub sijhawm. Muaj teeb meem thaum ua haujlwm nrog nws: lub roob ntawm qhov yuam kev tau nchuav rau hauv cov cav, thiab nws tsis paub meej tias yuav ua li cas rau lawv. Hauv kev sib tham nrog Roman Lomonosov (tus sau ntawm carbon-clickhouse, graphite-clickhouse thiab ntau, ntau ntxiv) tus laus tau xaiv tso 1.1.54236. Qhov yuam kev ploj mus - txhua yam pib ua haujlwm nrog lub suab nrov.

Xaiv los nyeem cov ntaub ntawv los ntawm ClickHouse graphite-slickhouse (kawg). Raws li API rau Graphite - carbonapi (kawg). ClickHouse tau siv los teeb tsa kev rov ua dua ntawm cov rooj tus zookeeper. Rau routing metrics, peb tso peb tus hlub carbon-c-relay (C) (saib tsab xov xwm dhau los).

Graphite + ClickHouse. Rooj qauv

"graphite" yog cov ntaub ntawv peb tsim los saib xyuas cov rooj.

"graphite.metrics" - rooj nrog ReplicatedReplacingMergeTree cav (replicated Hloov MergeTree). Cov lus no khaws cov npe ntawm cov ntsuas thiab cov kev rau lawv.

CREATE TABLE graphite.metrics ( Date Date, Level UInt32, Path String, Deleted UInt8, Version UInt32 ) ENGINE = ReplicatedReplacingMergeTree('/clickhouse/tables/replicator/graphite.metrics', ‘r1’, Date, (Level, Path), 8192, Version);

"graphite.data" - rooj nrog ReplicatedGraphiteMergeTree cav (replicated GraphiteMergeTree). Cov lus no khaws cov txiaj ntsig metric.

CREATE TABLE graphite.data ( Path String, Value Float64, Time UInt32, Date Date, Timestamp UInt32 ) ENGINE = ReplicatedGraphiteMergeTree('/clickhouse/tables/replicator/graphite.data', 'r1', Date, (Path, Time), 8192, 'graphite_rollup')

"graphite.date_metrics" yog ib lub rooj uas muaj cai nrog lub cav ReplicatedReplacingMergeTree. Cov lus no sau cov npe ntawm txhua qhov kev ntsuas uas tau ntsib thaum nruab hnub. Cov laj thawj rau nws txoj kev tsim tau piav qhia hauv ntu "Cov teeb meem" thaum kawg ntawm tsab xov xwm no.

CREATE MATERIALIZED VIEW graphite.date_metrics ( Path String,  Level UInt32,  Date Date) ENGINE = ReplicatedReplacingMergeTree('/clickhouse/tables/replicator/graphite.date_metrics', 'r1', Date, (Level, Path, Date), 8192) AS SELECT toUInt32(length(splitByChar('.', Path))) AS Level, Date, Path FROM graphite.data

"graphite.data_stat" - ib lub rooj sau raws li qhov xwm txheej, nrog ReplicatedAggregatingMergeTree cav (replicated AggregatingMergeTree). Cov lus no sau cov naj npawb ntawm cov kev ntsuas tuaj, tawg mus rau 4 qib zes.

CREATE MATERIALIZED VIEW graphite.data_stat ( Date Date,  Prefix String,  Timestamp UInt32,  Count AggregateFunction(count)) ENGINE = ReplicatedAggregatingMergeTree('/clickhouse/tables/replicator/graphite.data_stat', 'r1', Date, (Timestamp, Prefix), 8192) AS SELECT toStartOfMonth(now()) AS Date, replaceRegexpOne(Path, '^([^.]+.[^.]+.[^.]+).*$', '1') AS Prefix, toUInt32(toStartOfMinute(toDateTime(Timestamp))) AS Timestamp, countState() AS Count FROM graphite.data  GROUP BY Timestamp, Prefix

Graphite + ClickHouse. Daim duab qhia kev sib cuam tshuam

Graphite + ClickHouse. Cov ntaub ntawv tsiv teb tsaws

Raws li peb nco qab los ntawm qhov kev cia siab los ntawm qhov project no, kev hloov pauv mus rau ClickHouse yuav tsum tsis muaj kev poob qis; raws li, peb yuav tsum tau hloov pauv peb cov kev saib xyuas tag nrho mus rau qhov tshiab cia kom pom tseeb li sai tau rau peb cov neeg siv.
Nov yog peb ua li cas.

Ib txoj cai tau ntxiv rau carbon-c-relay kom xa cov kwj ntxiv ntawm kev ntsuas mus rau carbon-clickhouse ntawm ib qho ntawm cov servers koom nrog kev rov ua dua ntawm ClickHouse cov lus.
Peb tau sau ib tsab ntawv me me hauv python, uas, siv lub tsev qiv ntawv ntxhi-dum, nyeem tag nrho cov ntaub ntawv .wsp los ntawm peb qhov chaw khaws cia thiab xa cov ntaub ntawv no mus rau cov lus piav qhia saum toj no carbon-clickhouse hauv 24 xov. Tus naj npawb ntawm cov txiaj ntsig tau txais txiaj ntsig hauv carbon-clickhouse tau mus txog 125 lab / min, thiab ClickHouse tseem tsis tau tawg hws.
Peb tsim ib qho DataSource cais nyob rau hauv Grafana rau debug zog siv nyob rau hauv dashboards uas twb muaj lawm. Peb tau txheeb xyuas cov npe ntawm cov haujlwm uas peb siv, tab sis lawv tsis tau siv hauv carbonapi. Peb ntxiv cov haujlwm no thiab xa PRs rau cov neeg sau ntawv ntawm carbonapi (ua tsaug tshwj xeeb rau lawv).
Txhawm rau hloov qhov kev nyeem ntawv hauv cov khoom sib npaug, peb hloov cov ntsiab lus kawg ntawm graphite-api (API interface rau Graphite + Whisper) rau carbonapi.

Graphite + ClickHouse. tshwm sim

txo kev siv disk subsystem los ntawm 30% mus rau 1%;
txo qhov chaw nyob ntawm 1 TB rau 300 GB;
peb muaj peev xwm tau txais 125 lab metrics ib feeb mus rau hauv lub server (peaks thaum lub sij hawm tsiv teb tsaws);
pauv tag nrho cov kev ntsuas mus rau peb caug-ob lub sijhawm cia;
tau txais cov ntaub ntawv replication thiab ua txhaum cai;
hloov tsis muaj downtime;
Nws siv li 7 lub lis piam los ua kom tiav txhua yam.

Graphite + ClickHouse. Teeb meem

Hauv peb qhov xwm txheej, muaj qee qhov pitfalls. Nov yog qhov peb ntsib tom qab kev hloov pauv.

ClickHouse tsis tas yuav rov nyeem cov teeb tsa ntawm ya; qee zaum nws yuav tsum tau rov pib dua. Piv txwv li, nyob rau hauv cov ntaub ntawv ntawm cov lus piav qhia ntawm lub zookeeper pawg nyob rau hauv lub ClickHouse config, nws tsis tau siv kom txog thaum lub clickhouse-server twb rebooted.
Kev thov loj ClickHouse tsis dhau mus, yog li hauv graphite-clickhouse peb ClickHouse txoj hlua txuas zoo li no:
```
url = "http://localhost:8123/?max_query_size=268435456&max_ast_elements=1000000"
```
ClickHouse feem ntau tso tawm cov tshiab versions ruaj khov; lawv yuav muaj kev xav tsis thoob: ceev faj.
Dynamically tsim cov ntim hauv kubernetes xa ntau qhov kev ntsuas nrog lub neej luv luv thiab random. Tsis muaj ntau cov ntsiab lus rau cov kev ntsuas no, thiab tsis muaj teeb meem nrog qhov chaw. Tab sis thaum tsim cov lus nug, ClickHouse khaws ntau tus lej ntawm cov kev ntsuas tib yam no los ntawm 'metrics' rooj. Hauv 90% ntawm cov neeg mob, tsis muaj cov ntaub ntawv ntawm lawv dhau lub qhov rais (24 teev). Tab sis lub sij hawm siv los tshawb nrhiav cov ntaub ntawv no hauv 'cov ntaub ntawv' cov lus, thiab thaum kawg khiav mus rau lub sijhawm. Txhawm rau daws qhov teeb meem no, peb tau pib tswj hwm qhov kev pom cais nrog cov ntaub ntawv ntawm kev ntsuas uas tau ntsib thaum nruab hnub. Yog li, thaum tsim cov ntawv ceeb toom (graphs) rau cov thawv tsim kom muaj zog, peb nug tsuas yog cov kev ntsuas uas tau ntsib nyob rau hauv lub qhov rais muab, thiab tsis yog rau tag nrho lub sijhawm, uas ua rau kev tsim kho ntawm cov ntawv ceeb toom sai sai. Rau cov tshuaj tau piav qhia saum toj no, kuv sau graphite-clickhouse (hli), uas suav nrog kev ua haujlwm ntawm kev ua haujlwm nrog date_metrics table.

Graphite + ClickHouse. Cim npe

Nrog rau version 1.1.0 Graphite tau dhau los ua nom txhawb tag. Thiab peb nquag xav txog dab tsi thiab yuav ua li cas los txhawb txoj haujlwm no hauv graphite + clickhouse pawg.

Graphite + ClickHouse. Tsis pom kev detector

Raws li kev tsim kho vaj tse tau piav qhia saum toj no, peb tau siv cov qauv ntawm lub tshuab kuaj pom tsis zoo, thiab nws ua haujlwm! Tab sis ntau ntxiv txog nws nyob rau hauv tsab xov xwm tom ntej.

Subscribe, nias lub xub thiab zoo siab!

Tau qhov twg los: www.hab.com

Metrics cia: yuav ua li cas peb hloov ntawm Graphite + Whisper rau Graphite + ClickHouse