Mhoroi mose! Mune yake chinyorwa chekupedzisira Ndakanyora nezve kuronga modular yekutarisa system ye microservice architecture. Hapana chakamira, chirongwa chedu chiri kuramba chichikura, uye ndizvo zvakaitawo nhamba yemametric akachengetwa. Kuronga kwatakaita shanduko kubva kuGraphite+Whisper kuenda kuGraphite+ClickHouse pasi pemamiriro emutoro wepamusoro, verenga nezve tarisiro kubva kwairi uye nemhedzisiro yekutama pasi pekuchekwa.

Ndisati ndakuudza kuti takaronga sei shanduko kubva mukuchengetedza ma metrics muGraphite+Whisper kuenda kuGraphite+ClickHouse, ndinoda kupa ruzivo nezve zvikonzero zvekuita sarudzo yakadai uye nezvezvakaipira zveWhisper izvo takagara nazvo kwenguva yakareba.

Graphite+Whisper matambudziko

1. Yakakwirira mutoro pane disk subsystem

Panguva yeshanduko, angangoita miriyoni 1.5 metrics aisvika kwatiri paminiti. Nekuyerera kwakadaro, kushandiswa kwedhisiki pamaseva kwaive ~ 30%. Kazhinji, izvi zvakanga zvakagamuchirwa - zvinhu zvose zvakashanda zvakasimba, zvakanyorwa nokukurumidza, verenga nokukurumidza ... Kusvikira imwe yezvikwata zvekusimudzira yakabudisa chinhu chitsva uye yakatanga kutitumira mamiriyoni gumi emamiriyoni paminiti. Ndipo pakazosimba dhisiki subsystem, uye takaona 10% kushandiswa. Dambudziko rakagadziriswa nekukurumidza, asi zvakasara zvakasara.

2. Kushaikwa kwekudzokorora uye kuenderana

Zvingangoitika, senge munhu wese anoshandisa / akashandisa Graphite + Whisper, isu takadurura rwizi rwakafanana rwemetrics pane akati wandei maGraphite maseva kamwechete kuitira kugadzira kukanganisa kushivirira. Uye pakanga pasina matambudziko akakosha neizvi - kusvika panguva iyo imwe yemaseva yakadonha nekuda kwechimwe chikonzero. Dzimwe nguva isu takakwanisa kunhonga sevha yakadonha nekukurumidza zvakakwana, uye kabhoni-c-relay yakakwanisa kurodha metrics kubva kune cache yayo kupinda mairi, asi dzimwe nguva kwete. Uye ipapo pakanga paine gomba mumametrics, atakazadza ne rsync. Nzira yacho yakanga yakareba. Nyasha dzinoponesa dzaive dzekuti izvi zvakaitika kashoma. Isu taitorawo nguva nenguva seti yemametrics toaenzanisa nemamwe emhando imwechete pane dzakavakidzana node dze cluster. Munenge 5% yemakesi, akati wandei maitiro aive akasiyana, izvo zvatisina kufara nazvo.

3. Mutsindo mukuru

Sezvo isu tichinyora muGraphite kwete zvekuvaka chete, asiwo bhizinesi metrics (uye ikozvino zvakare metrics kubva Kubernetes), isu kazhinji tinowana mamiriro ayo metric ine mashoma mashoma maitiro, uye iyo .wsp faira inogadzirwa tichifunga nezve kuchengetwa kwese. nguva, uye inotora nzvimbo yakagoverwa kare, iyo kwatiri yaive ~ 2MB. Dambudziko rinowedzera kuwedzererwa nenyaya yekuti mafaera akawanda akafanana anoonekwa nekufamba kwenguva, uye pakuvaka mishumo paari, kuverenga mapoinzi asina chinhu kunotora nguva yakawanda uye zviwanikwa.

Ndinoda kukurumidza kuona kuti matambudziko anotsanangurwa pamusoro apa anogona kugadziriswa nekushandisa nzira dzakasiyana-siyana uye nemhando dzakasiyana-siyana dzekubudirira, asi iyo yakawanda data yaunotanga kugamuchira, inowedzera kuwedzera.

Kuve nezvose zviri pamusoro (tichifunga nezve yapfuura zvinyorwa), pamwe chete nekuwedzera nguva dzose kwenhamba yemetrics yakagamuchirwa, chido chekuendesa metrics yose kune imwe nguva yekuchengetedza ye30 seconds. (kusvika kumasekonzi gumi kana zvichidikanwa), takafunga kuyedza Graphite+ClickHouse seimwe inovimbisa imwe nzira yeWhisper.

Graphite+ClickHouse. Zvinotarisirwa

Kushanyira kwakati wandei kwevakomana kubva kuYandex, vachiverenga zvinyorwa zvishoma pamusoro paHabré, tapfuura nepakati pezvinyorwa uye tikawana zvine hungwaru zvikamu zvekusunga ClickHouse pasi peGraphite, takasarudza kutora danho!

Ndinoda kugamuchira zvinotevera:

kuderedza disk subsystem kushandiswa kubva ku30% kusvika ku5%;
kuderedza huwandu hwenzvimbo inotorwa kubva ku1TB kusvika ku100GB;
kukwanisa kugamuchira 100 miriyoni metrics paminiti museva;
kudzokorora data uye kukanganisa kushivirira kunze kwebhokisi;
usagara pachirongwa ichi kwegore uye kuita shanduko mukati menguva inonzwisisika;
chinja pasina downtime.

Kuda chinzvimbo, handiti?

Graphite+ClickHouse. Zvikamu

Kuti ugamuchire data kuburikidza neGraphite protocol uye wozorekodha muClickHouse, ndakasarudza carbon-clickhouse (golang).

Iyo yazvino kuburitswa kweClickHouse, yakagadzikana vhezheni 1.1.54253, yakasarudzwa sedhatabhesi yekuchengetedza nguva dzakatevedzana. Paiva nezvinetso pakushanda nayo: gomo rezvikanganiso rakadururirwa mumatanda, uye zvakanga zvisina kunyatsojeka kuti chii chaunofanira kuita navo. Mukukurukurirana na Roman Lomonosov (munyori wecarbon-clickhouse, graphite-clickhouse nezvimwe zvakawanda, zvakawanda) mukuru akasarudzwa kuburitsa 1.1.54236. Zvikanganiso zvakanyangarika - zvese zvakatanga kushanda ne bang.

Yakasarudzwa kuverenga data kubva kuClickHouse graphite-сlickhouse (golang). Sepurogiramu yeGraphite − carbonapi (golang). ClickHouse yakashandiswa kuronga kudzokorora pakati pematafura muchengeti wemhuka. Zvema routing metrics, takasiya mudiwa wedu carbon-c-relay (C) (ona chinyorwa chapfuura).

Graphite+ClickHouse. Tafura chimiro

"graphite" idatabase yatakagadzira yekutarisa matafura.

"graphite.metrics" - tafura ine ReplicatedReplacingMergeTree injini (yakadzokororwa KutsivaMergeTree) Tafura iyi inochengeta mazita emetrics uye nzira dzekuenda kwadzo.

CREATE TABLE graphite.metrics ( Date Date, Level UInt32, Path String, Deleted UInt8, Version UInt32 ) ENGINE = ReplicatedReplacingMergeTree('/clickhouse/tables/replicator/graphite.metrics', ‘r1’, Date, (Level, Path), 8192, Version);

"graphite.data" - tafura ine ReplicatedGraphiteMergeTree injini (yakadzokororwa GraphiteMergeTree) Tafura iyi inochengeta metric values.

CREATE TABLE graphite.data ( Path String, Value Float64, Time UInt32, Date Date, Timestamp UInt32 ) ENGINE = ReplicatedGraphiteMergeTree('/clickhouse/tables/replicator/graphite.data', 'r1', Date, (Path, Time), 8192, 'graphite_rollup')

"graphite.date_metrics" itafura yakazadzwa zvine mamiriro ine ReplicatedReplacingMergeTree injini. Tafura iyi inorekodha mazita emametric ese akasangana mukati mezuva. Zvikonzero zvekusikwa kwayo zvinotsanangurwa muchikamu "Matambudziko" pakupera kwechinyorwa chino.

CREATE MATERIALIZED VIEW graphite.date_metrics ( Path String,  Level UInt32,  Date Date) ENGINE = ReplicatedReplacingMergeTree('/clickhouse/tables/replicator/graphite.date_metrics', 'r1', Date, (Level, Path, Date), 8192) AS SELECT toUInt32(length(splitByChar('.', Path))) AS Level, Date, Path FROM graphite.data

"graphite.data_stat" - tafura yakazadzwa zvinoenderana nemamiriro, ine ReplicatedAggregatingMergeTree injini (yakadzokororwa AggregatingMergeTree) Tafura iyi inorekodha huwandu hwema metrics anouya, akakamurwa kusvika ku4 nesting mazinga.

CREATE MATERIALIZED VIEW graphite.data_stat ( Date Date,  Prefix String,  Timestamp UInt32,  Count AggregateFunction(count)) ENGINE = ReplicatedAggregatingMergeTree('/clickhouse/tables/replicator/graphite.data_stat', 'r1', Date, (Timestamp, Prefix), 8192) AS SELECT toStartOfMonth(now()) AS Date, replaceRegexpOne(Path, '^([^.]+.[^.]+.[^.]+).*$', '1') AS Prefix, toUInt32(toStartOfMinute(toDateTime(Timestamp))) AS Timestamp, countState() AS Count FROM graphite.data  GROUP BY Timestamp, Prefix

Graphite+ClickHouse. Chikamu chekudyidzana dhayagiramu

Graphite+ClickHouse. Kutama kwedata

Sezvatinorangarira kubva pane zvinotarisirwa kubva kupurojekiti iyi, shanduko kuenda kuClickHouse inofanira kunge isina nguva yekudzikira; nekudaro, taifanira kushandura hurongwa hwedu hwese hwekutarisisa kuenda kuchichengetedzo chitsva zviri pachena sezvinobvira kune vashandisi vedu.
Izvi ndizvo zvatakaita.

Mutemo wakawedzerwa kune kabhoni-c-relay kutumira imwe rwizi rwemetrics kune kabhoni-clickhouse yeimwe yemaseva ari kutora chikamu mukudzokorodza kweClickHouse matafura.
Takanyora script duku mu python, iyo, tichishandisa raibhurari yezeve-dump, verenga zvose .wsp mafaira kubva pakuchengetedza kwedu uye yakatumira iyi data kune yakatsanangurwa pamusoro-kabhoni-clickhouse mu 24 threads. Huwandu hweakagamuchirwa metric kukosha mucarbon-clickhouse yakasvika 125 miriyoni / min, uye ClickHouse haina kana kutyora dikita.
Isu takagadzira yakaparadzana DataSource muGrafana yekugadzirisa mabasa anoshandiswa mumadhibhodhi aripo. Isu takaona rondedzero yemabasa atakashandisa, asi haana kuitwa mucarbonapi. Takawedzera mabasa aya uye takatumira maPRs kune vanyori vecarbonapi (yakakosha kutenda kwavari).
Kuti tishandure mutoro wekuverenga muzvigadziriso zvebalancer, takashandura mapeji ekupedzisira kubva kugraphite-api (API interface yeGraphite + Whisper) kuita carbonapi.

Graphite+ClickHouse. mhinduro

yakaderedzwa disk subsystem kushandiswa kubva ku30% kusvika ku1%;
yakaderedza huwandu hwenzvimbo inotorwa kubva ku1 TB kusvika ku300 GB;
isu tine kugona kugashira 125 miriyoni metrics paminiti musevha (peaks panguva yekutama);
akatamisa ese metrics kune makumi matatu-sekondi ekuchengetedza nguva;
yakagamuchira kudzokororwa kwedata uye kukanganisa kushivirira;
kuchinjika pasina kupera nguva;
Zvakatora dzinenge 7 weeks kuti apedze zvese.

Graphite+ClickHouse. Matambudziko

Muchiitiko chedu, pakanga pane dzimwe misungo. Izvi ndizvo zvatakasangana nazvo mushure mekuchinja.

ClickHouse haiwanzo kuverengazve configs pane nhunzi; dzimwe nguva inoda kutangazve. Semuenzaniso, mune iyo tsananguro ye zookeeper cluster muClickHouse config, haina kushandiswa kudzamara iyo clickhouse-server yatangwazve.
Yakakura ClickHouse zvikumbiro hazvina kupfuura, saka mugraphite-clickhouse yedu ClickHouse yekubatanidza tambo inoita seizvi:
```
url = "http://localhost:8123/?max_query_size=268435456&max_ast_elements=1000000"
```
ClickHouse kazhinji inoburitsa mavhezheni matsva ekuburitswa kwakagadzikana; anogona kunge aine zvinoshamisa: chenjera.
Dynamically yakagadzirwa midziyo mukubernetes inotumira nhamba yakakura yemametrics nehupenyu hupfupi uye husina kurongeka. Iko hakuna mapoinzi akawanda emametric akadaro, uye hapana matambudziko nenzvimbo. Asi kana uchivaka mibvunzo, ClickHouse inotora huwandu hukuru hweaya metrics kubva patafura 'metrics'. Mu90% yezviitiko, hapana data pavari kunze kwehwindo (maawa makumi maviri nemana). Asi nguva inopedzwa kutsvaga iyi data mutafura ye 'data', uye inozopedzisira yapinda munguva yekubuda. Kuti tigadzirise dambudziko iri, takatanga kuchengetedza maonero akasiyana neruzivo rwemetrics yatakasangana nayo mukati mezuva. Nekudaro, kana tichivaka mishumo (magirafu) emidziyo yakagadzirwa zvine simba, isu tinobvunza chete iwo mametrics akasangana mukati mehwindo rakapihwa, uye kwete kwenguva yese, iyo yakamhanyisa zvakanyanya kuvakwa kwemishumo pazviri. Nokuda kwemhinduro yakatsanangurwa pamusoro apa, ndakaunganidza graphite-clickhouse (fork), iyo inosanganisira kushandiswa kwekushanda netafura yemazuva_metrics.

Graphite+ClickHouse. Tags

Neshanduro 1.1.0 Graphite yakave yepamutemo kutsigira tags. Uye isu tiri kushingairira kufunga nezve chii uye sei chekuita kutsigira chirongwa ichi mune graphite + clickhouse stack.

Graphite+ClickHouse. Anomaly detector

Kubva pane zvivakwa zvinotsanangurwa pamusoro, isu takaita prototype yeanomaly detector, uye inoshanda! Asi zvakawanda nezvake munyaya inotevera.

Nyorera, dzvanya museve wekumusoro uye ufare!

Source: www.habr.com

Metrics kuchengetedza: machinjiro atakaita kubva kuGraphite+Whisper kuenda kuGraphite+ClickHouse