Sanibonani nonke! Kweyakhe isihloko sokugcina Ngabhala mayelana nokuhlela uhlelo lokuqapha lwe-modular lwe-microservice architecture. Akukho okumile, iphrojekthi yethu ikhula njalo, kanjalo nenani lamamethrikhi agciniwe. Indlela esihlele ngayo inguquko ukusuka ku-Graphite+Whisper ukuya ku-Graphite+ClickHouse ngaphansi kwezimo zomthwalo ophezulu, funda mayelana nokulindelwe kuyo kanye nemiphumela yokufuduka ngaphansi kokusikwa.

Ngaphambi kokuthi ngikutshele ukuthi siluhlele kanjani ushintsho olusuka ekugcineni amamethrikhi ku-Graphite+Whisper kuya ku-Graphite+ClickHouse, ngithanda ukunikeza ulwazi mayelana nezizathu zokwenza leso sinqumo kanye nobubi be-Whisper esihlale nabo isikhathi eside.

Izinkinga ze-graphite+Whisper

1. Ukulayisha okuphezulu kusistimu engaphansi yediski

Ngesikhathi soshintsho, cishe ama-metrics ayizigidi ezingu-1.5 ayefika kithi ngomzuzu ngamunye. Ngokugeleza okunjalo, ukusetshenziswa kwediski kumaseva kwaba ~ 30%. Ngokuvamile, lokhu kwamukeleka impela - yonke into yasebenza ngokuzinza, yabhalwa ngokushesha, yafundwa ngokushesha... Kwaze kwaba yilapho elinye lamaqembu okuthuthukisa likhipha isici esisha futhi laqala ukusithumela amamethrikhi ayizigidi ezingu-10 ngomzuzu. Yilapho i-subsystem yediski iqina, futhi sabona ukusetshenziswa okungu-100%. Inkinga yaxazululwa ngokushesha, kodwa kwasala.

2. Ukuntuleka kokuphindaphinda nokuvumelana

Cishe, njengawo wonke umuntu osebenzisa/osebenzisa i-Graphite+Whisper, sithulule ukusakazwa okufanayo kwamamethrikhi kumaseva ambalwa e-Graphite ngesikhathi esisodwa ukuze sakhe ukubekezelela amaphutha. Futhi kwakungekho zinkinga ezikhethekile ngalokhu - kuze kube yisikhathi lapho enye yamaseva iphahlazeka ngesizathu esithile. Kwesinye isikhathi sikwazile ukucosha iseva ewile ngokushesha ngokwanele, futhi i-carbon-c-relay ikwazile ukulayisha amamethrikhi ukusuka kunqolobane yayo ukuya kuyo, kodwa kwesinye isikhathi akunjalo. Bese kuba khona imbobo kumamethrikhi, esiyigcwalise nge-rsync. Inqubo yayiyinde impela. Umusa osindisayo kuphela ukuthi lokhu kwenzeka kuyaqabukela. Siphinde futhi ngezikhathi ezithile sithathe isethi engahleliwe yamamethrikhi futhi siwaqhathanise namanye ohlobo olufanayo kumanodi angomakhelwane eqoqo. Ezimweni ezingaba ngu-5%, amanani amaningana ayehlukile, esasingajabule kakhulu ngawo.

3. Izinyathelo ezinkulu

Njengoba singabhali nge-Graphite hhayi ingqalasizinda kuphela, kodwa namamethrikhi ebhizinisi (futhi manje namamethrikhi avela ku-Kubernetes), sivame ukuthola isimo lapho imethrikhi iqukethe amanani ambalwa kuphela, futhi ifayela elithi .wsp liyadalwa kucatshangelwa konke ukugcinwa. isikhathi, futhi ithatha inani lesikhala esabelwe ngaphambili, thina esasingu-~2MB. Inkinga ibhebhethekiswa futhi yiqiniso lokuthi amafayela amaningi afanayo avela ngokuhamba kwesikhathi, futhi lapho wakha imibiko kuwo, ukufunda amaphuzu angenalutho kuthatha isikhathi esiningi nezinsiza.

Ngithanda ukuqaphela ngokushesha ukuthi izinkinga ezichazwe ngenhla zingabhekwana nazo kusetshenziswa izindlela ezihlukahlukene kanye namazinga ahlukene okusebenza, kodwa uma uqala ukuthola idatha eyengeziwe, ziba zimbi nakakhulu.

Ukuba nakho konke okungenhla (kucatshangelwa okwedlule izindatshana), kanye nokwenyuka okuqhubekayo kwenani lamamethrikhi atholiwe, isifiso sokudlulisela wonke amamethrikhi kusikhawu sokulondoloza samasekhondi angu-30. (kufika kumasekhondi angu-10 uma kudingeka), sinqume ukuzama i-Graphite+ClickHouse njengendlela ethembisayo esikhundleni se-Whisper.

I-Graphite+ClickHouse. Okulindelekile

Ngemva kokuvakashela imihlangano eminingana yabafana baseYandex, sebefundile izindatshana ezimbalwa ezikhuluma ngoHabré, ngemva kokudlula kumadokhumenti futhi sathola izingxenye ezihlakaniphile zokubopha i-ClickHouse ngaphansi kwe-Graphite, sanquma ukuthatha isinyathelo!

Ngingathanda ukuthola okulandelayo:

ukunciphisa ukusetshenziswa kwe-disk subsystem kusuka ku-30% kuya ku-5%;
yehlisa inani lesikhala esithathwayo sisuka ku-1TB siye ku-100GB;
ukwazi ukuthola amamethrikhi ayizigidi eziyi-100 ngomzuzu kuseva;
ukuphindaphindwa kwedatha nokubekezelela amaphutha ngaphandle kwebhokisi;
ungahlali kule phrojekthi unyaka futhi wenze ushintsho ngesikhathi esifanele;
shintsha ngaphandle kwesikhathi sokuphumula.

Isifiso sokuvelela, akunjalo?

I-Graphite+ClickHouse. Izingxenye

Ukwamukela idatha ngephrothokholi ye-Graphite bese uyiqopha ku-ClickHouse, ngikhethile i-carbon-clickhouse (ugogo).

Ukukhishwa kwakamuva kwe-ClickHouse, inguqulo ezinzile 1.1.54253, kukhethwe njengendawo egciniwe yokugcina uchungechunge lwesikhathi. Kwakukhona izinkinga lapho usebenza nayo: intaba yamaphutha athululelwa ezingodweni, futhi kwakungacaci ukuthi yini okufanele yenziwe ngabo. Engxoxweni no Roman Lomonosov (umbhali we-carbon-clickhouse, i-graphite-clickhouse nokunye okuningi, okuningi) omdala wakhethwa ukukhipha 1.1.54236. Amaphutha anyamalala - konke kwaqala ukusebenza nge-bang.

Ikhethelwe ukufunda idatha evela ku-ClickHouse i-graphite-сlickhouse (ugogo). Njenge-API ye-Graphite − carbonapi (ugogo). I-ClickHouse isetshenziselwe ukuhlela ukuphindaphinda phakathi kwamathebula umgcini wezilwane. Ukuze uthole amamethrikhi omzila, sishiye isithandwa sethu i-carbon-c-relay (NGANYE) (bona isihloko esidlule).

I-Graphite+ClickHouse. Isakhiwo sethebula

I-“graphite” isizindalwazi esisidalele amatafula okuqapha.

“graphite.metrics” - ithebula elinenjini ye-ReplicatedReplacingMergeTree (ephindaphindwe Ifaka esikhundleni seMergeTree). Leli thebula ligcina amagama amamethrikhi nezindlela eziya kuwo.

CREATE TABLE graphite.metrics ( Date Date, Level UInt32, Path String, Deleted UInt8, Version UInt32 ) ENGINE = ReplicatedReplacingMergeTree('/clickhouse/tables/replicator/graphite.metrics', ‘r1’, Date, (Level, Path), 8192, Version);

“graphite.data” - ithebula elinenjini ye-ReplicatedGraphiteMergeTree (okuphindaphindwe I-GraphiteMergeTree). Leli thebula ligcina amanani e-metric.

CREATE TABLE graphite.data ( Path String, Value Float64, Time UInt32, Date Date, Timestamp UInt32 ) ENGINE = ReplicatedGraphiteMergeTree('/clickhouse/tables/replicator/graphite.data', 'r1', Date, (Path, Time), 8192, 'graphite_rollup')

I-“graphite.date_metrics” iyithebula eligcwaliswe ngokombandela elinenjini ye-ReplicatedReplacingMergeTree. Leli thebula lirekhoda amagama awo wonke amamethrikhi okuhlangatshezwane nawo phakathi nosuku. Izizathu zokudalwa kwayo zichazwe esigabeni "Izinkinga" ekupheleni kwalesi sihloko.

CREATE MATERIALIZED VIEW graphite.date_metrics ( Path String,  Level UInt32,  Date Date) ENGINE = ReplicatedReplacingMergeTree('/clickhouse/tables/replicator/graphite.date_metrics', 'r1', Date, (Level, Path, Date), 8192) AS SELECT toUInt32(length(splitByChar('.', Path))) AS Level, Date, Path FROM graphite.data

“graphite.data_stat” - ithebula eligcwaliswe ngokuya ngesimo, ngenjini ye-ReplicatedAggregatingMergeTree (okuphindaphindwe I-AggregatingMergeTree). Leli thebula lirekhoda inani lamamethrikhi angenayo, ahlukaniswe amaleveli angu-4 okuzalela.

CREATE MATERIALIZED VIEW graphite.data_stat ( Date Date,  Prefix String,  Timestamp UInt32,  Count AggregateFunction(count)) ENGINE = ReplicatedAggregatingMergeTree('/clickhouse/tables/replicator/graphite.data_stat', 'r1', Date, (Timestamp, Prefix), 8192) AS SELECT toStartOfMonth(now()) AS Date, replaceRegexpOne(Path, '^([^.]+.[^.]+.[^.]+).*$', '1') AS Prefix, toUInt32(toStartOfMinute(toDateTime(Timestamp))) AS Timestamp, countState() AS Count FROM graphite.data  GROUP BY Timestamp, Prefix

I-Graphite+ClickHouse. Umdwebo wokusebenzelana kwengxenye

I-Graphite+ClickHouse. Ukuthuthwa kwedatha

Njengoba sikhumbula kulokho okulindelwe kule phrojekthi, ukushintshela ku-ClickHouse kufanele kube ngaphandle kwezikhathi zokuphumula; ngakho-ke, bekufanele ngandlela thile sishintshe lonke uhlelo lwethu lokuqapha siluyise endaweni entsha yokubeka ngokusobala ngangokunokwenzeka kubasebenzisi bethu.
Sikwenze kanjena.

Umthetho ungeziwe ku-carbon-c-relay ukuthumela ukusakazwa okwengeziwe kwamamethrikhi ku-carbon-clickhouse yeyodwa yamaseva abamba iqhaza ekuphindaphindweni kwamathebula e-ClickHouse.
Sibhale umbhalo omncane nge-python, okwathi, sisebenzisa umtapo wezincwadi wokulahla, safunda wonke amafayela we-.wsp kusitoreji sethu futhi sathumela le datha ku-carbon-clickhouse echazwe ngenhla ngemicu engu-24. Inani lamanani e-metric amukelwayo ku-carbon-clickhouse lifinyelele ku-125 million/min, futhi i-ClickHouse ayizange ikhuphule ngisho nomjuluko.
Sakhe i-DataSource ehlukile e-Grafana ukuze silungise imisebenzi esetshenziswa kumadeshibhodi akhona. Sihlonze uhlu lwemisebenzi esiyisebenzisile, kodwa ayizange isetshenziswe ku-carbonapi. Sengeze le misebenzi futhi sathumela ama-PRs kubabhali be-carbonapi (sibonga ngokukhethekile kubo).
Ukuze sishintshe umthwalo wokufunda kuzilungiselelo ze-balancer, siguqule amaphoyinti okugcina ukusuka ku-graphite-api (i-API interface ye-Graphite+Whisper) kuya ku-carbonapi.

I-Graphite+ClickHouse. imiphumela

ukunciphisa ukusetshenziswa kwe-disk subsystem kusuka ku-30% kuya ku-1%;
kuncishiswe inani lesikhala esithathiwe kusuka ku-1 TB kuya ku-300 GB;
sinekhono lokuthola amamethrikhi ayizigidi ezingu-125 ngomzuzu kuseva (iziqongo ngesikhathi sokuthutha);
idlulisele wonke amamethrikhi esikhawulweni sesitoreji samasekhondi angamashumi amathathu;
ithole ukuphindaphindwa kwedatha nokubekezelela amaphutha;
ishintshiwe ngaphandle kokuphumula;
Kuthathe cishe amasonto ayi-7 ukuqeda yonke into.

I-Graphite+ClickHouse. Izinkinga

Esimweni sethu, kwakunezihibe. Yilokhu esihlangabezane nakho ngemva koshintsho.

I-ClickHouse ayifundi kabusha njalo ukulungiselelwa endizeni; kwesinye isikhathi idinga ukuqaliswa kabusha. Isibonelo, endabeni yencazelo yeqoqo le-zookeeper ku-ClickHouse config, ayizange isetshenziswe kuze kube yilapho i-clickhouse-server iqalwa kabusha.
Izicelo ezinkulu ze-ClickHouse aziphumelelanga, ngakho-ke ku-graphite-clickhouse uchungechunge lwethu lokuxhuma lwe-ClickHouse lubukeka kanje:
```
url = "http://localhost:8123/?max_query_size=268435456&max_ast_elements=1000000"
```
I-ClickHouse ivamise ukukhipha izinguqulo ezintsha zokukhishwa okuzinzile; zingaqukatha lezimanga: qaphela.
Iziqukathi ezidalwe ngamandla ku-kubernetes zithumela inani elikhulu lamamethrikhi nempilo emfushane nengahleliwe. Awekho amaphuzu amaningi amamethrikhi anjalo, futhi azikho izinkinga ngesikhala. Kodwa uma kwakhiwa imibuzo, i-ClickHouse icosha inani elikhulu lawa mamethrikhi afanayo kuthebula 'lamamethrikhi'. Ezimweni ezingu-90%, ayikho idatha kuzo ngale kwewindi (amahora angu-24). Kodwa isikhathi sichithwa sicinga le datha kuthebula 'ledatha', futhi ekugcineni singena esikhathini sokuvala. Ukuze sixazulule le nkinga, siqale ukugcina umbono ohlukile ngolwazi lwamamethrikhi esihlangabezane nawo phakathi nosuku. Ngakho, lapho sakha imibiko (amagrafu) yeziqukathi ezidalwe ngamandla, sibuza kuphela lawo mamethrikhi esihlangabezane nawo ngaphakathi kwewindi elinikeziwe, hhayi isikhathi sonke, okusheshise kakhulu ukwakhiwa kwemibiko kuzo. Ngesixazululo esichazwe ngenhla, ngiqoqile i-graphite-clickhouse (imfoloko), okufaka ukusetshenziswa kokusebenza nethebula ledethi_metrics.

I-Graphite+ClickHouse. Omaka

Ngenguqulo 1.1.0 Graphite yaba ngokusemthethweni amathegi okusekela. Futhi sicabanga ngenkuthalo ukuthi yini nokuthi senzenjani ukuze sisekele lesi sinyathelo kusitaki se-graphite+clickhouse.

I-Graphite+ClickHouse. I-Anomaly detector

Ngokusekelwe kwingqalasizinda echazwe ngenhla, senze i-prototype yesitholi esingaqondakali, futhi siyasebenza! Kodwa okwengeziwe ngaye esihlokweni esilandelayo.

Bhalisa, cindezela umcibisholo oya phezulu futhi ujabule!

Source: www.habr.com

Isitoreji samamethrikhi: ukuthi sishintshe kanjani ukusuka ku-Graphite+Whisper kuya ku-Graphite+ClickHouse