Salaamu caleykum In uu maqaalkii ugu dambeeyay Waxa aan wax ka qoray habaynta nidaamka la socodka hab-socodka ee dhismaha adeeg-yaraha. Ma jiraan wax taagan, mashruucayagu si joogto ah ayuu u korayaa, iyo sidoo kale tirada cabbirada kaydsan. Sida aan u abaabulnay ka gudbida Graphite+whisper ilaa Graphite+ClickHouse ee xaaladaha culeyska sareeyo, ka akhri waxyaabaha laga filayo iyo natiijooyinka socdaalka hoostiisa.

Kahor intaanan kuu sheegin sida aan u abaabulnay ka gudubka kaydinta metrics ee Graphite+whisper ilaa Graphite+ClickHouse, waxaan jeclaan lahaa inaan bixiyo macluumaad ku saabsan sababaha go'aankan oo kale iyo faa'iido darrada Whisper ee aan la noolayn muddo dheer.

Graphite+Dhibaatooyinka waswaasiinta

1. Culays sare oo ku saabsan nidaamka hoose ee diskka

Waqtigii kala guurka, ku dhawaad 1.5 milyan oo mitir ayaa na soo gaadhayey daqiiqaddii. Socodka noocaas ah, isticmaalka diskka ee server-yada wuxuu ahaa ~ 30%. Guud ahaan, tani waxay ahayd mid la aqbali karo - wax walba waxay u shaqeeyeen si adag, si deg deg ah ayaa loo qoray, si deg deg ah loo akhriyo ... Ilaa mid ka mid ah kooxaha horumarinta ayaa soo bandhigay muuqaal cusub oo bilaabay inay noo soo diraan 10 milyan oo metrik daqiiqadii. Taasi waa markii nidaamka-hoosaadka diskku uu adkeeyay, waxaana aragnay 100% isticmaalka. Dhibaatada si degdeg ah ayaa loo xaliyay, laakiin hadhaaga ayaa hadhay.

2. Ku celcelin la'aanta iyo joogtaynta

Waxay u badan tahay, sida qof kasta oo isticmaala/isticmaalay Graphite+Whisper, waxaanu ku shubnay isla jaangooyooyin isku mid ah dhawr adeegayaal Graphite hal mar si loo abuuro dulqaadka khaladka ah. Oo ma jirin wax dhibaato ah oo gaar ah oo tan la xidhiidha - ilaa hadda markii mid ka mid ah server-yada uu burburay sabab qaar ka mid ah. Mararka qaarkood waxaan ku guuleysanay inaan si dhaqso ah u soo qaadno server-ka dhacay, iyo kaarboon-c-relay waxay u suurtagashay in ay ka soo shubto cabbirada khasnadeeda, laakiin mararka qaarkood ma dhicin. Ka dibna waxaa jiray dalool jaantusyada, kaas oo aan ka buuxinay rsync. Nidaamku wuxuu ahaa mid aad u dheer. Nimcada kaliya ee badbaadinta ayaa ah in tani ay dhacday si aad u yar. Waxa kale oo aanu si xilliyo ah u qaadanay jaangooyooyin bakhtiyaa nasiib ah oo aan barbar dhignay kuwa kale oo isku nooc ah oo ku yaal noodhka dariska ee kooxda. Qiyaastii 5% kiisaska, dhowr qiyam ayaa kala duwanaa, kuwaas oo aan aad ugu faraxsanayn.

3. Raad weyn

Maadaama aan ku qorno Graphite ma aha oo kaliya kaabayaasha, laakiin sidoo kale cabbirada ganacsiga (iyo hadda sidoo kale qiyaasaha Kubernetes), waxaan inta badan helnaa xaalad taas oo mitirku ka kooban yahay oo kaliya dhowr qiyam ah, iyo faylka .wsp ayaa la abuuray iyadoo la tixgelinayo dhammaan haynta. muddada, oo waxay qaadataa meel hore loo qoondeeyay, taasoo anaga ahayd ~ 2MB. Dhibka waxaa sii xumeynaya in faylal badan oo la mid ah ay soo baxaan muddo ka dib, marka la dhisayo warbixinnada iyaga, akhrinta dhibcaha madhan waxay qaadataa waqti iyo dhaqaale badan.

Waxaan jeclaan lahaa inaan isla markiiba ogaado in dhibaatooyinka kor lagu soo sheegay lagu xallin karo iyadoo la adeegsanayo habab kala duwan iyo heerar kala duwan oo waxtar leh, laakiin xogta badan ee aad bilowdo inaad hesho, way ka sii darayaan.

Haysashada dhammaan kuwa kor ku xusan (iyadoo la tixgelinayo kuwii hore qodobbada), iyo sidoo kale korodhka joogtada ah ee tirada mitirka la helay, rabitaanka in lagu wareejiyo dhammaan mitirka kaydinta 30 ilbiriqsi. (ilaa 10 ilbiriqsi haddii loo baahdo), waxaan go'aansanay inaan isku dayno Graphite+ClickHouse sidii beddel rajo leh oo Whisper ah.

Graphite+ClickHouse. Filashooyinka

Isagoo booqday dhowr kulan oo guys ka Yandex, isagoo akhriyay dhowr maqaal oo ku saabsan Habré, markaan dhex marnay dukumentiyada oo aan helnay qaybo miyir-qab ah oo loogu talagalay in lagu xidho ClickHouse oo hoos timaada Graphite, waxaan go'aansanay inaan tallaabo qaadno!

Waxaan jeclaan lahaa inaan helo kuwa soo socda:

hoos u dhigista ka faa'iidaysiga nidaamka hoosaadka 30% ilaa 5%;
in la dhimo xadiga boos ee 1TB ilaa 100GB;
inuu awoodo inuu helo 100 milyan oo mitir daqiiqaddii server-ka;
ku celcelinta xogta iyo u dulqaadashada qaladka ee ka baxsan sanduuqa;
ha ku fadhiisan mashruucan muddo hal sano ah oo ku beddelka wakhti macquul ah gudahood;
beddelo bilaa waqti

Aad hami u leh, sax?

Graphite+ClickHouse. Qaybaha

Si aan xogta uga helo hab-maamuuska Graphite oo dabadeed lagu duubo ClickHouse, waxaan doortay kaarboon-clickhouse (golang).

Siideynta ugu dambeysay ee ClickHouse, nooca xasilloon ee 1.1.54253, ayaa loo doortay kaydka kaydinta taxanaha wakhtiga. Waxaa jiray dhibaatooyin markii la shaqeynayey: buur khaladaad ah ayaa lagu shubay qoryaha, oo gebi ahaanba ma cadda waxa lagu sameeyo iyaga. Iyadoo laga doodayo Roman Lomonosov (qoraaga kaarboon-clickhouse, graphite-clickhouse iyo qaar badan oo badan) kii hore ayaa la doortay. sii daayo 1.1.54236. Khaladaadka ayaa la waayay - wax walba waxay bilaabeen inay la shaqeeyaan bang.

Loo doortay inuu akhriyo xogta ClickHouse graphite-сlickhouse (golang). Sida API ee Graphite - carbonapi (golang). ClickHouse waxa loo isticmaalay in lagu abaabulo ku celcelinta miisaska dhexdooda xannaaneeyaha. Halbeeg-wareejinta, waxaanu ka tagnay gacaliyahayaga kaarboon-c-gudbinta (C) (eeg maqaal hore).

Graphite+ClickHouse. Qaab dhismeedka miiska

"graphite" waa xog-ururin aanu u abuurnay miisaska la socodka.

"graphite.metrics" - miis leh ReplicatedReplacingMergeTree engine (la soo koobay BedelkaMergeTree). Jadwalkani waxa uu kaydiyaa magacyada halbeegyada iyo waddooyinka loo maro.

CREATE TABLE graphite.metrics ( Date Date, Level UInt32, Path String, Deleted UInt8, Version UInt32 ) ENGINE = ReplicatedReplacingMergeTree('/clickhouse/tables/replicator/graphite.metrics', ‘r1’, Date, (Level, Path), 8192, Version);

"graphite.data" - miis leh ReplicatedGraphiteMergeTree engine (la soo koobay GraphiteMergeTree). Jadwalkani waxa uu kaydiyaa qiyamka mitirka.

CREATE TABLE graphite.data ( Path String, Value Float64, Time UInt32, Date Date, Timestamp UInt32 ) ENGINE = ReplicatedGraphiteMergeTree('/clickhouse/tables/replicator/graphite.data', 'r1', Date, (Path, Time), 8192, 'graphite_rollup')

"graphite.date_metrics" waa miis shuruudaysan oo ay ku jiraan mishiinka ReplicatedReplacingMergeTree. Jadwalkani waxa uu diiwaangelinayaa dhammaan magacyada cabbiraadaha la kulmay maalintii. Sababaha loo abuuray ayaa lagu tilmaamay qaybta "Dhibaatooyinka" dhamaadka maqaalkan.

CREATE MATERIALIZED VIEW graphite.date_metrics ( Path String,  Level UInt32,  Date Date) ENGINE = ReplicatedReplacingMergeTree('/clickhouse/tables/replicator/graphite.date_metrics', 'r1', Date, (Level, Path, Date), 8192) AS SELECT toUInt32(length(splitByChar('.', Path))) AS Level, Date, Path FROM graphite.data

"graphite.data_stat" - miis loo buuxiyey si waafaqsan shuruuda, oo leh mashiinka ReplicatedAggregatingMergeTree (la soo koobay) AggregatingMergeTree). Jadwalkani waxa uu diiwaangeliyaa tirada mitir ee soo socda, oo loo jajabiyey 4 heerar buul ah.

CREATE MATERIALIZED VIEW graphite.data_stat ( Date Date,  Prefix String,  Timestamp UInt32,  Count AggregateFunction(count)) ENGINE = ReplicatedAggregatingMergeTree('/clickhouse/tables/replicator/graphite.data_stat', 'r1', Date, (Timestamp, Prefix), 8192) AS SELECT toStartOfMonth(now()) AS Date, replaceRegexpOne(Path, '^([^.]+.[^.]+.[^.]+).*$', '1') AS Prefix, toUInt32(toStartOfMinute(toDateTime(Timestamp))) AS Timestamp, countState() AS Count FROM graphite.data  GROUP BY Timestamp, Prefix

Graphite+ClickHouse. Jaantuska isdhexgalka qaybaha

Graphite+ClickHouse. guuritaanka xogta

Markaan xasuusanayno waxyaabaha laga filayo mashruucan, u gudubka ClickHouse waa in uu ahaado mid aan waqtigoodu hoos u dhicin; sidaas awgeed, waa in aan si uun ugu beddelno nidaamkayaga kormeerka kaydinta cusub sida hufan ee suurtogalka ah isticmaalayaashayada.
Sidan ayaanu ku samaynay.

Xeer ayaa lagu daray kaarboon-c-relay si loogu diro qulqulka mitirka dheeraadka ah ee kaarboon-clickhouse ee mid ka mid ah adeegayaasha ka qaybqaadanaya ku celcelinta miisaska ClickHouse.
Waxaan ku qornay qoraal yar oo Python ah, kaas oo, isticmaalaya maktabadda whisper-dump, akhriyay dhammaan faylasha .wsp ee kaydintayada oo u dirnay xogtan kor ku xusan ee kaarboon-clickhouse ee 24 dunta. Tirada mitirka mitir ee la aqbalay ee kaarboon-clickhouse waxay gaadhay 125 milyan/daqiiqo, ClickHouse xitaa ma jabin dhididka.
Waxaan ku abuurnay DataSource gaar ah gudaha Grafana si aan u tirtirno hawlaha loo isticmaalo dashboards-ka jira. Waxaan aqoonsannay liiska shaqooyinka aan isticmaalnay, laakiin laguma hirgelin carbonapi. Waxaan ku darnay hawlahan waxaanan u dirnay PRs qorayaasha carbonapi (mahadsanid gaar ah iyaga).
Si loo beddelo culayska akhriska ee goobaha dheellitirka, waxaanu ka beddelnay dhibcaha ugu dambeeya graphite-api (API interface for Graphite+ Whisper) una beddelnay carbonapi.

Graphite+ClickHouse. natiijooyin

hoos u dhac ku yimid isticmaalka nidaam hoosaadka diskooga 30% ilaa 1%;
waxa la dhimay xadiga boos ee lagu haayo 1 TB ilaa 300 GB;
waxaan awood u leenahay inaan helno 125 milyan oo mitir daqiiqadiiba server-ka (ugu sarreeya wakhtiga socdaalka);
loo wareejiyay dhammaan cabbirada illaa soddon iyo ilbiriqsi ee kaydinta;
helay ku celcelinta xogta iyo dulqaadka qaladka;
la beddelo iyada oo aan wakhtiga dhimin;
Waxay qaadatay ilaa 7 toddobaad in la dhammaystiro wax walba.

Graphite+ClickHouse. Dhibaatooyinka

Xaaladeena, waxaa jiray dabinno. Tani waa tii aan la kulanay is bedelkii ka dib.

ClickHouse had iyo jeer dib uma akhriyo qaabaynta duullimaadka; mararka qaarkood waxay u baahan tahay in dib loo bilaabo. Tusaale ahaan, marka laga hadlayo sharraxaadda kooxda zookeeper ee isku xidhka ClickHouse, lama isticmaalin ilaa adeegaha-clickhouse dib loo bilaabo.
Codsiyada ClickHouse ee waaweyn ma aysan soo marin, marka graphite-clickhouse xarriiqeena xiriirinta ClickHouse ayaa u eg sidan:
```
url = "http://localhost:8123/?max_query_size=268435456&max_ast_elements=1000000"
```
ClickHouse waxay inta badan sii deysaa noocyo cusub oo ah siideyn deggan; waxaa laga yaabaa inay ku jiraan waxyaabo yaab leh: taxaddar.
Weelasha si firfircoon loo abuuray ee kubernetes waxay soo diraan tiro badan oo cabbiro leh nolol gaaban oo aan toos ahayn. Ma jiraan dhibco badan oo loogu talagalay cabbirada noocaan ah, mana jiraan wax dhibaato ah oo ku saabsan booska. Laakin marka la dhisayo su'aalaha, ClickHouse waxay ka soo qaadaa tiro aad u badan oo ka mid ah cabbiradaas 'metrics' miiska. 90% kiisaska, ma jiraan wax xog ah oo iyaga ka baxsan daaqada (24 saacadood). Laakiin wakhtigu waxa uu ku baxaa raadinta xogtan miiska 'xogta', oo ugu dambayntii waxay ku dhacdaa wakhti go'an. Si loo xalliyo dhibaatadan, waxaan bilownay inaan ilaalino aragti gooni ah oo aan ku hayno macluumaadka ku saabsan cabbirada la kulmay maalintii. Markaa, marka la dhisayo warbixinnada (garaafyada) weelasha si firfircooni leh loo abuuray, waxaynu waydiinnaa kaliya cabbirada lagula kulmay daaqad la bixiyay, ee maahan wakhtiga oo dhan, taas oo si weyn u dedejisay dhisidda warbixinnada iyaga. Xalka kor lagu sharaxay, waxaan soo ururiyay graphite-clickhouse (fargeeto), oo ay ku jirto hirgelinta la shaqaynta shaxda date_metrics.

Graphite+ClickHouse. Tags

Iyadoo nooca 1.1.0 Graphite uu noqday mid rasmi ah tags taageero. Waxaannu si firfircoon uga fakaraynaa waxa iyo sida loo sameeyo si aan u taageerno hindisahan garaafyada+ kaydinta guriga.

Graphite+ClickHouse. Anomaly detector

Iyada oo ku saleysan kaabayaasha kor lagu sharaxay, waxaan hirgelinay prototype of anomaly detector, wuuna shaqeeyaa! Laakiin wax badan oo ku saabsan isaga maqaalka soo socda.

Isdiiwaangeli, taabo falaarta kor oo faraxsan!

Source: www.habr.com

Kaydinta cabbirrada: sida aan uga soo beddelnay Graphite+whisper una beddelnay Graphite+ClickHouse