Molweni nonke! Kweyakhe inqaku lokugqibela Ndabhala malunga nokuququzelela inkqubo yokubeka iliso yemodyuli ye-microservice architecture. Akukho nto imileyo, iprojekthi yethu ikhula rhoqo, kwaye kunjalo nenani leemethrikhi ezigciniweyo. Indlela esilungiselele ngayo utshintsho ukusuka kwiGraphite + Whisper ukuya kwiGraphite + ClickHouse phantsi kweemeko eziphezulu zomthwalo, funda malunga nokulindelwe kuyo kunye neziphumo zokufuduka phantsi kokusikwa.

Ngaphambi kokuba ndikuxelele indlela esiluququzelele ngayo utshintsho ukusuka ekugcinweni kweemetrics kwiGraphite+Whisper ukuya kwiGraphite+ClickHouse, ndingathanda ukunika ulwazi malunga nezizathu zokwenza eso sigqibo kunye nokungalunganga kwe-Whisper esiye sahlala nayo ixesha elide.

Iingxaki zeGraphite+Bebeza

1. Umthwalo ophezulu kwi-subsystem yedisk

Ngexesha lotshintsho, malunga ne-1.5 yezigidi zeemetrics bezifika kuthi ngomzuzu. Ngokuhamba okunjalo, ukusetyenziswa kwedisk kwiiseva kwakuyi ~ 30%. Ngokubanzi, oku kwamkelekile - yonke into yayisebenza ngokuzinzile, yabhalwa ngokukhawuleza, yafunda ngokukhawuleza ... Kwaze kwaba yilapho elinye lamaqela ophuhliso likhupha into entsha kwaye laqala ukusithumela i-10 yezigidi zeemitha ngomzuzu. Kulapho inkqubo esezantsi yediski iqinisiwe, kwaye sabona ukusetyenziswa kwe-100%. Ingxaki yasonjululwa ngokukhawuleza, kodwa intsalela yasala.

2. Ukunqongophala kokuphindaphinda kunye nokungaguquguquki

Okunokwenzeka, njengaye wonke umntu osebenzisa/osebenzisa iGraphite+Whisper, sigalele umjelo ofanayo weemethrikhi kwiiseva ezininzi zeGrafite ngaxeshanye ukuze sidale unyamezelo lweziphoso. Kwaye akukho zingxaki zikhethekileyo kule nto - de kube ngumzuzu xa enye yeeseva yaphuka ngenxa yesizathu esithile. Ngamanye amaxesha siye sakwazi ukuthabatha iseva ewileyo ngokukhawuleza ngokwaneleyo, kwaye i-carbon-c-relay ikwazi ukulayisha i-metrics ukusuka kwi-cache yayo kuyo, kodwa ngamanye amaxesha akunjalo. Kwaye ke kwakukho umngxuma kwiimethrikhi, esazizalisa nge-rsync. Inkqubo yayinde kakhulu. Ubabalo olusindisayo kuphela kukuba oku kwenzeka kunqabile. Siphinde ngamaxesha athile sithathe isethi engakhethiyo yeemetrics kwaye sizithelekise nezinye zohlobo olufanayo kwiindawo ezingabamelwane zeqela. Malunga ne-5% yamatyala, amaxabiso ahlukeneyo ayahluka, nto leyo esingavuyanga ngayo.

3. Umzobo omkhulu

Kuba sibhala kwiGraphite kungekuphela nje iziseko zophuhliso, kodwa neemetrics zeshishini (kwaye ngoku neemetrics ezivela Kubernetes), sihlala sifumana imeko apho imetric iqulathe amaxabiso ambalwa kuphela, kwaye ifayile ye.wsp yenziwe ngokuthathela ingqalelo konke ukugcinwa. ixesha, kwaye ithatha isixa-mali esabiwe kwangaphambili sesithuba, ebesiyi ~2MB. Ingxaki iyancipha ngakumbi kukuba ezininzi iifayile ezifanayo zivela ngexesha, kwaye xa ukwakhiwa kweengxelo kuzo, ukufunda amanqaku angenanto kuthatha ixesha elininzi kunye nezibonelelo.

Ndingathanda ukuqaphela ngokukhawuleza ukuba iingxaki ezichazwe ngasentla zinokujongwa ngokusetyenziswa kweendlela ezahlukeneyo kunye namazinga ahlukeneyo okusebenza, kodwa xa uqala ukuyifumana idatha eninzi, iyancipha ngakumbi.

Ukuba nakho konke oku kungasentla (uthathela ingqalelo okudlulileyo amanqaku), kunye nokunyuka okuqhubekayo kwinani leemetriki ezifunyenweyo, umnqweno wokudlulisa zonke iimethrikhi kwisithuba sokugcina imizuzwana engama-30. (ukuya kuthi ga kwimizuzwana eyi-10 ukuba kuyimfuneko), sigqibe kwelokuba sizame iGraphite+ClickHouse njengenye indlela ethembisayo kwiWhisper.

Graphite+ClickHouse. Okulindelekileyo

Ukundwendwela iindibano ezininzi zabafana abavela eYandex, befundile amanqaku ambalwa ngoHabré, emva kokuba sidlule kumaxwebhu kwaye sifumene amacandelo aphilileyo okubopha i-ClickHouse phantsi kweGraphite, sagqiba ekubeni sithathe inyathelo!

Ndingathanda ukufumana oku kulandelayo:

ukunciphisa ukusetyenziswa kwe-disk subsystem ukusuka kwi-30% ukuya kwi-5%;
ukunciphisa inani lendawo ehlala kwi-1TB ukuya kwi-100GB;
ukwazi ukufumana i-100 lezigidi zeemetrics ngomzuzu kwiseva;
ukuphindaphindwa kwedatha kunye nokunyamezela iimpazamo ngaphandle kwebhokisi;
ungahlali kule projekthi unyaka kwaye wenze utshintsho ngexesha elifanelekileyo;
tshintsha ngaphandle kwexesha lokuphumla.

Amabhongo kakhulu, akunjalo?

Graphite+ClickHouse. Amacandelo

Ukufumana idatha ngeGraphite protocol kwaye emva koko uyirekhode kwiClickHouse, ndiyikhethile i-carbon-clickhouse (golang).

Ukukhutshwa kwamva nje kweClickHouse, inguqulo ezinzileyo ye-1.1.54253, ikhethwe njengesiseko sedatha yokugcina uchungechunge lwexesha. Kwakukho iingxaki xa usebenza nayo: intaba yeempazamo ithululelwe kwizigodo, kwaye yayingacacanga ngokupheleleyo into enokuyenza ngayo. Kwingxoxo kunye Roman Lomonosov (umbhali we carbon-clickhouse, graphite-clickhouse and many, many more) omdala wakhethwa ukukhululwa 1.1.54236. Iimpazamo zanyamalala - yonke into yaqala ukusebenza nge-bang.

Ikhethwe ukufunda idatha esuka kwiClickHouse igraphite-сlickhouse (golang). Njenge-API yeGraphite - carbonapi (golang). I-ClickHouse isetyenziselwe ukulungelelanisa uphindaphindo phakathi kweetafile umgcini wezilwanyana. Ngokwenza iimethrikhi, sishiye intanda yethu carbon-c-relay (UKUSUKA) (jonga inqaku elidlulileyo).

Graphite+ClickHouse. Isakhiwo setafile

I-“graphite” yindawo egciniweyo esiyenzele iitafile zokubeka iliso.

"graphite.metrics" -itheyibhile ene-ReplicatedReplacingMergeTree injini (iphindaphindwe Ukubuyisela iMergeTree). Le theyibhile igcina amagama eemetrics kunye neendlela eziya kuzo.

CREATE TABLE graphite.metrics ( Date Date, Level UInt32, Path String, Deleted UInt8, Version UInt32 ) ENGINE = ReplicatedReplacingMergeTree('/clickhouse/tables/replicator/graphite.metrics', ‘r1’, Date, (Level, Path), 8192, Version);

"graphite.data" - itheyibhile ene-ReplicatedGraphiteMergeTree injini (ephindaphindwayo GraphiteMergeTree). Le theyibhile igcina amaxabiso emetric.

CREATE TABLE graphite.data ( Path String, Value Float64, Time UInt32, Date Date, Timestamp UInt32 ) ENGINE = ReplicatedGraphiteMergeTree('/clickhouse/tables/replicator/graphite.data', 'r1', Date, (Path, Time), 8192, 'graphite_rollup')

"graphite.date_metrics" yitafile ezaliswe ngokwemeko enenjini yeReplicatedReplaceMergeTree. Le theyibhile irekhoda amagama azo zonke ii-metrics ezifunyenwe ngexesha lasemini. Izizathu zokudalwa kwayo zichazwe kwicandelo "Iingxaki" ekupheleni kweli nqaku.

CREATE MATERIALIZED VIEW graphite.date_metrics ( Path String,  Level UInt32,  Date Date) ENGINE = ReplicatedReplacingMergeTree('/clickhouse/tables/replicator/graphite.date_metrics', 'r1', Date, (Level, Path, Date), 8192) AS SELECT toUInt32(length(splitByChar('.', Path))) AS Level, Date, Path FROM graphite.data

"graphite.data_stat" - itafile ezaliswe ngokwemeko, kunye ne-injini ye-ReplicatedAggregatingMergeTree (ephindwe kabini AggregatingMergeTree). Le theyibhile irekhoda inani leemethrikhi ezingenayo, zohlulwe ukuya kutsho kumanqanaba ama-4 okuzala.

CREATE MATERIALIZED VIEW graphite.data_stat ( Date Date,  Prefix String,  Timestamp UInt32,  Count AggregateFunction(count)) ENGINE = ReplicatedAggregatingMergeTree('/clickhouse/tables/replicator/graphite.data_stat', 'r1', Date, (Timestamp, Prefix), 8192) AS SELECT toStartOfMonth(now()) AS Date, replaceRegexpOne(Path, '^([^.]+.[^.]+.[^.]+).*$', '1') AS Prefix, toUInt32(toStartOfMinute(toDateTime(Timestamp))) AS Timestamp, countState() AS Count FROM graphite.data  GROUP BY Timestamp, Prefix

Graphite+ClickHouse. Umzobo wentsebenziswano yecandelo

Graphite+ClickHouse. Ukufuduka kwedatha

Njengoko sikhumbula kwizinto ezilindelekileyo kule projekthi, ukutshintshela kwi-ClickHouse kufuneka kube ngaphandle kwamaxesha aphantsi; ngokufanelekileyo, kwafuneka ukuba ngandlela-thile sitshintshe inkqubo yethu yokubeka iliso kwisitoreji esitsha ngokucacileyo ngangokunokwenzeka kubasebenzisi bethu.
Senze ngolu hlobo ke.

Umgaqo wongezwe kwi-carbon-c-relay ukuthumela umlambo owongezelelweyo we-metrics kwi-carbon-clickhouse enye yeeseva ezithatha inxaxheba ekuphindaphindweni kweetafile ze-ClickHouse.
Sabhala iskripthi esincinci kwi-python, ethi, ngokusebenzisa ilayibrari yokulahla i-whisper, funda zonke iifayile ze-wsp kwisitoreji sethu kwaye wathumela le datha kwi-carbon-clickhouse echazwe ngasentla kwimicu ye-24. Inani leemetriki ezamkelweyo kwi-carbon-clickhouse yafikelela kwi-125 yezigidi / min, kwaye iClickHouse ayizange ikhuphe ukubila.
Senze uMthombo weDatha owahlukileyo eGrafana ukulungisa iimpazamo ezisetyenziswa kwiideshibhodi ezikhoyo. Sichonge uluhlu lwemisebenzi esiyisebenzisayo, kodwa ayizange isetyenziswe kwi-carbonapi. Songeze le misebenzi kwaye sathumela i-PRs kubabhali be-carbonapi (umbulelo okhethekileyo kubo).
Ukutshintsha umthwalo wokufunda kwi-balancer setting, satshintsha isiphelo se-graphite-api (i-API interface ye-Graphite + Whisper) kwi-carbonapi.

Graphite+ClickHouse. iziphumo

ukunciphisa ukusetyenziswa kwe-disk subsystem ukusuka kwi-30% ukuya kwi-1%;
ukunciphisa inani lendawo ethathwe kwi-1 TB ukuya kwi-300 GB;
sinamandla okufumana i-125 yezigidi zeemetrics ngomzuzu kwi-server (iincopho ngexesha lokufuduka);
udlulisele zonke iimetrics kwisithuba sokugcinwa kwemizuzu engamashumi amathathu anesibini;
ifumene ukuphindaphindwa kwedatha kunye nokunyamezela iimpazamo;
itshintshe ngaphandle kwexesha;
Kuthathe malunga neeveki ezisi-7 ukugqiba yonke into.

Graphite+ClickHouse. Iingxaki

Kwimeko yethu, kwakukho imigibe. Yile nto siye sadibana nayo emva kwenguqu.

I-ClickHouse ayisoloko ifunda kwakhona uqwalaselo kubhabho; ngamanye amaxesha kufuneka iqalwe ngokutsha. Ngokomzekelo, kwimeko yenkcazo yeqela lezookeeper kwiClickHouse config, ayizange isetyenziswe de i-clickhouse-server iqaliswe kwakhona.
Izicelo ezinkulu zeClickHouse azikhange zigqithe, ke kwigraphite-clickhouse yethu umtya woqhagamshelo lweClickHouse ujongeka ngolu hlobo:
```
url = "http://localhost:8123/?max_query_size=268435456&max_ast_elements=1000000"
```
I-ClickHouse ihlala ikhupha iinguqulelo ezintsha zokhupho oluzinzileyo; zinokuqulatha izinto ezothusayo: lumka.
Izikhongozeli ezenziwe ngamandla kwi-kubernetes zithumela inani elikhulu leemetrikhi ngexesha elifutshane kunye nobomi obungenamkhethe. Akukho manqaku amaninzi kwiimetriki ezinjalo, kwaye akukho ngxaki ngesithuba. Kodwa xa kusakhiwa imibuzo, iClickHouse ithatha inani elikhulu lezi metrics ezifanayo kwitafile 'yemetrics'. Kwi-90% yamatyala, akukho datha kubo ngaphaya kwefestile (iiyure ezingama-24). Kodwa ixesha lichithwa kukhangelwa le datha kwitheyibhile 'yedatha', kwaye ekugqibeleni ibaleka kwixesha lokuphuma. Ukuze sisombulule le ngxaki, saqala ukugcina imbono eyahlukileyo ngolwazi malunga neemetrics esiye sadibana nazo emini. Ngaloo ndlela, xa kwakha iingxelo (iigrafu) zemigqomo eyenziwe ngamandla, sibuza kuphela ezo metrics eziye zadibana nazo ngaphakathi kwefestile enikeziweyo, kwaye kungekhona ixesha lonke, eliye lakhawulezisa ukwakhiwa kweengxelo kuzo. Kwisisombululo esichazwe ngasentla, ndiqokelele graphite-clickhouse (ifolokhwe), ebandakanya ukuphunyezwa kokusebenza kunye netheyibhile ye-date_metrics.

Graphite+ClickHouse. Iithegi

Ngenguqulo 1.1.0 Graphite yaba ngokusemthethweni iithegi zenkxaso. Kwaye sicinga ngenkuthalo malunga nokuba yintoni kwaye senze njani ukuxhasa eli nyathelo kwi-graphite+clickhouse stack.

Graphite+ClickHouse. Isibonisi esingaqhelekanga

Ngokusekelwe kwiziseko ezingundoqo ezichazwe ngasentla, siphumeze iprototype ye-anomaly detector, kwaye iyasebenza! Kodwa okungakumbi ngaye kwinqaku elilandelayo.

Bhalisa, cofa utolo oluphezulu kwaye wonwabe!

umthombo: www.habr.com

Ugcino lweeMetrics: indlela esitshintshe ngayo ukusuka kwigraphite+Whisper ukuya kwiGraphite+ClickHouse