Aloha kākou! I kona ʻatikala hope loa Ua kākau wau e pili ana i ka hoʻonohonoho ʻana i kahi ʻōnaehana nānā modular no ka hoʻolālā microservice. ʻAʻohe mea kū mālie, ke ulu mau nei kā mākou papahana, a pēlā pū ka helu o nā metric i mālama ʻia. Pehea mākou i hoʻonohonoho ai i ka hoʻololi ʻana mai Graphite+Whisper to Graphite+ClickHouse ma lalo o nā kūlana haʻahaʻa kiʻekiʻe, heluhelu e pili ana i nā manaʻolana mai ia mea a me nā hopena o ka neʻe ʻana ma lalo o ka ʻoki.

Ma mua o koʻu haʻi ʻana iā ʻoe pehea mākou i hoʻonohonoho ai i ka hoʻololi ʻana mai ka mālama ʻana i nā metric ma Graphite+Whisper a Graphite+ClickHouse, makemake wau e hāʻawi i ka ʻike e pili ana i nā kumu o ka hoʻoholo ʻana a me nā hemahema o Whisper a mākou i noho ai no ka manawa lōʻihi.

Nā pilikia Graphite+Whisper

1. Hoʻouka kiʻekiʻe ma ka subsystem disk

I ka manawa o ka hoʻololi ʻana, ma kahi o 1.5 miliona metric e hiki mai ana iā mākou i kēlā me kēia minuke. Me kēlā kahe, ʻo ka hoʻohana ʻana i ka disk ma nā kikowaena he ~ 30%. Ma keʻano laulā, ua ʻae ʻia kēia - ua hana paʻa nā mea a pau, ua kākau wikiwiki ʻia, heluhelu wikiwiki ... A hiki i kahi o nā hui hoʻomohala i ʻōwili i kahi hiʻohiʻona hou a hoʻomaka i ka hoʻouna ʻana iā mākou i 10 miliona metric i kēlā me kēia minuke. ʻO ia ka manawa i paʻa ai ka subsystem disk, a ʻike mākou i ka 100% hoʻohana. Ua hoʻoholo koke ʻia ka pilikia, akā ua koe ke koena.

2. Ka nele o ka hana hou ʻana a me ke kūlike

ʻO ka mea nui paha, e like me ka poʻe a pau e hoʻohana/hoʻohana i ka Graphite+Whisper, ua ninini mākou i ke kahawai like o nā ana ma luna o nā kikowaena Graphite i ka manawa hoʻokahi i mea e hana ai i ka hoʻomanawanui hewa. A ʻaʻohe pilikia kūikawā me kēia - a hiki i ka manawa i hāʻule ai kekahi o nā kikowaena no kekahi kumu. I kekahi manawa hiki iā mākou ke kiʻi i kahi kikowaena hāʻule wikiwiki, a hiki i ka carbon-c-relay ke hoʻouka i nā metrics mai kāna cache i loko, akā i kekahi manawa ʻaʻole. A laila aia kahi puka i loko o nā metric, a mākou i hoʻopiha ai me rsync. He lōʻihi loa ke kaʻina hana. ʻO ka pōmaikaʻi hoʻopakele wale nō, ʻaʻole i loaʻa pinepine kēia. Lawe pū mākou i kekahi mau ʻano helu maʻamau a hoʻohālikelike iā lākou me nā ʻano like ʻole ma nā node kokoke o ka puʻupuʻu. Ma kahi o 5% o nā hihia, ʻokoʻa nā waiwai, ʻaʻole mākou i hauʻoli nui.

3. Ka wāwae nui

Ma muli o kā mākou kākau ʻana ma Graphite ʻaʻole wale i ka ʻenehana, akā ʻo nā metric ʻoihana pū kekahi (a i kēia manawa nā metric mai Kubernetes), loaʻa pinepine mākou i kahi kūlana i loaʻa i ka metric he mau waiwai liʻiliʻi wale nō, a ua hana ʻia ka faila .wsp me ka mālama ʻana i nā mea āpau. manawa, a lawe i ka nui o ka manawa i hoʻokaʻawale mua ʻia, no mākou he ~ 2MB. Hoʻonui hou ʻia ka pilikia ma muli o ka nui o nā faila like e ʻike ʻia i ka manawa, a i ke kūkulu ʻana i nā hōʻike ma luna o lākou, ʻo ka heluhelu ʻana i nā kiko kikoʻī he nui ka manawa a me nā kumuwaiwai.

Makemake wau e hoʻomaopopo koke i nā pilikia i hōʻike ʻia ma luna nei e hiki ke hoʻoponopono ʻia me ka hoʻohana ʻana i nā ʻano like ʻole a me nā pae like ʻole o ka pono, akā ʻoi aku ka nui o ka ʻikepili āu e hoʻomaka ai e loaʻa, ʻoi aku ka nui o lākou.

Loaʻa i nā mea a pau i luna (e noʻonoʻo ana i ka mua nāʻatikala), a me ka hoʻonui mau ʻana i ka helu o nā metric i loaʻa, ka makemake e hoʻoneʻe i nā metric āpau i kahi manawa mālama o 30 kekona. (a hiki i ka 10 kekona inā pono), ua hoʻoholo mākou e hoʻāʻo iā Graphite+ClickHouse ma ke ʻano he mea hoʻohiki i ka Whisper.

Graphite+ClickHouse. Nā manaʻolana

Ma hope o ke kipa ʻana i kekahi mau hui o nā kāne mai Yandex, i ka heluhelu ʻana ʻelua mau ʻatikala ma Habré, i ka hele ʻana i nā palapala a loaʻa nā ʻāpana kūpono no ka hoʻopaʻa ʻana iā ClickHouse ma lalo o Graphite, ua hoʻoholo mākou e hana!

Makemake au e loaʻa kēia mau mea:

e hoʻemi i ka hoʻohana ʻana o ka subsystem disk mai 30% a i 5%;
e hoʻemi i ka nui o ka lumi i noho ʻia mai 1TB a i 100GB;
hiki ke loaʻa iā 100 miliona metrics i kēlā me kēia minuke i loko o ke kikowaena;
ka hoʻopili ʻana i ka ʻikepili a me ka hoʻomanawanui hewa mai ka pahu;
mai noho ma kēia papahana no hoʻokahi makahiki a hana i ka hoʻololi i loko o kahi manawa kūpono;
hoʻololi me ka wā hoʻomaha.

Ambisious loa, pololei?

Graphite+ClickHouse. Nā ʻāpana

No ka loaʻa ʻana o ka ʻikepili ma o ka protocol Graphite a hoʻopaʻa ʻia ma ClickHouse, ua koho wau hale kaomi kalapona (golang).

ʻO ka hoʻokuʻu hou loa o ClickHouse, stable version 1.1.54253, ua koho ʻia ʻo ia ka ʻikepili no ka mālama ʻana i ka manawa. Aia nā pilikia i ka hana ʻana me ia: ua ninini ʻia kahi mauna o nā hewa i loko o nā lāʻau, a ʻaʻole maopopo loa i ka mea e hana ai me lākou. Ma ke kūkākūkā me Roma Lomonosov (ka mea kākau o carbon-clickhouse, graphite-clickhouse a me nā mea he nui aku) ua koho ʻia ka mea kahiko hoʻokuʻu 1.1.54236. Ua nalowale nā hewa - ua hoʻomaka nā mea a pau e hana me kahi bang.

Koho ʻia e heluhelu i ka ʻikepili mai ClickHouse graphite-сlickhouse (golang). Ma ke ʻano he API no ka Graphite − carbonapi (golang). Ua hoʻohana ʻia ʻo ClickHouse e hoʻonohonoho i ka replication ma waena o nā papa kiai holoholona. No ka metric routing, haʻalele mākou i kā mākou mea aloha kalapona-c-relay (C) (e nānā i ka ʻatikala mua).

Graphite+ClickHouse. Hoʻolālā papa

ʻO "graphite" kahi waihona a mākou i hana ai no ka nānā ʻana i nā papa.

“graphite.metrics” - papaʻaina me ReplicatedReplacingMergeTree engine (replicated Ke pani nei iMergeTree). Mālama kēia pākaukau i nā inoa o nā ana a me nā ala iā lākou.

CREATE TABLE graphite.metrics ( Date Date, Level UInt32, Path String, Deleted UInt8, Version UInt32 ) ENGINE = ReplicatedReplacingMergeTree('/clickhouse/tables/replicator/graphite.metrics', ‘r1’, Date, (Level, Path), 8192, Version);

"graphite.data" - papaʻaina me ReplicatedGraphiteMergeTree engine (replicated GraphiteMergeTree). Mālama kēia pākaukau i nā waiwai metric.

CREATE TABLE graphite.data ( Path String, Value Float64, Time UInt32, Date Date, Timestamp UInt32 ) ENGINE = ReplicatedGraphiteMergeTree('/clickhouse/tables/replicator/graphite.data', 'r1', Date, (Path, Time), 8192, 'graphite_rollup')

ʻO "graphite.date_metrics" kahi papa i hoʻopiha piha ʻia me ka ʻenekini ReplicatedReplacingMergeTree. Hoʻopaʻa kēia papa i nā inoa o nā ana a pau i loaʻa i ka lā. Hōʻike ʻia nā kumu o kāna hana ʻana ma ka ʻāpana "Nā pilikia" ma ka hope o keia 'atikala.

CREATE MATERIALIZED VIEW graphite.date_metrics ( Path String,  Level UInt32,  Date Date) ENGINE = ReplicatedReplacingMergeTree('/clickhouse/tables/replicator/graphite.date_metrics', 'r1', Date, (Level, Path, Date), 8192) AS SELECT toUInt32(length(splitByChar('.', Path))) AS Level, Date, Path FROM graphite.data

"graphite.data_stat" - he papa i hoʻopiha ʻia e like me ke kūlana, me ka ʻenekini ReplicatedAggregatingMergeTree (hoʻopiʻi ʻia. HuihuiMergeTree). Hoʻopaʻa kēia papa ʻaina i ka helu o nā metric e hiki mai ana, i hoʻokaʻawale ʻia i 4 pae pūnana.

CREATE MATERIALIZED VIEW graphite.data_stat ( Date Date,  Prefix String,  Timestamp UInt32,  Count AggregateFunction(count)) ENGINE = ReplicatedAggregatingMergeTree('/clickhouse/tables/replicator/graphite.data_stat', 'r1', Date, (Timestamp, Prefix), 8192) AS SELECT toStartOfMonth(now()) AS Date, replaceRegexpOne(Path, '^([^.]+.[^.]+.[^.]+).*$', '1') AS Prefix, toUInt32(toStartOfMinute(toDateTime(Timestamp))) AS Timestamp, countState() AS Count FROM graphite.data  GROUP BY Timestamp, Prefix

Graphite+ClickHouse. Kiʻikuhi hui pū

Graphite+ClickHouse. Ka neʻe ʻana o ka ʻikepili

Ke hoʻomanaʻo nei mākou mai nā mea i manaʻo ʻia mai kēia papahana, ʻaʻole pono ka hoʻololi ʻana i ClickHouse me ka ʻole o nā manawa haʻahaʻa; no laila, pono mākou e hoʻololi i kā mākou ʻōnaehana kiaʻi holoʻokoʻa i ka waihona hou e like me ka hiki ke hiki i kā mākou mea hoʻohana.
Penei kā mākou i hana ai.

Ua hoʻohui ʻia kahi lula i ka carbon-c-relay e hoʻouna i kahi kahawai hou o nā ana i ka hale kaomi kalapona o kekahi o nā kikowaena e komo ana i ka replication o nā papa ClickHouse.
Ua kākau mākou i kahi palapala liʻiliʻi i loko o ka python, me ka hoʻohana ʻana i ka waihona whisper-dump, heluhelu i nā faila .wsp a pau mai kā mākou waihona a hoʻouna i kēia ʻikepili i ka carbon-clickhouse i hōʻike ʻia ma luna ma 24 mau kaula. ʻO ka nui o nā waiwai metric i ʻae ʻia i ka carbon-clickhouse i hōʻea i 125 miliona / min, a ʻaʻole i haki ʻo ClickHouse i kahi hou.
Ua hana mākou i kahi DataSource ʻokoʻa ma Grafana e hoʻopau i nā hana i hoʻohana ʻia i nā dashboards. Ua ʻike mākou i kahi papa inoa o nā hana a mākou i hoʻohana ai, akā ʻaʻole i hoʻokō ʻia i carbonapi. Hoʻohui mākou i kēia mau hana a hoʻouna aku i nā PR i nā mea kākau o carbonapi (mahalo kūikawā iā lākou).
No ka hoʻololi ʻana i ka haʻahaʻa heluhelu ma nā hoʻonohonoho kaulike, ua hoʻololi mākou i nā helu hope mai graphite-api (API interface no Graphite+Whisper) i carbonapi.

Graphite+ClickHouse. hopena

hoʻemi ʻia ka hoʻohana ʻana o ka subsystem disk mai 30% a i 1%;
ua ho'ēmi i ka nui o ka lumi i nohoʻia mai ka 1 TB a hiki i ka 300 GB;
hiki iā mākou ke loaʻa ka 125 miliona metric i kēlā me kēia minuke i loko o ke kikowaena (nā piko i ka manawa o ka neʻe ʻana);
hoʻololi i nā ana a pau i kahi manawa mālama kanakolukūmālua;
loaʻa ka hoʻopiʻi ʻikepili a me ka hoʻomanawanui hewa;
hoʻololi ʻia me ka manawa ʻole;
Ma kahi o 7 pule e hoʻopau ai i nā mea a pau.

Graphite+ClickHouse. Nā pilikia

I ko mākou hihia, aia kekahi mau pilikia. ʻO kēia ka mea a mākou i hālāwai ai ma hope o ka hoʻololi.

ʻAʻole heluhelu hou ʻo ClickHouse i nā configs ma ka lele; i kekahi manawa pono e hoʻomaka hou. No ka laʻana, ma ke ʻano o ka wehewehe ʻana i ka pūʻulu zookeeper i ka ClickHouse config, ʻaʻole i hoʻohana ʻia a hiki i ka hoʻomaka hou ʻana o ka clickhouse-server.
ʻAʻole i hele nā noi ClickHouse nui, no laila, ma ka graphite-clickhouse ko mākou ClickHouse kaula pili e like me kēia:
```
url = "http://localhost:8123/?max_query_size=268435456&max_ast_elements=1000000"
```
Hoʻokuʻu pinepine ʻo ClickHouse i nā mana hou o nā hoʻokuʻu paʻa; aia paha lākou i nā mea kupanaha: e akahele.
Hoʻouna ʻia nā pahu i hana ikaika ʻia i nā kubernetes i nā helu he nui me ka pōkole a me ke ola ʻole. ʻAʻole nui nā helu no ia mau ana, a ʻaʻohe pilikia me ka lewa. Akā i ke kūkulu ʻana i nā nīnau, ʻohi ʻo ClickHouse i kahi helu nui o kēia mau ana like mai ka papa 'metrics'. Ma 90% o nā hihia, ʻaʻohe ʻikepili iā lākou ma waho o ka puka makani (24 mau hola). Akā, ua pau ka manawa i ka ʻimi ʻana i kēia ʻikepili ma ka papa 'ikepili', a i ka hopena e holo i kahi manawa. I mea e hoʻoponopono ai i kēia pilikia, hoʻomaka mākou e mālama i kahi ʻike kaʻawale me ka ʻike e pili ana i nā metric i loaʻa i ka lā. No laila, i ke kūkulu ʻana i nā hōʻike (kiʻi) no nā ipu i hana ʻia me ka ikaika, e nīnau wale mākou i kēlā mau ana i ʻike ʻia i loko o ka puka aniani i hāʻawi ʻia, ʻaʻole no ka manawa holoʻokoʻa, ka mea i wikiwiki i ka hana ʻana i nā hōʻike ma luna o lākou. No ka hoʻonā i hōʻike ʻia ma luna, ua hōʻiliʻili wau graphite-clickhouse (fork), e pili ana i ka hoʻokō ʻana i ka hana ʻana me ka pākaukau date_metrics.

Graphite+ClickHouse. Nā huaʻōlelo

Me ka mana 1.1.0 ua lilo ʻo Graphite i mana kākoʻo nā hōʻailona. A ke noʻonoʻo nei mākou i ka mea a pehea e hana ai e kākoʻo i kēia hana ma ka graphite+clickhouse stack.

Graphite+ClickHouse. Mea ʻike anomaly

Ma muli o ka ʻōnaehana i hōʻike ʻia ma luna nei, ua hoʻokō mākou i kahi prototype o kahi mea ʻike anomaly, a hana ia! Akā e pili ana iā ia ma ka ʻatikala aʻe.

E kau inoa, e kaomi i ka pua i luna a hauʻoli!

Source: www.habr.com

Ka mālama ʻana i nā metric: pehea mākou i hoʻololi ai mai Graphite+Whisper i Graphite+ClickHouse