I kēia lā, ma waho aʻe o ka monolithic code, ʻo kā mākou papahana e pili ana i nā hanana microservices. Pono kēlā me kēia o lākou e nānā. ʻO ka hana ʻana i kēia ma kēlā ʻano me ka hoʻohana ʻana i nā ʻenekini DevOps he pilikia. Ua hoʻomohala mākou i kahi ʻōnaehana nānā e hana ma ke ʻano he lawelawe no nā mea hoʻomohala. Hiki iā lākou ke kākau kūʻokoʻa i nā metric i loko o ka ʻōnaehana nānā, hoʻohana iā lākou, kūkulu i nā dashboards e pili ana iā lākou, a hoʻopili i nā mākaʻikaʻi iā lākou e hoʻoulu ʻia i ka wā e loaʻa ai nā waiwai paepae. No nā ʻenekini DevOps, ʻo ka ʻenehana a me nā palapala wale nō.
ʻO kēia pou kahi kope o kaʻu ʻōlelo me kā mākou ma RIT++. Nui ka poʻe i noi mai iā mākou e hana i nā mana kikokikona o nā hōʻike mai laila mai. Inā ʻoe ma ka ʻaha kūkā a nānā paha i ke wikiō, ʻaʻole ʻoe e ʻike i kahi mea hou. A me nā mea ʻē aʻe - aloha i ka pōpoki. E haʻi wau iā ʻoe pehea mākou i hele mai ai i kahi ʻōnaehana, pehea e hana ai a pehea mākou e hoʻolālā ai e hoʻonui.
ʻO ka wā i hala: nā papahana a me nā hoʻolālā
Pehea mākou i hōʻea ai i ka ʻōnaehana nānā i kēia manawa? I mea e pane ai i kēia nīnau, pono ʻoe e hele i 2015. Penei ke ano o ia manawa:
Loaʻa iā mākou ma kahi o 24 nodes i kuleana no ka nānā ʻana. Aia kekahi pūʻulu o nā lei aliʻi like ʻole, nā palapala, nā daemons e nānā i kekahi mea, hoʻouna i nā leka, a hana i nā hana. Ua manaʻo mākou ʻo ka hele ʻana aku, ʻoi aku ka liʻiliʻi o ia ʻōnaehana. ʻAʻohe kumu o ka hoʻomohala ʻana: paʻakikī loa.
Ua hoʻoholo mākou e koho i kēlā mau mea nānā a mākou e mālama a hoʻomohala, a me nā mea a mākou e haʻalele ai. He 19 o lākou. ʻO nā graphites, aggregators a me Grafana ma ke ʻano he dashboard i koe. Akā pehea ke ʻano o ka ʻōnaehana hou? E like me kēia:
Loaʻa iā mākou kahi waihona metric: he mau graphites kēia, e hoʻokumu ʻia ma nā drive SSD wikiwiki, ʻo ia kekahi mau aggregators no metrics. Aʻe - Grafana no ka hōʻike ʻana i nā dashboards a me Moira no ka makaʻala. Ua makemake pū mākou e hoʻomohala i kahi ʻōnaehana no ka ʻimi ʻana i nā anomalies.
Kūlana: Nānā 2.0
ʻO kēia ke ʻano o nā hoʻolālā i ka makahiki 2015. Akā, pono mākou e hoʻomākaukau ʻaʻole wale i ka ʻōnaehana a me ka lawelawe ponoʻī, akā i nā palapala no ia mea. Ua hoʻomohala mākou i kahi maʻamau ʻoihana no mākou iho, a mākou e kapa nei i ka nānā ʻana 2.0. He aha nā koi no ka ʻōnaehana?
- loaʻa mau;
- anana kaawale waiho = 10 kekona;
- ka mālama ʻana i nā metric a me nā dashboards;
- SLA > 99,99%
- ka hōʻiliʻili o nā ana hanana ma o UDP (!).
Pono mākou i ka UDP no ka mea he nui kā mākou kahe o nā kaʻa a me nā hanana e hana ana i nā metric. Inā kākau ʻoe iā lākou a pau i ka graphite i ka manawa hoʻokahi, e hāʻule ka waihona. Ua koho mākou i nā prefix pae mua no nā ana a pau.
He waiwai ko kēlā me kēia prefix. Aia nā metric no nā kikowaena, nā pūnaewele, nā ipu, nā kumuwaiwai, nā noi, a pēlā aku. Ua hoʻokō ʻia ke kānana akaka, koʻikoʻi, paʻi ʻia, kahi mākou e ʻae ai i nā metric pae mua a waiho wale i ke koena. Pēlā mākou i hoʻolālā ai i kēia ʻōnaehana ma 2015. He aha ka mea i kēia manawa?
I kēia manawa: ke kiʻikuhi o ka pilina o nā mea nānā
ʻO ka mea mua, nānā mākou i nā noi: kā mākou PHP code, nā noi a me nā microservices - i ka pōkole, nā mea a pau a kā mākou mea hoʻomohala e kākau ai. Hoʻouna nā noi āpau i nā ana ma o UDP i ka Brubeck aggregator (statsd, kākau hou ʻia ma C). Ua lilo ia i mea wikiwiki loa i nā hoʻokolohua synthetic. A hoʻouna ʻo ia i nā metric i hōʻuluʻulu ʻia iā Graphite ma o TCP.
Loaʻa iā ia kahi ʻano metric i kapa ʻia ʻo timers. He mea maʻalahi loa kēia. No ka laʻana, no kēlā me kēia mea hoʻohana pili i ka lawelawe, hoʻouna ʻoe i kahi metric me ka manawa pane iā Brubeck. Hoʻokahi miliona pane i komo mai, akā ua hoʻihoʻi ka mea hōʻuluʻulu i 10 metric wale nō. Loaʻa iā ʻoe ka helu o ka poʻe i hele mai, ka palena kiʻekiʻe, ka liʻiliʻi a me ka awelika o ka manawa pane, ka median a me 4 pakeneka. A laila hoʻoili ʻia ka ʻikepili i ka Graphite a ʻike mākou i ke ola āpau.
Loaʻa iā mākou ka hōʻuluʻulu ʻana no nā metric ma ka ʻenehana, lako polokalamu, nā ʻōnaehana ʻōnaehana a me kā mākou ʻōnaehana nānā Munin kahiko (ua hana ia no mākou a hiki i 2015). Hōʻiliʻili mākou i kēia ma o ka C daemon CollectD (he pūʻulu holoʻokoʻa o nā plugins ʻokoʻa i kūkulu ʻia i loko, hiki iā ia ke koho i nā kumuwaiwai āpau o ka ʻōnaehana hoʻokipa kahi i hoʻokomo ʻia ai, e kuhikuhi wale i ka hoʻonohonoho i kahi e kākau ai i ka ʻikepili) a e kākau i ka ʻikepili i ka Graphite ma o ia mea. Kākoʻo pū ʻo ia i nā plugins python a me nā palapala shell, no laila hiki iā ʻoe ke kākau i kāu mau hoʻoponopono maʻamau: E hōʻiliʻili ʻo CollectD i kēia ʻikepili mai kahi host kūloko a mamao paha (manaʻo ʻia ʻo Curl) a hoʻouna iā ia i Graphite.
A laila hoʻouna mākou i nā ana a pau a mākou i hōʻiliʻili ai iā Carbon-c-relay. ʻO kēia ka hopena Carbon Relay mai Graphite, i hoʻololi ʻia ma C. ʻO kēia kahi alalai e hōʻiliʻili ai i nā metric a pau a mākou e hoʻouna ai mai kā mākou aggregators a alakaʻi iā lākou i nā nodes. Ma ke kahua hoʻokele, e nānā i ka pono o nā ana. ʻO ka mea mua, pono lākou e like me ka prefix scheme aʻu i hōʻike mua ai a, ʻo ka lua, kūpono lākou no ka graphite. A i ʻole e hāʻule lākou.
Hoʻouna ʻo Carbon-c-relay i nā ana i ka hui Graphite. Hoʻohana mākou i ka Carbon-cache, kākau hou ʻia ma Go, ʻo ia ka waihona nui o nā ana. ʻO Go-carbon, ma muli o kāna multithreading, ʻoi aku ka maikaʻi ma mua o Carbon-cache. Loaʻa iā ia ka ʻikepili a kākau iā ia i nā disks me ka hoʻohana ʻana i ka pūʻolo hāwanawana (maʻamau, kākau ʻia ma ka python). I mea e heluhelu ai i ka ʻikepili mai kā mākou waihona, hoʻohana mākou i ka Graphite API. ʻOi aku ka wikiwiki ma mua o ka Graphite WEB maʻamau. He aha ka hopena i ka ʻikepili ma hope?
Hele lākou i Grafana. Hoʻohana mākou i kā mākou mau pūʻulu graphite ma ke kumu nui o ka ʻikepili, a loaʻa iā mākou ʻo Grafana ma ke ʻano he kikowaena pūnaewele no ka hōʻike ʻana i nā metric a me ke kūkulu ʻana i nā dashboards. No kēlā me kēia o kā lākou lawelawe, hana nā mea hoʻomohala i kā lākou dashboard ponoʻī. A laila kūkulu lākou i nā pakuhi e pili ana iā lākou, e hōʻike ana i nā metric a lākou e kākau ai mai kā lākou noi. Ma waho aʻe o Grafana, loaʻa iā mākou ka SLAM. He daimonio python kēia e helu ana i ka SLA ma muli o ka ʻikepili mai ka graphite. E like me kaʻu i ʻōlelo ai, loaʻa iā mākou he mau kaukani microservices, ʻo kēlā me kēia me kāna pono ponoʻī. Ke hoʻohana nei i ka SLAM, hele mākou i ka palapala a hoʻohālikelike me ka mea i loko o ka Graphite a hoʻohālikelike i ka maikaʻi o nā koi i ka loaʻa o kā mākou lawelawe.
E hele hou kāua: makaʻala. Hoʻonohonoho ʻia me ka hoʻohana ʻana i kahi ʻōnaehana ikaika - Moira. He kūʻokoʻa ia no ka mea he Graphite kona ma lalo o ka puʻu. Hoʻokumu ʻia e nā kāne mai SKB "Kontur", i kākau ʻia ma ka python a me Go, ke kumu wehe loa. Loaʻa iā Moira ke kahe like e hele ai i nā graphites. Inā no kekahi kumu i make ai kāu waihona, e mau ana kāu makaʻala.
Ua kau mākou iā Moira ma Kubernetes; hoʻohana ia i kahi pūʻulu o nā kikowaena Redis ma ke ʻano he waihona nui. ʻO ka hopena he ʻōnaehana hoʻomanawanui hewa. Hoʻohālikelike ia i ke kahawai o nā ana me ka papa inoa o nā mea hoʻomaka: inā ʻaʻohe ʻōlelo i loko, a laila e hāʻule i ka metric. No laila hiki iā ia ke ʻeli i nā gigabytes o nā metric i kēlā me kēia minuke.
Hoʻopili pū mākou i kahi LDAP hui iā ia, me ke kōkua o kēlā me kēia mea hoʻohana o ka ʻōnaehana hui e hiki ke hana i nā leka no lākou iho e pili ana i nā mea hoʻomaka (a i ʻole i hana hou ʻia). No ka mea aia ʻo Moira i ka Graphite, kākoʻo ia i kāna mau hiʻohiʻona āpau. No laila e lawe mua ʻoe i ka laina a kope iā ia i loko o Grafana. E ʻike pehea e hōʻike ʻia ai ka ʻikepili ma nā pakuhi. A laila lawe ʻoe i ka laina like a kope iā Moira. Kau ʻoe iā ia me nā palena a loaʻa i kahi makaʻala i ka hopena. No ka hana ʻana i kēia, ʻaʻole pono ʻoe i kahi ʻike kikoʻī. Hiki iā Moira ke makaʻala ma o SMS, leka uila, Jira, Slack ... Kākoʻo pū ʻo ia i ka hoʻokō ʻana i nā palapala maʻamau. Ke loaʻa iā ia kahi kumu, a kau inoa ʻo ia i kahi palapala maʻamau a i ʻole binary, holo ʻo ia a hoʻouna iā JSON i stdin no kēia binary. No laila, pono kāu polokalamu e hoʻokaʻawale iā ia. ʻO kāu mea e hana ai me kēia JSON aia iā ʻoe. Inā makemake ʻoe, e hoʻouna iā Telegram, inā makemake ʻoe, wehe i nā hana ma Jira, e hana i nā mea āpau.
Hoʻohana pū mākou i kā mākou kūkulu ponoʻī no ka makaʻala - Imagotag. Ua hoʻololi mākou i ka panel, i hoʻohana mau ʻia no nā hōʻailona kumukūʻai uila ma nā hale kūʻai, e kūpono i kā mākou pono. Lawe mākou i nā mea hoʻokani mai Moira iā ia. Hōʻike ia i ko lākou kūlana a i ka wā i hiki mai ai. Ua haʻalele kekahi o nā mea hoʻomohala i nā leka hoʻomaopopo ma Slack a me ka leka uila i makemake ʻia i kēia papa.
ʻAe, ʻoiai he hui holomua mākou, ua nānā pū mākou i nā Kubernetes ma kēia ʻōnaehana. Hoʻokomo mākou iā ia i loko o ka ʻōnaehana me ka hoʻohana ʻana iā Heapster, a mākou i hoʻokomo ai i loko o ka pūʻulu, ʻohi ia i ka ʻikepili a hoʻouna iā ia i ka Graphite. ʻO ka hopena, ua like ke kiʻikuhi penei:
Nā ʻāpana kiaʻi
Eia ka papa inoa o nā loulou i nā ʻāpana a mākou i hoʻohana ai no kēia hana. ʻO lākou a pau he open source.
graphite:
- kalapona hele:
- hawanawana.
- graphite-api:
Kalapona-c-relay:
Brubeck:
ʻohi ʻia:
Moira:
Grafana:
Heapster:
Lakeponahelu
A eia kekahi mau helu e pili ana i ka hana o ka ʻōnaehana no mākou.
Mea hōʻuluʻulu (brubeck)
Ka helu o nā ana: ~300/sec
Wawa no ka hoʻouna ʻana i nā ana i ka Graphite: 30 sec
Hoʻohana waiwai kikowaena: ~ 6% CPU (ke kamaʻilio nei mākou e pili ana i nā kikowaena piha); ~ 1Gb RAM; ~3 Mbps LAN
Graphite (go-carbon)
Ka helu o nā ana: ~ 1 / min
ʻO ka manawa hoʻopou hou o nā ana: 30 kekona
ʻO ka hoʻolālā mālama ʻana: 30sec 35d, 5min 90d, 10min 365d (hāʻawi iā ʻoe i ka hoʻomaopopo ʻana i ka hopena o ka lawelawe no ka manawa lōʻihi)
Hoʻohana waiwai kikowaena: ~10% CPU; ~ 20Gb RAM; ~30 Mbps LAN
ʻĀkau
Mahalo nui mākou ma Avito i ka maʻalahi i kā mākou lawelawe nānā. No ke aha ʻo ia i hele ai e like me kēia? ʻO ka mea mua, hiki ke hoʻololi ʻia kāna mau ʻāpana: ʻo nā ʻāpana ponoʻī a me kā lākou mau mana. ʻO ka lua, kākoʻo. No ka mea he kumu wehe ka papahana holoʻokoʻa, hiki iā ʻoe ke hoʻoponopono i ke code iā ʻoe iho, hana i nā loli, a hoʻokō i nā hana i loaʻa ʻole ma waho o ka pahu. Hoʻohana ʻia nā ʻōpala maʻamau, ʻo Go a me Python, no laila ua hana maʻalahi kēia.
Eia kekahi laʻana o ka pilikia maoli. ʻO kahi metric ma Graphite he faila. He inoa kona. inoa waihona = inoa metric. A aia kahi ala e hiki ai i laila. Ua kaupalena ʻia nā inoa faila ma Linux i 255 mau huaʻōlelo. A loaʻa iā mākou (ma ke ʻano he "mea kūʻai aku kūloko") mai ka ʻoihana waihona. 'Ōlelo lākou iā mākou: "Makemake mākou e nānā i kā mākou mau nīnau SQL. ʻAʻole lākou he 255 mau huaʻōlelo, akā 8 MB kēlā me kēia. Makemake mākou e hōʻike iā lākou ma Grafana, ʻike i nā ʻāpana no kēia noi, a ʻoi aku ka maikaʻi, makemake mākou e ʻike i ka piko o ia mau noi. He mea maikaʻi inā hōʻike ʻia i ka manawa maoli. He mea maikaʻi loa ke kau ʻana iā lākou i ka makaʻala. "
Lawe ʻia ka laʻana nīnau SQL ma ke ʻano he laʻana mai
Hoʻonohonoho mākou i kahi kikowaena Redis a hoʻohana i kā mākou Collectd plugins, e hele i Postgres a lawe i nā ʻikepili āpau mai laila, e hoʻouna ana i nā metric i Graphite. Akā hoʻololi mākou i ka inoa metric me nā hashes. Hoʻouna mākou i ka hash like iā Redis ma ke kī, a ʻo ka nīnau SQL holoʻokoʻa he waiwai. ʻO nā mea a mākou e hana ai, ʻo ia ka mea hiki iā Grafana ke hele i Redis a lawe i kēia ʻike. Ke wehe nei mākou i ka Graphite API no ka mea... ʻO kēia ke kumu nui no ka launa pū ʻana o nā mea nānā āpau me ka graphite, a komo mākou i kahi hana hou ma laila i kapa ʻia ʻo aliasByHash () - mai Grafana loaʻa mākou i ka inoa o ka metric, a hoʻohana iā ia i kahi noi iā Redis ma ke kī, ma pane mākou i ka waiwai o ke kī, ʻo ia kā mākou "Query SQL" " No laila, ua hōʻike mākou ma Grafana i kahi hōʻike o kahi nīnau SQL, ʻaʻole hiki ke hōʻike ʻia ma laila, me nā helu helu ma luna o ia (nā kelepona, nā lālani, total_time, ...).
Nā hopena
Loaʻa Loaʻa kā mākou lawelawe kiaʻi 24/7 mai kekahi noi a me kekahi code. Inā hiki iā ʻoe ke komo i nā keʻena mālama, hiki iā ʻoe ke kākau i ka ʻikepili i ka lawelawe. ʻAʻole koʻikoʻi ka ʻōlelo, ʻaʻole koʻikoʻi nā hoʻoholo. Pono ʻoe e ʻike pehea e wehe ai i kahi kumu, kau i kahi metric ma laila a pani i ke kumu.
Pono ʻO nā ʻāpana a pau he ʻae hewa a mālama maikaʻi i kā mākou ukana.
Pale haʻahaʻa i ke komo ʻana. I mea e hoʻohana ai i kēia ʻōnaehana, ʻaʻole pono ʻoe e aʻo i nā ʻōlelo papahana a me nā nīnau ma Grafana. E wehe wale i kāu noi, e hoʻokomo i kahi kumu i loko e hoʻouna i nā metric iā Graphite, pani iā ia, wehe iā Grafana, hana i nā dashboards ma laila a nānā i ke ʻano o kāu metric, e loaʻa ana nā leka ma o Moira.
Kūʻokoʻa. Hiki iā ʻoe ke hana i kēia mau mea iā ʻoe iho, me ke kōkua ʻole o nā ʻenekini DevOps. A he mea pono kēia, no ka mea hiki iā ʻoe ke nānā i kāu papahana i kēia manawa, ʻaʻole pono ʻoe e nīnau i kekahi - e hoʻomaka i ka hana a i ʻole e hoʻololi.
He aha kā mākou e ʻimi nei?
ʻO nā mea a pau i helu ʻia ma lalo nei ʻaʻole ia he mau manaʻo abstract wale nō, akā he mea e pili ana i ka liʻiliʻi o nā hana mua i lawe ʻia.
- Mea ʻike anomaly. Makemake mākou e hana i kahi lawelawe e hele ai i kā mākou waihona Graphite a nānā i kēlā me kēia metric me ka hoʻohana ʻana i nā algorithms like ʻole. Aia nā algorithms a mākou e makemake ai e nānā, aia ka ʻikepili, ʻike mākou pehea e hana ai me ia.
- Metadata. He nui kā mākou lawelawe, hoʻololi lākou i ka manawa, e like me ka poʻe e hana pū me lākou. ʻAʻole koho ka mālama mau ʻana i nā palapala me ka lima. ʻO ia ke kumu ke hoʻokomo nei mākou i nā metadata i kā mākou microservices. Hōʻike ʻo ia i ka mea nāna ia i hoʻomohala, nā ʻōlelo e launa pū me ia, nā koi SLA, kahi a me wai e hoʻouna ʻia ai nā leka. Ke kau ʻia kahi lawelawe, hana kaʻawale ʻia nā ʻikepili hui āpau. ʻO ka hopena, loaʻa iā ʻoe ʻelua mau loulou - hoʻokahi i nā mea hoʻoulu, ʻo kekahi i nā dashboards ma Grafana.
- Ka nānā ʻana i kēlā me kēia hale. Manaʻo mākou e hoʻohana nā mea hoʻomohala āpau i ia ʻōnaehana. I kēia hihia, hoʻomaopopo mau ʻoe i kahi o kāu huakaʻi, he aha ka mea e hana ai, kahi e hāʻule ai, kahi o kona mau nāwaliwali. Inā, no ka laʻana, hiki mai kekahi mea a hāʻule i kāu lawelawe, a laila e aʻo ʻoe e pili ana iā ia ʻaʻole i ke kelepona ʻana mai ka luna, akā mai kahi makaʻala, a hiki iā ʻoe ke wehe koke i nā lāʻau hou loa a ʻike i ka mea i hana ʻia ma laila.
- Hana kiʻekiʻe. Ke ulu mau nei kā mākou papahana, a i kēia lā ke hana nei ʻo ia ma kahi o 2 metric waiwai i kēlā me kēia minuke. I hoʻokahi makahiki i hala aku nei, he 000 kēia helu. E like me kaʻu i ʻōlelo ai, ʻo kēia ʻōnaehana kiaʻi ma ke ao holoʻokoʻa ma muli o ka hoʻololi ʻana o nā ʻāpana. Mālama kekahi a hoʻonui mau i kā lākou ʻoihana no Graphite, akā ua hoʻoholo mākou e hele i kahi ala ʻē aʻe: hoʻohana ma ke ʻano he waihona no kā mākou metric. Ua aneane pau kēia hoʻololi, a ʻaʻole koke wau e haʻi hou aku iā ʻoe pehea i hana ʻia ai: he aha nā pilikia i loaʻa a pehea i lanakila ai lākou, pehea ka hele ʻana o ka neʻe ʻana, e wehewehe wau i nā mea i koho ʻia e like me ka hoʻopaʻa ʻana a me kā lākou hoʻonohonoho.
Mahalo i kou nānā ʻana! E nīnau i kāu mau nīnau e pili ana i ke kumuhana, e hoʻāʻo wau e pane ma ʻaneʻi a i ʻole ma kēia mau pou. Ua ʻike paha kekahi i ke kūkulu ʻana i kahi ʻōnaehana nānā like a i ʻole ke hoʻololi ʻana iā Clickhouse i kahi kūlana like - kaʻana like i nā manaʻo.
Source: www.habr.com