Masiku ano, kuwonjezera pa code monolithic, polojekiti yathu ikuphatikiza ma microservices ambiri. Aliyense wa iwo amafuna kuyang'aniridwa. Kuchita izi pamlingo wotere pogwiritsa ntchito mainjiniya a DevOps ndikovuta. Tapanga njira yowunikira yomwe imagwira ntchito ngati ntchito kwa omanga. Amatha kulemba paokha ma metric mumayendedwe owunikira, kuwagwiritsa ntchito, kupanga ma dashboard potengera iwo, ndikuyika zidziwitso kwa iwo zomwe zingayambitsidwe zikafika pofika. Kwa mainjiniya a DevOps, zomangamanga zokha ndi zolemba.
Cholemba ichi ndi cholembedwa cha zolankhula zanga ndi zathu pa RIT++. Anthu ambiri adatipempha kuti tipange zolemba za malipoti kuchokera kumeneko. Mukadakhala pamsonkhano kapena kuwonera kanema, simupeza chatsopano. Ndipo ena onse - kulandiridwa kwa mphaka. Ndikuuzani momwe tinafikira dongosolo loterolo, momwe limagwirira ntchito komanso momwe timakonzekera kukonzanso.
Zakale: ndondomeko ndi mapulani
Kodi tinafika bwanji pa dongosolo loyang'anira panopa? Kuti muyankhe funso ili, muyenera kupita ku 2015. Izi ndi momwe zinkawonekera panthawiyo:
Tinali ndi ma node pafupifupi 24 omwe anali ndi udindo woyang'anira. Pali paketi yonse ya akorona osiyanasiyana, zolembedwa, ma daemoni omwe mwanjira ina yake amawunika, kutumiza mauthenga, ndikuchita ntchito. Tinkaganiza kuti tikapita patsogolo, dongosolo loterolo silikhala lothandiza. Palibe chifukwa chokulitsa: ndizovuta kwambiri.
Tinaganiza zosankha zinthu zowunikira zomwe tizisunga ndikuzikulitsa, ndi zomwe tidzazisiya. Panali 19. Ma graphite, ma aggregator ndi Grafana monga dashboard okha ndi omwe adatsala. Koma kodi dongosolo latsopanoli lidzakhala lotani? Ngati chonchi:
Tili ndi ma metrics osungira: awa ndi ma graphite, omwe azitengera ma drive a SSD othamanga, awa ndi ophatikiza ma metric. Chotsatira - Grafana powonetsa ma dashboards ndi Moira kuti achenjeze. Tinkafunanso kupanga njira yofufuzira zolakwika.
Muyezo: Monitoring 2.0
Izi ndi zomwe mapulaniwo ankawoneka mu 2015. Koma sitinayenera kukonzekera zokhazokha ndi ntchito yokha, komanso zolemba zake. Tadzipangira tokha muyezo wamakampani, womwe timawutcha kuti monitoring 2.0. Kodi zofunika pa dongosololi zinali zotani?
- kupezeka kosalekeza;
- metrics yosungirako nthawi = masekondi 10;
- kusungidwa kwadongosolo kwa metrics ndi dashboards;
- SLA> 99,99%
- kusonkhanitsa ma metric a zochitika kudzera pa UDP (!).
Tidafunikira UDP chifukwa tili ndi kuchuluka kwa magalimoto ndi zochitika zomwe zimapanga ma metric. Mukawalemba onse mu graphite nthawi imodzi, zosungirako zidzagwa. Tidasankhanso ma prefixes amtundu woyamba wamametric onse.
Iliyonse mwa ma prefixes ili ndi katundu wake. Pali ma metric a maseva, ma network, zotengera, zothandizira, mapulogalamu, ndi zina zotero. Zosefera zomveka bwino, zokhwima, zotayidwa zakhazikitsidwa, pomwe timavomereza ma metric a mulingo woyamba ndikungosiya zina zonse. Umu ndi momwe tinakonzera dongosololi mu 2015. Kodi panopa ndi chiyani?
Panopa: chithunzi cha kuyanjana kwa zigawo zowunikira
Choyamba, timayang'anira ntchito: nambala yathu ya PHP, mapulogalamu ndi ma microservices - mwachidule, zonse zomwe opanga athu amalemba. Mapulogalamu onse amatumiza ma metrics kudzera ku UDP kupita ku Brubeck aggregator (statsd, olembedwanso mu C). Zinapezeka kuti ndizothamanga kwambiri pamayeso opangira. Ndipo imatumiza ma metric ophatikizidwa kale ku Graphite kudzera pa TCP.
Ili ndi mtundu wa ma metric otchedwa zowerengera nthawi. Ichi ndi chinthu chothandiza kwambiri. Mwachitsanzo, pakugwiritsa ntchito aliyense wogwiritsa ntchito, mumatumiza metric yokhala ndi nthawi yoyankha ku Brubeck. Mayankho miliyoni adabwera, koma ophatikiza adabweza ma metric 10 okha. Muli ndi chiwerengero cha anthu amene anabwera, pazipita, osachepera ndi pafupifupi nthawi kuyankha, apakatikati ndi 4 percentiles. Kenako deta imasamutsidwa ku Graphite ndipo tikuwona zonse zamoyo.
Tilinso ndi ma metrics pa hardware, mapulogalamu, ma metrics a dongosolo ndi makina athu akale a Munin (zinatithandiza mpaka 2015). Timasonkhanitsa zonsezi kudzera mu C daemon CollectD (ili ndi gulu lonse la mapulagini osiyanasiyana omwe amamangidwa mmenemo, amatha kufufuza zonse zomwe zili ndi makina osungira omwe aikidwapo, tangofotokozani momwe mungalembere deta) ndi lembani deta ku Graphite kupyolera mu izo. Imathandiziranso mapulagini a python ndi zolemba za zipolopolo, kotero mutha kulemba mayankho anu omwe mumakonda: CollectD idzasonkhanitsa deta iyi kuchokera kwa munthu wamba kapena wakutali (potengera Curl) ndikutumiza ku Graphite.
Kenako timatumiza ma metric onse omwe tidasonkhanitsa ku Carbon-c-relay. Iyi ndi njira ya Carbon Relay yochokera ku Graphite, yosinthidwa mu C. Iyi ndi rauta yomwe imasonkhanitsa ma metrics onse omwe timatumiza kuchokera kwa ophatikiza athu ndikuwapititsa ku ma node. Komanso pa siteji yolowera, imayang'ana kutsimikizika kwa ma metric. Choyamba, ziyenera kugwirizana ndi chiwembu choyambira chomwe ndidawonetsa kale ndipo, chachiwiri, ndizovomerezeka ku graphite. Apo ayi adzagwa.
Carbon-c-relay kenako imatumiza ma metrics ku gulu la Graphite. Timagwiritsa ntchito Carbon-cache, yolembedwanso mu Go, monga chosungira chachikulu cha ma metric. Go-carbon, chifukwa cha kuchuluka kwake, imaposa Carbon-cache. Imalandira deta ndikuyilembera ku disks pogwiritsa ntchito phukusi la whisper (muyezo, wolembedwa mu python). Kuti tiwerenge deta kuchokera muzosungira zathu, timagwiritsa ntchito Graphite API. Ndiwothamanga kwambiri kuposa muyezo wa Graphite WEB. Chimachitika ndi chiyani pa data kenako?
Iwo amapita ku Grafana. Timagwiritsa ntchito magulu athu a graphite monga gwero lalikulu la data, kuphatikiza tili ndi Grafana ngati mawonekedwe apaintaneti owonetsera ma metric ndi ma dashboards omangira. Pa mautumiki awo aliwonse, opanga amapanga dashboard yawo. Kenako amapanga ma graph potengera iwo, omwe amawonetsa ma metric omwe amalemba kuchokera pamapulogalamu awo. Kuphatikiza pa Grafana, tilinso ndi SLAM. Ichi ndi chiwanda cha python chomwe chimawerengera SLA kutengera deta kuchokera ku graphite. Monga ndanenera kale, tili ndi ma microservices angapo, iliyonse ili ndi zofunikira zake. Pogwiritsa ntchito SLAM, timapita ku zolembazo ndikuzifanizitsa ndi zomwe zili mu Graphite ndikufanizira momwe zofunikira zimayenderana ndi kupezeka kwa mautumiki athu.
Tiyeni tipitirire: kuchenjeza. Imakonzedwa pogwiritsa ntchito dongosolo lolimba - Moira. Ndiwodziyimira pawokha chifukwa ili ndi Graphite yake pansi pa hood. Yopangidwa ndi anyamata ochokera ku SKB "Kontur", olembedwa mu python ndi Go, gwero lotseguka. Moira amalandira kuyenda komweko komwe kumapita ku ma graphite. Ngati pazifukwa zina zosungira zanu zifa, kuchenjeza kwanu kudzagwirabe ntchito.
Tidatumiza Moira ku Kubernetes; imagwiritsa ntchito gulu la ma seva a Redis ngati nkhokwe yayikulu. Chotsatira chake chinali dongosolo lololera zolakwika. Imafananiza mtsinje wa ma metric ndi mndandanda wazoyambitsa: ngati palibe zotchulidwa mmenemo, ndiye kuti imatsitsa metric. Chifukwa chake imatha kukumba ma gigabytes a metrics pamphindi.
Tidaphatikizanso LDAP yamakampani kwa iyo, mothandizidwa ndi omwe aliyense wogwiritsa ntchito kampaniyo atha kudzipangira zidziwitso kutengera zoyambitsa zomwe zilipo (kapena zatsopano). Popeza Moira ili ndi Graphite, imathandizira mawonekedwe ake onse. Chifukwa chake mumayamba mwatenga mzere ndikuwukopera ku Grafana. Onani momwe deta imasonyezedwera pazithunzi. Kenako mutenge mzere womwewo ndikuwukopera ku Moira. Mumapachika ndi malire ndikupeza chenjezo pazotulutsa. Kuti muchite zonsezi, simukusowa chidziwitso china chilichonse. Moira akhoza kuchenjeza kudzera pa SMS, imelo, Jira, Slack ... Imathandizanso kuchitidwa kwa zolemba zachizolowezi. Chiwombankhanga chikachitika kwa iye, ndipo amalembetsa ku script kapena binary, amayendetsa ndikutumiza JSON ku stdin pa binary iyi. Chifukwa chake, pulogalamu yanu iyenera kuwongolera. Zomwe mungachite ndi JSON iyi zili ndi inu. Ngati mukufuna, tumizani ku Telegalamu, ngati mukufuna, tsegulani ntchito ku Jira, chitani chilichonse.
Timagwiritsanso ntchito chitukuko chathu pochenjeza - Imagotag. Tinasintha gulu, lomwe nthawi zambiri limagwiritsidwa ntchito pamitengo yamagetsi m'masitolo, kuti ligwirizane ndi zosowa zathu. Tinabweretsa zoyambitsa kuchokera ku Moira kupita nazo. Zimawonetsa momwe zilili komanso nthawi yomwe zidachitika. Ena mwa anyamata otukuka adasiya zidziwitso ku Slack ndi imelo mokomera gululi.
Chabwino, popeza ndife kampani yomwe ikupita patsogolo, tidayang'aniranso Kubernetes m'dongosolo lino. Tidaziphatikiza mudongosolo pogwiritsa ntchito Heapster, yomwe tidayika mugulu, imasonkhanitsa deta ndikuitumiza ku Graphite. Chifukwa chake, chithunzichi chikuwoneka motere:
Monitoring Components
Nawu mndandanda wamalumikizidwe azinthu zomwe tidagwiritsa ntchito pa ntchitoyi. Onsewa ndi open source.
Graphite:
- go-carbon:
- kunong'oneza:
- graphite-api:
Carbon-c-relay:
Brubeck:
Zosonkhanitsidwa:
Moira:
Grafana:
Heapster:
Amabala
Ndipo apa pali manambala a momwe dongosololi limagwirira ntchito kwa ife.
Aggregator (brubeck)
Chiwerengero cha ma metrics: ~ 300/mphindikati
Nthawi yotumiza ma metric ku Graphite: 30 sec
Kugwiritsa ntchito kwa seva: ~ 6% CPU (tikulankhula za ma seva athunthu); ~ 1GB RAM; ~3 Mbps LAN
Graphite (go-carbon)
Chiwerengero cha ma metric: ~ 1 / min
Nthawi yosinthira ma metrics: 30 sec
Dongosolo losungira ma metrics: 30sec 35d, 5min 90d, 10min 365d (imakupatsani kumvetsetsa zomwe zimachitika pautumiki pakanthawi yayitali)
Kugwiritsa ntchito kwa seva: ~ 10% CPU; ~ 20Gb RAM; ~30 Mbps LAN
Kusintha
Ife ku Avito timayamikira kwambiri kusinthasintha mu ntchito yathu yowunikira. Chifukwa chiyani adakhala chonchi? Choyamba, zigawo zake zimasinthana: zigawo zonsezo ndi matembenuzidwe awo. Kachiwiri, chithandizo. Popeza pulojekiti yonseyi ndi yotseguka, mukhoza kusintha code nokha, kusintha, ndikugwiritsanso ntchito zomwe sizikupezeka m'bokosi. Minda wamba imagwiritsidwa ntchito, makamaka Go ndi Python, kotero izi zimachitika mosavuta.
Pano pali chitsanzo cha vuto lenileni. Metric mu Graphite ndi fayilo. Ili ndi dzina. Dzina lafayilo = metric name. Ndipo pali njira yopitira kumeneko. Mayina afayilo mu Linux ali ndi zilembo 255 zokha. Ndipo tili (monga "makasitomala amkati") anyamata ochokera ku dipatimenti ya database. Amatiuza kuti: "Tikufuna kuyang'anira mafunso athu a SQL. Ndipo iwo si zilembo 255, koma 8 MB aliyense. Tikufuna kuwawonetsa ku Grafana, onani magawo a pempholi, ndipo ngakhale bwino, tikufuna kuwona pamwamba pazopempha zotere. Zidzakhala zabwino ngati zikuwonetsedwa mu nthawi yeniyeni. Zingakhale zabwino kwambiri kuwayika m'maso. "
Chitsanzo cha funso la SQL limatengedwa ngati chitsanzo kuchokera
Timakhazikitsa seva ya Redis ndikugwiritsa ntchito mapulagini athu a Collectd, omwe amapita ku Postgres ndikutenga deta yonse kuchokera kumeneko, kutumiza ma metrics ku Graphite. Koma timasintha dzina la metric ndi ma hashes. Nthawi yomweyo timatumiza hashi yomweyo ku Redis ngati kiyi, ndi funso lonse la SQL ngati mtengo. Zomwe tiyenera kuchita ndikuwonetsetsa kuti Grafana atha kupita ku Redis ndikutenga chidziwitsochi. Tikutsegula Graphite API chifukwa... Ili ndilo mawonekedwe akuluakulu a kuyanjana kwa zigawo zonse zowunikira ndi graphite, ndipo timalowetsa ntchito yatsopano kumeneko yotchedwa aliasByHash () - kuchokera ku Grafana timapeza dzina la metric, ndikuigwiritsa ntchito popempha Redis ngati kiyi, mu kuyankha timapeza mtengo wa kiyi, yomwe ndi "funso lathu la SQL" " Chifukwa chake, tidawonetsa ku Grafana chiwonetsero cha funso la SQL, lomwe mwalingaliro silinali zotheka kuwonetsa pamenepo, pamodzi ndi ziwerengero zake (mayitanidwe, mizere, total_time, ...).
Zotsatira
Kupezeka Ntchito yathu yowunikira imapezeka 24/7 kuchokera ku pulogalamu iliyonse ndi nambala iliyonse. Ngati muli ndi mwayi wosungirako, mukhoza kulemba deta ku utumiki. Chilankhulo sichofunikira, zosankha sizofunikira. Mukungofunika kudziwa momwe mungatsegulire socket, ikani metric pamenepo ndikutseka socket.
Kudalirika Zigawo zonse ndizololera zolakwika ndipo zimanyamula katundu wathu bwino.
Chotchinga chochepa kulowa. Kuti mugwiritse ntchito dongosololi, simuyenera kuphunzira zilankhulo zamapulogalamu ndi mafunso ku Grafana. Ingotsegulani pulogalamu yanu, lowetsani socket momwe mungatumizire ma metric ku Graphite, kutseka, tsegulani Grafana, pangani ma dashboards pamenepo ndikuyang'ana machitidwe a ma metrics anu, kulandira zidziwitso kudzera ku Moira.
Kudziimira. Mutha kuchita zonsezi nokha, popanda kuthandizidwa ndi mainjiniya a DevOps. Ndipo uwu ndi mwayi, chifukwa mutha kuyang'anira ntchito yanu pakali pano, simukuyenera kufunsa aliyense - kuti ayambe ntchito kapena kusintha.
Kodi tikufuna chiyani?
Chilichonse chomwe chalembedwa m'munsimu sichimangoganizira chabe, koma china chake chomwe masitepe oyamba adatengerapo.
- Anomaly detector. Tikufuna kupanga ntchito yomwe ipita ku Graphite storages ndikuyang'ana metric iliyonse pogwiritsa ntchito ma aligorivimu osiyanasiyana. Pali kale ma algorithms omwe tikufuna kuwona, pali deta, timadziwa momwe tingagwiritsire ntchito.
- Metadata. Tili ndi mautumiki ambiri, amasintha pakapita nthawi, monga anthu omwe amagwira nawo ntchito. Kusunga zolemba nthawi zonse pamanja sichosankha. Ichi ndichifukwa chake tsopano tikuyika metadata mu ma microservices athu. Imatchula yemwe adayipanga, zilankhulo zomwe zimalumikizana nazo, zofunikira za SLA, komwe zidziwitso ziyenera kutumizidwa ndi kwa ndani. Mukatumiza ntchito, data yonse yamabungwe imapangidwa palokha. Zotsatira zake, mumapeza maulalo awiri - imodzi yoyambitsa, ina yopita ku dashboards ku Grafana.
- Kuwunika m'nyumba iliyonse. Tikukhulupirira kuti opanga onse ayenera kugwiritsa ntchito dongosolo lotere. Pankhaniyi, nthawi zonse mumamvetsetsa komwe magalimoto anu ali, zomwe zimachitika kwa izo, kumene zimagwera, kumene zofooka zake zili. Ngati, mwachitsanzo, chinachake chikubwera ndikusokoneza ntchito yanu, ndiye kuti simungaphunzirepo pa kuyitana kuchokera kwa woyang'anira, koma kuchokera ku chenjezo, ndipo mukhoza kutsegula zipika zaposachedwa ndikuwona zomwe zidachitika pamenepo.
- Kuchita kwakukulu. Pulojekiti yathu ikukula mosalekeza, ndipo lero ikupanga pafupifupi 2 metric values ββpamphindi. Chaka chapitacho, chiwerengerochi chinali 000 000. Ndipo kukula kukupitirirabe, ndipo izi zikutanthauza kuti patapita nthawi Graphite (minong'ono) idzayamba kudzaza kwambiri disk subsystem. Monga ndanenera kale, njira yowunikirayi ndi yapadziko lonse lapansi chifukwa cha kusinthasintha kwa zigawo. Winawake amasunga ndikukulitsa nthawi zonse zida zawo za Graphite, koma tinaganiza zopita njira ina: gwiritsani ntchito ngati malo osungira ma metric athu. Kusinthaku kwatsala pang'ono kutha, ndipo posachedwa ndikuwuzani mwatsatanetsatane momwe izi zidachitikira: zovuta zomwe zidalipo komanso momwe zidagonjetsedwera, momwe kusamuka kudayendera, ndikufotokozerani zigawo zomwe zasankhidwa kukhala zomangirira ndi masanjidwe awo.
Zikomo chifukwa chakumvetsera! Funsani mafunso anu pamutuwu, ndiyesera kuyankha pano kapena m'makalata otsatirawa. Mwina wina ali ndi chidziwitso chopanga njira yowunikira yofananira kapena kusinthana ndi Clickhouse mumkhalidwe womwewo - gawanani mu ndemanga.
Source: www.habr.com