Ngokunokwenzeka, namhlanje akukho mntu ubuza ukuba kutheni kuyimfuneko ukuqokelela iimetriki zenkonzo. Isinyathelo esilandelayo esinengqiqo kukuseta isilumkiso kwiimetriki eziqokelelweyo, eziza kwazisa malunga naluphi na ukutenxa kwidatha kwiitshaneli ezikulungeleyo (i-imeyile, i-Slack, iTelegram). Kwinkonzo yokubhukisha ihotele kwi-intanethi zonke iimetrics zeenkonzo zethu zigalelwe kwi-InfluxDB kwaye ziboniswe eGrafana, kwaye isilumkiso esisisiseko sikwaqwalaselwe apho. Kwimisebenzi efana "kufuneka ubale into kwaye uthelekise nayo," sisebenzisa iKapacitor.
I-Kapacitor yinxalenye ye-TICK stack enokuqhuba iimetrics ezivela kwi-InfluxDB. Inokudibanisa imilinganiselo emininzi kunye (dibanisa), ibale into eluncedo kwidatha efunyenweyo, bhala umphumo ubuyele kwi-InfluxDB, uthumele isilumkiso kwi-Slack/Telegram/imeyile.
Isitaki sonke sipholile kwaye sineenkcukacha , kodwa kuya kuhlala kukho izinto eziluncedo ezingaboniswanga ngokucacileyo kwiincwadi. Kweli nqaku, ndigqibe ekubeni ndiqokelele amacebiso anjalo aluncedo, angacacanga (i-syntax esisiseko ye-TICKscipt ichaziwe. ) uze ubonise indlela ezinokusetyenziswa ngayo usebenzisa umzekelo wokusombulula enye yeengxaki zethu.
Masihambe!
i-float & int, iimpazamo zokubala
Ingxaki esemgangathweni ngokupheleleyo, isonjululwe ngee-castes:
var alert_float = 5.0
var alert_int = 10
data|eval(lambda: float("value") > alert_float OR float("value") < float("alert_int"))
Ukusebenzisa okumiselweyo()
Ukuba ithegi/ibala aligcwaliswanga, iimpazamo zokubala ziyakwenzeka:
|default()
.tag('status', 'empty')
.field('value', 0)
Gcwalisa ukujoyina (ngaphakathi vs ngaphandle)
Ngokungagqibekanga, ukujoyina kuya kulahla amanqaku apho kungekho datha (yangaphakathi).
Ngokugcwalisa('null'), udibaniso lwangaphandle luyakwenziwa, emva koko kufuneka wenze okungagqibekanga () kwaye ugcwalise amaxabiso angenanto:
var data = res1
|join(res2)
.as('res1', 'res2)
.fill('null')
|default()
.field('res1.value', 0.0)
.field('res2.value', 100.0)
Kusekho i-nuance apha. Kulo mzekelo ungentla, ukuba enye yoluhlu (res1 okanye i-res2) ayinanto, uchungechunge oluphumayo (idatha) nalo luya kuba lungenanto. Kukho amatikiti amaninzi kwesi sihloko kwi-Github (, , ) - silindele ukulungiswa kunye nokubandezeleka kancinci.
Ukusebenzisa iimeko kubalo (ukuba kwi lambda)
|eval(lambda: if("value" > 0, true, false)
Imizuzu emihlanu yokugqibela ukusuka kumbhobho welo xesha
Umzekelo, kufuneka uthelekise amaxabiso emizuzu emihlanu yokugqibela kunye neveki ephelileyo. Ungathatha iibhetshi ezimbini zedatha kwiibhetshi ezimbini ezahlukeneyo okanye ukhuphe inxalenye yedatha kwixesha elikhulu:
|where(lambda: duration((unixNano(now()) - unixNano("time"))/1000, 1u) < 5m)
Enye indlela kwimizuzu emihlanu yokugqibela iya kuba kukusebenzisa iBarrierNode, esika idatha ngaphambi kwexesha elichaziweyo:
|barrier()
.period(5m)
Imizekelo yokusebenzisa iGo templates kumyalezo
Izakhelo zihambelana nefomathi evela kwiphakheji Apha ngezantsi kukho iiphazili odibana nazo rhoqo.
ukuba-enye
Sibeka izinto ngolungelelwano kwaye asibangeli abantu ngetekisi kwakhona:
|alert()
...
.message(
'{{ if eq .Level "OK" }}It is ok now{{ else }}Chief, everything is broken{{end}}'
)
Amanani amabini emva kwenqaku lokugqibela kumyalezo
Ukuphucula ukufundeka komyalezo:
|alert()
...
.message(
'now value is {{ index .Fields "value" | printf "%0.2f" }}'
)
Ukwandisa iinguqu kumyalezo
Sibonisa ulwazi oluthe kratya kumyalezo ukuze siphendule umbuzo othi "Kutheni ikhwaza"?
var warnAlert = 10
|alert()
...
.message(
'Today value less then '+string(warnAlert)+'%'
)
Isazisi esisodwa sesilumkiso
Le yinto eyimfuneko xa kukho ngaphezu kweqela elinye kwidatha, kungenjalo isilumkiso esinye kuphela siya kwenziwa:
|alert()
...
.id('{{ index .Tags "myname" }}/{{ index .Tags "myfield" }}')
Umphathi wesiko
Uluhlu olukhulu lwabaphathi lubandakanya i-exec, ekuvumela ukuba uphumeze iskripthi sakho ngeeparamitha ezigqithisiweyo (stdin) - uyilo kwaye akukho nto ingaphezulu!
Elinye lamasiko ethu sisikripthi esincinci sePython sokuthumela izaziso ukuba ziyeke.
Ekuqaleni, besifuna ukuthumela umfanekiso wegrafana okhuselwe ngogunyaziso kumyalezo. Emva koko, bhala u-Kulungile kumsonto kwisilumkiso sangaphambili kwiqela elinye, kwaye hayi njengomyalezo owahlukileyo. Kancinane emva kwexesha-yongeza kumyalezo eyona mpazamo ixhaphakileyo kwimizuzu engu-X yokugqibela.
Isihloko esahlukileyo lunxibelelwano nezinye iinkonzo kunye nazo naziphi na izenzo eziqalwe sisilumkiso (kuphela ukuba ukujonga kwakho kusebenza ngokwaneleyo).
Umzekelo wenkcazo yomphathi, apho i-slack_handler.py sisikripthi sethu esizibhalelayo:
topic: slack_graph
id: slack_graph.alert
match: level() != INFO AND changed() == TRUE
kind: exec
options:
prog: /sbin/slack_handler.py
args: ["-c", "CHANNELID", "--graph", "--search"]
Ukulungisa njani ingxaki?
Inketho enemveliso yelog
|log()
.level("error")
.prefix("something")
Bukela (cli): kapacitor -url :9092 iilog lvl=impazamo
Ukhetho nge httpOut
Ibonisa idatha kumbhobho wangoku:
|httpOut('something')
Bukela (fumana): :9092/kapacitor/v1/tasks/task_name/something
Umzobo wokwenziwa
- Umsebenzi ngamnye ubuyisela umthi wophumezo onamanani aluncedo kwifomati .
- Thatha ibhloko .
- Yincamathisele kwisibonisi, .
Ungayifumana phi enye iraki?
isitampu sexesha kwi-influxdb ekubhaleni kwakhona
Umzekelo, simisela isilumkiso semali yezicelo ngeyure (groupBy(1h)) kwaye sifuna ukurekhoda isilumkiso esenzeke kwi-influxdb (ukubonisa kakuhle inyani yengxaki kwigrafu kwigrafana).
influxDBOut() iya kubhala ixabiso lexesha ukusuka kwisilumkiso ukuya kwisitampu sexesha ngokufanelekileyo, indawo ekwitshati iya kubhalwa ngaphambili/emva kokuba isivuseleli sifikile.
Xa ukuchaneka kufunwa: sisebenza malunga nale ngxaki ngokufowunela isiphathi esiqhelekileyo, esiya kubhala idatha kwi-influxdb kunye nesitampu sexesha sangoku.
docker, ukwakha kunye nokusasazwa
Ekuqalisweni, i-kapacitor inokulayisha imisebenzi, iitemplates kunye nabaphathi ukusuka kulawulo oluchazwe kwi-config kwi-block [yomthwalo].
Ukwenza ngokuchanekileyo umsebenzi, kufuneka ezi zinto zilandelayo:
- Igama lefayile-yandiswe kwi-id/name yeskripthi
- Uhlobo – umsinga/ibhetshi
- dbrp - igama elingundoqo ukubonisa ukuba yeyiphi isiseko sedatha + umgaqo-nkqubo osebenza kuwo (dbrp "umthengisi." "autogen")
Ukuba umsebenzi othile webhetshi awunalo umgca kunye ne-dbrp, yonke inkonzo iya kunqaba ukuqala kwaye iya kubhala ngokunyanisekileyo ngayo kwilogi.
Kwi-chronograf, ngokuchaseneyo, lo mgca awufanele ubekho;
I-Hack xa isakha isikhongozeli: I-Dockerfile iphuma nge -1 ukuba kukho imigca ene // + dbrp, eya kukuvumela ukuba uqonde ngoko nangoko isizathu sokungaphumeleli xa uhlanganisa isakhiwo.
dibanisa enye kwabaninzi
Umzekelo womsebenzi: kufuneka uthathe ipesenti ze-95 zexesha lokusebenza kwenkonzo ngeveki, thelekisa umzuzu ngamnye we-10 yokugqibela kunye neli xabiso.
Awunako ukwenza udibaniso olunye ukuya kwabaninzi, okokugqibela/okuthetha/umdibaniso ngaphezulu kweqela lamanqaku ujika i-node ibe ngumsinga, impazamo “ayinakongeza imiphetho engahambelaniyo yomntwana: ibhetshi -> umsinga” iya kubuyiselwa.
Isiphumo sebhetshi, njengento eguquguqukayo kwintetho ye-lambda, nayo ayifakwa endaweni yayo.
Kukho ukhetho lokugcina amanani ayimfuneko ukusuka kwibhetshi yokuqala ukuya kwifayile nge-udf kwaye ulayishe le fayile ngokulayisha ecaleni.
Sisombulule ntoni ngale nto?
Sinabanikezeli behotele abamalunga ne-100, ngamnye kubo unokuba noqhagamshelo oluninzi, masiyibize itshaneli. Kukho malunga nama-300 ala majelo, itshaneli nganye inokuwa. Kuzo zonke iimethrikhi ezirekhodiweyo, siya kubeka iliso kwireyithi yempazamo (izicelo kunye neempazamo).
Kutheni kungeyo grafana?
Izaziso zemposiso ziqwalaselwe eGrafana zineengxaki ezininzi. Ezinye zibalulekile, ezinye unokuvala amehlo akho kuzo, kuxhomekeke kwimeko.
I-Grafana ayikwazi ukubala phakathi kwemilinganiselo + ukwazisa, kodwa sifuna ireyithi (izicelo-iimpazamo)/izicelo.
Iimpazamo zikhangeleka zimbi:
Kwaye ububi obuncinci xa bujongwa ngezicelo eziyimpumelelo:
Kulungile, sinokubala kwangaphambili izinga kwinkonzo phambi kwegrafana, kwaye kwezinye iimeko oku kuya kusebenza. Kodwa hayi kwethu, kuba... kwitshaneli nganye umlinganiselo wayo uthathwa "njengesiqhelo", kwaye izilumkiso zisebenza ngokwexabiso elingatshintshiyo (sijonge ngamehlo ethu, sitshintshe ukuba kukho izilumkiso rhoqo).
Le yimizekelo "yesiqhelo" yamajelo ahlukeneyo:
Asiyihoyi ingongoma yangaphambili kwaye sicinge ukuba umfanekiso "oqhelekileyo" ufana nabo bonke ababoneleli. Ngoku yonke into ilungile, kwaye singadlula ngezilumkiso kwigrafana?
Sinako, kodwa asifuni ngokwenene, kuba kufuneka sikhethe enye yeenketho:
a) yenza iigrafu ezininzi kwitshaneli nganye ngokwahlukeneyo (kwaye kabuhlungu uhambe nazo)
b) shiya itshati enye enazo zonke iitshaneli (kwaye ulahleke kwimigca enemibala kunye nezilumkiso ezilungiselelwe wena)
Wenze njani?
Kwakhona, kukho umzekelo omhle wokuqala kuxwebhu (), inokujongwa okanye ithathwe njengesiseko kwiingxaki ezifanayo.
Yintoni esiyenzileyo ekugqibeleni:
- Joyina iiseries ezimbini kwiiyure ezimbalwa, ukudibanisa ngamajelo;
- gcwalisa uthotho ngeqela ukuba bekungekho datha;
- thelekisa i-median yemizuzu eyi-10 yokugqibela kunye nedatha yangaphambili;
- siyakhwaza ukuba sifumene into;
- sibhala iireyithi ezibaliweyo kunye nezilumkiso ezenzeke kwi-influxdb;
- thumela umyalezo oluncedo ukuba ucothe.
Ngokombono wam, sikwazile ukufezekisa yonke into ebesifuna ukuyifumana ekugqibeleni (kwaye nangakumbi nabaphathi besiko) ngobuhle kangangoko kunokwenzeka.
Unokujonga kwi-github.com и iskripthi esisiphumo.
Umzekelo wekhowudi yesiphumo:
dbrp "supplier"."autogen"
var name = 'requests.rate'
var grafana_dash = 'pczpmYZWU/mydashboard'
var grafana_panel = '26'
var period = 8h
var todayPeriod = 10m
var every = 1m
var warnAlert = 15
var warnReset = 5
var reqQuery = 'SELECT sum("count") AS value FROM "supplier"."autogen"."requests"'
var errQuery = 'SELECT sum("count") AS value FROM "supplier"."autogen"."errors"'
var prevErr = batch
|query(errQuery)
.period(period)
.every(every)
.groupBy(1m, 'channel', 'supplier')
var prevReq = batch
|query(reqQuery)
.period(period)
.every(every)
.groupBy(1m, 'channel', 'supplier')
var rates = prevReq
|join(prevErr)
.as('req', 'err')
.tolerance(1m)
.fill('null')
// заполняем значения нулями, если их не было
|default()
.field('err.value', 0.0)
.field('req.value', 0.0)
// if в lambda: считаем рейт, только если ошибки были
|eval(lambda: if("err.value" > 0, 100.0 * (float("req.value") - float("err.value")) / float("req.value"), 100.0))
.as('rate')
// записываем посчитанные значения в инфлюкс
rates
|influxDBOut()
.quiet()
.create()
.database('kapacitor')
.retentionPolicy('autogen')
.measurement('rates')
// выбираем данные за последние 10 минут, считаем медиану
var todayRate = rates
|where(lambda: duration((unixNano(now()) - unixNano("time")) / 1000, 1u) < todayPeriod)
|median('rate')
.as('median')
var prevRate = rates
|median('rate')
.as('median')
var joined = todayRate
|join(prevRate)
.as('today', 'prev')
|httpOut('join')
var trigger = joined
|alert()
.warn(lambda: ("prev.median" - "today.median") > warnAlert)
.warnReset(lambda: ("prev.median" - "today.median") < warnReset)
.flapping(0.25, 0.5)
.stateChangesOnly()
// собираем в message ссылку на график дашборда графаны
.message(
'{{ .Level }}: {{ index .Tags "channel" }} err/req ratio ({{ index .Tags "supplier" }})
{{ if eq .Level "OK" }}It is ok now{{ else }}
'+string(todayPeriod)+' median is {{ index .Fields "today.median" | printf "%0.2f" }}%, by previous '+string(period)+' is {{ index .Fields "prev.median" | printf "%0.2f" }}%{{ end }}
http://grafana.ostrovok.in/d/'+string(grafana_dash)+
'?var-supplier={{ index .Tags "supplier" }}&var-channel={{ index .Tags "channel" }}&panelId='+string(grafana_panel)+'&fullscreen&tz=UTC%2B03%3A00'
)
.id('{{ index .Tags "name" }}/{{ index .Tags "channel" }}')
.levelTag('level')
.messageField('message')
.durationField('duration')
.topic('slack_graph')
// "today.median" дублируем как "value", также пишем в инфлюкс остальные филды алерта (keep)
trigger
|eval(lambda: "today.median")
.as('value')
.keep()
|influxDBOut()
.quiet()
.create()
.database('kapacitor')
.retentionPolicy('autogen')
.measurement('alerts')
.tag('alertName', name)
Siyintoni isigqibo?
I-Kapacitor inkulu ekwenzeni izilumkiso zokubeka iliso kunye neqela lamaqela, ukwenza izibalo ezongezelelweyo ezisekelwe kwi-metrics esele irekhodiwe, ukwenza izenzo zesiko kunye nokuqhuba izikripthi (udf).
Umqobo wokungena awuphezulu kakhulu - zama ukuba i-grafana okanye ezinye izixhobo azinelisekanga ngokupheleleyo iminqweno yakho.
umthombo: www.habr.com
