E foliga mai, i aso nei e leai se tasi e fesili pe aisea e manaʻomia ai le aoina o metotia o auaunaga. O le isi laasaga talafeagai o le faʻatulagaina lea o se mataala mo metotia aoina, lea o le a logoina e uiga i soʻo se suiga i faʻamatalaga i auala talafeagai mo oe (meli, Slack, Telegram). I totonu o le initaneti faletalimalo tautua tautua metric uma oa matou auaunaga e sasaa atu i totonu o le InfluxDB ma faʻaalia i le Grafana, ma o loʻo faʻapipiʻiina foʻi iina le mataala faʻavae. Mo galuega e pei o le "e tatau ona e fuafuaina se mea ma faʻatusatusa i ai," matou te faʻaaogaina Kapacitor.
Kapacitor ose vaega ole fa'aputuga TICK e mafai ona fa'agasolo metric mai le InfluxDB. E mafai ona faʻafesoʻotaʻi le tele o fua faʻatasi (faʻatasi), fuafua se mea aoga mai faʻamaumauga na maua, tusi le iʻuga i tua i InfluxDB, lafo se mataala i Slack / Telegram / meli.
O le faaputuga atoa e malu ma auiliili , ae o le a i ai pea lava mea aoga e le o faailoaina manino mai i tusi lesona. I lenei tusiga, na ou filifili e aoina se numera o fautuaga aoga, e le o manino (o loʻo faʻamatalaina le syntax faavae o TICKscipt. ) ma faʻaali le auala e mafai ona faʻaaogaina e ala i se faʻataʻitaʻiga o le foia o se tasi o a tatou faafitauli.
Tatou o!
float & int, fa'atatau mea sese
O se faʻafitauli faʻapitoa, e foia e ala i castes:
var alert_float = 5.0
var alert_int = 10
data|eval(lambda: float("value") > alert_float OR float("value") < float("alert_int"))
Fa'aaogaina fa'aletonu()
Afai e le fa'atumuina se pine/fa'afanua, o le a tupu fa'aletonu fa'atatau:
|default()
.tag('status', 'empty')
.field('value', 0)
fa'atumu fa'atasi (i totonu vs fafo)
Ona o le faaletonu, o le auai o le a lafoa'i vaega e leai ni fa'amaumauga (i totonu).
Faatasi ai ma le faʻatumu ('null'), o le a faia se faʻatasi i fafo, a maeʻa ona e manaʻomia le faia o se faaletonu () ma faʻatumu tulaga gaogao:
var data = res1
|join(res2)
.as('res1', 'res2)
.fill('null')
|default()
.field('res1.value', 0.0)
.field('res2.value', 100.0)
O loʻo i ai pea se faʻaaliga iinei. I le faʻataʻitaʻiga o loʻo i luga, afai o se tasi o faʻasologa (res1 poʻo res2) e gaogao, o le faʻasologa o faʻasologa (faʻamaumauga) o le a gaogao foi. E tele tiketi i lenei autu ile Github (, , ) - o loʻo matou faʻatali mo faʻaleleia ma mafatia teisi.
Fa'aaogā tulaga ile fa'atatauga (pe a fai ile lambda)
|eval(lambda: if("value" > 0, true, false)
Lima minute mulimuli mai le paipa mo le vaitaimi
Mo se faʻataʻitaʻiga, e tatau ona e faʻatusatusa le tau o le lima minute mulimuli ma le vaiaso talu ai. E mafai ona e ave ni vaega se lua o faʻamaumauga i ni vaega eseese se lua pe aveese se vaega o faʻamaumauga mai se vaitaimi tele:
|where(lambda: duration((unixNano(now()) - unixNano("time"))/1000, 1u) < 5m)
O se isi mea mo le lima minute mulimuli o le faʻaaogaina lea o le BarrierNode, lea e tipi ese ai faʻamatalaga aʻo leʻi oʻo i le taimi faʻapitoa:
|barrier()
.period(5m)
Fa'ata'ita'iga o le fa'aogaina o Go templates ile fe'au
O faʻataʻitaʻiga e fetaui ma le faatulagaga mai le afifi O lo'o i lalo nisi o paso e masani ona maua.
afai-a leai
Matou te fa'atonu mea ma e le toe fa'aoso tagata i tusitusiga:
|alert()
...
.message(
'{{ if eq .Level "OK" }}It is ok now{{ else }}Chief, everything is broken{{end}}'
)
E lua numera pe a uma le numera tesimale i le fe'au
Fa'aleleia atili le faitau o le fe'au:
|alert()
...
.message(
'now value is {{ index .Fields "value" | printf "%0.2f" }}'
)
Fa'alauteleina fesuiaiga i le fe'au
Matou te faʻaalia nisi faʻamatalaga i le feʻau e tali ai le fesili "Aisea ua ee ai"?
var warnAlert = 10
|alert()
...
.message(
'Today value less then '+string(warnAlert)+'%'
)
Tulaga fa'ailoa mata'utia
O se mea tatau lea pe a sili atu ma le tasi le vaega i faʻamaumauga, a le o lea e naʻo le tasi le mataala o le a faia:
|alert()
...
.id('{{ index .Tags "myname" }}/{{ index .Tags "myfield" }}')
Tagata fa'aaganu'u
O le lisi tele o tagata e faʻaaogaina e aofia ai le exec, lea e mafai ai ona e faʻatinoina lau faʻamaumauga faʻatasi ma tapulaʻa pasia (stdin) - fatufatuga ma leai se isi mea!
O se tasi oa tatou aganuʻu o se tamaʻi tusi Python mo le tuʻuina atu o faʻamatalaga i le vaivai.
I le taimi muamua, na matou mananaʻo e lafo se ata faʻapolopolo puipuia-puipuia i se feʻau. A uma, tusi le OK i le filo i le mataala muamua mai le vaega lava e tasi, ae le o se savali ese. I se taimi mulimuli ane - faʻaopoopo i le feʻau le mea sese masani i le X minute mulimuli.
Ole isi autu ole feso'ota'iga ma isi au'aunaga ma so'o se gaioiga na amataina e se mataala (pe'ā lelei lau mata'ituina).
O se faʻataʻitaʻiga o se faʻamatalaga faʻamatalaga, lea o le slack_handler.py o la matou tusitusiga na tusia e le tagata lava ia:
topic: slack_graph
id: slack_graph.alert
match: level() != INFO AND changed() == TRUE
kind: exec
options:
prog: /sbin/slack_handler.py
args: ["-c", "CHANNELID", "--graph", "--search"]
E fa'afefea ona debug?
Filifiliga fa'atasi ai ma ogalaau fa'apipi'i
|log()
.level("error")
.prefix("something")
Matamata (cli): kapacitor -url :9092 ogalaau lvl=sese
Filifiliga ma le httpOut
Fa'aalia fa'amaumauga i le paipa o lo'o iai nei:
|httpOut('something')
Matamata (maua): :9092/kapacitor/v1/tasks/task_name/something
Faiga fa'atino
- O galuega ta'itasi e toe fa'afo'i mai ai se la'au fa'atino ma ni numera aoga i le faatulagaga .
- Ave se poloka .
- Faapipii i le tagata matamata, .
O fea se isi mea e te maua ai se salu?
timestamp i influxdb ile toe tusi
Mo se faʻataʻitaʻiga, matou te setiina se mataala mo le aofaʻi o talosaga i le itula (groupBy(1h)) ma manaʻo e faʻamaumau le mataala na tupu i influxdb (ia faʻaalia matagofie le mea moni o le faʻafitauli i luga o le kalafi i le grafana).
influxDBOut() o le a tusia le tau o le taimi mai le mataala i le timestamp; e tusa ai, o le mea i luga o le siata o le a tusia muamua / mulimuli ane nai lo le mataala na taunuu.
Pe a manaʻomia le saʻo: matou te faʻaogaina lenei faʻafitauli e ala i le valaʻau i se faʻataʻitaʻiga masani, lea o le a tusia faʻamatalaga i influxdb ma le taimi nei.
docker, fausiaina ma le faʻapipiʻiina
I le amataga, e mafai e le kapacitor ona utaina galuega, faʻataʻitaʻiga ma faʻatautaia mai le lisi o loʻo faʻamaonia i le config i le poloka [load].
Ina ia faia saʻo se galuega, e te manaʻomia mea nei:
- Igoa o le faila – fa'alautele ile fa'ailoga id/igoa
- Ituaiga – stream/batch
- dbrp - upu autu e faʻaalia ai le database + faiga faʻavae o loʻo faʻaogaina ai le tusitusiga (dbrp "supplier." "autogen")
Afai o nisi galuega faʻapipiʻi e le o iai se laina ma le dbrp, o le a mumusu le auaunaga atoa e amata ma o le a tusi faʻamaoni e uiga i ai i le ogalaau.
I le chronograf, i se isi itu, e le tatau ona i ai lenei laina; e le taliaina e ala i le atinaʻe ma faʻatupuina se mea sese.
Hack pe a fausia se atigipusa: Dockerfile alu ese ma -1 pe a iai laina ma //.+dbrp, lea e mafai ai ona e vave malamalama i le mafuaʻaga o le toilalo pe a faʻapipiʻi le fale.
auai tasi i le tele
Fa'ata'ita'iga galuega: e te mana'omia le ave o le 95th percentile o le taimi fa'agaioiga o le tautua mo le vaiaso, fa'atusatusa minute ta'itasi o le 10 mulimuli ma lenei tau.
E le mafai ona e faia se tasi-i-tele e auai, mulimuli/mean/median i luga o se vaega o togi e liliu ai le node i se vaitafe, o le mea sese "e le mafai ona faʻaopoopo le tamaititi le fetaui pito: batch -> stream" o le a toe faafoi mai.
O le taunuuga o se vaega, e pei o se fesuiaiga i se faʻamatalaga lambda, e le suia foi.
O loʻo i ai se filifiliga e teu ai numera manaʻomia mai le vaega muamua i se faila e ala i le udf ma utaina lenei faila e ala i le sideload.
O le a le mea na matou foia i lenei mea?
E tusa ma le 100 a matou faleoloa faletalimalo, e mafai e i latou taʻitoʻatasi ona maua ni fesoʻotaʻiga, seʻi taʻua o se auala. E tusa ma le 300 o nei alāleo, e mafai ona pa'u ese ia alalaupapa ta'itasi. Mai fua fa'amaumau uma, o le a matou mata'ituina le fua faatatau o mea sese (talosaga ma mea sese).
Aisea e le mafai ai ona tusi ata?
Fa'ailoga fa'aletonu ua fa'atulagaina i Grafana e tele ni fa'aletonu. O nisi e faitio, o nisi e mafai ona e moeiini ou mata, e fuafua i le tulaga.
E le iloa e Grafana pe faʻafefea ona faʻatatau i le va o fua + mataala, ae matou te manaʻomia se fua (talosaga-sese) / talosaga.
O mea sese e foliga leaga:
Ma faʻaitiitia le leaga pe a vaʻai i talosaga manuia:
Lelei, e mafai ona tatou muai fuafuaina le fua faatatau i le tautua aʻo leʻi oʻo i le grafana, ma i nisi tulaga o le a aoga. Ae le o tatou, aua... mo auala taʻitasi o lona lava fua faatatau e manatu "masani", ma mataala e galue e tusa ai ma tulaga taua (tatou te suʻeina i latou ma o tatou mata, suia pe a iai faʻasalalauga masani).
O faʻataʻitaʻiga ia o le "masani" mo auala eseese:
Matou te le amanaʻia le manatu muamua ma manatu o le ata "masani" e tutusa mo faʻatau uma. O lea ua lelei mea uma, ma e mafai ona tatou fa'aola ma fa'ailo i grafana?
Matou te mafaia, ae matou te matua le manaʻo lava, aua e tatau ona matou filifilia se tasi o filifiliga:
a) faia le tele o kalafi mo auala taʻitasi eseese (ma faʻatasi ma latou tiga)
e) tu'u se siata e tasi ma ala uma (ma leiloa i laina felanulanua'i ma fa'ailoga fa'apitoa)
Na faapefea ona e faia?
Toe, o loʻo i ai se faʻataʻitaʻiga lelei amata i totonu o faʻamaumauga (), e mafai ona va'ava'ai pe ave e fai ma fa'avae i fa'afitauli fa'apena.
Le mea na matou faia i le faaiuga:
- auai i ni fa'asologa se lua i ni nai itula, fa'avasega i alalaupapa;
- fa'atumu le fa'asologa i vaega pe a leai ni fa'amaumauga;
- fa'atusatusa le vaeluaga o le 10 minute mulimuli ma fa'amaumauga muamua;
- matou te alalaga pe a matou maua se mea;
- matou te tusia fua faʻatatau ma faʻaaliga na tupu i le influxdb;
- auina atu se feau aoga i le paie.
I loʻu manatu, na mafai ona matou ausia mea uma matou te manaʻo e maua i le faaiuga (ma e oʻo lava i sina mea sili atu i tagata faʻaleaganuʻu masani) i le matagofie e mafai ai.
E mafai ona e vaʻai ile github.com и o le tusitusiga e maua.
O se faʻataʻitaʻiga o le faʻailoga e maua ai:
dbrp "supplier"."autogen"
var name = 'requests.rate'
var grafana_dash = 'pczpmYZWU/mydashboard'
var grafana_panel = '26'
var period = 8h
var todayPeriod = 10m
var every = 1m
var warnAlert = 15
var warnReset = 5
var reqQuery = 'SELECT sum("count") AS value FROM "supplier"."autogen"."requests"'
var errQuery = 'SELECT sum("count") AS value FROM "supplier"."autogen"."errors"'
var prevErr = batch
|query(errQuery)
.period(period)
.every(every)
.groupBy(1m, 'channel', 'supplier')
var prevReq = batch
|query(reqQuery)
.period(period)
.every(every)
.groupBy(1m, 'channel', 'supplier')
var rates = prevReq
|join(prevErr)
.as('req', 'err')
.tolerance(1m)
.fill('null')
// заполняем значения нулями, если их не было
|default()
.field('err.value', 0.0)
.field('req.value', 0.0)
// if в lambda: считаем рейт, только если ошибки были
|eval(lambda: if("err.value" > 0, 100.0 * (float("req.value") - float("err.value")) / float("req.value"), 100.0))
.as('rate')
// записываем посчитанные значения в инфлюкс
rates
|influxDBOut()
.quiet()
.create()
.database('kapacitor')
.retentionPolicy('autogen')
.measurement('rates')
// выбираем данные за последние 10 минут, считаем медиану
var todayRate = rates
|where(lambda: duration((unixNano(now()) - unixNano("time")) / 1000, 1u) < todayPeriod)
|median('rate')
.as('median')
var prevRate = rates
|median('rate')
.as('median')
var joined = todayRate
|join(prevRate)
.as('today', 'prev')
|httpOut('join')
var trigger = joined
|alert()
.warn(lambda: ("prev.median" - "today.median") > warnAlert)
.warnReset(lambda: ("prev.median" - "today.median") < warnReset)
.flapping(0.25, 0.5)
.stateChangesOnly()
// собираем в message ссылку на график дашборда графаны
.message(
'{{ .Level }}: {{ index .Tags "channel" }} err/req ratio ({{ index .Tags "supplier" }})
{{ if eq .Level "OK" }}It is ok now{{ else }}
'+string(todayPeriod)+' median is {{ index .Fields "today.median" | printf "%0.2f" }}%, by previous '+string(period)+' is {{ index .Fields "prev.median" | printf "%0.2f" }}%{{ end }}
http://grafana.ostrovok.in/d/'+string(grafana_dash)+
'?var-supplier={{ index .Tags "supplier" }}&var-channel={{ index .Tags "channel" }}&panelId='+string(grafana_panel)+'&fullscreen&tz=UTC%2B03%3A00'
)
.id('{{ index .Tags "name" }}/{{ index .Tags "channel" }}')
.levelTag('level')
.messageField('message')
.durationField('duration')
.topic('slack_graph')
// "today.median" дублируем как "value", также пишем в инфлюкс остальные филды алерта (keep)
trigger
|eval(lambda: "today.median")
.as('value')
.keep()
|influxDBOut()
.quiet()
.create()
.database('kapacitor')
.retentionPolicy('autogen')
.measurement('alerts')
.tag('alertName', name)
O le a le faaiuga?
Kapacitor e lelei tele i le faʻatinoina o le mataʻituina-talatala faʻatasi ai ma le tele o faʻalapotopotoga, faia faʻatusatusaga faʻaopoopo e faʻavae i luga o fua faʻamaumau, faʻatino gaioiga masani ma faʻasologa o tusitusiga (udf).
Ole pa puipui ile ulufale e le maualuga tele - taumafai pe a le faʻamalieina atoatoa e le grafana poʻo isi meafaigaluega ou manaʻoga.
puna: www.habr.com