Mateki ekugadzirisa metrics muKapacitor

Zvingangodaro, nhasi hakuna munhu anobvunza kuti nei zvakakosha kuunganidza sevhisi metrics. Nhanho inotevera ine musoro ndeyekumisikidza yambiro kune yakaunganidzwa metrics, iyo inozivisa nezve chero kutsauka mune data mumatashi akakunakira iwe (tsamba, Slack, Teregiramu). Mubasa rekubhucha hotera yepamhepo Ostrovok.ru ese metrics emasevhisi edu anodururwa muInfluxDB uye anoratidzwa muGrafana, uye yekutanga yambiro inogadziriswa ipapo. Kune mabasa akaita se "iwe unofanirwa kuverenga chimwe chinhu uye kuenzanisa nacho," isu tinoshandisa Kapacitor.

Kapacitor chikamu cheTICK stack inogona kugadzirisa metrics kubva kuInfluxDB. Inogona kubatanidza zviyero zvakati wandei pamwechete (joinha), kuverenga chimwe chinhu chinobatsira kubva kune yakagamuchirwa data, nyora mhedzisiro kudzokera kuInfluxDB, tumira yambiro kuSlack/Teregiramu/mail.

Iyo yese stack inotonhorera uye yakadzama zvinyorwa, asi pachagara paine zvinhu zvinobatsira zvisina kuratidzwa mumabhuku ekushandisa. Muchinyorwa chino, ndafunga kuunganidza akati wandei anobatsira, asiri-pachena matipi (iyo yekutanga syntax yeTICKscipt inotsanangurwa. pano) uye ratidza kuti angashandiswa sei uchishandisa muenzaniso wokugadzirisa chimwe chezvinetso zvedu.

Ngatitangei!

float & int, zvikanganiso zvekuverenga

Dambudziko rakazara, rinogadziriswa kuburikidza ne castes:

var alert_float = 5.0
var alert_int = 10
data|eval(lambda: float("value") > alert_float OR float("value") < float("alert_int"))

Kushandisa default ()

Kana tag/munda usina kuzadzwa mukati, zvikanganiso zvekuverenga zvichaitika:

|default()
        .tag('status', 'empty')
        .field('value', 0)

zadza kujoina (mukati vs kunze)

Nekumisikidza, kujoinha kunorasa mapoinzi pasina data (mukati).
Nekuzadza('null'), kubatana kwekunze kuchaitwa, mushure mezvo iwe unofanirwa kuita default () uye zadza zvisina chinhu kukosha:

var data = res1
    |join(res2)
        .as('res1', 'res2)
        .fill('null')
    |default()
        .field('res1.value', 0.0)
        .field('res2.value', 100.0)

Pachine nuance pano. Mumuenzaniso uri pamusoro, kana imwe yenhevedzano (res1 kana res2) isina chinhu, mhedzisiro inoteedzana (data) ichave isinawo. Kune akati wandei matikiti pane iyi nyaya paGithub (1633, 1871, 6967) - isu takamirira kugadzirisa uye kutambura zvishoma.

Kushandisa mamiriro mukuverenga (kana mune lambda)

|eval(lambda: if("value" > 0, true, false)

Chengeta maminitsi mashanu kubva paipi yenguva

Semuenzaniso, iwe unofanirwa kuenzanisa kukosha kwemaminetsi mashanu ekupedzisira nevhiki yapfuura. Iwe unogona kutora mabheti maviri e data mumabheji maviri akaparadzana kana kubvisa chikamu che data kubva panguva yakakura:

 |where(lambda: duration((unixNano(now()) - unixNano("time"))/1000, 1u) < 5m)

Imwe nzira yemaminetsi mashanu ekupedzisira ingave yekushandisa BarrierNode, iyo inocheka data nguva isati yatarwa:

|barrier()
        .period(5m)

Mienzaniso yekushandisa Go templates mumeseji

Matemplate anoenderana nefomati kubva pasuru text.templatePazasi pane mamwe mapuzzle anowanzo sangana.

kana-zvimwe

Isu tinoisa zvinhu muhurongwa uye hatikonzerese vanhu nemavara zvakare:

|alert()
    ...
    .message(
        '{{ if eq .Level "OK" }}It is ok now{{ else }}Chief, everything is broken{{end}}'
    )

Nhamba mbiri mushure meiyo decimal mumeseji

Kuvandudza kuverenga kwemeseji:

|alert()
    ...
    .message(
        'now value is {{ index .Fields "value" | printf "%0.2f" }}'
    )

Kuwedzera mabhii emashoko

Isu tinoratidza rumwe ruzivo mumeseji kupindura mubvunzo wekuti "Sei iri kupopota"?

var warnAlert = 10
  |alert()
    ...
    .message(
       'Today value less then '+string(warnAlert)+'%'
    )

Unique yambiro identifier

Ichi chinhu chinodiwa kana paine boka rinopfuura rimwe mu data, zvikasadaro imwe chete yambiro ichagadzirwa:

|alert()
      ...
      .id('{{ index .Tags "myname" }}/{{ index .Tags "myfield" }}')

Custom handler's

Rondedzero yakakura yevanobata inosanganisira exec, iyo inokutendera iwe kuti uite script yako neyakapfuura paramita (stdin) - hunyanzvi uye hapana chimwe!

Imwe yetsika dzedu idiki Python script yekutumira zviziviso kunonoka.
Pakutanga, taida kutumira pikicha-yakachengetedzwa-yakachengetedzwa grafana mune meseji. Pashure, nyora OK mushinda kune yakare yambiro kubva muboka rimwechete, uye kwete semeseji yakaparadzana. Mushure mechinguva - wedzera kune meseji chikanganiso chakajairika mumaminetsi ekupedzisira eX.

Musoro wakasiyana kutaurirana nemamwe masevhisi uye chero zviito zvinotangwa neyambiro (chete kana kutarisa kwako kuchishanda zvakakwana).
Muenzaniso wetsananguro yemubati, uko slack_handler.py ndiyo yedu-yakanyorwa script:

topic: slack_graph
id: slack_graph.alert
match: level() != INFO AND changed() == TRUE
kind: exec
options:
  prog: /sbin/slack_handler.py
  args: ["-c", "CHANNELID", "--graph", "--search"]

Nzira yekugadzirisa sei?

Sarudzo ine log inobuda

|log()
      .level("error")
      .prefix("something")

Tarisa (cli): kapacitor -url host-kana-ip:9092 logs lvl= kukanganisa

Sarudzo ne httpOut

Inoratidza data mune yazvino pombi:

|httpOut('something')

Tarisa (tora): host-kana-ip:9092/kapacitor/v1/tasks/task_name/chimwe chinhu

Execution scheme

• Basa rega rega rinodzorera muti wekuuraya une nhamba dzinobatsira mufomati graphviz.
• Tora block dot.
• Namatidza mukuona, enjoy.

Ndekupi kumwe kwaungawana rake?

timestamp mu influxdb pane kunyora shure

Semuyenzaniso, isu tinoseta yambiro yehuwandu hwezvikumbiro paawa (groupBy(1h)) uye tinoda kurekodha yambiro yakaitika influxdb (kuratidza zvinoyevedza chokwadi chedambudziko pagirafu mugrafana).

influxDBOut() ichanyora kukosha kwenguva kubva payambiro kuenda kuchitambi chenguva; saizvozvo, poindi iri pachati ichanyorwa kare/kuzotevera yasvika.

Kana huroyi huchidiwa: isu tinoshanda kutenderedza dambudziko iri nekufonera mugadziri wetsika, uyo anonyora data kuinfluxdb neiyo timestamp.

docker, kuvaka uye kutumira

Pakutanga, kapacitor inogona kurodha mabasa, matemplate uye zvibatiso kubva pane dhairekitori inotsanangurwa mu config mu [load] block.

Kuti uite basa nemazvo, unoda zvinhu zvinotevera:

1. Zita refaira - rakawedzerwa kuita script id/zita
2. Rudzi – rukova/batch
3. dbrp - kiyi kiyi kuratidza kuti ndeipi dhatabhesi + mutemo iyo script inomhanya mairi (dbrp "mutengesi." "autogen")

Kana rimwe batch basa risina mutsara une dbrp, iyo yese sevhisi inoramba kutanga uye ichanyora nekutendeka nezvayo murogi.

Mu chronograf, pane zvinopesana, iyi mutsara haifanirwe kuvepo; haigamuchirwe kuburikidza neiyo interface uye inogadzira kukanganisa.

Hack paunenge uchivaka mudziyo: Dockerfile inobuda ne -1 kana paine mitsara ine // + dbrp, izvo zvinokutendera iwe kuti unzwisise nekukurumidza chikonzero chekutadza pakuunganidza kuvaka.

batanidza imwe kune vakawanda

Basa remuenzaniso: unofanirwa kutora 95th percentile yenguva yekushanda kwesevhisi kwevhiki, enzanisa imwe neimwe miniti yegumi yekupedzisira neichi kukosha.

Iwe haugone kuita imwe-kune-vazhinji kujoinha, yekupedzisira/kureva/median pamusoro peboka remapoinzi inoshandura node kuita rukova, iko kukanganisa "hakugone kuwedzera mipendero isina kuwirirana yemwana: batch -> rukova" inodzoserwa.

Mhedzisiro yebatch, sechichinjiko mukutaura kwe lambda, hainawo kutsiviwa.

Pane sarudzo yekuchengetedza nhamba dzinodiwa kubva pabatch yekutanga kuenda kufaira kuburikidza ne udf uye kurodha iyi faira kuburikidza nesideload.

Chii chatagadzirisa neizvi?

Tine vangangosvika zana vatengesi vehotera, mumwe nemumwe wavo anogona kuve akati wandei, ngatizvidaidze chiteshi. Kune angangoita mazana matatu ematanho aya, imwe neimwe yenzira inogona kudonha. Pane ese akarekodhwa metrics, isu tichatarisa chiyero chekukanganisa (zvikumbiro uye zvikanganiso).

Sei kwete grafana?

Zvikanganiso zviziviso zvakagadzirirwa muGrafana zvine zvakati kuti zvakaipa. Mamwe anotsoropodza, mamwe unogona kuvhara maziso ako, zvichienderana nemamiriro ezvinhu.

Grafana haazive kuverenga pakati pezviyero + kunyevera, asi isu tinoda chiyero (zvikumbiro-zvikanganiso)/zvikumbiro.

Zvikanganiso zvinotaridzika zvakashata:

Uye zvishoma zvakaipa kana zvichionekwa nezvikumbiro zvakabudirira:

Zvakanaka, isu tinogona kufanoverengera chiyero mubasa pamberi pegrafana, uye mune dzimwe nguva izvi zvichashanda. Asi kwete kwedu, nekuti... kune imwe neimwe chiteshi chiyero chayo chinoonekwa se "chakajairika", uye chenjedzo dzinoshanda zvinoenderana neyakaomesesa tsika (tinozvitsvaga nemeso edu, tishandure kana paine chenjedzo dzinowanzoitwa).

Iyi mienzaniso ye "zvakajairika" kune akasiyana chiteshi:

Isu tinofuratira iyo yapfuura poindi uye tinofungidzira kuti "yakajairika" mufananidzo wakafanana kune vese vatengesi. Iye zvino zvinhu zvese zvakanaka, uye isu tinogona kupfuura nekuzivisa mugrafana?
Tinogona, asi isu hatidi, nekuti isu tinofanirwa kusarudza imwe yesarudzo:
a) gadzira magirafu akawanda kune yega yega chiteshi zvakasiyana (uye zvinorwadza uende nawo)
b) siya chati imwe ine ese machaneli (uye kurasika mumitsetse ine mavara uye yakasarudzika ziviso)

Wakazviita sei?

Zvekare, kune muenzaniso wakanaka wekutanga mune zvinyorwa (Kuverengera mitengo muzvikamu zvakabatanidzwa), inogona kutariswa kana kutorwa sehwaro mumatambudziko akafanana.

Zvatakaita pakupedzisira:

• batanidza nhevedzano mbiri mumaawa mashoma, uchiunganidza nemachaneli;
• zadza zvakatevedzana neboka kana pakanga pasina data;
• enzanisa pakati pemaminiti ekupedzisira e10 nemashoko apfuura;
• tinoshevedzera kana tawana chimwe chinhu;
• tinonyora zviyero zvakaverengerwa uye zviziviso zvakaitika muinfluxdb;
• tumira meseji inobatsira kunonoka.

Sekuona kwangu, isu takakwanisa kuita zvese zvataida kuwana kumagumo (uye kunyangwe zvishoma nevabati vetsika) zvinoyevedza sezvinobvira.

Unogona kutarisa github.com kodhi muenzaniso и diki redunhu (graphviz) script inoguma.

Muenzaniso wekodhi inobuda:

dbrp "supplier"."autogen"
var name = 'requests.rate'
var grafana_dash = 'pczpmYZWU/mydashboard'
var grafana_panel = '26'
var period = 8h
var todayPeriod = 10m
var every = 1m
var warnAlert = 15
var warnReset = 5
var reqQuery = 'SELECT sum("count") AS value FROM "supplier"."autogen"."requests"'
var errQuery = 'SELECT sum("count") AS value FROM "supplier"."autogen"."errors"'

var prevErr = batch
    |query(errQuery)
        .period(period)
        .every(every)
        .groupBy(1m, 'channel', 'supplier')

var prevReq = batch
    |query(reqQuery)
        .period(period)
        .every(every)
        .groupBy(1m, 'channel', 'supplier')

var rates = prevReq
    |join(prevErr)
        .as('req', 'err')
        .tolerance(1m)
        .fill('null')
    // заполняем значения нулями, если их не было
    |default()
        .field('err.value', 0.0)
        .field('req.value', 0.0)
    // if в lambda: считаем рейт, только если ошибки были
    |eval(lambda: if("err.value" > 0, 100.0 * (float("req.value") - float("err.value")) / float("req.value"), 100.0))
        .as('rate')

// записываем посчитанные значения в инфлюкс
rates
    |influxDBOut()
        .quiet()
        .create()
        .database('kapacitor')
        .retentionPolicy('autogen')
        .measurement('rates')

// выбираем данные за последние 10 минут, считаем медиану
var todayRate = rates
    |where(lambda: duration((unixNano(now()) - unixNano("time")) / 1000, 1u) < todayPeriod)
    |median('rate')
        .as('median')

var prevRate = rates
    |median('rate')
        .as('median')

var joined = todayRate
    |join(prevRate)
        .as('today', 'prev')
    |httpOut('join')

var trigger = joined
    |alert()
        .warn(lambda: ("prev.median" - "today.median") > warnAlert)
        .warnReset(lambda: ("prev.median" - "today.median") < warnReset)
        .flapping(0.25, 0.5)
        .stateChangesOnly()
        // собираем в message ссылку на график дашборда графаны
        .message(
            '{{ .Level }}: {{ index .Tags "channel" }} err/req ratio ({{ index .Tags "supplier" }})
{{ if eq .Level "OK" }}It is ok now{{ else }}
'+string(todayPeriod)+' median is {{ index .Fields "today.median" | printf "%0.2f" }}%, by previous '+string(period)+' is {{ index .Fields "prev.median" | printf "%0.2f" }}%{{ end }}
http://grafana.ostrovok.in/d/'+string(grafana_dash)+
'?var-supplier={{ index .Tags "supplier" }}&var-channel={{ index .Tags "channel" }}&panelId='+string(grafana_panel)+'&fullscreen&tz=UTC%2B03%3A00'
        )
        .id('{{ index .Tags "name" }}/{{ index .Tags "channel" }}')
        .levelTag('level')
        .messageField('message')
        .durationField('duration')
        .topic('slack_graph')

// "today.median" дублируем как "value", также пишем в инфлюкс остальные филды алерта (keep)
trigger
    |eval(lambda: "today.median")
        .as('value')
        .keep()
    |influxDBOut()
        .quiet()
        .create()
        .database('kapacitor')
        .retentionPolicy('autogen')
        .measurement('alerts')
        .tag('alertName', name)

Chii chiri mhedziso?

Kapacitor yakanaka pakuita monitoring-alerts neboka remapoka, kuita mamwe masvomhu anoenderana nemametric akarekodhwa, kuita zviito zvetsika uye kumhanya zvinyorwa (udf).

Chipingamupinyi chekupinda hachina kunyanya kukwirira - edza kana grafana kana mamwe maturusi asingagutse zvizere zvishuwo zvako.

Source: www.habr.com