🥇Amaqhinga okucubungula amamethrikhi ku-Kapacitor

Cishe, namuhla akekho obuza ukuthi kungani kudingekile ukuqoqa amamethrikhi wesevisi. Isinyathelo esilandelayo esinengqondo siwukusetha isexwayiso samamethrikhi aqoqiwe, esizokwazisa nganoma yikuphi ukuchezuka kudatha eziteshini ezikulungele wena (i-imeyili, i-Slack, iTelegramu). Kusevisi yokubhukha amahhotela eku-inthanethi Ostrovok.ru wonke ama-metrics wamasevisi ethu athululelwa ku-InfluxDB futhi aboniswa e-Grafana, futhi izexwayiso eziyisisekelo nazo zilungiswa lapho. Ngemisebenzi efana nokuthi "udinga ukubala okuthile bese uqhathanisa nakho," sisebenzisa i-Kapacitor.

I-Kapacitor iyingxenye yesitaki sika-TICK engacubungula amamethrikhi avela ku-InfluxDB. Ingakwazi ukuxhuma izilinganiso ezimbalwa ndawonye (ihlanganise), ibale okuthile okuwusizo kudatha eyamukelwe, ibhale umphumela ibuyele ku-InfluxDB, ithumele isexwayiso ku-Slack/Telegram/mail.

Sonke isitaki sipholile futhi sinemininingwane imibhalo, kodwa kuyohlale kunezinto eziwusizo ezingavezwanga ngokucacile kumanyuwali. Kulesi sihloko, nginqume ukuqoqa amathiphu anjalo awusizo, angabonakali (i-syntax eyisisekelo ye-TICKscipt ichazwe lapha) futhi ubonise ukuthi zingasetshenziswa kanjani kusetshenziswa isibonelo sokuxazulula enye yezinkinga zethu.

Hamba!

ukuntanta & int, amaphutha okubala

Inkinga ejwayelekile ngokuphelele, ixazululwe ngama-castes:

var alert_float = 5.0
var alert_int = 10
data|eval(lambda: float("value") > alert_float OR float("value") < float("alert_int"))

Ukusebenzisa okuzenzakalelayo()

Uma umaka/inkambu ingagcwaliswanga, amaphutha ezibalo azokwenzeka:

|default()
        .tag('status', 'empty')
        .field('value', 0)

gcwalisa ukujoyina (ngaphakathi vs ngaphandle)

Ngokuzenzakalelayo, ukujoyina kuzolahla amaphuzu lapho ingekho khona idatha (ngaphakathi).
Ngokugcwalisa('null'), ukujoyina kwangaphandle kuzokwenziwa, ngemuva kwalokho udinga ukwenza okuzenzakalelayo() bese ugcwalisa amanani angenalutho:

var data = res1
    |join(res2)
        .as('res1', 'res2)
        .fill('null')
    |default()
        .field('res1.value', 0.0)
        .field('res2.value', 100.0)

Kusene-nuance lapha. Esibonelweni esingenhla, uma uchungechunge olulodwa (res1 noma res2) lungenalutho, uchungechunge oluphumayo (idatha) nalo luzongabi nalutho. Kunamathikithi amaningana kulesi sihloko ku-Github (1633, 1871, 6967) - silindele ukulungiswa nokuhlupheka kancane.

Ukusebenzisa izimo ekubaleni (uma ku-lambda)

|eval(lambda: if("value" > 0, true, false)

Imizuzu emihlanu edlule ukusuka epayipini lesikhathi

Isibonelo, udinga ukuqhathanisa amanani emizuzu emihlanu yokugcina neviki eledlule. Ungathatha amaqoqo amabili edatha ngamaqoqo amabili ahlukene noma ukhiphe ingxenye yedatha esikhathini esikhudlwana:

 |where(lambda: duration((unixNano(now()) - unixNano("time"))/1000, 1u) < 5m)

Enye indlela yemizuzu emihlanu yokugcina kungaba ukusebenzisa i-BarrierNode, enqamula idatha ngaphambi kwesikhathi esishiwo:

|barrier()
        .period(5m)

Izibonelo zokusebenzisa izifanekiso ze-Go kumlayezo

Izifanekiso zihambisana nefomethi evela kuphakheji text.templateNgezansi kukhona ama-puzzle avame ukuhlangana nawo.

uma-okunye

Sibeka izinto ngononina futhi asibangeli abantu ngombhalo futhi:

|alert()
    ...
    .message(
        '{{ if eq .Level "OK" }}It is ok now{{ else }}Chief, everything is broken{{end}}'
    )

Amadijithi amabili ngemva kwephoyinti ledesimali emlayezweni

Ukwenza ngcono ukufundeka komlayezo:

|alert()
    ...
    .message(
        'now value is {{ index .Fields "value" | printf "%0.2f" }}'
    )

Ukwandisa okuguquguqukayo kumlayezo

Sibonisa ulwazi olwengeziwe emlayezo ukuze siphendule umbuzo othi “Kungani kumemeza”?

var warnAlert = 10
  |alert()
    ...
    .message(
       'Today value less then '+string(warnAlert)+'%'
    )

Isihlonzi sesexwayiso esiyingqayizivele

Lokhu kuyinto edingekayo uma kunamaqembu angaphezu kwelilodwa kudatha, ngaphandle kwalokho kuzokwenziwa isaziso esisodwa:

|alert()
      ...
      .id('{{ index .Tags "myname" }}/{{ index .Tags "myfield" }}')

Izibambi ngokwezifiso

Uhlu olukhulu lwabaphathi lufaka i-exec, ekuvumela ukuthi usebenzise iskripthi sakho ngamapharamitha adlulile (stdin) - ubuhlakani futhi akukho okunye!

Elinye lamasiko ethu umbhalo omncane wePython wokuthumela izaziso ukuxega.
Ekuqaleni, besifuna ukuthumela isithombe se-grafana esivikelwe ngokugunyazwa kumlayezo. Ngemva kwalokho, bhala okuthi KULUNGILE ochungechungeni lwesexwayiso sangaphambilini esivela eqenjini elifanayo, hhayi njengomyalezo ohlukile. Kamuva kancane - engeza kumlayezo iphutha elivame kakhulu emizuzwini engu-X yokugcina.

Isihloko esihlukile ukuxhumana namanye amasevisi nanoma yiziphi izenzo eziqalwa yisixwayiso (kuphela uma ukuqapha kwakho kusebenza kahle ngokwanele).
Isibonelo sencazelo yesibambi, lapho i-slack_handler.py kuwumbhalo wethu esizibhalele wona:

topic: slack_graph
id: slack_graph.alert
match: level() != INFO AND changed() == TRUE
kind: exec
options:
  prog: /sbin/slack_handler.py
  args: ["-c", "CHANNELID", "--graph", "--search"]

Indlela yokulungisa iphutha?

Inketho enokukhishwa kwelogi

|log()
      .level("error")
      .prefix("something")

Buka (cli): kapacitor -url umphathi-noma-ip:9092 izingodo lvl=iphutha

Inketho nge-httpOut

Ibonisa idatha emgqeni wamanje:

|httpOut('something')

Buka (thola): umphathi-noma-ip:9092/kapacitor/v1/tasks/task_name/something

Uhlelo lokusebenza

Umsebenzi ngamunye ubuyisela isihlahla sokwenza esinezinombolo eziwusizo ngefomethi grafsi.
Thatha ibhulokhi dowry.
Yinamathisele kusibukeli, ujabulele.

Ungalitholaphi elinye iraki?

isitembu sesikhathi ku-influxdb ekubhaleni emuva

Isibonelo, simisa isexwayiso sesamba sezicelo ngehora (groupBy(1h)) futhi sifuna ukurekhoda isexwayiso esenzeke ku-influxdb (ukukhombisa kahle iqiniso lenkinga kugrafu kugrafana).

i-influxDBOut() izobhala inani lesikhathi ukusuka kusaziso kuye kusitembu sesikhathi; ngokufanelekile, iphuzu eshadini lizobhalwa ngaphambi kwesikhathi/kamuva kunokufika kwesixwayiso.

Lapho kudingeka ukunemba: sixazulula le nkinga ngokushayela isibambi sangokwezifiso, esizobhala idatha ku-influxdb ngesitembu sesikhathi samanje.

i-docker, ukwakha kanye nokuthunyelwa

Ekuqaleni, i-kapacitor ingalayisha imisebenzi, izifanekiso nezibambi kusukela kuhla lwemibhalo olucaciswe ku-config kubhulokhi [yokulayisha].

Ukuze udale umsebenzi ngendlela efanele, udinga izinto ezilandelayo:

Igama lefayela – lanwetshwa laba ngumazisi/igama leskripthi
Uhlobo – ukusakaza/inqwaba
dbrp - igama elingukhiye ukukhombisa ukuthi iyiphi isizindalwazi + inqubomgomo umbhalo osebenza kuyo (dbrp “umphakeli.”“autogen”)

Uma umsebenzi othile we-batch ungenawo umugqa one-dbrp, yonke isevisi izokwenqaba ukuqala futhi izobhala ngokwethembeka ngakho kulogi.

Ku-chronograf, ngokuphambene nalokho, lo mugqa akufanele ube khona; awamukelwa ngesixhumi esibonakalayo futhi udala iphutha.

Hack lapho wakha isiqukathi: I-Dockerfile iphuma ngo -1 uma kukhona imigqa eno-//+dbrp, okuzokuvumela ukuthi uqonde ngokushesha isizathu sokwehluleka lapho uhlanganisa isakhiwo.

hlanganisa eyodwa kwabaningi

Umsebenzi oyisibonelo: udinga ukuthatha iphesenti elingu-95 lesikhathi sokusebenza sesevisi iviki lonke, uqhathanise iminithi ngalinye kwangu-10 okugcina naleli nani.

Awukwazi ukuhlanganisa okukodwa kuya kokuningi, okokugcina/okushoyo/okumaphakathi phezu kweqembu lamaphuzu kushintsha inodi ibe ukusakaza, iphutha elithi “ayikwazi ukwengeza imiphetho engafanele yengane: inqwaba -> ukusakaza” izobuyiswa.

Umphumela wenqwaba, njengokuguquguquka kusisho se-lambda, nawo awushintshwa.

Kukhona inketho yokulondoloza izinombolo ezidingekayo kusukela kuqeqebana lokuqala ukuya efayeleni nge-udf futhi ulayishe leli fayela ngokulayisha okuseceleni.

Yini esiyixazulule ngalokhu?

Sinabahlinzeki bamahhotela abangaba ngu-100, ngamunye wabo angaba nokuxhumana okuningana, ake sikubize ngesiteshi. Cishe zingu-300 zalezi ziteshi, isiteshi ngasinye singawa. Kuwo wonke amamethrikhi arekhodiwe, sizoqapha izinga lamaphutha (izicelo namaphutha).

Kungani hhayi grafana?

Izaziso zephutha ezilungiselelwe e-Grafana zinokungalungi okumbalwa. Ezinye zibucayi, ezinye ungazivala amehlo, kuye ngesimo.

U-Grafana akazi ukuthi kubalwa kanjani phakathi kwezilinganiso + nokuxwayisa, kodwa sidinga isilinganiso (izicelo-amaphutha)/izicelo.

Amaphutha abukeka emabi:

Nobubi buncane uma bubhekwa ngezicelo eziyimpumelelo:

Kulungile, singabala kusengaphambili inani kusevisi ngaphambi kwe-grafana, futhi kwezinye izimo lokhu kuzosebenza. Kodwa hhayi kithi, ngoba... esiteshini ngasinye isilinganiso saso sibhekwa "esivamile", futhi izexwayiso zisebenza ngokuvumelana namanani amile (sizibheka ngamehlo ethu, sizishintshe uma kunezixwayiso ezivamile).

Lezi izibonelo "zokuvamile" eziteshini ezahlukene:

Siziba iphuzu langaphambilini bese sicabanga ukuthi isithombe "esivamile" siyefana kubo bonke abahlinzeki. Manje konke kuhamba kahle, futhi singakwazi ukudlula ngezexwayiso ku-grafana?
Singakwazi, kodwa asifuni ngempela, ngoba kufanele sikhethe enye yezinketho:
a) yenza amagrafu amaningi esiteshi ngasinye ngokuhlukana (futhi uwaphelezele kabuhlungu)
b) shiya ishadi elilodwa elinawo wonke amashaneli (futhi ulahleke emigqeni emibalabala nezexwayiso ezenziwe ngezifiso)

Ukwenze kanjani?

Futhi, kunesibonelo esihle sokuqala emibhalweni (Ibala amanani kulo lonke uchungechunge oluhlanganisiwe), ingabhekisiswa noma ithathwe njengesisekelo sezinkinga ezifanayo.

Esikwenzile ekugcineni:

joyina uchungechunge olubili emahoreni ambalwa, uqoqe ngeziteshi;
gcwalisa uchungechunge ngeqembu uma bekungekho idatha;
qhathanisa i-median yemizuzu engu-10 yokugcina nedatha yangaphambilini;
siyamemeza uma sithola okuthile;
sibhala amanani abaliwe kanye nezixwayiso ezenzeke ku-influxdb;
thumela umlayezo owusizo ukudebeselela.

Ngokubona kwami, sikwazile ukuzuza konke ebesifuna ukukuthola ekugcineni (ngisho nokungaphezulu nabaphathi bangokwezifiso) ngobuhle ngangokunokwenzeka.

Ungabheka ku-github.com isibonelo sekhodi и isekethe encane (graphviz) umbhalo owumphumela.

Isibonelo sekhodi yomphumela:

dbrp "supplier"."autogen"
var name = 'requests.rate'
var grafana_dash = 'pczpmYZWU/mydashboard'
var grafana_panel = '26'
var period = 8h
var todayPeriod = 10m
var every = 1m
var warnAlert = 15
var warnReset = 5
var reqQuery = 'SELECT sum("count") AS value FROM "supplier"."autogen"."requests"'
var errQuery = 'SELECT sum("count") AS value FROM "supplier"."autogen"."errors"'

var prevErr = batch
    |query(errQuery)
        .period(period)
        .every(every)
        .groupBy(1m, 'channel', 'supplier')

var prevReq = batch
    |query(reqQuery)
        .period(period)
        .every(every)
        .groupBy(1m, 'channel', 'supplier')

var rates = prevReq
    |join(prevErr)
        .as('req', 'err')
        .tolerance(1m)
        .fill('null')
    // заполняем значения нулями, если их не было
    |default()
        .field('err.value', 0.0)
        .field('req.value', 0.0)
    // if в lambda: считаем рейт, только если ошибки были
    |eval(lambda: if("err.value" > 0, 100.0 * (float("req.value") - float("err.value")) / float("req.value"), 100.0))
        .as('rate')

// записываем посчитанные значения в инфлюкс
rates
    |influxDBOut()
        .quiet()
        .create()
        .database('kapacitor')
        .retentionPolicy('autogen')
        .measurement('rates')

// выбираем данные за последние 10 минут, считаем медиану
var todayRate = rates
    |where(lambda: duration((unixNano(now()) - unixNano("time")) / 1000, 1u) < todayPeriod)
    |median('rate')
        .as('median')

var prevRate = rates
    |median('rate')
        .as('median')

var joined = todayRate
    |join(prevRate)
        .as('today', 'prev')
    |httpOut('join')

var trigger = joined
    |alert()
        .warn(lambda: ("prev.median" - "today.median") > warnAlert)
        .warnReset(lambda: ("prev.median" - "today.median") < warnReset)
        .flapping(0.25, 0.5)
        .stateChangesOnly()
        // собираем в message ссылку на график дашборда графаны
        .message(
            '{{ .Level }}: {{ index .Tags "channel" }} err/req ratio ({{ index .Tags "supplier" }})
{{ if eq .Level "OK" }}It is ok now{{ else }}
'+string(todayPeriod)+' median is {{ index .Fields "today.median" | printf "%0.2f" }}%, by previous '+string(period)+' is {{ index .Fields "prev.median" | printf "%0.2f" }}%{{ end }}
http://grafana.ostrovok.in/d/'+string(grafana_dash)+
'?var-supplier={{ index .Tags "supplier" }}&var-channel={{ index .Tags "channel" }}&panelId='+string(grafana_panel)+'&fullscreen&tz=UTC%2B03%3A00'
        )
        .id('{{ index .Tags "name" }}/{{ index .Tags "channel" }}')
        .levelTag('level')
        .messageField('message')
        .durationField('duration')
        .topic('slack_graph')

// "today.median" дублируем как "value", также пишем в инфлюкс остальные филды алерта (keep)
trigger
    |eval(lambda: "today.median")
        .as('value')
        .keep()
    |influxDBOut()
        .quiet()
        .create()
        .database('kapacitor')
        .retentionPolicy('autogen')
        .measurement('alerts')
        .tag('alertName', name)

Siyini isiphetho?

I-Kapacitor inhle ekwenzeni izexwayiso zokuqapha ngenqwaba yamaqembu, yenza izibalo ezengeziwe ngokusekelwe kumamethrikhi asevele erekhodiwe, ukwenza izenzo zangokwezifiso kanye nemibhalo esebenzayo (udf).

Isithiyo sokungena asiphezulu kakhulu - sizame uma i-grafana noma amanye amathuluzi enganeliseki ngokugcwele izifiso zakho.

Source: www.habr.com

Amacebo okucubungula amamethrikhi ku-Kapacitor