🥇Zidule za kukonza ma metric ku Kapacitor

Mwinamwake, lero palibe amene akufunsa chifukwa chake kuli kofunikira kusonkhanitsa ma metrics a utumiki. Chotsatira chotsatira ndikukhazikitsa chenjezo la ma metric omwe asonkhanitsidwa, omwe angadziwitse zopatuka zilizonse mumayendedwe omwe angakuthandizireni (makalata, Slack, Telegraph). Muntchito yosungitsa mahotelo pa intaneti Ostrovok.ru ma metric onse a ntchito zathu amatsanuliridwa mu InfluxDB ndikuwonetsedwa ku Grafana, ndipo zidziwitso zoyambira zimakonzedwanso pamenepo. Pantchito monga "muyenera kuwerengera china chake ndikufananiza nacho," timagwiritsa ntchito Kapacitor.

Kapacitor ndi gawo la TICK stack yomwe imatha kukonza ma metric kuchokera ku InfluxDB. Itha kulumikiza miyeso ingapo palimodzi (kujowina), kuwerengera china chake chofunikira kuchokera pazomwe mwalandira, kulemba zotsatira kubwerera ku InfluxDB, kutumiza chenjezo ku Slack/Telegraph/mail.

Chigawo chonsecho ndi chozizira komanso chatsatanetsatane zolemba, koma nthawi zonse padzakhala zinthu zothandiza zomwe sizinasonyezedwe mwatsatanetsatane m'mabuku. Munkhaniyi, ndidaganiza zotolera malangizo angapo othandiza, osawonekera (mawu oyambira a TICKscipt akufotokozedwa. apa) ndikuwonetsa momwe angagwiritsire ntchito pogwiritsa ntchito chitsanzo cha kuthetsa limodzi mwamavuto athu.

Tiyeni tipite!

zoyandama & int, zolakwika zowerengera

Vuto lokhazikika, lothetsedwa kudzera mwa ma castes:

var alert_float = 5.0
var alert_int = 10
data|eval(lambda: float("value") > alert_float OR float("value") < float("alert_int"))

Kugwiritsa ntchito default ()

Ngati tag/munda sunadzazidwe, zolakwika zowerengera zidzachitika:

|default()
        .tag('status', 'empty')
        .field('value', 0)

lembani kulowa (mkati vs kunja)

Mwachikhazikitso, kujowina kudzataya malo omwe mulibe deta (mkati).
Ndi fill('null'), kujowina kwakunja kudzachitika, pambuyo pake muyenera kuchita zosasintha () ndikudzaza zopanda kanthu:

var data = res1
    |join(res2)
        .as('res1', 'res2)
        .fill('null')
    |default()
        .field('res1.value', 0.0)
        .field('res2.value', 100.0)

Pakali pano. Mu chitsanzo pamwambapa, ngati imodzi mwa mndandanda (res1 kapena res2) ilibe kanthu, mndandanda wotsatira (deta) udzakhala wopanda kanthu. Pali matikiti angapo pamutuwu pa Github (1633, 1871, 6967) - tikuyembekezera kukonza ndi kuvutika pang'ono.

Kugwiritsa ntchito mikhalidwe powerengera (ngati mu lambda)

|eval(lambda: if("value" > 0, true, false)

Mphindi zisanu zomaliza kuchokera paipi yanthawiyo

Mwachitsanzo, muyenera kufananiza mfundo za mphindi zisanu zapitazi ndi sabata yapitayi. Mutha kutenga magulu awiri a data m'magulu awiri osiyana kapena kuchotsa gawo la data mu nthawi yokulirapo:

 |where(lambda: duration((unixNano(now()) - unixNano("time"))/1000, 1u) < 5m)

Njira ina kwa mphindi zisanu zapitazi ingakhale kugwiritsa ntchito BarrierNode, yomwe imadula deta nthawi isanakwane:

|barrier()
        .period(5m)

Zitsanzo zogwiritsira ntchito ma Go templates mu uthenga

Ma templates amafanana ndi mawonekedwe a phukusi text.templateM'munsimu muli ma puzzles omwe amakumana nawo kawirikawiri.

ngati-china

Timakonza zinthu ndipo osayambitsanso anthu kulembanso mameseji:

|alert()
    ...
    .message(
        '{{ if eq .Level "OK" }}It is ok now{{ else }}Chief, everything is broken{{end}}'
    )

Manambala awiri pambuyo pa nambala ya decimal mu uthenga

Kupititsa patsogolo kuwerengeka kwa uthenga:

|alert()
    ...
    .message(
        'now value is {{ index .Fields "value" | printf "%0.2f" }}'
    )

Kukulitsa zosinthika mu uthenga

Tikuwonetsa zambiri mu uthengawo kuti tiyankhe funso lakuti "Chifukwa chiyani ukufuula"?

var warnAlert = 10
  |alert()
    ...
    .message(
       'Today value less then '+string(warnAlert)+'%'
    )

Chizindikiritso chazidziwitso chapadera

Ichi ndi chinthu chofunikira pakakhala gulu lopitilira limodzi mu data, apo ayi chenjezo limodzi lokha lidzapangidwa:

|alert()
      ...
      .id('{{ index .Tags "myname" }}/{{ index .Tags "myfield" }}')

Custom handler's

Mndandanda waukulu wa othandizira akuphatikiza exec, omwe amakulolani kuti mugwiritse ntchito script yanu ndi magawo omwe adadutsa (stdin) - mwanzeru komanso china chilichonse!

Chimodzi mwazochita zathu ndi kalembedwe kakang'ono ka Python potumiza zidziwitso kuti zichepe.
Poyamba, tinkafuna kutumiza chithunzi cha grafana chotetezedwa mu uthenga. Pambuyo pake, lembani Chabwino mu ulusi ku chenjezo lapitalo kuchokera ku gulu lomwelo, osati monga uthenga wosiyana. Pambuyo pake - onjezani ku uthenga cholakwika chofala kwambiri pamphindi X zapitazi.

Mutu wina ndi kulankhulana ndi mautumiki ena ndi zochita zilizonse zoyambitsidwa ndi chenjezo (pokhapo ngati kuwunika kwanu kukuyenda bwino mokwanira).
Chitsanzo cha kufotokozera kwa wothandizira, komwe slack_handler.py ndizomwe timalemba tokha:

topic: slack_graph
id: slack_graph.alert
match: level() != INFO AND changed() == TRUE
kind: exec
options:
  prog: /sbin/slack_handler.py
  args: ["-c", "CHANNELID", "--graph", "--search"]

Kodi debug bwanji?

Njira yokhala ndi log linanena bungwe

|log()
      .level("error")
      .prefix("something")

Watch (cli): kapacitor -url host-or-ip:9092 zipika lvl=zolakwika

Njira ndi httpOut

Ikuwonetsa zomwe zikuchitika pano:

|httpOut('something')

Penyani (tenga): host-or-ip:9092/kapacitor/v1/tasks/task_name/something

Chithunzi chokonzekera

Ntchito iliyonse imabweretsa mtengo wophera wokhala ndi manambala othandiza pamawonekedwe zoye.
Tengani chipika dontho.
Ikani mu owonera, sangalalani.

Ndi kuti komwe mungapezeko chotengera?

timestamp mu influxdb pa writeback

Mwachitsanzo, timakhazikitsa chenjezo la kuchuluka kwa zopempha pa ola limodzi (groupBy(1h)) ndipo tikufuna kujambula chenjezo lomwe lidachitika mu influxdb (kuti tiwonetse bwino vuto lomwe lili pa grafana).

influxDBOut() idzalemba mtengo wanthawi kuchokera ku chenjezo kupita ku sitempu yanthawi; motero, mfundo yomwe ili pa tchati idzalembedwa kale/mochedwa kuposa pamene chenjezo lidafika.

Pakafunika kulondola bwino: timathetsa vutoli poyimbira foni munthu amene amatsatira makonda, yemwe amalemba data ku influxdb ndi chidindo chapano.

docker, kumanga ndi kutumiza

Poyambitsa, kapacitor imatha kuyika ntchito, ma templates ndi othandizira kuchokera ku bukhu lofotokozedwa mu config mu [load] block.

Kuti mupange bwino ntchito, muyenera zinthu zotsatirazi:

Dzina lafayilo - lokulitsidwa kukhala script id/name
Mtundu - mtsinje / batch
dbrp - mawu ofunikira kuti asonyeze kuti ndi database yanji + ndondomeko yomwe script ikulowera (dbrp "wopereka.""autogen")

Ngati ntchito ina ya batch ilibe mzere ndi dbrp, ntchito yonseyo ikana kuyamba ndipo ilemba moona mtima mu chipikacho.

Mu chronograf, m'malo mwake, mzerewu suyenera kukhalapo; sichivomerezedwa kudzera mu mawonekedwe ndipo imapanga cholakwika.

Kuthyolako pomanga chidebe: Dockerfile imatuluka ndi -1 ngati pali mizere ndi // + dbrp, zomwe zidzakuthandizani kuti mumvetsetse chifukwa chake mukulephera kusonkhanitsa kumanga.

phatikizani chimodzi kwa ambiri

Ntchito yachitsanzo: muyenera kutenga 95 peresenti ya nthawi yogwira ntchito kwa sabata, yerekezerani mphindi iliyonse ya 10 yomaliza ndi mtengo uwu.

Simungathe kujowina mmodzi-mpaka-ambiri, omaliza/wotanthauza/wapakatikati pa gulu la mfundo amasandutsa mfundo kukhala mtsinje, cholakwika "sichingawonjezere malire osagwirizana ndi ana: batch -> mtsinje" ibwezedwa.

Zotsatira za batch, monga kusintha kwa mawu a lambda, sizimalowetsedwanso.

Pali njira yosungira manambala ofunikira kuchokera pagulu loyamba kupita ku fayilo kudzera pa udf ndikuyika fayiloyi kudzera pa sideload.

Tinathetsa chiyani ndi izi?

Tili ndi pafupifupi 100 ogulitsa mahotelo, aliyense wa iwo akhoza kukhala ndi maulumikizidwe angapo, tiyeni titchule njira. Pali pafupifupi 300 mwa mayendedwe awa, iliyonse mwa njira imatha kugwa. Mwa ma metric onse ojambulidwa, tiziwunika kuchuluka kwa zolakwika (zopempha ndi zolakwika).

Bwanji osachita grafana?

Zidziwitso zolakwika zokhazikitsidwa ku Grafana zili ndi zovuta zingapo. Zina ndizovuta, zina mukhoza kutseka maso anu, malingana ndi momwe zinthu zilili.

Grafana sadziwa kuwerengera pakati pa miyeso + kuchenjeza, koma timafunikira mulingo (zopempha-zolakwa)/zopempha.

Zolakwazo zikuwoneka zoipa:

Ndipo zoyipa zochepa zikawonedwa ndi zopempha zopambana:

Chabwino, titha kuwerengeratu mtengo wautumiki pamaso pa grafana, ndipo nthawi zina izi zitha kugwira ntchito. Koma osati kwathu, chifukwa... pa njira iliyonse chiŵerengero chake chimatengedwa ngati "chabwinobwino", ndipo zidziwitso zimagwira ntchito molingana ndi mfundo zokhazikika (timayang'ana ndi maso athu, sinthani ngati pali zochenjeza pafupipafupi).

Izi ndi zitsanzo za "zabwinobwino" pamakanema osiyanasiyana:

Timanyalanyaza mfundo yapitayi ndikuganiza kuti chithunzi "chabwinobwino" chikufanana ndi onse ogulitsa. Tsopano zonse zili bwino, ndipo titha kudutsa ndi zidziwitso mu grafana?
Titha, koma sitikufuna, chifukwa tiyenera kusankha imodzi mwazosankha:
a) pangani ma graph ambiri panjira iliyonse padera (ndikuwatsatira mopweteka)
b) siyani tchati chimodzi chokhala ndi matchanelo onse (ndipo tayikani mizere yokongola ndi zidziwitso zosinthidwa makonda)

Munapanga bwanji?

Apanso, pali chitsanzo chabwino choyambira muzolemba (Kuwerengera mitengo pamindandanda yolumikizana), imatha kuyang'aniridwa kapena kutengedwa ngati maziko amavuto ofanana.

Zomwe tidachita pomaliza:

lowetsani mndandanda wazinthu ziwiri m'maola ochepa, ndikugawaniza ndi ma channel;
lembani mndandanda ndi gulu ngati panalibe deta;
yerekezerani wapakati wa mphindi 10 zapitazi ndi deta yam'mbuyo;
timakuwa ngati tipeza chinachake;
timalemba mitengo yowerengeredwa ndi zidziwitso zomwe zidachitika mu influxdb;
tumiza uthenga wothandiza kuti uchepe.

M'malingaliro anga, tinakwanitsa kukwaniritsa zonse zomwe timafuna kuti tipeze kumapeto (komanso zochulukirapo ndi osamalira mwambo) mokongola momwe tingathere.

Mutha kuyang'ana pa github.com kodi chitsanzo и dera locheperako (graphviz) script zotsatira.

Chitsanzo cha zotsatira zake:

dbrp "supplier"."autogen"
var name = 'requests.rate'
var grafana_dash = 'pczpmYZWU/mydashboard'
var grafana_panel = '26'
var period = 8h
var todayPeriod = 10m
var every = 1m
var warnAlert = 15
var warnReset = 5
var reqQuery = 'SELECT sum("count") AS value FROM "supplier"."autogen"."requests"'
var errQuery = 'SELECT sum("count") AS value FROM "supplier"."autogen"."errors"'

var prevErr = batch
    |query(errQuery)
        .period(period)
        .every(every)
        .groupBy(1m, 'channel', 'supplier')

var prevReq = batch
    |query(reqQuery)
        .period(period)
        .every(every)
        .groupBy(1m, 'channel', 'supplier')

var rates = prevReq
    |join(prevErr)
        .as('req', 'err')
        .tolerance(1m)
        .fill('null')
    // заполняем значения нулями, если их не было
    |default()
        .field('err.value', 0.0)
        .field('req.value', 0.0)
    // if в lambda: считаем рейт, только если ошибки были
    |eval(lambda: if("err.value" > 0, 100.0 * (float("req.value") - float("err.value")) / float("req.value"), 100.0))
        .as('rate')

// записываем посчитанные значения в инфлюкс
rates
    |influxDBOut()
        .quiet()
        .create()
        .database('kapacitor')
        .retentionPolicy('autogen')
        .measurement('rates')

// выбираем данные за последние 10 минут, считаем медиану
var todayRate = rates
    |where(lambda: duration((unixNano(now()) - unixNano("time")) / 1000, 1u) < todayPeriod)
    |median('rate')
        .as('median')

var prevRate = rates
    |median('rate')
        .as('median')

var joined = todayRate
    |join(prevRate)
        .as('today', 'prev')
    |httpOut('join')

var trigger = joined
    |alert()
        .warn(lambda: ("prev.median" - "today.median") > warnAlert)
        .warnReset(lambda: ("prev.median" - "today.median") < warnReset)
        .flapping(0.25, 0.5)
        .stateChangesOnly()
        // собираем в message ссылку на график дашборда графаны
        .message(
            '{{ .Level }}: {{ index .Tags "channel" }} err/req ratio ({{ index .Tags "supplier" }})
{{ if eq .Level "OK" }}It is ok now{{ else }}
'+string(todayPeriod)+' median is {{ index .Fields "today.median" | printf "%0.2f" }}%, by previous '+string(period)+' is {{ index .Fields "prev.median" | printf "%0.2f" }}%{{ end }}
http://grafana.ostrovok.in/d/'+string(grafana_dash)+
'?var-supplier={{ index .Tags "supplier" }}&var-channel={{ index .Tags "channel" }}&panelId='+string(grafana_panel)+'&fullscreen&tz=UTC%2B03%3A00'
        )
        .id('{{ index .Tags "name" }}/{{ index .Tags "channel" }}')
        .levelTag('level')
        .messageField('message')
        .durationField('duration')
        .topic('slack_graph')

// "today.median" дублируем как "value", также пишем в инфлюкс остальные филды алерта (keep)
trigger
    |eval(lambda: "today.median")
        .as('value')
        .keep()
    |influxDBOut()
        .quiet()
        .create()
        .database('kapacitor')
        .retentionPolicy('autogen')
        .measurement('alerts')
        .tag('alertName', name)

Kodi mapeto ake ndi otani?

Kapacitor ndi wabwino pakuchita zowunikira-zochenjeza ndi gulu lamagulu, kuchita mawerengedwe owonjezera kutengera ma metrics ojambulidwa kale, kuchita zochitika ndi zolemba (udf).

Cholepheretsa kulowa sichokwera kwambiri - yesani ngati grafana kapena zida zina sizikukwaniritsa zokhumba zanu.

Source: www.habr.com

Njira zopangira ma metrics ku Kapacitor