Loki - log muunganidzwa uchishandisa nzira yePrometheus

Kwazisai, vagari veKhabrovsk! Manheru ekutanga kwekunyoresa kutsva kwekosi "DevOps maitiro uye maturusi" Takagadzirira shanduro yezvinyorwa zvinonakidza kwauri.

Chinyorwa ichi sumo pfupi kuna Loki. Project Loki inotsigirwa naGrafana uye yakanangana nepakati pekuunganidza matanda (kubva kumaseva kana midziyo).

Kufuridzira kukuru kwaLoki kwaive Prometheus nepfungwa yekushandisa nzira dzake dzekuronga manejimendi:

• kushandisa mavara kuchengetedza data
• kushandiswa kwezviwanikwa zvishoma

Tichadzokera kune mashandiro anoita Prometheus uye nekupa mimwe mienzaniso yekushandiswa kwayo mumamiriro eKubernetes.

Mashoko mashoma pamusoro pePrometheus

Kuti unzwisise zvizere mashandiro anoita Loki, zvakakosha kutora nhanho kumashure uye kurangarira zvishoma nezvePrometheus.

Chimwe chezvakasiyana zvePrometheus kutorwa kwemametrics kubva kunzvimbo dzekuunganidza (kuburikidza nevanotengesa kunze) uye kuachengeta muTSDB (Time Series Data Base), pamwe nekuwedzera metadata muchimiro chemavara.

Nei uchiida

Munguva pfupi yapfuura, Prometheus yave iyo de facto chiyero munyika yemidziyo uye Kubernetes: kuisirwa kwayo kuri nyore kwazvo, uye Kubernetes cluster inouya neinogumira Prometheus. Prometheus inogona zvakare kubvisa metrics kubva kumashandisirwo akaiswa mumudziyo uchichengeta chaiwo mavara. Naizvozvo, yekutarisa application iri nyore kwazvo kuita.

Nehurombo, hapasati pasisina turnkey mhinduro yelog manejimendi, uye iwe unofanirwa kuzvitsvagira mhinduro:

• yakachengetedzwa gore sevhisi yekuisa centralizing matanda (AWS, Azure kana Google)
• yekutarisa sevhisi "kutarisa sevhisi" (semuenzaniso, Datadog)
• kugadzira yako yega log yekuunganidza sevhisi.

Kune yechitatu sarudzo, ini ndakagara ndichishandisa Elasticsearch, kunyangwe hazvo ndanga ndisiri kugara ndichifara nayo (kunyanya kuremerwa kwayo uye kuoma kwekugadzirisa).

Loki yakagadzirirwa kurerutsa kuita maererano nemisimboti inotevera:

• kuve nyore kutanga
• shandisa zviwanikwa zvishoma
• shanda wakazvimiririra pasina gadziriso yakakosha
• shanda sekuwedzera kuPrometheus kubatsira nekuferefeta kwebug

Zvisinei, izvi zviri nyore zvinouya nekurasikirwa kwezvimwe zvibvumirano. Imwe yacho haisi yekunongedza zviri mukati. Naizvozvo, kutsvaga kwemavara hakuna kunyatso shanda kana kupfuma uye hakubvumidze nhamba dzezviri mukati mezvinyorwa. Asi sezvo Loki achida kuenzana negrep uye mubatsiri wePrometheus, izvi hazvisi zvisizvo.

Chiitiko Kuongorora

Kuti unzwisise zviri nani kuti sei Loki asingade indexing, ngatidzokerei kunzira yekuferefeta chiitiko yakashandiswa nevagadziri veLoki:

1 Yambiro → 2 Dashboard → 3 Adhoc Mubvunzo → 4 Rog Aggregation → 5 Distributed Tracing → 6 Gadzirisa!
(1 Yambiro → 2 Dashboard → 3 Adhoc Query → 4 Log Aggregation → 5 Distributed Tracing → 6 Kugadzirisa!)

Pfungwa ndeyekuti tinogashira imwe mhando yechenjedzo (Slack Notification, SMS, etc.) uye mushure meizvozvo:

• tarisa Grafana dashboards
• tarisa masevhisi metrics (semuenzaniso, muPrometheus)
• tarisa zvinyorwa zvelogi (semuenzaniso, muElasticsearch)
• pamwe tarisa maitiro akagoverwa (Jaeger, Zipkin, nezvimwewo)
• uye pakupedzisira kugadzirisa dambudziko rekutanga.

Pano, munyaya yeGrafana + Prometheus + Elasticsearch + Zipkin stack, uchafanira kushandisa maturusi mana akasiyana. Kuti uderedze nguva, zvingave zvakanaka kukwanisa kuita matanho ese aya uchishandisa chishandiso chimwe: Grafana. Zvakakosha kuziva kuti nzira iyi yekutsvakurudza yakashandiswa muGrafana kubva mushanduro 6. Nokudaro, zvinova zvinokwanisika kuwana Prometheus data zvakananga kubva kuGrafana.

Explorer skrini yakakamurwa pakati pePrometheus naLoki

Kubva pane ino skrini unogona kuona Loki matanda ane hukama nePrometheus metrics uchishandisa iyo split screen pfungwa. Kubva vhezheni 6.5, Grafana inokutendera kuti utarise id yekutsvaga muLoki log entries kutevedzera zvinongedzo kune ako aunofarira akagoverwa ekutsvaga maturusi (Jaeger).

Yemunharaunda Loki bvunzo

Nzira iri nyore yekuyedza Loki munharaunda ndeye kushandisa docker-compose. Iyo docker-compose faira iri muLoki repository. Iwe unogona kuwana repository uchishandisa murairo unotevera git:

$ git clone https://github.com/grafana/loki.git

Ipapo iwe unofanirwa kuenda kune dhairekitori rekugadzira:

$ cd production

Mushure meizvi unogona kuwana yazvino vhezheni yeDocker mifananidzo:

$ docker-compose pull

Pakupedzisira, iyo Loki stack inotangwa nemurairo unotevera:

$ docker-compose up

Loki architecture

Heino dhayagiramu diki ine Loki architecture:

Loki Architecture Mitemo

Webhu mutengi anomhanyisa zvikumbiro pane sevha, Promtail inounganidza matanda uye inoatumira kuLoki, mutengi wewebhu anotumirawo metadata kuLoki. Loki anounganidza zvese uye ozvitumira kuGrafana.
Loki inotangwa. Kuti uone zvikamu zviripo, shandisa murairo unotevera:

$ docker ps

Panyaya yeDocker ichangobva kuisirwa, iwo murairo unofanirwa kudzosera inotevera mhedzisiro:

IMAGE               PORTS                  NAMES
grafana/promtail:                          production_promtail_1
grafana/grafana: m  0.0.0.0:3000->3000/tcp production_grafana_1
grafana/loki: late  80/tcp,0.0.0.0:3100... production_loki_1

Tinoona zvinotevera zvikamu:

• Promtail: mumiriri ane mutoro wepakati pekuisa matanda
• Grafana: chishandiso chakakurumbira chedhibhodhi
• Loki: Data Centralization Daemon

Sechikamu chemhando yezvivakwa (semuenzaniso, zvichibva pamakina chaiwo), mumiriri wePromtail anofanira kuiswa pamushini wega wega. Grafana naLoki vanogona kuiswa pamushini mumwe chete.

Kuendeswa kuKubernetes

Kuisa Loki zvikamu paKubernetes zvichave sezvizvi:

• daemonSet yekuisa iyo Promtail mumiriri pane yega yega michina iri muserver cluster.
• Deployment Loki
• uye yekupedzisira ndeyekutumirwa kweGrafana.

Neraki, Loki inowanikwa seHelm package, zvichiita kuti zvive nyore kuendesa.

Kuiswa kuburikidza neHeml

Iwe unofanirwa kunge uine Heml yakaiswa. Inogona kutorwa kubva kupurojekiti yeGitHub repository. Iyo inoiswa nekusunungura iyo archive inoenderana neyako dhizaini uye nekuwedzera helm kune $PATH.

Cherechedza: vhezheni 3.0.0 yeHelm yakaburitswa nguva pfupi yadarika. Sezvo pakave neshanduko dzakawanda pairi, muverengi anorayirwa kuti ambomira zvishoma asati aishandisa..

Kuwedzera kunobva Helm

Danho rekutanga nderekuwedzera "loki" repository uchishandisa murairo unotevera:

$ helm add loki https://grafana.github.io/loki/charts

Mushure meizvi, unogona kutsvaga mapakeji anonzi "loki":

$ helm search loki

Mhinduro:

loki/loki       0.17.2 v0.4.0 Loki: like Prometheus, but for logs.
loki/loki-stack 0.19.1 v0.4.0 Loki: like Prometheus, but for logs.
loki/fluent-bit 0.0.2  v0.0.1 Uses fluent-bit Loki go plugin for...
loki/promtail   0.13.1 v0.4.0 Responsible for gathering logs and...

Aya mapakeji ane zvinotevera zvinhu:

• the package loki/loki inofanana neLoki server chete
• the package loki/fluent-bit inokutendera kuti utumire DaemonSet uchishandisa fluent-bin kuunganidza matanda panzvimbo yePromtail
• the package loki/promtail ine log file collection agent
• the package loki/loki-stack, inokutendera kuti uise Loki nekukurumidza pamwe nePromtail.

Kuisa Loki

Kuendesa Loki kuKubernetes, mhanyisa unotevera kuraira mu "monitoring" namespace:

$ helm upgrade --install loki loki/loki-stack --namespace monitoring

Kuti uchengetedze ku diski, wedzera parameter --set loki.persistence.enabled = true:

$ helm upgrade --install loki loki/loki-stack 
              --namespace monitoring 
              --set loki.persistence.enabled=true

Cherechedza: kana iwe uchida kuendesa Grafana panguva imwe chete, wobva wawedzera parameter --set grafana.enabled = true

Paunomhanyisa uyu murairo unofanirwa kuwana zvinotevera zvinobuda:

LAST DEPLOYED: Tue Nov 19 15:56:54 2019
NAMESPACE: monitoring
STATUS: DEPLOYED
RESOURCES:
==> v1/ClusterRole
NAME AGE
loki-promtail-clusterrole 189d
…
NOTES:
The Loki stack has been deployed to your cluster. Loki can now be added as a datasource in Grafana.
See <a href="http://docs.grafana.org/features/datasources/loki/">http://docs.grafana.org/features/datasources/loki/</a> for more details.

Tichitarisa mamiriro emapods mu "monitoring" namespace, tinogona kuona kuti zvese zvakaiswa:

$ kubectl -n monitoring get pods -l release=loki

Mhinduro:

NAME                 READY  STATUS   RESTARTS  AGE
loki-0               1/1    Running  0         147m
loki-promtail-9zjvc  1/1    Running  0         3h25m
loki-promtail-f6brf  1/1    Running  0         11h
loki-promtail-hdcj7  1/1    Running  0         3h23m
loki-promtail-jbqhc  1/1    Running  0         11h
loki-promtail-mj642  1/1    Running  0         62m
loki-promtail-nm64g  1/1    Running  0         24m

Mapodhi ese ari kushanda. Iye zvino yave nguva yekuita mimwe miedzo!

Kubatana kuGrafana

Kuti ubatanidze kuGrafana pasi peKubernetes, unofanirwa kuvhura tunnel kune pod yayo. Pazasi pane murairo wekuvhura port 3000 yeGrafana pod:

$ kubectl -n port-forward monitoring svc/loki-grafana 3000:80

Imwe pfungwa yakakosha ndiko kudiwa kwekudzorera iyo Grafana administrator password. Pasiwedhi inochengetwa yakavanzika loki-grafana mumunda .data.admin-user mu base64 format.

Kuti uidzorere, unofanirwa kumhanyisa murairo unotevera:

$ kubectl -n monitoring get secret loki-grafana 
 --template '{{index .data "admin-password" | base64decode}}'; echo

Shandisa iyi password yakabatana neiyo default administrator account (admin).

Kutsanangura nzvimbo yeLoki data muGrafana

Chekutanga pane zvese, ita shuwa kuti Loki data sosi yakagadzirwa (Configuration/Datasource).
Heuno muenzaniso:

Muenzaniso wekugadzira data sosi yeLoki

Nekudzvanya pa "Muedzo" unogona kutarisa kubatana neLoki.

Kuita zvikumbiro kuna Loki

Zvino enda kuGrafana kune "Ongorora" chikamu. Paunenge uchigamuchira matanda kubva mumidziyo, Loki anowedzera metadata kubva Kubernetes. Saka, zvinokwanisika kuona matanda eimwe mudziyo.

Semuenzaniso, kusarudza iyo promtail mudziyo matanda, unogona kushandisa unotevera mubvunzo: {container_name = "promtail"}.
Pano zvakare yeuka kusarudza iyo Loki data sosi.

Muvhunzo uyu uchadzosa zviitwa zvemudziyo sezvinotevera:

Mhinduro yemubvunzo muGrafana

Wedzera kune dashboard

Kutanga neGrafana 6.4, unogona kuisa ruzivo rwelogi zvakananga pane dashboard. Mushure meizvi, mushandisi anozokwanisa kukurumidza kushandura pakati pehuwandu hwezvikumbiro pane yake saiti uye maitiro ekushandisa.

Pazasi pane muenzaniso wedashboard inoshandisa iyi kudyidzana:

Sample dashboard ine Prometheus metrics uye Loki matanda

Ramangwana reLoki

Ndakatanga kushandisa Loki kumashure muna Chivabvu/Chikumi neshanduro 0.1. Nhasi vhezheni 1, uye kunyangwe 1.1 uye 1.2 zvakatoburitswa.

Zvinofanira kubvumwa kuti vhezheni 0.1 yakanga isina kugadzikana zvakakwana. Asi 0.3 yakatoratidza zviratidzo zvechokwadi zvekukura, uye shanduro dzinotevera (0.4, ipapo 1.0) dzakasimbisa pfungwa iyi chete.

Mushure me1.0.0, hapana angave nechikonzero chekusashandisa chishandiso ichi chinoshamisa.

Kumwe kuvandudzwa hakufanirwe kunge kuri pamusoro peLoki, asi kusanganisa kwayo neiyo yakanakisa Grafana. Muchokwadi, Grafana 6.4 yatove nekubatana kwakanaka nemadhibhodhi.

Grafana 6.5, iyo yakaburitswa nguva pfupi yadarika, inovandudza kubatanidzwa uku nekuziva otomatiki zviri mukati meJSON fomati.

Vhidhiyo iri pazasi inoratidza muenzaniso mudiki weiyi michina:

Kushandisa Loki tambo dzakafumurwa muGrafana

Izvo zvinogoneka kushandisa imwe yeJSON minda, semuenzaniso, ye:

• zvinongedzo kune chekunze chishandiso
• kusefa zviri mukati

Semuenzaniso, unogona kudzvanya pane traceId kuenda kuZipkin kana Jaeger.

Semazuva ese, tinotarisira mhinduro dzako uye tinokukoka iwe kuti udaro vhura webinar, kwatichataura nezvekuti indasitiri yeDevOps yakagadziridzwa sei muna 2019 uye tokurukura nzira dzebudiriro dze2020.

Source: www.habr.com