ProHoster > Π‘Π»ΠΎΠ³ > Kev tswj hwm > Loki - log sau siv Prometheus mus kom ze

Loki - log sau siv Prometheus mus kom ze

Ua tsaug, cov neeg nyob hauv Khabrovsk! Nyob rau hnub ua ntej ntawm qhov pib ntawm kev cuv npe tshiab rau chav kawm "DevOps kev coj ua thiab cov cuab yeej" Peb tau npaj ib qho kev txhais cov ntaub ntawv nthuav dav rau koj.

Kab lus no yog ib qho kev qhia luv luv rau Loki. Qhov project Loki txhawb nqa los ntawm Grafana thiab yog aimed ntawm centralized sau cov cav (los ntawm servers los yog ntim).

Lub ntsiab kev tshoov siab rau Loki yog Prometheus nrog lub tswv yim ntawm kev siv nws txoj hauv kev rau kev tswj xyuas cov cav:

  • siv cov ntawv sau los khaws cov ntaub ntawv
  • siv ob peb yam kev pab

Peb yuav rov qab mus rau Prometheus ua haujlwm li cas thiab muab qee qhov piv txwv ntawm nws siv hauv cov ntsiab lus ntawm Kubernetes.

Ob peb lo lus hais txog Prometheus

Yuav kom nkag siab meej tias Loki ua haujlwm li cas, nws yog ib qho tseem ceeb uas yuav tsum tau ua ib kauj ruam rov qab thiab nco ntsoov me ntsis txog Prometheus.

Ib qho ntawm cov yam ntxwv tshwj xeeb ntawm Prometheus yog kev rho tawm cov kev ntsuas los ntawm kev sau cov ntsiab lus (ntawm cov xa tawm) thiab khaws cia rau hauv TSDB (Lub Sijhawm Series Data Base), nrog rau kev ntxiv cov metadata hauv daim ntawv lo.

Vim li cas nws thiaj xav tau

Tsis ntev los no, Prometheus tau dhau los ua tus txheej txheem de facto nyob rau hauv lub ntiaj teb no ntim thiab Kubernetes: nws installation yog heev yooj yim, thiab cov Kubernetes pawg los nrog ib haiv neeg kawg point rau Prometheus. Prometheus tseem tuaj yeem rho tawm cov kev ntsuas los ntawm cov ntawv thov xa mus rau hauv lub thawv thaum khaws cov ntawv tshwj xeeb. Yog li ntawd, kev soj ntsuam daim ntawv thov yog yooj yim heev rau kev siv.

Hmoov tsis zoo, tseem tsis muaj kev daws teeb meem rau kev tswj xyuas cav, thiab koj yuav tsum nrhiav kev daws teeb meem rau koj tus kheej:

  • tswj huab kev pab cuam rau centralizing cav (AWS, Azure los yog Google)
  • saib xyuas kev pabcuam "kev saib xyuas raws li kev pabcuam" (piv txwv li, Datadog)
  • tsim koj tus kheej cov kev pabcuam sau cia.

Rau qhov kev xaiv thib peb, kuv ib txwm siv Elasticsearch, txawm tias qhov tseeb tias kuv tsis zoo siab nrog nws (tshwj xeeb tshaj yog nws hnyav thiab nyuaj ntawm kev teeb tsa).

Loki tau tsim los ua kom yooj yim rau kev siv raws li cov hauv qab no:

  • pib yooj yim
  • siv ob peb yam kev pab
  • ua haujlwm ntawm nws tus kheej yam tsis muaj kev saib xyuas tshwj xeeb
  • ua ib qho ntxiv rau Prometheus los pab nrog kev tshawb nrhiav kab

Txawm li cas los xij, qhov kev simplicity no los ntawm kev siv qee qhov kev cuam tshuam. Ib qho ntawm lawv tsis yog los ntsuas cov ntsiab lus. Yog li ntawd, kev tshawb nrhiav cov ntawv nyeem tsis zoo heev lossis nplua nuj thiab tsis tso cai rau txheeb xyuas cov ntsiab lus ntawm cov ntawv nyeem. Tab sis txij li Loki xav ua qhov sib npaug ntawm grep thiab ntxiv rau Prometheus, qhov no tsis yog qhov tsis zoo.

Kev tshawb nrhiav xwm txheej

Yuav kom nkag siab zoo dua yog vim li cas Loki tsis xav tau kev ntsuas ntsuas, cia peb rov qab mus rau txoj kev tshawb nrhiav qhov xwm txheej uas Loki cov neeg tsim khoom siv:

Loki - log sau siv Prometheus mus kom ze
1 Alert β†’ 2 Dashboard β†’ 3 Adhoc Query β†’ 4 Log Aggregation β†’ 5 Distributed Tracing β†’ 6 Kho!
(1 Ceeb Toom β†’ 2 Dashboard β†’ 3 Adhoc Query β†’ 4 Log Aggregation β†’ 5 Distributed Tracing β†’ 6 Kho!)

Lub tswv yim yog tias peb tau txais qee yam kev ceeb toom (Slack Notification, SMS, thiab lwm yam) thiab tom qab ntawd:

  • saib Grafana dashboards
  • saib cov kev ntsuas kev pabcuam (piv txwv li, hauv Prometheus)
  • saib cov log nkag (piv txwv li, hauv Elasticsearch)
  • Tej zaum saib cov kab sib faib (Jaeger, Zipkin, thiab lwm yam)
  • thiab thaum kawg kho qhov teeb meem qub.

Ntawm no, nyob rau hauv rooj plaub ntawm Grafana + Prometheus + Elasticsearch + Zipkin pawg, koj yuav tau siv plaub yam sib txawv. Txhawm rau txo lub sijhawm, nws yuav zoo rau ua tau tag nrho cov kauj ruam no siv ib lub cuab yeej: Grafana. Nws yog ib qho tsim nyog sau cia tias txoj hauv kev rau kev tshawb fawb no tau ua tiav hauv Grafana txij li version 6. Yog li, nws tuaj yeem nkag mus rau Prometheus cov ntaub ntawv ncaj qha los ntawm Grafana.

Loki - log sau siv Prometheus mus kom ze
Explorer screen cais ntawm Prometheus thiab Loki

Los ntawm qhov screen no koj tuaj yeem saib Loki cov cav muaj feem xyuam nrog Prometheus metrics siv lub tswv yim sib cais. Txij li thaum version 6.5, Grafana tso cai rau koj los txheeb xyuas tus lej cim hauv Loki cov ntawv nkag mus ua raws cov kev txuas mus rau koj nyiam cov cuab yeej taug qab (Jaeger).

Local Loki xeem

Txoj hauv kev yooj yim tshaj plaws los sim Loki hauv zos yog siv docker-compose. Cov ntaub ntawv docker-compose nyob rau hauv Loki repository. Koj tuaj yeem tau txais qhov chaw cia khoom siv cov lus txib hauv qab no git:

$ git clone https://github.com/grafana/loki.git

Tom qab ntawd koj yuav tsum mus rau qhov chaw tsim khoom:

$ cd production

Tom qab no koj tuaj yeem tau txais qhov tseeb version ntawm Docker cov duab:

$ docker-compose pull

Thaum kawg, Loki pawg tau pib nrog cov lus txib hauv qab no:

$ docker-compose up

Loki architecture

Nov yog daim duab me me nrog Loki architecture:

Loki - log sau siv Prometheus mus kom ze
Loki Architecture Principles

Tus neeg siv lub vev xaib khiav cov ntawv thov ntawm lub server, Promtail sau cov cav thiab xa lawv mus rau Loki, tus neeg siv lub vev xaib kuj xa metadata rau Loki. Loki sib sau ua ke txhua yam thiab xa mus rau Grafana.
Loki yog launched. Txhawm rau saib cov khoom muaj, khiav cov lus txib hauv qab no:

$ docker ps

Nyob rau hauv cov ntaub ntawv ntawm lub tshiab ntsia Docker, cov lus txib yuav tsum rov qab tau cov txiaj ntsig hauv qab no:

IMAGE               PORTS                  NAMES
grafana/promtail:                          production_promtail_1
grafana/grafana: m  0.0.0.0:3000->3000/tcp production_grafana_1
grafana/loki: late  80/tcp,0.0.0.0:3100... production_loki_1

Peb pom cov khoom hauv qab no:

  • Promtail: tus neeg saib xyuas lub luag haujlwm rau centralizing cav
  • Grafana: lub npe nrov dashboard cuab yeej
  • Loki: Data Centralization Daemon

Raws li ib feem ntawm cov txheej txheem classic (piv txwv li, raws li cov tshuab virtual), tus neeg sawv cev Promtail yuav tsum tau xa mus rau txhua lub tshuab. Grafana thiab Loki tuaj yeem ntsia tau rau ntawm tib lub tshuab.

Kev xa mus rau Kubernetes

Txhim kho Loki Cheebtsam ntawm Kubernetes yuav ua raws li hauv qab no:

  • daemonSet rau deploying Promtail tus neeg saib xyuas ntawm txhua lub tshuab hauv pawg neeg rau zaub mov
  • Kev xa tawm Loki
  • thiab qhov kawg yog kev xa tawm ntawm Grafana.

Hmoov zoo, Loki muaj nyob rau hauv Helm pob, ua kom yooj yim rau kev xa mus.

Installation ntawm Heml

Koj yuav tsum tau teeb tsa Heml. Nws tuaj yeem rub tawm los ntawm qhov project GitHub repository. Nws yog ntsia los ntawm unpacking lub archive coj mus rau koj architecture thiab ntxiv helm rau $PATH.

Nco ntsoov: version 3.0.0 ntawm Helm tau tso tawm tsis ntev los no. Txij li thaum muaj ntau yam kev hloov pauv rau nws, tus nyeem ntawv tau qhia kom tos me ntsis ua ntej siv nws..

Ntxiv ib qhov chaw rau Helm

Thawj kauj ruam yog ntxiv qhov "loki" repository siv cov lus txib hauv qab no:

$ helm add loki https://grafana.github.io/loki/charts

Tom qab ntawd, koj tuaj yeem tshawb nrhiav cov pob khoom hu ua "loki":

$ helm search loki

Tshwm sim:

loki/loki       0.17.2 v0.4.0 Loki: like Prometheus, but for logs.
loki/loki-stack 0.19.1 v0.4.0 Loki: like Prometheus, but for logs.
loki/fluent-bit 0.0.2  v0.0.1 Uses fluent-bit Loki go plugin for...
loki/promtail   0.13.1 v0.4.0 Responsible for gathering logs and...

Cov pob no muaj cov yam ntxwv hauv qab no:

  • pob loki/loki phim Loki server nkaus xwb
  • pob loki/fluent-bit tso cai rau koj xa mus rau DaemonSet siv fluent-bin los sau cov cav es tsis txhob Promtail
  • pob loki/promtail muaj tus neeg sawv cev sau cov ntaub ntawv
  • pob loki/loki-stack, tso cai rau koj tam sim xa Loki ua ke nrog Promtail.

Txhim kho Loki

Txhawm rau xa Loki rau Kubernetes, khiav cov lus txib hauv qab no hauv "saib xyuas" namespace:

$ helm upgrade --install loki loki/loki-stack --namespace monitoring

Txhawm rau txuag rau disk, ntxiv qhov parameter --set loki.persistence.enabled = true:

$ helm upgrade --install loki loki/loki-stack 
              --namespace monitoring 
              --set loki.persistence.enabled=true

Nco ntsoov: Yog tias koj xav xa Grafana tib lub sijhawm, tom qab ntawd ntxiv qhov ntsuas --set grafana.enabled = true

Thaum koj khiav cov lus txib no koj yuav tsum tau txais cov zis hauv qab no:

LAST DEPLOYED: Tue Nov 19 15:56:54 2019
NAMESPACE: monitoring
STATUS: DEPLOYED
RESOURCES:
==> v1/ClusterRole
NAME AGE
loki-promtail-clusterrole 189d
…
NOTES:
The Loki stack has been deployed to your cluster. Loki can now be added as a datasource in Grafana.
See <a href="http://docs.grafana.org/features/datasources/loki/">http://docs.grafana.org/features/datasources/loki/</a> for more details.

Saib ntawm lub xeev ntawm cov pods hauv "kev saib xyuas" namespace, peb tuaj yeem pom tias txhua yam yog siv:

$ kubectl -n monitoring get pods -l release=loki

Tshwm sim:

NAME                 READY  STATUS   RESTARTS  AGE
loki-0               1/1    Running  0         147m
loki-promtail-9zjvc  1/1    Running  0         3h25m
loki-promtail-f6brf  1/1    Running  0         11h
loki-promtail-hdcj7  1/1    Running  0         3h23m
loki-promtail-jbqhc  1/1    Running  0         11h
loki-promtail-mj642  1/1    Running  0         62m
loki-promtail-nm64g  1/1    Running  0         24m

Tag nrho cov pods khiav. Tam sim no nws yog lub sijhawm los ua qee qhov kev sim!

Txuas rau Grafana

Txhawm rau txuas mus rau Grafana nyob rau hauv Kubernetes, koj yuav tsum qhib lub qhov rau nws cov pod. Hauv qab no yog cov lus txib kom qhib chaw nres nkoj 3000 rau Grafana pod:

$ kubectl -n port-forward monitoring svc/loki-grafana 3000:80

Lwm qhov tseem ceeb yog qhov yuav tsum tau rov qab Grafana tus thawj tswj password. Tus password yog khaws cia zais cia loki-grafana hauv teb .data.admin-user hauv base64 format.

Txhawm rau kho nws, koj yuav tsum khiav cov lus txib hauv qab no:

$ kubectl -n monitoring get secret loki-grafana 
 --template '{{index .data "admin-password" | base64decode}}'; echo

Siv tus password no ua ke nrog tus thawj tswj hwm tus account (admin).

Txhais Loki cov ntaub ntawv hauv Grafana

Ua ntej tshaj plaws, xyuas kom meej tias Loki cov ntaub ntawv tau raug tsim (Configuration/Datasource).
Ntawm no yog ib qho piv txwv:

Loki - log sau siv Prometheus mus kom ze
Piv txwv ntawm kev teeb tsa cov ntaub ntawv rau Loki

Los ntawm txhaj rau "Test" koj tuaj yeem tshawb xyuas qhov txuas nrog Loki.

Ua kev thov rau Loki

Tam sim no mus rau Grafana mus rau ntu "Xav". Thaum tau txais cov cav los ntawm cov thawv, Loki ntxiv cov metadata los ntawm Kubernetes. Yog li, nws tuaj yeem pom cov cav ntawm ib lub thawv tshwj xeeb.

Piv txwv li, txhawm rau xaiv cov thawv ntim khoom promtail, koj tuaj yeem siv cov lus nug hauv qab no: {container_name = "promtail"}.
Ntawm no kuj tseem nco ntsoov xaiv Loki cov ntaub ntawv qhov chaw.

Cov lus nug no yuav rov qab ntim cov dej num raws li hauv qab no:

Loki - log sau siv Prometheus mus kom ze
Cov lus nug tshwm sim hauv Grafana

Ntxiv rau dashboard

Pib nrog Grafana 6.4, koj tuaj yeem tso cov ntaub ntawv nkag ncaj qha rau ntawm lub dashboard. Tom qab ntawd, tus neeg siv yuav tuaj yeem hloov pauv sai ntawm tus lej ntawm kev thov ntawm nws qhov chaw thiab daim ntawv thov.

Hauv qab no yog ib qho piv txwv ntawm dashboard uas siv qhov kev sib tshuam no:

Loki - log sau siv Prometheus mus kom ze
Qauv dashboard nrog Prometheus metrics thiab Loki cav

Yav tom ntej ntawm Loki

Kuv pib siv Loki rov qab rau lub Tsib Hlis / Lub Rau Hli nrog version 0.1. Niaj hnub no version 1, thiab txawm 1.1 thiab 1.2 twb tau tso tawm.

Nws yuav tsum raug lees paub tias version 0.1 tsis ruaj khov txaus. Tab sis 0.3 twb pom cov cim tiag tiag ntawm kev loj hlob, thiab cov ntawv txuas ntxiv (0.4, tom qab ntawd 1.0) tsuas yog txhawb nqa qhov kev xav no.

Tom qab 1.0.0, tsis muaj leej twg tuaj yeem muaj kev zam txim tsis siv cov cuab yeej zoo no.

Kev txhim kho ntxiv yuav tsum tsis yog hais txog Loki, tab sis nws txoj kev koom ua ke nrog Grafana zoo heev. Qhov tseeb, Grafana 6.4 twb muaj kev koom ua ke zoo nrog dashboards.

Grafana 6.5, uas tau tso tawm tsis ntev los no, ntxiv txhim kho qhov kev sib koom ua ke los ntawm kev lees paub cov ntsiab lus hauv JSON hom.

Cov yeeb yaj kiab hauv qab no qhia txog qhov piv txwv me me ntawm cov txheej txheem no:

Loki - log sau siv Prometheus mus kom ze
Siv Loki cov hlua nthuav tawm hauv Grafana

Nws tuaj yeem siv ib qho ntawm JSON teb, piv txwv li, rau:

  • txuas mus rau lwm lub cuab yeej
  • lim cov ntsiab lus

Piv txwv li, koj tuaj yeem nyem rau ntawm traceId mus rau Zipkin lossis Jaeger.

Raws li ib txwm, peb tos ntsoov rau koj cov lus thiab caw koj mus qhib webinar, qhov twg peb yuav tham txog yuav ua li cas DevOps kev lag luam tau tsim nyob rau hauv 2019 thiab sib tham txog txoj kev txhim kho rau xyoo 2020.

Tau qhov twg los: www.hab.com