ProHoster > Pūnaewele > Nā Administration > Loki - ka ʻohi ʻana i nā lāʻau me ka hoʻohana ʻana i ke ala Prometheus

Loki - ka ʻohi ʻana i nā lāʻau me ka hoʻohana ʻana i ke ala Prometheus

Aloha ʻoe, e nā Khabrovites! I ka kali ʻana i ka hoʻomaka ʻana o kahi kākau inoa hou no ka papa "Nā hana a me nā mea hana DevOps" ua hoʻomākaukau i kahi unuhi o nā mea hoihoi no ʻoe.

He hoʻolauna pōkole kēia ʻatikala iā Loki. Papahana Loki kākoʻo ʻia e Grafana a ke kuhikuhi ʻia nei i ka hōʻiliʻili kikowaena o nā lāʻau (mai nā kikowaena a i ʻole nā ​​​​mea pahu).

ʻO ka manaʻo nui no Loki IHILA me ka manaʻo o ka hoʻohana ʻana i kāna mau ala i ka hoʻokele log:

  • me ka hoʻohana ʻana i nā lepili e mālama i ka ʻikepili
  • haʻahaʻa hoʻohana waiwai

E hoʻi mākou i nā loina o Prometheus a hāʻawi i kekahi mau hiʻohiʻona o kona hoʻohana ʻana i ka pōʻaiapili o Kubernetes.

ʻO kekahi mau ʻōlelo e pili ana iā Prometheus

No ka hoʻomaopopo piha ʻana i ka hana ʻana o Loki, he mea nui e hoʻi i hope a kipa hou iā Prometheus.

ʻO kekahi o nā hiʻohiʻona ʻokoʻa o Prometheus ʻo ia ka unuhi ʻana o nā metric mai nā wahi hōʻiliʻili (ma o nā mea hoʻopukapuka) a mālama iā lākou i kahi TSDB (Time Series Data Base, time series database) me ka hoʻohui ʻana i nā metadata ma ke ʻano o nā lepili.

No ke aha e pono ai

I kēia mau lā, ua lilo ʻo Prometheus i ka maʻamau de facto i ka honua o nā ipu a me nā Kubernetes: maʻalahi loa kona hoʻokomo ʻana, a ʻo kahi hui Kubernetes i hoʻomaka i kahi hopena no Prometheus. Hiki iā Prometheus ke unuhi i nā metric mai nā noi i kau ʻia i loko o kahi pahu ʻoiai e mālama ana i nā lepili kikoʻī. No laila, maʻalahi loa ka hoʻokō ʻana i ka nānā ʻana i ka noi.

ʻO ka mea pōʻino, ʻaʻohe hopena turnkey no ka hoʻokele log, a pono ʻoe e ʻimi i kahi hopena no ʻoe iho:

  • lawelawe ʻia ke ao no ka hoʻonohonoho ʻana i nā lāʻau (AWS, Azure a i ʻole Google)
  • lawelawe nānā "ka nānā ʻana ma ke ʻano he lawelawe" (no ka laʻana, Datadog)
  • hana i kāu lawelawe hōʻiliʻili lāʻau ponoʻī.

No ke kolu o ka koho, ua hoʻohana maʻamau wau iā Elasticsearch, ʻoiai ʻaʻole wau i hauʻoli mau me ia (ʻoi aku ka kaumaha a me ka paʻakikī o ka hoʻonohonoho).

Ua hoʻolālā ʻia ʻo Loki e maʻalahi e hoʻokō e like me kēia mau loina:

  • e maʻalahi e hoʻomaka
  • e hoopau i na mea waiwai
  • hana kūʻokoʻa me ka mālama ʻole
  • e lilo i mea hoʻohui iā Prometheus e kōkua i ka hoʻokolokolo ʻana i nā bug

Eia naʻe, hiki mai kēia maʻalahi ma muli o kekahi mau hoʻohālikelike. ʻO kekahi o lākou ʻaʻole e kuhikuhi i ka ʻike. No laila, ʻaʻole maikaʻi loa ka ʻimi kikokikona a ʻaʻole e ʻae iā ʻoe e mālama i nā ʻikepili i ka ʻike o ka kikokikona. Akā no ka mea makemake ʻo Loki e like me ka grep a hoʻokō iā Prometheus, ʻaʻole ia he mea pōʻino.

Ka hoʻokolokolo ʻana

No ka hoʻomaopopo maikaʻi ʻana i ke kumu ʻaʻole pono ai ʻo Loki i ka kuhikuhi ʻana, e hoʻi kāua i ke ʻano hana noiʻi hanana i hoʻohana ʻia e nā mea hoʻomohala Loki:

Loki - ka ʻohi ʻana i nā lāʻau me ka hoʻohana ʻana i ke ala Prometheus
1 Alert → 2 Dashboard → 3 Adhoc Query → 4 Log Aggregation → 5 Distributed Tracing → 6 Fix!
(1 ʻŌlelo Aʻo → 2 Dashboard → 3 Adhoc Query → 4 Log Aggregation → 5 Distributed Tracing → 6 Fix!)

ʻO ka manaʻo e loaʻa iā mākou kekahi ʻano makaʻala (Slack Notification, SMS, etc.) a ma hope o kēlā:

  • e nānā i nā papa kuhikuhi Grafana
  • e nānā i nā metric lawelawe (no ka laʻana, ma Prometheus)
  • e nānā i nā hoʻokomo log (no ka laʻana, ma Elasticsearch)
  • e nānā paha i nā ʻāpana i hoʻolaha ʻia (Jaeger, Zipkin, etc.)
  • a hoʻoponopono hope i ka pilikia kumu.

Eia, ma ke ʻano o ka Grafana + Prometheus + Elasticsearch + Zipkin stack, pono ʻoe e hoʻohana i ʻehā mau mea hana like ʻole. No ka mālama ʻana i ka manawa, ʻoi aku ka maikaʻi o ka hiki ke hana i kēia mau ʻanuʻu me hoʻokahi mea hana: Grafana. Pono e hoʻomaopopo ʻia ua hoʻokō ʻia kēia ʻano noiʻi ma Grafana mai ka mana 6. No laila, hiki ke komo pololei i ka ʻikepili Prometheus mai Grafana.

Loki - ka ʻohi ʻana i nā lāʻau me ka hoʻohana ʻana i ke ala Prometheus
Ua hoʻokaʻawale ʻia ka pale Explorer ma waena o Prometheus a me Loki

Mai kēia pale, hiki iā ʻoe ke nānā i nā lāʻau ma Loki e pili ana i nā metric Prometheus me ka hoʻohana ʻana i ka manaʻo ʻāpana ʻāpana. Mai ka mana 6.5, ʻae ʻo Grafana iā ʻoe e hoʻokaʻawale i ka trace id ma Loki log entries e hahai i nā loulou i kāu mau mea hoʻohana punahele punahele (Jaeger).

Loki hoao kūloko

ʻO ke ala maʻalahi loa e hoʻāʻo iā Loki ma ka ʻāina ʻo ka hoʻohana ʻana i ka docker-compose. Aia ka faila docker-compose ma ka waihona Loki. Hiki iā ʻoe ke kiʻi i ka waihona me kēia kauoha git:

$ git clone https://github.com/grafana/loki.git

A laila pono ʻoe e hoʻololi i ka papa kuhikuhi hana:

$ cd production

Ma hope o kēlā, hiki iā ʻoe ke kiʻi i nā kiʻi Docker hou loa:

$ docker-compose pull

ʻO ka hope, ua hoʻomaka ka Loki stack me kēia kauoha:

$ docker-compose up

hale hoʻolālā Loki

Eia kekahi kiʻi liʻiliʻi me ka hoʻolālā Loki:

Loki - ka ʻohi ʻana i nā lāʻau me ka hoʻohana ʻana i ke ala Prometheus
Loki Architecture Principles

Hoʻohana ka mea kūʻai pūnaewele i nā noi ma ke kikowaena, ʻohi ʻo Promtail i nā lāʻau a hoʻouna iā lākou i Loki, hoʻouna pū ka mea kūʻai pūnaewele i nā metadata iā Loki. Hoʻohui ʻo Loki i nā mea a pau a hāʻawi iā Grafana.
Ke holo nei ʻo Loki. No ka ʻike ʻana i nā ʻāpana i loaʻa, e holo i kēia kauoha:

$ docker ps

I ka hihia o kahi Docker hou, pono ke kauoha e hoʻihoʻi i kēia huahana:

IMAGE               PORTS                  NAMES
grafana/promtail:                          production_promtail_1
grafana/grafana: m  0.0.0.0:3000->3000/tcp production_grafana_1
grafana/loki: late  80/tcp,0.0.0.0:3100... production_loki_1

ʻIke mākou i kēia mau ʻāpana:

  • Promtail: luna kuleana no ka hoʻonohonoho ʻana i nā lāʻau
  • ʻO Grafana: ka mea hana dashboard kaulana
  • Loki: daemon kikowaena data

Ma ke ʻano he ʻano hana maʻamau (no ka laʻana, e pili ana i nā mīkini virtual), pono e kau ʻia ka ʻelele Promtail ma kēlā me kēia mīkini. Hiki ke hoʻokomo ʻia ʻo Grafana a me Loki ma ka mīkini like.

Hoʻolālā i nā Kubernetes

ʻO ka hoʻokomo ʻana i nā ʻāpana Loki ma Kubernetes penei:

  • daemonSet e hoʻolālā i ka ʻelele Promtail ma kēlā me kēia mīkini i ka pūʻulu kikowaena
  • Hoʻokomo Loki
  • a ʻo ka mea hope loa, ʻo ia ke kau ʻana o Grafana.

ʻO ka mea pōmaikaʻi, loaʻa ʻo Loki ma ke ʻano he pūʻolo Helm, e maʻalahi ai ke kau ʻana.

Hoʻokomo ʻia ma o Heml

Pono ʻoe i hoʻokomo iā Heml. Hiki ke hoʻoiho ʻia mai ka waihona waihona ʻo GitHub. Hoʻokomo ʻia ia ma ka unuhi ʻana i ka waihona i kūpono i kāu hoʻolālā a hoʻohui i ka helm i $PATH.

'Ōlelo Aʻo: Ua hoʻokuʻu ʻia ka mana 3.0.0 o Helm. No ka mea he nui nā loli i loko o ia mea, ʻōlelo ʻia ka mea heluhelu e kali iki ma mua o ka hoʻomaka ʻana e hoʻohana..

Pākuʻi kumu no Helm

ʻO ka hana mua e hoʻohui i ka waihona "loki" me kēia kauoha:

$ helm add loki https://grafana.github.io/loki/charts

Ma hope o kēlā, hiki iā ʻoe ke ʻimi i nā pūʻolo i kapa ʻia ʻo "loki":

$ helm search loki

Hualoaʻa:

loki/loki       0.17.2 v0.4.0 Loki: like Prometheus, but for logs.
loki/loki-stack 0.19.1 v0.4.0 Loki: like Prometheus, but for logs.
loki/fluent-bit 0.0.2  v0.0.1 Uses fluent-bit Loki go plugin for...
loki/promtail   0.13.1 v0.4.0 Responsible for gathering logs and...

Loaʻa i kēia mau pūʻolo nā hiʻohiʻona penei:

  • pūʻolo loki/loki pili wale i ke kikowaena Loki
  • pūʻolo loki/fluent-bit hiki iā ʻoe ke kau iā DaemonSet me ka fluent-bin e hōʻiliʻili i nā lāʻau ma kahi o Promtail
  • pūʻolo loki/promtail loaʻa i kahi luna hōʻiliʻili lāʻau
  • pūʻolo loki/loki-stack, hiki iā ʻoe ke kau koke iā Loki me Promtail.

Ke hoʻokomo nei iā Loki

No ka hoʻokomo ʻana iā Loki i nā Kubernetes, e holo i kēia kauoha ma ka inoa "nānā":

$ helm upgrade --install loki loki/loki-stack --namespace monitoring

No ka mālama ʻana i ka disk, hoʻohui i ke koho --set loki.persistence.enabled = true:

$ helm upgrade --install loki loki/loki-stack 
              --namespace monitoring 
              --set loki.persistence.enabled=true

'Ōlelo Aʻo: inā makemake ʻoe e kau iā Grafana i ka manawa like, a laila e hoʻohui i ka ʻāpana --set grafana.enabled = true

Ke holo ʻoe i kēia kauoha, pono ʻoe e loaʻa i kēia huahana:

LAST DEPLOYED: Tue Nov 19 15:56:54 2019
NAMESPACE: monitoring
STATUS: DEPLOYED
RESOURCES:
==> v1/ClusterRole
NAME AGE
loki-promtail-clusterrole 189d
…
NOTES:
The Loki stack has been deployed to your cluster. Loki can now be added as a datasource in Grafana.
See <a href="http://docs.grafana.org/features/datasources/loki/">http://docs.grafana.org/features/datasources/loki/</a> for more details.

Ke nānā nei i ke kūlana o nā pods i ka inoa inoa "nānā", hiki iā mākou ke ʻike ua kau ʻia nā mea āpau:

$ kubectl -n monitoring get pods -l release=loki

Hualoaʻa:

NAME                 READY  STATUS   RESTARTS  AGE
loki-0               1/1    Running  0         147m
loki-promtail-9zjvc  1/1    Running  0         3h25m
loki-promtail-f6brf  1/1    Running  0         11h
loki-promtail-hdcj7  1/1    Running  0         3h23m
loki-promtail-jbqhc  1/1    Running  0         11h
loki-promtail-mj642  1/1    Running  0         62m
loki-promtail-nm64g  1/1    Running  0         24m

Ke holo nei nā pahu a pau. ʻO ka manawa kēia e hana ai i kekahi mau hoʻāʻo!

Hoʻopili iā Grafana

I mea e hoʻopili ai iā Grafana ma lalo o Kubernetes, pono ʻoe e wehe i kahi tunnel i kona pod. Eia ke kauoha e wehe i ke awa 3000 no kahi pod Grafana:

$ kubectl -n port-forward monitoring svc/loki-grafana 3000:80

ʻO kekahi mea koʻikoʻi ʻo ka pono e hoʻihoʻi i ka ʻōlelo huna luna o Grafana. Ua hūnā ʻia ka ʻōlelo huna loki-grafana ma ke kula .data.admin-user ma ka waihona base64.

No ka hoʻihoʻi ʻana, pono ʻoe e holo i kēia kauoha:

$ kubectl -n monitoring get secret loki-grafana 
 --template '{{index .data "admin-password" | base64decode}}'; echo

E hoʻohana i kēia ʻōlelo huna me ka mooolelo hoʻokele paʻamau (admin).

ʻO ka wehewehe ʻikepili Loki ma Grafana

ʻO ka mea mua, e hōʻoia i ka hoʻokumu ʻia ʻana o ka ʻikepili Loki (Configuration / Datasource).
Eia kekahi hiʻohiʻona:

Loki - ka ʻohi ʻana i nā lāʻau me ka hoʻohana ʻana i ke ala Prometheus
He laʻana o ka hoʻonohonoho ʻana i kahi kumu ʻikepili no Loki

Ma ke kaomi ʻana i ka "Test" hiki iā ʻoe ke hoʻāʻo i ka pilina me Loki.

Ke noi aku nei iā Loki

E hele i Grafana a hele i ka ʻāpana "Explore". I ka loaʻa ʻana o nā lāʻau mai nā pahu, hoʻohui ʻo Loki i nā metadata mai Kubernetes. No laila, hiki ke nānā i nā lāʻau o kahi pahu kikoʻī.

No ka laʻana, e koho i nā pahu pahu promtail, hiki iā ʻoe ke hoʻohana i kēia nīnau: {container_name = "promtail"}.
Mai poina e koho i ke kumu ikepili Loki maanei.

E hoʻihoʻi ana kēia nīnau i ka hana ipu e like me kēia:

Loki - ka ʻohi ʻana i nā lāʻau me ka hoʻohana ʻana i ke ala Prometheus
Ka hopena nīnau ma Grafana

Hoʻohui i ka dashboard

E hoʻomaka me Grafana 6.4, hiki ke kau pololei i ka ʻike log ma ka dashboard. Ma hope o kēlā, hiki i ka mea hoʻohana ke hoʻololi koke i waena o ka helu o nā noi ma kāna pūnaewele i nā ʻōkuhi noi.

Aia ma lalo kahi laʻana dashboard e hoʻokō i kēia pilina:

Loki - ka ʻohi ʻana i nā lāʻau me ka hoʻohana ʻana i ke ala Prometheus
Laʻana dashboard me Prometheus metrics a me Loki logs

ʻO ka wā e hiki mai ana o Loki

Ua hoʻomaka wau e hoʻohana iā Loki i Mei / Iune me ka mana 0.1. Ua hoʻokuʻu ʻia ka version 1 i kēia lā, a ʻo 1.1 a me 1.2.

Pono e ʻae ʻia ʻaʻole paʻa ka mana 0.1. Akā ua hōʻike mua ʻo 0.3 i nā hōʻailona maoli o ke oʻo, a ʻo nā mana hou (0.4, a laila 1.0) i hoʻoikaika i kēia manaʻo.

Ma hope o 1.0.0, ʻaʻole hiki i kekahi ke loaʻa ke kumu e hoʻohana ʻole ai i kēia mea hana kupanaha.

ʻAʻole pono nā hoʻomaikaʻi hou e pili ana iā Loki, akā ʻo kona hoʻohui pū ʻana me ka Grafana maikaʻi loa. ʻOiaʻiʻo, ua loaʻa iā Grafana 6.4 kahi hoʻohui maikaʻi me nā dashboards.

ʻO Grafana 6.5, i hoʻokuʻu ʻia i kēia manawa, hoʻomaikaʻi hou i kēia hoʻohui ʻana ma o ka ʻike ʻana i nā ʻike o nā logs ma JSON format.

Hōʻike ka wikiō ma lalo nei i kahi hiʻohiʻona liʻiliʻi o kēia mīkini:

Loki - ka ʻohi ʻana i nā lāʻau me ka hoʻohana ʻana i ke ala Prometheus
Ke hoʻohana nei i nā kaula Loki i unuhi ʻia ma Grafana

Hiki ke hoʻohana i kekahi o nā kahua JSON, no ka laʻana, i:

  • nā loulou i kahi mea hana waho
  • kānana maʻiʻo log

No ka laʻana, hiki iā ʻoe ke kaomi ma ka traceId e hele i Zipkin a i ʻole Jaeger.

E like me ka mea maʻamau, ke kakali nei mākou i kāu mau manaʻo a kono iā ʻoe e wehe i ka webinar, kahi e kamaʻilio ai mākou e pili ana i ka ulu ʻana o ka ʻoihana DevOps i ka makahiki 2019 a kūkākūkā i nā ala hoʻomohala hiki no 2020.

Source: www.habr.com