Loki - kusonkhanitsa zipika pogwiritsa ntchito njira ya Prometheus

Moni, okhala ku Khabrovsk! Madzulo oyambira kulembetsa kwatsopano kwa maphunzirowa "Zochita ndi zida za DevOps" Takukonzerani kumasulira kwazinthu zosangalatsa kwa inu.

Nkhaniyi ndi mawu oyamba achidule a Loki. Project Loki mothandizidwa ndi Grafana ndipo cholinga chake ndi kusonkhanitsa zipika zapakati (kuchokera ku maseva kapena zotengera).

Kudzoza kwakukulu kwa Loki kunali Prometheus ndi lingaliro logwiritsa ntchito njira zake pakuwongolera mitengo:

• kugwiritsa ntchito zilembo kusunga deta
• kugwiritsa ntchito zinthu zochepa

Tibwereranso momwe Prometheus amagwirira ntchito ndikupereka zitsanzo za momwe amagwiritsidwira ntchito potengera Kubernetes.

Mawu ochepa za Prometheus

Kuti mumvetsetse momwe Loki amagwirira ntchito, ndikofunikira kuti mubwerere ndikukumbukira pang'ono za Prometheus.

Chimodzi mwazinthu zodziwika bwino za Prometheus ndikuchotsa ma metric kuchokera kumalo osonkhanitsira (kudzera mwa otumiza kunja) ndikusunga mu TSDB (Time Series Data Base), ndikuwonjezera metadata mu mawonekedwe a zilembo.

Chifukwa chiyani chikufunika?

Posachedwapa, Prometheus wakhala muyeso wapadziko lonse wa zotengera ndi Kubernetes: kukhazikitsa kwake ndikosavuta, ndipo gulu la Kubernetes limabwera ndi mathero a Prometheus. Prometheus imathanso kuchotsa ma metrics kuchokera kuzinthu zomwe zayikidwa mu chidebe ndikusunga zolemba zina. Chifukwa chake, kuyang'anira ntchito ndikosavuta kugwiritsa ntchito.

Tsoka ilo, palibe yankho la turnkey pakuwongolera chipika, ndipo muyenera kupeza yankho lanu:

• ntchito yoyang'anira mitambo yoyika zipika (AWS, Azure kapena Google)
• ntchito yowunikira "kuyang'anira ngati ntchito" (mwachitsanzo, Datadog)
• kupanga ntchito yanu yotolera zipika.

Pachisankho chachitatu, ndimakonda kugwiritsa ntchito Elasticsearch, ngakhale sindinkasangalala nayo nthawi zonse (makamaka kulemera kwake komanso kusinthika kwake).

Loki idapangidwa kuti ikhale yosavuta kugwiritsa ntchito motsatira mfundo izi:

• kukhala kosavuta kuyamba
• amawononga zinthu zochepa
• ntchito paokha popanda kukonza mwapadera
• amagwira ntchito ngati chowonjezera ku Prometheus kuti athandizire pakufufuza kwa cholakwika

Komabe, kuphweka kumeneku kumabwera chifukwa cha kusagwirizana kwina. Chimodzi mwa izo ndikusalozera zomwe zili. Choncho, kufufuza malemba sikothandiza kwambiri kapena kolemera ndipo sikulola ziwerengero za zomwe zili m'mawuwo. Koma popeza Loki akufuna kukhala wofanana ndi grep komanso wothandizira Prometheus, izi sizoyipa.

Kufufuza kwa Zochitika

Kuti timvetse bwino chifukwa chomwe Loki safunikira kulondolera, tiyeni tibwerere ku njira yofufuzira zomwe okonza Loki adagwiritsa ntchito:

1 Chenjezo → 2 Dashboard → 3 Adhoc Query → 4 Log Aggregation → 5 Kutsata Kugawa → 6 Konzani!
(Chenjezo 1 → Dashboard 2 → 3 Adhoc Query → 4 Log Aggregation → 5 Distributed tracing → 6 Kukonza!)

Lingaliro ndiloti timalandira chenjezo lamtundu wina (Slack Notification, SMS, etc.) ndipo pambuyo pake:

• yang'anani pa Grafana dashboards
• yang'anani ma metric a utumiki (mwachitsanzo, ku Prometheus)
• yang'anani zolemba (mwachitsanzo, mu Elasticsearch)
• mwina yang'anani zotsatiridwa (Jaeger, Zipkin, etc.)
• ndipo potsiriza kukonza vuto loyamba.

Apa, pankhani ya Grafana + Prometheus + Elasticsearch + Zipkin stack, muyenera kugwiritsa ntchito zida zinayi zosiyanasiyana. Kuti muchepetse nthawi, zingakhale bwino kuti mutha kuchita zonsezi pogwiritsa ntchito chida chimodzi: Grafana. Ndikoyenera kudziwa kuti njira iyi yofufuzira yakhazikitsidwa ku Grafana kuyambira mtundu wa 6. Choncho, zimakhala zotheka kupeza deta ya Prometheus mwachindunji kuchokera ku Grafana.

Screen Explorer idagawanika pakati pa Prometheus ndi Loki

Kuchokera pazenerali mutha kuwona zipika za Loki zokhudzana ndi ma metrics a Prometheus pogwiritsa ntchito lingaliro logawanika. Popeza mtundu wa 6.5, Grafana amakulolani kuti muwonetse id muzolemba za Loki kuti mutsatire maulalo a zida zomwe mumakonda zotsatiridwa (Jaeger).

Mayeso a Local Loki

Njira yosavuta yoyesera Loki kwanuko ndikugwiritsa ntchito docker-compose. Fayilo ya docker-compose ili m'malo a Loki. Mutha kupeza chosungira pogwiritsa ntchito lamulo ili git:

$ git clone https://github.com/grafana/loki.git

Kenako muyenera kupita ku chikwatu chopanga:

$ cd production

Pambuyo pake mutha kupeza zithunzi zaposachedwa za Docker:

$ docker-compose pull

Pomaliza, stack ya Loki imayambitsidwa ndi lamulo ili:

$ docker-compose up

Zomangamanga za Loki

Nachi chithunzi chaching'ono chokhala ndi kamangidwe ka Loki:

Mfundo Zomangamanga za Loki

Wothandizira pa intaneti amayendetsa mapulogalamu pa seva, Promtail amasonkhanitsa zipika ndikuzitumiza ku Loki, kasitomala amatumizanso metadata ku Loki. Loki amaphatikiza zonse ndikuzitumiza ku Grafana.
Loki inayambika. Kuti muwone zigawo zomwe zilipo, yesani lamulo ili:

$ docker ps

Pankhani ya Docker yokhazikitsidwa kumene, lamulo liyenera kubweza zotsatirazi:

IMAGE               PORTS                  NAMES
grafana/promtail:                          production_promtail_1
grafana/grafana: m  0.0.0.0:3000->3000/tcp production_grafana_1
grafana/loki: late  80/tcp,0.0.0.0:3100... production_loki_1

Timawona zigawo zotsatirazi:

• Promtail: wothandizira omwe ali ndi udindo woyang'anira zipika
• Grafana: chida chodziwika bwino cha dashboard
• Loki: Data Centralization Daemon

Monga gawo lachitukuko chapamwamba (mwachitsanzo, kutengera makina enieni), wothandizira wa Promtail ayenera kutumizidwa pamakina aliwonse. Grafana ndi Loki akhoza kuikidwa pa makina omwewo.

Kutumizidwa ku Kubernetes

Kuyika zida za Loki pa Kubernetes zikhala motere:

• daemonSet potumiza wothandizira wa Promtail pamakina aliwonse omwe ali mugulu la seva
• Kutumiza Loki
• ndipo chomaliza ndi kutumizidwa kwa Grafana.

Mwamwayi, Loki ikupezeka ngati phukusi la Helm, ndikupangitsa kuti ikhale yosavuta kugwiritsa ntchito.

Kuyika kudzera pa Heml

Muyenera kukhala ndi Heml yoyikiratu. Itha kutsitsidwa kuchokera kumalo osungirako a GitHub. Imayikidwa ndikutsegula zakale zomwe zikugwirizana ndi zomangamanga zanu ndikuwonjezera helm $PATH.

Taonani: mtundu 3.0.0 wa Helm adatulutsidwa posachedwa. Popeza pakhala kusintha kochuluka kwa izo, wowerenga akulangizidwa kuti adikire pang'ono asanagwiritse ntchito..

Kuwonjezera gwero la Helm

Gawo loyamba ndikuwonjezera posungira "loki" pogwiritsa ntchito lamulo ili:

$ helm add loki https://grafana.github.io/loki/charts

Pambuyo pake, mutha kusaka phukusi lotchedwa "loki":

$ helm search loki

Zotsatira:

loki/loki       0.17.2 v0.4.0 Loki: like Prometheus, but for logs.
loki/loki-stack 0.19.1 v0.4.0 Loki: like Prometheus, but for logs.
loki/fluent-bit 0.0.2  v0.0.1 Uses fluent-bit Loki go plugin for...
loki/promtail   0.13.1 v0.4.0 Responsible for gathering logs and...

Maphukusiwa ali ndi izi:

• phukusi loki/loki ikufanana ndi seva ya Loki yokha
• phukusi loki/fluent-bit imakupatsani mwayi wotumiza DaemonSet pogwiritsa ntchito bwino-bin kuti mutenge zipika m'malo mwa Promtail
• phukusi loki/promtail ili ndi cholozera mafayilo a log
• phukusi loki/loki-stack, imakulolani kuti mutumize Loki nthawi yomweyo ndi Promtail.

Kukhazikitsa Loki

Kuti mutumize Loki ku Kubernetes, yendetsani lamulo ili mu "monitoring" namespace:

$ helm upgrade --install loki loki/loki-stack --namespace monitoring

Kuti musunge ku disk, yonjezerani parameter --set loki.persistence.enabled = true:

$ helm upgrade --install loki loki/loki-stack 
              --namespace monitoring 
              --set loki.persistence.enabled=true

Taonani: ngati mukufuna kutumiza Grafana nthawi yomweyo, yonjezerani parameter --set grafana.enabled = true

Mukayendetsa lamulo ili muyenera kupeza zotsatirazi:

LAST DEPLOYED: Tue Nov 19 15:56:54 2019
NAMESPACE: monitoring
STATUS: DEPLOYED
RESOURCES:
==> v1/ClusterRole
NAME AGE
loki-promtail-clusterrole 189d
…
NOTES:
The Loki stack has been deployed to your cluster. Loki can now be added as a datasource in Grafana.
See <a href="http://docs.grafana.org/features/datasources/loki/">http://docs.grafana.org/features/datasources/loki/</a> for more details.

Kuyang'ana momwe ma pod ali mu "monitoring" namespace, titha kuwona kuti zonse zayikidwa:

$ kubectl -n monitoring get pods -l release=loki

Zotsatira:

NAME                 READY  STATUS   RESTARTS  AGE
loki-0               1/1    Running  0         147m
loki-promtail-9zjvc  1/1    Running  0         3h25m
loki-promtail-f6brf  1/1    Running  0         11h
loki-promtail-hdcj7  1/1    Running  0         3h23m
loki-promtail-jbqhc  1/1    Running  0         11h
loki-promtail-mj642  1/1    Running  0         62m
loki-promtail-nm64g  1/1    Running  0         24m

Masamba onse akutha. Tsopano ndi nthawi yoti muyese!

Kulumikizana ndi Grafana

Kuti mulumikizane ndi Grafana pansi pa Kubernetes, muyenera kutsegula ngalande ku pod yake. Pansipa pali lamulo loti mutsegule doko 3000 la Grafana pod:

$ kubectl -n port-forward monitoring svc/loki-grafana 3000:80

Mfundo ina yofunika ndikufunika kuti achire Grafana administrator password. Mawu achinsinsi amasungidwa mwachinsinsi loki-grafana m'munda .data.admin-user mu mawonekedwe a base64.

Kuti mubwezeretse, muyenera kuyendetsa lamulo ili:

$ kubectl -n monitoring get secret loki-grafana 
 --template '{{index .data "admin-password" | base64decode}}'; echo

Gwiritsani ntchito mawu achinsinsiwa molumikizana ndi akaunti ya woyang'anira (admin).

Kufotokozera gwero la data la Loki ku Grafana

Choyamba, onetsetsani kuti gwero la data la Loki lapangidwa (Configuration/Datasource).
Nachi chitsanzo:

Chitsanzo chokhazikitsa gwero la data la Loki

Mwa kuwonekera pa "Mayeso" mutha kuyang'ana kulumikizana ndi Loki.

Kufunsira kwa Loki

Tsopano pitani ku Grafana ku gawo la "Explore". Mukalandira zipika kuchokera m'mitsuko, Loki amawonjezera metadata kuchokera Kubernetes. Chifukwa chake, zimakhala zotheka kuwona zipika za chidebe china.

Mwachitsanzo, kuti musankhe zolemba za promtail, mutha kugwiritsa ntchito funso ili: {container_name = "promtail"}.
Apanso kumbukirani kusankha gwero la data la Loki.

Funsoli libweza ntchito zotengera zinthu motere:

Zotsatira za funso ku Grafana

Onjezani ku dashboard

Kuyambira ndi Grafana 6.4, mutha kuyika zambiri za chipika pa dashboard. Pambuyo pake, wogwiritsa ntchitoyo adzatha kusintha mofulumira pakati pa chiwerengero cha zopempha pa tsamba lake ndi kufufuza ntchito.

Pansipa pali chitsanzo cha dashboard yomwe imagwiritsa ntchito izi:

Zitsanzo za dashboard yokhala ndi ma Prometheus metrics ndi Loki logs

Tsogolo la Loki

Ndidayamba kugwiritsa ntchito Loki m'mwezi wa Meyi / Juni ndi mtundu wa 0.1. Masiku ano mtundu 1, ndipo ngakhale 1.1 ndi 1.2 zatulutsidwa kale.

Ziyenera kuvomerezedwa kuti mtundu wa 0.1 sunali wokhazikika mokwanira. Koma 0.3 kale anasonyeza zizindikiro zenizeni za kukhwima, ndipo Mabaibulo wotsatira (0.4, ndiye 1.0) anangolimbitsa maganizo.

Pambuyo pa 1.0.0, palibe amene angakhale ndi chowiringula kuti asagwiritse ntchito chida chodabwitsachi.

Kuwongolera kwina sikuyenera kukhala za Loki, koma kuphatikiza kwake ndi Grafana wabwino kwambiri. M'malo mwake, Grafana 6.4 ili kale ndi kuphatikiza kwabwino ndi ma dashboard.

Grafana 6.5, yomwe idatulutsidwa posachedwa, imathandiziranso kuphatikiza uku pozindikira zokha zomwe zili mumtundu wa JSON.

Kanema pansipa akuwonetsa chitsanzo chaching'ono cha makina awa:

Kugwiritsa ntchito zingwe za Loki zowululidwa ku Grafana

Zimakhala zotheka kugwiritsa ntchito imodzi mwamagawo a JSON, mwachitsanzo, pa:

• maulalo ku chida chakunja
• kusefa zolembalemba

Mwachitsanzo, mutha kudina traceId kupita ku Zipkin kapena Jaeger.

Monga mwachizolowezi, tikuyembekezera ndemanga zanu ndikukuitanani tsegulani webinar, komwe tikambirana momwe makampani a DevOps adapangidwira mu 2019 ndikukambirana njira zomwe zingatheke chitukuko cha 2020.

Source: www.habr.com