ProHoster > Ibhulogi > Ulawulo > Loki - log collection usebenzisa indlela Prometheus

Loki - log collection usebenzisa indlela Prometheus

Molweni, bahlali baseKhabrovsk! Ngobusuku bangaphambi kokuqala kobhaliso olutsha lwekhosi "Iinkqubo zeDevOps kunye nezixhobo" Sikulungiselele inguqulelo yezinto ezinomdla kuwe.

Eli nqaku liyintshayelelo emfutshane yeLoki. Iprojekthi Loki ixhaswa nguGrafana kwaye ijolise ekuqokeleleni okuphakathi kweelog (ezisuka kwiiseva okanye kwizikhongozeli).

Eyona nto yayiphefumlelwe nguLoki Prometheus ngombono wokusebenzisa iindlela zakhe zolawulo lokungena:

  • usebenzisa iilebhile ukugcina idatha
  • ukusetyenziswa kwemithombo embalwa

Siza kubuyela kwindlela i-Prometheus esebenza ngayo kwaye sinike imizekelo yokusetyenziswa kwayo kumxholo we-Kubernetes.

Amagama ambalwa malunga Prometheus

Ukuqonda ngokupheleleyo indlela uLoki asebenza ngayo, kubalulekile ukuthatha inyathelo umva kwaye ukhumbule kancinci malunga nePrometheus.

Enye yezinto eziphawulekayo zePrometheus kukukhutshwa kweemetriki kwiindawo zokuqokelela (ngokuthunyelwa kwamanye amazwe) kwaye zigcinwe kwi-TSDB (i-Time Series Data Base), kunye nokongezwa kwemethadatha ngendlela yeelebula.

Kutheni ufuna

Kutshanje, i-Prometheus ibe ngumgangatho we-de facto kwihlabathi leekhonteyina kunye ne-Kubernetes: ukufakwa kwayo kulula kakhulu, kunye neqela le-Kubernetes liza nesiphelo semveli se-Prometheus. I-Prometheus inokukhupha i-metrics kwizicelo ezifakwe kwi-container ngelixa igcina iilebhile ezithile. Ngoko ke, esweni isicelo kulula kakhulu ukuphumeza.

Ngelishwa, akukabikho sisombululo sesitshixo solawulo lwelogi, kwaye kuya kufuneka uzifumanele isisombululo:

  • Inkonzo yelifu elawulwayo yokufakwa phakathi kweelog (AWS, Azure okanye Google)
  • inkonzo yokubeka iliso “yokubeka iliso njengenkonzo” (umzekelo, iDatadog)
  • ukwenza eyakho inkonzo yengqokelela yelog.

Kukhetho lwesithathu, ngokwesiko ndandisebenzisa i-Elasticsearch, nangona ndandingasoloko ndonwabile ngayo (ingakumbi ubunzima bayo kunye nobunzima boqwalaselo).

I-Loki yenzelwe ukwenza lula ukuphunyezwa ngokungqinelana nale migaqo ilandelayo:

  • kube lula ukuqalisa
  • sebenzisa izixhobo ezimbalwa
  • sebenza ngokuzimeleyo ngaphandle kokugcinwa okukhethekileyo
  • sebenza njengokongeza kwiPrometheus ukunceda kuphando lwebug

Nangona kunjalo, oku kulula kuza ngenxa yokulalanisa okuthile. Enye yazo ayikuko ukukhomba umxholo. Ngoko ke, ukukhangela okubhaliweyo akusebenzi kakuhle okanye kutyebile kwaye akuvumeli izibalo kumxholo wombhalo. Kodwa ekubeni uLoki efuna ukufana ne-grep kunye nomphelelisi we-Prometheus, oku ayisiyonto ingalunganga.

UPhando lwezehlo

Ukuqonda ngcono ukuba kutheni uLoki engadingi indexing, masibuyele kwindlela yophando lwesiganeko esetyenziswe ngabaphuhlisi beLoki:

Loki - log collection usebenzisa indlela Prometheus
1 Isilumkiso → 2 Dashboard → 3 Adhoc Umbuzo → 4 Udityaniso lweLog → 5 Ukuhanjiswa Komkhondo → 6 Lungisa!
(1 Isilumkiso → 2 Dashboard → 3 Adhoc Umbuzo → 4 Udibaniso lweLog → 5 Ukulandelela Ukuhanjiswa → 6 Ukulungisa!)

Umbono kukuba sifumana uhlobo oluthile lwesilumkiso (Isaziso seSlack, iSMS, njl.njl.) kwaye emva koko:

  • jonga kwiideshibhodi zaseGrafana
  • jonga kwiimetrics zenkonzo (umzekelo, kwiPrometheus)
  • jonga amangeniso elog (umzekelo, kwi-Elasticsearch)
  • mhlawumbi ujonge imikhondo esasaziweyo (uJaeger, uZipkin, njl.)
  • kwaye ekugqibeleni ukulungisa ingxaki yokuqala.

Apha, kwimeko yeGrafana + Prometheus + Elasticsearch + Zipkin stack, kuya kufuneka usebenzise izixhobo ezine ezahlukeneyo. Ukunciphisa ixesha, kuya kuba kuhle ukuba ukwazi ukwenza onke la manyathelo usebenzisa isixhobo esinye: Grafana. Kubalulekile ukuqaphela ukuba le ndlela yophando iphunyezwe eGrafana ukususela kwinguqulo 6. Ngaloo ndlela, kuba lula ukufikelela kwidatha ye-Prometheus ngokuthe ngqo eGrafana.

Loki - log collection usebenzisa indlela Prometheus
Isikrini somhloli sahlulahlulwe phakathi kwePrometheus kunye neLoki

Ukusuka kwesi sikrini unokujonga iilog zeLoki ezinxulumene neemetrics zePrometheus usebenzisa ingqikelelo yesikrini sokwahlula. Ukusukela kwinguqulo 6.5, iGrafana ikuvumela ukuba ucazulule i-id yomkhondo kumangenelo elogi ye-Loki ukulandela amakhonkco kwizixhobo zakho zomkhondo ezisasaziweyo (Jaeger).

Uvavanyo lweLoki yendawo

Eyona ndlela ilula yokuvavanya uLoki ekuhlaleni kukusebenzisa idocker-compose. Ifayile yokuqamba idokhi ibekwe kwindawo yokugcina iLoki. Unokufumana indawo yokugcina usebenzisa lo myalelo ulandelayo git:

$ git clone https://github.com/grafana/loki.git

Emva koko kufuneka uye kuluhlu lwemveliso:

$ cd production

Emva koku unokufumana inguqulelo yamva nje yemifanekiso yeDocker:

$ docker-compose pull

Ekugqibeleni, isitaki seLoki siqaliswa ngalo myalelo ulandelayo:

$ docker-compose up

Uyilo lweLoki

Nanku umzobo omncinci onoyilo lweLoki:

Loki - log collection usebenzisa indlela Prometheus
Loki Architecture Imigaqo

Umxhasi wewebhu uqhuba izicelo kumncedisi, i-Promtail iqokelela iilogi kwaye ithumele kwi-Loki, umxhasi wewebhu uphinde uthumele imethadatha kwi-Loki. ULoki udibanisa yonke into aze ayidlulisele eGrafana.
I-Loki iqalisiwe. Ukujonga amacandelo akhoyo, sebenzisa lo myalelo ulandelayo:

$ docker ps

Kwimeko yeDocker esanda kufakwa, umyalelo kufuneka ubuyisele ezi ziphumo zilandelayo:

IMAGE               PORTS                  NAMES
grafana/promtail:                          production_promtail_1
grafana/grafana: m  0.0.0.0:3000->3000/tcp production_grafana_1
grafana/loki: late  80/tcp,0.0.0.0:3100... production_loki_1

Sibona amacandelo alandelayo:

  • I-Promtail: i-arhente enoxanduva lokubeka kumbindi welog
  • Grafana: isixhobo esidumileyo sedeshibhodi
  • Loki: Data Centralization Daemon

Njengenxalenye yeziseko zophuhliso zakudala (umzekelo, ngokusekelwe koomatshini bokwenene), i-arhente ye-Promtail kufuneka isetyenziswe kumatshini ngamnye. IGrafana kunye neLoki zinokufakelwa kumatshini omnye.

Ukusasazwa kwi-Kubernetes

Ukufaka izinto zeLoki kwiKubernetes ziya kuba ngolu hlobo lulandelayo:

  • daemonSet yokusasaza iarhente yePromtail kumatshini ngamnye kwiqela leseva.
  • Ukusasazwa Loki
  • kwaye eyokugqibela kukusasazwa kweGrafana.

Ngethamsanqa, uLoki uyafumaneka njengephakheji yeHelm, eyenza kube lula ukuyihambisa.

Ukufakela ngeHeml

Kuya kufuneka ukuba sele unayo iHeml efakiweyo. Inokukhutshelwa kwindawo yokugcina iprojekthi yeGitHub. Ifakwe ngokukhupha i-archive ehambelana ne-architecture yakho kunye nokongeza i-helm kuyo $PATH.

Qaphela: inguqulo 3.0.0 yeHelm ikhutshwe kutshanje. Kuba kukho utshintsho oluninzi kuyo, umfundi uyacetyiswa ukuba alinde kancinci ngaphambi kokuyisebenzisa..

Ukongeza umthombo weHelm

Inyathelo lokuqala kukongeza indawo yokugcina "loki" usebenzisa lo myalelo ulandelayo:

$ helm add loki https://grafana.github.io/loki/charts

Emva koku, unokukhangela iipakethe ezinegama elithi "loki":

$ helm search loki

Isiphumo:

loki/loki       0.17.2 v0.4.0 Loki: like Prometheus, but for logs.
loki/loki-stack 0.19.1 v0.4.0 Loki: like Prometheus, but for logs.
loki/fluent-bit 0.0.2  v0.0.1 Uses fluent-bit Loki go plugin for...
loki/promtail   0.13.1 v0.4.0 Responsible for gathering logs and...

Ezi phakheji zinezi mpawu zilandelayo:

  • ipakethe loki/loki ihambelana nomncedisi we Loki kuphela
  • ipakethe i-loki/i-fluent-bit ikuvumela ukuba usebenzise i-DaemonSet usebenzisa umgqomo otyibilikayo ukuqokelela iilog endaweni ye-Promtail
  • ipakethe loki/promtail iqulathe umenzeli wokuqokelela ifayile yelog
  • ipakethe loki/loki-stack, ikuvumela ukuba uthumele ngokukhawuleza i-Loki kunye ne-Promtail.

Kufakwe iLoki

Ukuhambisa i-Loki ukuya ku-Kubernetes, sebenzisa lo myalelo ulandelayo kwindawo "yokubeka iliso" kwindawo yamagama:

$ helm upgrade --install loki loki/loki-stack --namespace monitoring

Ukugcina kwidiski, yongeza iparamitha --set loki.persistence.enabled = true:

$ helm upgrade --install loki loki/loki-stack 
              --namespace monitoring 
              --set loki.persistence.enabled=true

Qaphela: ukuba ufuna ukusasaza iGrafana ngexesha elinye, yongeza iparameter --set grafana.enabled = true

Xa usenza lo myalelo kufuneka ufumane imveliso elandelayo:

LAST DEPLOYED: Tue Nov 19 15:56:54 2019
NAMESPACE: monitoring
STATUS: DEPLOYED
RESOURCES:
==> v1/ClusterRole
NAME AGE
loki-promtail-clusterrole 189d
…
NOTES:
The Loki stack has been deployed to your cluster. Loki can now be added as a datasource in Grafana.
See <a href="http://docs.grafana.org/features/datasources/loki/">http://docs.grafana.org/features/datasources/loki/</a> for more details.

Ukujonga imeko yeepods kwindawo "yokubeka iliso", sinokubona ukuba yonke into ibekiwe:

$ kubectl -n monitoring get pods -l release=loki

Isiphumo:

NAME                 READY  STATUS   RESTARTS  AGE
loki-0               1/1    Running  0         147m
loki-promtail-9zjvc  1/1    Running  0         3h25m
loki-promtail-f6brf  1/1    Running  0         11h
loki-promtail-hdcj7  1/1    Running  0         3h23m
loki-promtail-jbqhc  1/1    Running  0         11h
loki-promtail-mj642  1/1    Running  0         62m
loki-promtail-nm64g  1/1    Running  0         24m

Zonke iipods ziyasebenza. Ngoku lixesha lokwenza ezinye iimvavanyo!

Ukuqhagamshela eGrafana

Ukuqhagamshela kwiGrafana phantsi kweKubernetes, kufuneka uvule itonela kwipod yayo. Ngezantsi ngumyalelo wokuvula izibuko 3000 leGrafana pod:

$ kubectl -n port-forward monitoring svc/loki-grafana 3000:80

Enye ingongoma ebalulekileyo sisidingo sokufumana kwakhona igama eligqithisiweyo lomlawuli weGrafana. Igama eliyimfihlo ligcinwa liyimfihlo loki-grafana ebaleni .data.admin-user kwifomathi ye-base64.

Ukuyibuyisela, kufuneka usebenzise lo myalelo ulandelayo:

$ kubectl -n monitoring get secret loki-grafana 
 --template '{{index .data "admin-password" | base64decode}}'; echo

Sebenzisa eli gama lokugqithisa ngokudibanisa neakhawunti yomlawuli omiselweyo (admin).

Ukuchaza umthombo wedatha yeLoki eGrafana

Okokuqala, qiniseka ukuba umthombo wedatha weLoki udaliwe (Uqwalaselo/umthombo weDatha).
Nasi umzekelo:

Loki - log collection usebenzisa indlela Prometheus
Umzekelo wokumisela umthombo wedatha weLoki

Ngokucofa ku-"Vavanyo" ungajonga uqhagamshelo noLoki.

Ukwenza izicelo kuLoki

Ngoku yiya eGrafana kwindawo ethi "Jonga". Xa ufumana iilogi ezisuka kwizikhongozeli, uLoki wongeza imetadata evela Kubernetes. Ngaloo ndlela, kunokwenzeka ukujonga iilogi zesikhongozeli esithile.

Umzekelo, ukukhetha iilogi zesikhongozeli se-promtail, ungasebenzisa lo mbuzo ulandelayo: {container_name = "promtail"}.
Apha kwakhona khumbula ukukhetha umthombo wedatha yeLoki.

Lo mbuzo uzakubuyisela umsebenzi wesikhongozeli ngolu hlobo lulandelayo:

Loki - log collection usebenzisa indlela Prometheus
Iziphumo zombuzo eGrafana

Yongeza kwideshibhodi

Ukuqala ngeGrafana 6.4, ungabeka iinkcukacha zelog ngqo kwideshibhodi. Emva koko, umsebenzisi uya kuba nako ukutshintsha ngokukhawuleza phakathi kwenani lezicelo kwindawo yakhe kunye nesicelo sokulandelela.

Apha ngezantsi ngumzekelo wedeshbhodi esebenzisa olu nxibelelwano:

Loki - log collection usebenzisa indlela Prometheus
Isampulu yedeshibhodi ene-Prometheus metrics kunye neelog zeLoki

Ikamva likaLoki

Ndiqale ukusebenzisa i-Loki emva ngoMeyi/Juni ngenguqulo 0.1. Namhlanje inguqulo 1, kunye ne-1.1 kunye ne-1.2 sele zikhutshiwe.

Makuvunywe ukuba inguqulelo 0.1 ibingazinzanga ngokwaneleyo. Kodwa i-0.3 sele ibonise iimpawu zangempela zokuvuthwa, kwaye iinguqulelo ezilandelayo (0.4, ngoko-1.0) zomeleza kuphela le ngcamango.

Emva kwe-1.0.0, akukho mntu unokuba nesizathu sokungasisebenzisi esi sixhobo simangalisayo.

Uphuculo olongezelelweyo akufanele lube malunga neLoki, kodwa kunoko ukuhlanganiswa kwayo kunye neGrafana ebalaseleyo. Enyanisweni, iGrafana 6.4 sele inokudibanisa okulungileyo kunye needeshibhodi.

I-Grafana 6.5, ekhutshwe kutshanje, iphucula ngakumbi oku kudityaniswa ngokuqonda ngokuzenzekelayo imixholo yelogi kwifomathi ye-JSON.

Le vidiyo ingezantsi ibonisa umzekelo omncinci wale ndlela:

Loki - log collection usebenzisa indlela Prometheus
Ukusebenzisa imitya yeLoki evezwe eGrafana

Kuyenzeka ukuba usebenzise enye yemihlaba ye-JSON, umzekelo, ku:

  • amakhonkco kwisixhobo sangaphandle
  • imixholo yelog yokucoca

Umzekelo, ungacofa kwi traceId ukuya Zipkin okanye Jaeger.

Njengesiqhelo, sijonge phambili kwizimvo zakho kwaye sikumema ukuba vula i-webinar, apho siza kuthetha malunga nendlela imboni ye-DevOps ephuhliswe ngayo ngexesha le-2019 kwaye sixoxe ngeendlela ezinokwenzeka zophuhliso lwe-2020.

umthombo: www.habr.com