Ukuqapha sekuyingxenye ebaluleke kakhulu yezixazululo zamafu ezikhulayo njengoba ubunkimbinkimbi bezinhlelo ezisabalalisiwe bukhula. Kuyadingeka ukuqonda ukuziphatha kwabo. Sidinga amathuluzi angakala angaqoqa idatha kuwo wonke amasevisi - futhi anikeze ochwepheshe ngesixhumi esibonakalayo esisodwa esinokuhlaziya ukusebenza, ukuboniswa kwamaphutha, ukutholakala namalogi.
La mathuluzi afanayo kufanele asebenze kahle futhi akhiqize. Kulesi sihloko, sizobheka izitaki ezimbili zobuchwepheshe ezidumile: i-EFK (Elasticsearch) ne-PLG (Loki) futhi sihlole izakhiwo zabo kanye nokwehluka.
Isitaki se-EFK
Kungenzeka ukuthi usuzwile kakade nge-ELK noma i-EFK ethandwa kakhulu. Isitaki siqukethe izingxenye ezimbalwa ezihlukene: I-Elasticsearch (isitoreji sento), i-Logstash noma i-FluentD (ukuqoqwa kwelogi nokuhlanganisa), kanye ne-Kibana yokubuka ngeso lengqondo.
Uhlelo lokusebenza olujwayelekile lubukeka kanje:
Islastiki - isitoreji sento esabalalisiwe ngokusesha kanye nezibalo zesikhathi sangempela. Isixazululo esihle kakhulu sedatha esakhiwe kancane njengamalogi. Ulwazi lulondolozwa njengamadokhumenti e-JSON, akhonjwa ngesikhathi sangempela futhi lusatshalaliswa kuwo wonke ama-cluster node. Kusetshenziswa inkomba ehlanekezelwe, equkethe wonke amagama ayingqayizivele kanye nemibhalo ehlobene nosesho lombhalo ogcwele, nawo osuselwe kunjini yokusesha ye-Apache Lucene.
FluentD ingumqoqi wedatha ohlanganisa idatha lapho iqoqwa futhi isetshenziswa. Izama ukuhlela idatha ku-JSON ngangokunokwenzeka. Ukwakheka kwayo kuyanwebeka, kukhona nokunye , esekelwa umphakathi, kuzo zonke izikhathi.
Kibana - ithuluzi lokubuka idatha le-Elasticsearch elinamakhono ahlukahlukene angeziwe, isibonelo, ukuhlaziya uchungechunge lwesikhathi, ukuhlaziya igrafu, ukufunda ngomshini nokunye.
I-Elasticsearch Architecture
Idatha yeqoqo le-Elasticsearch igcinwa isabalale kuzo zonke izindawo zayo. Iqoqo liqukethe amanodi amaningi ukuze kuthuthukiswe ukutholakala nokuqina. Noma iyiphi i-node ingenza zonke izindima zeqoqo, kodwa ekusetshenzisweni kwesilinganiso esikhulu, ama-node ngokuvamile anikezwa imisebenzi ngayinye.
Izinhlobo ze-Cluster node:
- i-master node - ilawula iqoqo, okungenani ezintathu ziyadingeka, eyodwa ihlala isebenza;
- idatha yedatha - igcina idatha ekhonjiwe futhi yenza imisebenzi ehlukahlukene ngayo;
- i-ingest node - ihlela amapayipi okuguqula idatha ngaphambi kokufakwa ohlwini;
- i-node yokuxhumanisa - izicelo zomzila, ukunciphisa isigaba sokucutshungulwa kokusesha, ukuxhumanisa ukukhomba ngobuningi;
- i-node yokuxwayisa - ukwethula imisebenzi yokuxwayisa;
- indawo yokufunda yomshini - ukucubungula imisebenzi yokufunda yomshini.
Umdwebo ongezansi ubonisa ukuthi idatha igcinwa futhi iphindaphindwe kanjani kuwo wonke ama-node ukuze kuzuzwe ukutholakala kwedatha okuphezulu.
Idatha yekhophi ngayinye igcinwa kunkomba ehlanekezelwe, umdwebo ongezansi ubonisa ukuthi lokhu kwenzeka kanjani:
setting
Imininingwane ingabukwa , ngizosebenzisa ishadi lesigqoko:
$ helm install efk-stack stable/elastic-stack --set logstash.enabled=false --set fluentd.enabled=true --set fluentd-elasticsIsitaki se-PLG
Ungamangali uma ungasitholi lesi sifinyezo, njengoba saziwa kangcono ngeleGrafana Loki. Kunoma yikuphi, lesi sitaki sithola ukuthandwa ngoba sisebenzisa izixazululo zobuchwepheshe ezifakazelwe. Kungenzeka ukuthi usuzwile kakade nge-Grafana, ithuluzi lokubuka elidumile. Abadali bayo, abagqugquzelwe u-Prometheus, bathuthukise i-Loki, uhlelo lokuhlanganisa amalogi oluvundlile, olusebenza kahle kakhulu. I-Loki ikhomba imethadatha kuphela, hhayi amajenali ngokwawo, isisombululo sobuchwepheshe esivumela ukuthi kube lula ukuyisebenzisa futhi kungabizi.
I-Promtail - i-ejenti yokuthumela amalogi kusukela kusistimu yokusebenza kuya kuqoqo le-Loki. UGrafana iyithuluzi lokubuka ngeso elisekelwe kudatha evela ku-Loki.
I-Loki yakhelwe phezu kwezimiso ezifanayo neka-Prometheus, okuyenza ifaneleke kahle ukugcina nokuhlaziya izingodo ze-Kubernetes.
Loki Architecture
I-Loki ingaqhutshwa njengenqubo eyodwa noma njengezinqubo eziningi, okuvumela ukukala okuvundlile.
Ingasebenza futhi njengohlelo lokusebenza lwe-monolithic noma njenge-microservice. Ukugijima njengenqubo eyodwa kungaba usizo ekuthuthukisweni kwendawo noma ekuqapheni okuncane. Ukuze kusetshenziswe izimboni kanye nomthwalo okhulayo wokusebenza, kunconywa ukusebenzisa inketho ye-microservice. Izindlela zokubhala nokufunda idatha zihlukanisiwe, ngakho-ke ingashunwa kahle futhi ikalwe njengoba kudingeka.
Ake sibheke ukwakheka kohlelo lokuqoqwa kwelogi ngaphandle kokungena emininingwaneni:
Futhi nansi incazelo (i-microservice architecture):
Izingxenye:
I-Promtail - i-ejenti efakwe kuma-node (njengeqoqo lezinsizakalo), isusa amalogi emisebenzini futhi ifinyelele i-Kubernetes API ukuze ithole imethadatha ezomaka izingodo. Ibese ithumela ilogu kusevisi ye-Loki eyinhloko. Imephu yemethadatha isekela imithetho yokumaka efanayo ne-Prometheus.
Ukusabalalisa - umsabalalisi wesevisi osebenza njengesilondolozi. Ukuze kucutshungulwe izigidi zamarekhodi, ipakisha idatha engenayo, iyicindezele ngamabhulokhi njengoba ifika. Osinki bedatha abambalwa basebenza ngesikhathi esisodwa, kodwa amalogu okugeleza kwedatha okukodwa okungenayo kufanele avele kuphela kokukodwa kwawo kuwo wonke amabhulokhi awo. Lokhu kuhlelwe kwaba indandatho yamasinki kanye ne-hashing elandelanayo. Ngokubekezelela amaphutha nokuphindaphinda, lokhu kwenziwa izikhathi ezingu-n (3 uma kungalungiselelwe).
Ingester - umamukeli wesevisi. Amabhulokhi edatha afika ecindezelwe ngamalogi engeziwe. Lapho ibhulokhi isinosayizi owanele, ibhulokhi idluliselwa ku-database. Imethadatha iya kunkomba, futhi idatha evela ku-log block iya ku-Chunks (imvamisa isitoreji sento). Ngemva kokusetha kabusha, umamukeli udala ibhulokhi entsha lapho okufakiwe okusha kuzongezwa khona.
Index - database, DynamoDB, Cassandra, Google BigTable, njll.
Izinqamu β amabhlogo okungena efomini elicindezelwe, ngokuvamile agcinwa endaweni yokugcina into, isibonelo, i-S3.
Umbuzo - indlela yokufunda eyenza wonke umsebenzi ongcolile. Ibheka ibanga lesikhathi nesitembu sesikhathi, bese ibheka inkomba ukuze ithole okufanayo. Okulandelayo, ifunda amabhlogo edatha futhi iwahlunge ukuze ithole umphumela.
Manje ake sibone konke kusebenza.
setting
Indlela elula yokufaka ku-Kubernetes ukusebenzisa i-helm. Sicabanga ukuthi usuvele uyifakile futhi wayimisa ( cishe. umhumushi)
Engeza indawo yokugcina bese ufaka isitaki.
$ helm repo add loki https://grafana.github.io/loki/charts
$ helm repo update
$ helm upgrade --install loki loki/loki-stack --set grafana.enabled=true,prometheus.enabled=true,prometheus.alertmanager.persistentVolume.enabled=false,prometheus.server.persistentVolume.enabled=falseNgezansi kunesibonelo sedeshibhodi ebonisa idatha esuka ku-Prometheus ye-Etcd metrics kanye ne-Loki yamalogi we-Etcd pod.
Manje ake sixoxe ngokwakhiwa kwazo zombili izinhlelo, futhi siphinde siqhathanise amakhono azo nomunye nomunye.
Ukuqhathanisa
Ulimi lombuzo
I-Elasticsearch isebenzisa i-Query DSL kanye nolimi lombuzo lwe-Lucene ukuze inikeze amakhono okusesha ombhalo ogcwele. Kuyinjini yokusesha emisiwe, enamandla enosekelo olubanzi lomsebenzisi. Ngayo, ungasesha ngokomongo futhi uhlunge ngokuhambisana.
Ngakolunye uhlangothi lwendandatho i-LogQL, esetshenziswe ku-Loki, umlandeli we-PromQL (ulimi lombuzo we-Prometheus). Isebenzisa omaka belogi ukuze ihlunge futhi ikhethe idatha yelogi. Kungenzeka ukusebenzisa abanye opharetha kanye ne-arithmetic njengoba kuchaziwe , kodwa ngokwamakhono isalele ngemuva kolimi olunwebekayo.
Njengoba imibuzo ku-Loki ihlotshaniswa namathegi, kulula ukuhlotshaniswa namamethrikhi, futhi ngenxa yalokho, kulula ukuhlela ukuqapha kokusebenza nakho.
I-Scalability
Zombili izitaki ziyakala ngokuvundlile, kodwa i-Loki ikwenza kube lula ngoba inezindlela ezihlukene zokufunda nokubhala kanye ne-microservice architecture. I-Loki ingenziwa ngendlela oyifisayo ukuze ihambisane nezidingo zakho futhi ingasetshenziselwa imiqulu emikhulu kakhulu yedatha yelogi.
I-Multitenncy
I-Cluster multitenancy iyitimu evamile kusifinyezo se-OPEX, zombili izitaki zihlinzeka nge-multitenancy. Kukhona eziningana ze-Elasticsearch ukuhlukaniswa kweklayenti: inkomba ehlukene yeklayenti ngalinye, umzila osuselwe kumakhasimende, izinkambu zamaklayenti eziyingqayizivele, izihlungi zokusesha. U-Loki uye ngendlela yesihloko se-HTTP X-Scope-OrgID.
izindleko
I-Loki ibiza kakhulu ngenxa yokuthi ayikhombi idatha, i-metadata kuphela. Lokhu kuphumelela kanye nenkumbulo (inqolobane), njengoba ukugcinwa kwento ishibhile kunokugcinwa kwebhulokhi, esetshenziswa kumaqoqo e-Elasticsearch.
isiphetho
Isitaki se-EFK singasetshenziselwa izinjongo ezahlukahlukene, sinikeze ukuguquguquka okuphezulu kanye nesixhumi esibonakalayo esicebile se-Kibana sokuhlaziya, ukubonwa, kanye nemibuzo. Ingathuthukiswa ngokwengeziwe ngamakhono okufunda omshini.
Isitaki se-Loki siwusizo ku-Kubernetes ecosystem ngenxa yendlela yaso yokuthola imethadatha. Ungakwazi ukuhlobanisa kalula idatha yokuqapha ngokusekelwe ochungechungeni lwesikhathi e-Grafana namalogi.
Uma kukhulunywa ngezindleko nokugcinwa kwelogi yesikhathi eside, i-Loki iyindawo enhle kakhulu yokungena ezixazululweni zamafu.
Kukhona ezinye izindlela emakethe - ezinye zingaba ngcono kuwe. Isibonelo, i-GKE inokuhlanganiswa kwe-Stackdriver enikeza isisombululo esihle kakhulu sokuqapha. Asizange sibafake ekuhlaziyeni kwethu kulesi sihloko.
Izinkomba:
Isihloko sahunyushwa futhi salungiselelwa uHabr ngabasebenzi - izifundo ezijulile, izifundo zevidiyo kanye nokuqeqeshwa kwebhizinisi okuvela kochwepheshe (Kubernetes, DevOps, Docker, Ansible, Ceph, SRE, Agile)
Source: www.habr.com