Kev saib xyuas tau dhau los ua ib qho tseem ceeb ntawm cov kev daws teeb meem huab cua loj zuj zus vim qhov nyuaj ntawm kev faib tshuab nce ntxiv. Nws yog ib qho tsim nyog kom nkag siab txog lawv tus cwj pwm. Peb xav tau cov cuab yeej ua kom muaj peev xwm uas tuaj yeem sau cov ntaub ntawv los ntawm txhua qhov kev pabcuam - thiab muab cov kws tshaj lij nrog ib qho kev sib txuas nrog kev tshuaj xyuas kev ua haujlwm, kev ua yuam kev, kev muaj thiab cov cav.
Cov cuab yeej tib yam no yuav tsum ua kom muaj txiaj ntsig thiab tsim tau. Hauv tsab xov xwm no, peb yuav saib ob lub tshuab nrov nrov: EFK (Elasticsearch) thiab PLG (Loki) thiab tshuaj xyuas lawv cov qauv tsim thiab qhov sib txawv.
EFK pawg
Tej zaum koj twb tau hnov ββtxog ELK lossis EFK nrov heev. Cov pawg muaj ntau qhov sib txawv: Elasticsearch (khoom khaws cia), Logstash lossis FluentD (log sau thiab sib sau), thiab Kibana rau kev pom.
Ib qho kev ua haujlwm zoo li no:
Elasticsearch - faib khoom cia nrog kev tshawb nrhiav thiab tshawb xyuas lub sijhawm. Cov kev daws teeb meem zoo heev rau cov ntaub ntawv semi-structured xws li cav. Cov ntaub ntawv tau txais kev cawmdim raws li JSON cov ntaub ntawv, indexed nyob rau hauv lub sij hawm ntawm lub sij hawm thiab muab faib thoob plaws pawg nodes. Ib qho inverted index yog siv, muaj tag nrho cov lus tshwj xeeb thiab cov ntaub ntawv txuam rau kev tshawb nrhiav cov ntawv nyeem, uas nyob rau hauv lem yog raws li lub Apache Lucene search engine.
FluentD yog tus sau cov ntaub ntawv uas sib sau cov ntaub ntawv thaum sau thiab siv nws. Nws sim npaj cov ntaub ntawv hauv JSON kom ntau li ntau tau. Nws architecture yog extensible, muaj ntau , txhawb zej zog, rau txhua lub sijhawm.
Kibana - cov cuab yeej pom cov ntaub ntawv rau Elasticsearch nrog ntau lub peev xwm ntxiv, piv txwv li, lub sijhawm tsom xam, ntsuas duab, kev kawm tshuab thiab lwm yam.
Elasticsearch architecture
Elasticsearch pawg cov ntaub ntawv khaws cia kis thoob plaws tag nrho nws cov nodes. Ib pawg muaj ntau cov nodes los txhim kho qhov muaj thiab muaj peev xwm. Ib lub node tuaj yeem ua tau tag nrho cov luag haujlwm ntawm pawg, tab sis hauv kev xa tawm loj, cov nodes feem ntau yog muab rau ib tus neeg ua haujlwm.
Cluster node hom:
- master node - tswj cov pawg, tsawg kawg peb xav tau, ib qho yog ib txwm ua haujlwm;
- cov ntaub ntawv node - khaws cov ntaub ntawv indexed thiab ua ntau yam haujlwm nrog nws;
- ingest node - npaj cov kav dej rau kev hloov cov ntaub ntawv ua ntej indexing;
- coordinating node - routing thov, txo cov kev tshawb fawb ua theem, kev sib koom tes indexing loj;
- ceeb toom ntawm - pib ua haujlwm ceeb toom;
- machine learning node - ua cov hauj lwm tshuab kawm.
Daim duab hauv qab no qhia tau hais tias yuav khaws cov ntaub ntawv li cas thiab rov ua dua ntawm cov nodes kom ua tiav cov ntaub ntawv muaj ntau dua.
Txhua replica cov ntaub ntawv khaws cia nyob rau hauv ib qho inverted index, daim duab hauv qab no qhia tau hais tias qhov no tshwm sim:
chaw
Cov ntsiab lus tuaj yeem pom , Kuv yuav siv daim ntawv qhia kev cai:
$ helm install efk-stack stable/elastic-stack --set logstash.enabled=false --set fluentd.enabled=true --set fluentd-elasticsPLG pawg
Tsis txhob xav tsis thoob yog tias koj nrhiav tsis tau cov lus no, raws li nws zoo dua lub npe Grafana Loki. Txawm li cas los xij, pawg no tau txais koob meej vim nws siv cov kev daws teeb meem pov thawj. Tej zaum koj twb tau hnov ββββtxog Grafana, cov cuab yeej pom kev nrov. Nws cov neeg tsim, kev tshoov siab los ntawm Prometheus, tsim Loki, kab rov tav scalable, kev ua tau zoo ntawm kev sib sau ua ke. Loki indexes tsuas yog cov metadata, tsis yog cov ntawv xov xwm lawv tus kheej, kev daws teeb meem uas tso cai rau nws siv tau yooj yim thiab siv tau zoo.
Promtail - tus neeg sawv cev xa cov cav los ntawm kev khiav haujlwm mus rau Loki pawg. ua grafana yog cov cuab yeej pom kev pom raws li cov ntaub ntawv los ntawm Loki.
Loki yog tsim los ntawm tib lub hauv paus ntsiab lus raws li Prometheus, ua kom nws zoo haum rau khaws cia thiab tshuaj xyuas Kubernetes cav.
Loki architecture
Loki tuaj yeem ua haujlwm raws li ib qho txheej txheem lossis ntau yam txheej txheem, tso cai rau kev ntsuas kab rov tav.
Nws tuaj yeem ua haujlwm xws li daim ntawv thov monolithic lossis ua microservice. Kev khiav raws li ib qho txheej txheem yuav pab tau rau kev loj hlob hauv zos lossis rau kev saib xyuas me me. Rau kev ua lag luam thiab kev ua haujlwm scalable, nws raug nquahu kom siv cov kev xaiv microservice. Txoj hauv kev rau kev sau ntawv thiab nyeem cov ntaub ntawv raug cais tawm, yog li nws tuaj yeem raug kho kom zoo thiab ntsuas raws li xav tau.
Cia peb saib cov architecture ntawm lub log sau system yam tsis tau mus rau hauv kev nthuav dav:
Thiab ntawm no yog cov lus piav qhia (microservice architecture):
Cheebtsam:
Promtail - tus neeg sawv cev tau teeb tsa ntawm cov nodes (raws li cov kev pabcuam), nws tshem tawm cov cav los ntawm kev ua haujlwm thiab nkag mus rau Kubernetes API kom tau txais cov ntaub ntawv metadata uas yuav sau cov ntawv teev npe. Nws mam li xa lub cav mus rau lub ntsiab Loki kev pabcuam. Metadata daim ntawv qhia txhawb nqa tib txoj cai tagging raws li Prometheus.
Distributor - ib tug kev pab cuam distributor uas ua hauj lwm raws li ib tug tsis. Txhawm rau ua cov ntaub ntawv ntau lab, nws ntim cov ntaub ntawv tuaj, compressing hauv blocks thaum nws tuaj txog. Ob peb cov ntaub ntawv dab dej tau khiav ib txhij, tab sis cov cav uas muaj nyob rau hauv cov ntaub ntawv nkag mus yuav tsum tsuas yog tshwm sim hauv ib qho ntawm lawv rau tag nrho nws cov blocks. Qhov no yog teem rau hauv lub nplhaib ntawm dab dej thiab sequential hashing. Rau qhov ua txhaum cai thiab rov ua dua, qhov no ua tiav n zaug (3 yog tias tsis teeb tsa).
Ingester - tus txais kev pabcuam. Cov ntaub ntawv thaiv tuaj txog compressed nrog cov cav ntxiv. Thaum lub block yog qhov loj txaus, lub block yog flushed rau hauv database. Metadata mus rau qhov ntsuas, thiab cov ntaub ntawv los ntawm lub cav thaiv mus rau Chunks (feem ntau yog cov khoom khaws cia). Tom qab rov pib dua, tus txais yuav tsim ib qho kev thaiv tshiab uas cov ntawv nkag tshiab yuav raug ntxiv.
Index - database, DynamoDB, Cassandra, Google BigTable, thiab lwm yam.
Pob tawv - log blocks nyob rau hauv daim ntawv compressed, feem ntau muab cia rau hauv cov khoom cia, piv txwv li, S3.
Querier - txoj kev nyeem ntawv uas ua txhua txoj haujlwm qias neeg. Nws saib ntawm lub sijhawm thiab sijhawm, thiab tom qab ntawd saib qhov ntsuas kom pom qhov sib tw. Tom ntej no, nws nyeem cov ntaub ntawv thaiv thiab lim lawv kom tau txais qhov tshwm sim.
Tam sim no cia saib txhua yam hauv kev nqis tes ua.
chaw
Txoj kev yooj yim tshaj plaws rau nruab hauv Kubernetes yog siv lub kaus mom hlau. Peb xav tias koj twb tau teeb tsa thiab teeb tsa nws ( kwv yees. tus txhais lus)
Ntxiv ib lub repository thiab nruab ib pawg.
$ helm repo add loki https://grafana.github.io/loki/charts
$ helm repo update
$ helm upgrade --install loki loki/loki-stack --set grafana.enabled=true,prometheus.enabled=true,prometheus.alertmanager.persistentVolume.enabled=false,prometheus.server.persistentVolume.enabled=falseHauv qab no yog ib qho piv txwv dashboard uas qhia cov ntaub ntawv los ntawm Prometheus rau Etcd metrics thiab Loki rau Etcd pod cav.
Tam sim no cia peb tham txog cov architecture ntawm ob lub tshuab, thiab kuj sib piv lawv lub peev xwm nrog ib leeg.
Sib piv
Lus nug
Elasticsearch siv Query DSL thiab Lucene cov lus nug kom muaj peev xwm tshawb nrhiav cov ntawv nyeem. Nws yog ib qho tsim, muaj zog tshawb nrhiav cav nrog kev txhawb nqa dav dav. Nrog nws, koj tuaj yeem tshawb nrhiav los ntawm cov ntsiab lus thiab txheeb los ntawm qhov tseeb.
Nyob rau sab nraud ntawm lub nplhaib yog LogQL, siv hauv Loki, tus ua tiav rau PromQL (Prometheus query language). Nws siv cov cim npe los lim thiab xaiv cov ntaub ntawv teev tseg. Nws tuaj yeem siv qee tus neeg teb xov tooj thiab lej lej raws li tau piav qhia , tab sis nyob rau hauv cov nqe lus ntawm muaj peev xwm nws lags qab Elastic lus.
Txij li cov lus nug hauv Loki yog txuam nrog cov cim npe, lawv yooj yim rau kev sib raug zoo nrog cov ntsuas, thiab yog li ntawd, lawv yooj yim dua los tswj xyuas kev ua haujlwm nrog.
Scalability
Ob pawg yog kab rov tav scalable, tab sis Loki ua kom yooj yim dua vim tias nws tau cais kev nyeem thiab sau cov kab thiab microservice architecture. Loki tuaj yeem hloov kho kom haum rau koj cov kev xav tau thiab tuaj yeem siv rau ntau qhov loj ntawm cov ntaub ntawv cav.
Multitenancy
Cluster multitenancy yog ib lub ntsiab lus nyob rau hauv OPEX abbreviation, ob pawg muab multitenancy. Muaj ntau ntau rau Elasticsearch Kev sib cais ntawm cov neeg siv khoom: cais qhov ntsuas rau txhua tus neeg siv khoom, cov neeg siv khoom raws li txoj kev, cov neeg siv khoom tshwj xeeb, cov ntxaij lim dej tshawb nrhiav. Loki muaj nyob rau hauv daim ntawv ntawm HTTP X-Scope-OrgID header.
nqi ntawm
Loki yog tus nqi zoo heev vim qhov tseeb tias nws tsis ntsuas cov ntaub ntawv, tsuas yog cov metadata xwb. Qhov no ua tiav thiab nco (cache), txij li cov khoom cia yog pheej yig dua li thaiv cia, uas yog siv hauv Elasticsearch pawg.
xaus
Cov pawg EFK tuaj yeem siv rau ntau lub hom phiaj, muab kev hloov pauv siab tshaj plaws thiab muaj txiaj ntsig zoo-nplua nuj Kibana interface rau kev tshuaj xyuas, kev pom, thiab cov lus nug. Nws tuaj yeem txhim kho ntxiv los ntawm kev kawm tshuab.
Cov pawg Loki yog qhov muaj txiaj ntsig zoo hauv Kubernetes ecosystem vim nws cov metadata nrhiav pom. Koj tuaj yeem yooj yim sib txheeb cov ntaub ntawv rau kev saib xyuas raws li lub sijhawm series hauv Grafana thiab cov cav.
Thaum nws los txog rau tus nqi thiab kev khaws cia ntev ntev, Loki yog qhov zoo tshaj plaws nkag mus rau hauv huab kev daws teeb meem.
Muaj ntau lwm txoj hauv kev ua lag luam - qee qhov yuav zoo dua rau koj. Piv txwv li, GKE muaj kev sib koom ua ke ntawm Stackdriver uas muab kev tshuaj xyuas zoo heev. Peb tsis tau suav nrog lawv hauv peb qhov kev tshuaj xyuas hauv kab lus no.
Links:
Zaj lus tau txhais thiab npaj rau Habr los ntawm cov neeg ua haujlwm - Cov kev kawm hnyav, cov chav kawm video thiab kev cob qhia koom nrog los ntawm cov kws tshaj lij (Kubernetes, DevOps, Docker, Ansible, Ceph, SRE, Agile)
Tau qhov twg los: www.hab.com