Sau cov cav los ntawm Loki

Ntawm Badoo, peb niaj hnub saib xyuas cov thev naus laus zis tshiab thiab ntsuas seb lawv puas tsim nyog siv hauv peb lub cev. Peb xav qhia ib qho ntawm cov kev tshawb fawb no rau cov zej zog. Nws yog nplooj siab rau Loki, lub cav sib sau ua ke.

Loki yog ib qho kev daws teeb meem rau khaws cia thiab saib cov cav, thiab pawg no kuj tseem muab cov txheej txheem hloov tau yooj yim rau kev tshuaj xyuas lawv thiab xa cov ntaub ntawv mus rau Prometheus. Thaum lub Tsib Hlis, lwm qhov hloov tshiab tau tso tawm, uas tau txhawb nqa los ntawm cov neeg tsim khoom. Peb xav paub seb Loki tuaj yeem ua li cas, nws muaj peev xwm ua tau dab tsi, thiab qhov twg nws tuaj yeem ua lwm yam rau ELK, pawg uas peb siv tam sim no.

Loki yog dab tsi

Grafana Loki yog ib txheej ntawm cov khoom siv ua tiav rau kev ua haujlwm nrog cov cav. Tsis zoo li lwm cov tshuab zoo sib xws, Loki yog raws li lub tswv yim ntawm kev txheeb xyuas tsuas yog cov ntawv teev lus metadata - cov ntawv sau (ib yam li hauv Prometheus), thiab compressing cov cav lawv tus kheej mus rau hauv nyias nyias.

Tsev nplooj ntawv, GitHub

Ua ntej peb nkag mus rau hauv qhov koj tuaj yeem ua nrog Loki, Kuv xav qhia meej tias peb txhais li cas los ntawm "lub tswv yim ntawm kev txheeb xyuas tsuas yog metadata." Cia peb muab piv rau Loki txoj hauv kev thiab txoj hauv kev rau kev ntsuas hauv cov kev daws teeb meem ib txwm muaj xws li Elasticsearch, siv cov piv txwv ntawm kab los ntawm nginx cav:

172.19.0.4 - - [01/Jun/2020:12:05:03 +0000] "GET /purchase?user_id=75146478&item_id=34234 HTTP/1.1" 500 8102 "-" "Stub_Bot/3.0" "0.001"

Cov txheej txheem ib txwm cais tag nrho cov kab, suav nrog cov teb uas muaj ntau tus lej user_id thiab item_id qhov tseem ceeb, thiab khaws txhua yam hauv qhov ntsuas loj. Qhov kom zoo dua ntawm txoj kev no yog tias koj tuaj yeem khiav cov lus nug nyuaj sai sai, vim tias yuav luag tag nrho cov ntaub ntawv nyob hauv qhov ntsuas. Tab sis qhov no los ntawm tus nqi nyob rau hauv uas qhov Performance index ua loj, uas txhais tau tias yuav tsum nco. Raws li qhov tshwm sim, cov ntawv sau tag nrho cov ntawv ntsuas qhov sib piv yog qhov loj me rau cov cav lawv tus kheej. Yuav kom tshawb nrhiav sai sai los ntawm nws, qhov ntsuas yuav tsum tau muab tso rau hauv lub cim xeeb. Thiab cov cav ntau dua, qhov Performance index loj hlob sai thiab ntau lub cim xeeb nws noj.

Loki txoj kev xav kom tsuas yog cov ntaub ntawv tsim nyog tau muab rho tawm los ntawm ib txoj hlua, tus naj npawb ntawm qhov tseem ceeb yog me me. Txoj kev no peb tau txais qhov ntsuas me me thiab tuaj yeem tshawb xyuas cov ntaub ntawv los ntawm kev lim nws los ntawm lub sijhawm thiab los ntawm cov teb indexed, thiab tom qab ntawd luam theej tawm nrog cov kab lus tsis tu ncua lossis kev tshawb nrhiav substring. Cov txheej txheem tsis zoo li qhov nrawm tshaj plaws, tab sis Loki faib qhov kev thov mus rau ntau qhov chaw thiab ua rau lawv ua tib yam, ua tiav cov ntaub ntawv loj hauv lub sijhawm luv luv. Tus naj npawb ntawm shards thiab parallel thov nyob rau hauv lawv yog configurable; yog li, tus nqi ntawm cov ntaub ntawv uas tuaj yeem ua tiav ib chav tsev ntawm lub sijhawm nyob ntawm linearly ntawm cov peev txheej muab.

Qhov kev lag luam tawm ntawm qhov loj, nrawm nrawm thiab qhov me me, sib npaug brute-force index tso cai rau Loki tswj tus nqi ntawm lub cev. Nws tuaj yeem hloov kho tau yooj yim thiab nthuav dav raws li qhov xav tau.

Loki pawg muaj peb yam: Promtail, Loki, Grafana. Promtail sau cov cav, ua lawv thiab xa mus rau Loki. Loki khaws lawv. Thiab Grafana tuaj yeem thov cov ntaub ntawv los ntawm Loki thiab tso nws. Feem ntau, Loki tuaj yeem siv tsis tau tsuas yog khaws cov cav thiab tshawb xyuas lawv. Tag nrho pawg muab sijhawm zoo rau kev ua haujlwm thiab tshuaj xyuas cov ntaub ntawv nkag los siv txoj kev Prometheus.
Cov lus piav qhia ntawm cov txheej txheem installation tuaj yeem pom no.

Nrhiav los ntawm cov cav

Koj tuaj yeem tshawb xyuas cov cav hauv qhov tshwj xeeb Grafana interface - Explorer. Cov lus nug siv LogQL lus, uas zoo ib yam li PromQL siv hauv Prometheus. Hauv txoj ntsiab cai, nws tuaj yeem xav tias yog kev faib grep.

Qhov kev tshawb nrhiav interface zoo li no:

Qhov kev thov nws tus kheej muaj ob ntu: selector thiab lim. Selector yog ib qho kev tshawb fawb siv indexed metadata (labels) uas tau muab rau cov cav, thiab lim yog ib txoj hlua nrhiav los yog regexp uas lim tawm cov ntaub ntawv teev tseg los ntawm tus xaiv. Hauv qhov piv txwv muab: Hauv curly braces muaj tus xaiv, txhua yam tom qab yog lim.

{image_name="nginx.promtail.test"} |= "index"

Vim txoj kev Loki ua haujlwm, koj tsis tuaj yeem ua cov lus nug yam tsis muaj tus xaiv, tab sis cov ntawv sau tuaj yeem ua raws li qhov koj nyiam.

Ib qho selector yog tus nqi tseem ceeb hauv cov curly braces. Koj tuaj yeem sib txuas cov neeg xaiv thiab qhia meej cov kev tshawb fawb sib txawv uas siv cov neeg ua haujlwm =, != lossis cov kab lus tsis tu ncua:

{instance=~"kafka-[23]",name!="kafka-dev"} 
// Найдёт логи с лейблом instance, имеющие значение kafka-2, kafka-3, и исключит dev 

Lub lim yog ntawv lossis regexp uas yuav lim tawm tag nrho cov ntaub ntawv tau txais los ntawm tus xaiv.

Nws yog ua tau kom tau txais ad-hoc graphs raws li cov ntaub ntawv tau txais hauv hom ntsuas. Piv txwv li, koj tuaj yeem nrhiav tau ntau npaum li cas ib qho kev nkag uas muaj cov hlua index tshwm nyob rau hauv nginx cav:

Cov lus piav qhia tag nrho ntawm cov peev txheej tuaj yeem pom hauv cov ntaub ntawv LogQL.

Kev txheeb xyuas

Muaj ntau txoj hauv kev los sau cov cav:

• Siv Promtail, ib qho qauv ntawm cov pawg rau kev sau cov cav.
• Ncaj nraim los ntawm lub thawv docker siv Loki Docker Logging Driver.
• Siv Fluentd lossis Fluent Bit, uas tuaj yeem xa cov ntaub ntawv mus rau Loki. Tsis zoo li Promtail, lawv muaj cov parsers npaj txhij rau yuav luag txhua hom cav thiab tuaj yeem tswj tau ntau lub cav.

Feem ntau Promtail yog siv los txheeb xyuas. Nws ua peb yam:

• Nrhiav cov ntaub ntawv.
• Txuas cov ntawv rau lawv.
• Xa cov ntaub ntawv rau Loki.

Tam sim no Promtail tuaj yeem nyeem cov cav los ntawm cov ntaub ntawv hauv zos thiab los ntawm cov ntawv xov xwm systemd. Nws yuav tsum tau nruab rau ntawm txhua lub tshuab los ntawm cov cav uas tau sau.

Muaj kev koom ua ke nrog Kubernetes: Promtail tau txais, dhau los ntawm Kubernetes REST API, lees paub lub xeev ntawm pawg thiab sau cov cav los ntawm lub node, kev pabcuam lossis pod, tam sim ntawd tshaj tawm cov ntawv sau raws li metadata los ntawm Kubernetes (pob lub npe, cov ntaub ntawv npe, thiab lwm yam) .

Koj tuaj yeem dai daim ntawv lo raws li cov ntaub ntawv los ntawm lub cav siv Pipeline. Pipeline Promtail tuaj yeem muaj plaub hom theem. Xav paub ntxiv hauv cov ntaub ntawv raug cai, Kuv mam li nco dheev qee qhov nuances.

1. Parsing theem. Qhov no yog theem RegEx thiab JSON. Nyob rau theem no, peb rho tawm cov ntaub ntawv los ntawm cov cav mus rau hauv lub npe hu ua rho tawm daim ntawv qhia. Peb tuaj yeem rho tawm los ntawm JSON los ntawm kev luam cov teb uas peb xav tau rau hauv daim ntawv qhia tawm, lossis los ntawm cov kab lus tsis tu ncua (RegEx), qhov twg cov npe hu ua "mapped" rau hauv daim ntawv qhia tawm. Daim ntawv qhia Extracted yog lub khw muag khoom tseem ceeb, qhov tseem ceeb yog lub npe ntawm daim teb, thiab tus nqi yog nws cov nqi ntawm cov cav.
2. Hloov cov theem. Cov theem no muaj ob txoj kev xaiv: hloov pauv, qhov twg peb teeb tsa txoj cai hloov pauv, thiab qhov chaw - cov ntaub ntawv hloov pauv los ntawm daim ntawv qhia rho tawm. Yog tias tsis muaj daim teb no hauv daim ntawv qhia tawm, nws yuav raug tsim. Txoj kev no nws tuaj yeem tsim cov ntawv sau uas tsis yog raws li daim ntawv qhia tawm. Nyob rau theem no peb tuaj yeem tswj cov ntaub ntawv hauv daim ntawv qhia rho tawm siv lub zog muaj zog Cov Template. Tsis tas li ntawd, peb yuav tsum nco ntsoov tias daim ntawv qhia tau muab rho tawm tag nrho thaum lub sijhawm txheeb xyuas, uas ua rau nws ua tau, piv txwv li, txhawm rau txheeb xyuas tus nqi hauv nws: “{{yog .tag}tag tus nqi tshwm sim{end}}”. Template txhawb cov xwm txheej, loops, thiab qee txoj hlua ua haujlwm xws li Hloov thiab Txiav.
3. Cov theem ua. Ntawm no koj tuaj yeem ua qee yam nrog cov ntsiab lus rho tawm:
   • Tsim ib daim ntawv lo los ntawm cov ntaub ntawv rho tawm, uas yuav raug ntsuas los ntawm Loki.
   • Hloov lossis teeb tsa lub sijhawm tshwm sim los ntawm lub cav.
   • Hloov cov ntaub ntawv (log ntawv) uas yuav mus rau Loki.
   • Tsim kev ntsuas.
4. Lim theem. Cov theem sib tw, qhov twg peb tuaj yeem xa cov ntawv nkag uas peb tsis tas yuav / dev / null lossis xa lawv mus rau kev ua haujlwm ntxiv.

Siv ib qho piv txwv ntawm kev ua cov cav nginx li niaj zaus, kuv yuav qhia koj li cas tuaj yeem txheeb xyuas cov cav siv Promtail.

Rau qhov kev sim, cia peb ua raws li nginx-proxy hloov kho nginx duab jwilder/nginx-proxy: alpine thiab daemon me me uas tuaj yeem nug nws tus kheej ntawm HTTP. Lub daemon muaj ob peb lub ntsiab lus kawg, uas nws tuaj yeem muab cov lus teb ntawm ntau qhov sib txawv, nrog cov xwm txheej HTTP sib txawv thiab muaj qhov sib txawv qeeb.

Peb yuav sau cov cav los ntawm cov thawv docker, uas tuaj yeem pom nyob hauv txoj kev /var/lib/docker/containers//-json.log

Hauv docker-compose.yml peb configure Promtail thiab qhia txoj hauv kev rau config:

promtail:
  image: grafana/promtail:1.4.1
 // ...
 volumes:
   - /var/lib/docker/containers:/var/lib/docker/containers:ro
   - promtail-data:/var/lib/promtail/positions
   - ${PWD}/promtail/docker.yml:/etc/promtail/promtail.yml
 command:
   - '-config.file=/etc/promtail/promtail.yml'
 // ...

Ntxiv txoj hauv kev rau cov cav mus rau promtail.yml (muaj ib qho "docker" kev xaiv hauv config, uas ua tib yam hauv ib kab, tab sis nws yuav tsis meej):

scrape_configs:
 - job_name: containers

   static_configs:
       labels:
         job: containerlogs
         __path__: /var/lib/docker/containers/*/*log  # for linux only

Thaum qhov kev teeb tsa no tau qhib, cov cav los ntawm txhua lub thawv yuav raug xa mus rau Loki. Txhawm rau zam qhov no, peb hloov qhov chaw ntawm qhov kev sim nginx hauv docker-compose.yml - ntxiv cov ntawv sau npe:

proxy:
 image: nginx.test.v3
//…
 logging:
   driver: "json-file"
   options:
     tag: "{{.ImageName}}|{{.Name}}"

Hloov kho promtail.yml thiab teeb tsa Pipeline. Cov tswv yim suav nrog cov cav ntawm hom hauv qab no:

{"log":"u001b[0;33;1mnginx.1    | u001b[0mnginx.test 172.28.0.3 - - [13/Jun/2020:23:25:50 +0000] "GET /api/index HTTP/1.1" 200 0 "-" "Stub_Bot/0.1" "0.096"n","stream":"stdout","attrs":{"tag":"nginx.promtail.test|proxy.prober"},"time":"2020-06-13T23:25:50.66740443Z"}
{"log":"u001b[0;33;1mnginx.1    | u001b[0mnginx.test 172.28.0.3 - - [13/Jun/2020:23:25:50 +0000] "GET /200 HTTP/1.1" 200 0 "-" "Stub_Bot/0.1" "0.000"n","stream":"stdout","attrs":{"tag":"nginx.promtail.test|proxy.prober"},"time":"2020-06-13T23:25:50.702925272Z"}

Pipeline theem:

 - json:
     expressions:
       stream: stream
       attrs: attrs
       tag: attrs.tag

Peb rho tawm cov kwj dej, attrs, attrs.tag (yog tias lawv muaj) los ntawm JSON tuaj thiab muab tso rau hauv daim ntawv qhia rho tawm.

 - regex:
     expression: ^(?P<image_name>([^|]+))|(?P<container_name>([^|]+))$
     source: "tag"

Yog tias peb tswj tau muab cov ntawv sau rau hauv daim ntawv qhia tawm, tom qab ntawd siv regexp peb rho tawm cov npe ntawm daim duab thiab ntim.

 - labels:
     image_name:
     container_name:

Peb muab cov ntawv sau. Yog tias cov duab_name thiab container_name yuam sij pom nyob rau hauv cov ntaub ntawv rho tawm, ces lawv cov txiaj ntsig yuav raug muab rau cov ntawv sau sib xws.

 - match:
     selector: '{job="docker",container_name="",image_name=""}'
     action: drop

Peb muab pov tseg tag nrho cov cav uas tsis tau teeb tsa cov ntawv cim image_name thiab container_name.

  - match:
     selector: '{image_name="nginx.promtail.test"}'
     stages:
       - json:
           expressions:
             row: log

Rau tag nrho cov cav uas nws image_name yog nginx.promtail.test, rho tawm lub cav teb los ntawm lub hauv paus cav thiab muab tso rau hauv daim ntawv qhia rho tawm nrog cov kab yuam sij.

  - regex:
         # suppress forego colors
         expression: .+nginx.+|.+[0m(?P<virtual_host>[a-z_.-]+) +(?P<nginxlog>.+)
         source: logrow

Peb tshem cov kab lus nkag nrog cov kab lus tsis tu ncua thiab rub tawm nginx virtual host thiab nginx cav kab.

     - regex:
         source: nginxlog
         expression: ^(?P<ip>[w.]+) - (?P<user>[^ ]*) [(?P<timestamp>[^ ]+).*] "(?P<method>[^ ]*) (?P<request_url>[^ ]*) (?P<request_http_protocol>[^ ]*)" (?P<status>[d]+) (?P<bytes_out>[d]+) "(?P<http_referer>[^"]*)" "(?P<user_agent>[^"]*)"( "(?P<response_time>[d.]+)")?

Parse nginx cav siv cov lus hais tsis tu ncua.

    - regex:
           source: request_url
           expression: ^.+.(?P<static_type>jpg|jpeg|gif|png|ico|css|zip|tgz|gz|rar|bz2|pdf|txt|tar|wav|bmp|rtf|js|flv|swf|html|htm)$
     - regex:
           source: request_url
           expression: ^/photo/(?P<photo>[^/?.]+).*$
       - regex:
           source: request_url
           expression: ^/api/(?P<api_request>[^/?.]+).*$

Wb parse request_url. Siv regexp peb txiav txim siab lub hom phiaj ntawm qhov kev thov: rau cov ntaub ntawv zoo li qub, cov duab, rau API thiab teeb tsa tus yuam sij sib xws hauv daim ntawv qhia tawm.

       - template:
           source: request_type
           template: "{{if .photo}}photo{{else if .static_type}}static{{else if .api_request}}api{{else}}other{{end}}"

Siv cov neeg ua haujlwm raws cai hauv Template, peb tshawb xyuas cov chaw teeb tsa hauv daim ntawv qhia rho tawm thiab teeb tsa cov txiaj ntsig xav tau rau daim ntawv thov_type: duab, zoo li qub, API. Muab lwm tus yog tias nws ua tsis tiav. request_type tam sim no muaj hom thov.

       - labels:
           api_request:
           virtual_host:
           request_type:
           status:

Peb teeb tsa cov ntawv api_request, virtual_host, request_type thiab xwm txheej (HTTP xwm txheej) raws li qhov peb tau tswj kom muab tso rau hauv daim ntawv qhia rho tawm.

       - output:
           source: nginx_log_row

Hloov cov zis. Tam sim no lub cav nginx ntxuav los ntawm daim ntawv qhia tawm mus rau Loki.

Tom qab khiav lub config saum toj no, koj tuaj yeem pom tias txhua qhov nkag tau muab cov ntawv sau raws li cov ntaub ntawv los ntawm lub cav.

Ib yam uas yuav tsum nco ntsoov yog tias khaws cov ntawv nrog ntau tus nqi (cardinality) tuaj yeem ua rau Loki qeeb. Ntawd yog, koj yuav tsum tsis txhob tso, piv txwv li, user_id hauv qhov ntsuas. Nyeem ntxiv txog qhov no hauv kab lus "Yuav ua li cas cov ntawv lo hauv Loki tuaj yeem ua rau cov lus nug nkag sai thiab yooj yim dua" Tab sis qhov no tsis tau txhais hais tias koj tsis tuaj yeem tshawb nrhiav los ntawm user_id yam tsis muaj indexes. Koj yuav tsum siv cov ntxaij lim dej thaum tshawb nrhiav ("lob" cov ntaub ntawv), thiab qhov Performance index ntawm no ua tus cim kwj.

Visualization ntawm cov cav

Loki tuaj yeem ua raws li cov ntaub ntawv rau Grafana graphs siv LogQL. Cov yam ntxwv hauv qab no tau txais kev txhawb nqa:

• tus nqi - tus naj npawb ntawm cov ntaub ntawv ib ob;
• suav lub sijhawm - tus naj npawb ntawm cov ntaub ntawv nyob rau hauv qhov kev teev tseg.

Kuj tseem muaj cov haujlwm sib sau ua ke Sum, Avg thiab lwm yam. Koj tuaj yeem tsim cov duab nyuaj, piv txwv li ib daim duab ntawm tus lej HTTP yuam kev:

Tus qauv cov ntaub ntawv qhov chaw Loki yog me ntsis txo qis hauv kev ua haujlwm piv rau Prometheus cov ntaub ntawv qhov chaw (piv txwv li, koj tsis tuaj yeem hloov cov lus dab neeg), tab sis Loki tuaj yeem txuas nrog qhov chaw nrog Prometheus hom. Kuv tsis paub tseeb tias qhov no yog cov ntaub ntawv tus cwj pwm, tab sis txiav txim los ntawm cov neeg tsim khoom teb "Yuav ua li cas rau configure Loki li Prometheus datasource? · Issue #1222 · grafana/loki”, piv txwv li, yog tag nrho raws li txoj cai, thiab Loki tau ua tiav nrog PromQL.

Ntxiv Loki ua cov ntaub ntawv qhov chaw nrog hom Prometheus thiab ntxiv URL / loki:

Thiab peb tuaj yeem ua cov duab kos, zoo li peb tau ua haujlwm nrog kev ntsuas los ntawm Prometheus:

Kuv xav tias qhov tsis sib xws hauv kev ua haujlwm yog ib ntus thiab cov neeg tsim khoom yuav kho qhov no yav tom ntej.

Kev ntsuas

Loki muab lub peev xwm los rho tawm cov lej ntsuas ntawm cov cav thiab xa mus rau Prometheus. Piv txwv li, nginx cav muaj pes tsawg tus bytes ib lo lus teb, nrog rau, nrog rau ib qho kev hloov kho ntawm tus qauv cav hom, lub sij hawm nyob rau hauv vib nas this uas nws coj los teb. Cov ntaub ntawv no tuaj yeem muab rho tawm thiab xa mus rau Prometheus.

Ntxiv lwm ntu rau promtail.yml:

- match:
   selector: '{request_type="api"}'
   stages:
     - metrics:
         http_nginx_response_time:
           type: Histogram
           description: "response time ms"
           source: response_time
           config:
             buckets: [0.010,0.050,0.100,0.200,0.500,1.0]
- match:
   selector: '{request_type=~"static|photo"}'
   stages:
     - metrics:
         http_nginx_response_bytes_sum:
           type: Counter
           description: "response bytes sum"
           source: bytes_out
           config:
             action: add
         http_nginx_response_bytes_count:
           type: Counter
           description: "response bytes count"
           source: bytes_out
           config:
             action: inc

Qhov kev xaiv tso cai rau koj los txhais thiab hloov kho cov ntsuas ntsuas raws li cov ntaub ntawv los ntawm daim ntawv qhia rho tawm. Cov kev ntsuas no tsis raug xa mus rau Loki - lawv tshwm sim hauv Promtail / ntsuas qhov kawg. Prometheus yuav tsum tau teeb tsa kom tau txais cov ntaub ntawv tau txais nyob rau theem no. Hauv qhov piv txwv saum toj no, rau request_type = "api" peb sau ib histogram metric. Nrog hom kev ntsuas no nws yooj yim kom tau txais feem pua. Rau qhov zoo li qub thiab duab, peb sau cov lej ntawm bytes thiab cov kab uas peb tau txais bytes los xam qhov nruab nrab.

Nyeem ntxiv txog metrics no.

Qhib qhov chaw nres nkoj ntawm Promtail:

promtail:
     image: grafana/promtail:1.4.1
     container_name: monitoring.promtail
     expose:
       - 9080
     ports:
       - "9080:9080"

Nco ntsoov tias kev ntsuas nrog promtail_custom prefix tshwm:

Teeb tsa Prometheus. Ntxiv txoj haujlwm promtail:

- job_name: 'promtail'
 scrape_interval: 10s
 static_configs:
   - targets: ['promtail:9080']

Thiab peb kos ib daim duab:

Txoj kev no koj tuaj yeem pom, piv txwv li, plaub cov lus nug qeeb tshaj plaws. Koj tuaj yeem teeb tsa kev saib xyuas rau cov ntsuas no.

Scaling

Loki tuaj yeem yog hauv ib hom binary lossis hauv hom sharded (horizontalally-scalable mode). Hauv qhov thib ob, nws tuaj yeem txuag cov ntaub ntawv rau huab, thiab cov chunks thiab qhov ntsuas tau muab cais. Version 1.5 qhia txog lub peev xwm khaws cia hauv ib qho chaw, tab sis nws tseem tsis tau pom zoo kom siv nws hauv kev tsim khoom.

Cov chunks tuaj yeem khaws cia hauv S3-tshaj qhov chaw cia, thiab kab rov tav scalable databases tuaj yeem siv los khaws cov ntsuas: Cassandra, BigTable lossis DynamoDB. Lwm qhov chaw ntawm Loki - Distributors (rau kev sau ntawv) thiab Querier (rau cov lus nug) - yog cov neeg tsis muaj lub xeev thiab kuj ntsuas kab rov tav.

Ntawm DevOpsDays Vancouver 2019 lub rooj sib tham, ib tus neeg koom nrog Callum Styan tshaj tawm tias nrog Loki nws txoj haujlwm muaj petabytes ntawm cov cav nrog qhov ntsuas tsawg dua 1% ntawm tag nrho qhov loj: "Yuav Ua Li Cas Loki Correlates Metrics thiab Logs - Thiab Txuag Koj Cov Nyiaj".

Kev sib piv ntawm Loki thiab ELK

Index loj

Txhawm rau ntsuas qhov ntsuas qhov ntsuas qhov loj me, kuv coj cov cav los ntawm lub thawv nginx uas lub Pipeline saum toj no tau teeb tsa. Cov ntaub ntawv teev tseg muaj 406 kab nrog tag nrho ntim ntawm 624 MB. Cov cav tau tsim nyob rau hauv ib teev, kwv yees li 109 nkag rau ib ob.

Piv txwv ntawm ob kab ntawm lub log:

Thaum ntsuas los ntawm ELK, qhov no tau muab qhov ntsuas qhov loj ntawm 30,3 MB:

Nyob rau hauv rooj plaub ntawm Loki, qhov no ua rau kwv yees li 128 KB ntawm qhov ntsuas thiab kwv yees li 3,8 MB ntawm cov ntaub ntawv hauv chunks. Nws yog ib qho tsim nyog sau cia tias lub cav tau tsim tawm thiab tsis muaj ntau yam ntaub ntawv. Ib qho yooj yim gzip ntawm tus thawj Docker JSON cav nrog cov ntaub ntawv muab compression ntawm 95,4%, thiab coj mus rau hauv tus account qhov tseeb tias tsuas yog lub cav nginx ntxuav tau raug xa mus rau Loki nws tus kheej, compression txog li 4 MB yog nkag siab. Tag nrho cov txiaj ntsig tshwj xeeb rau Loki cov ntawv lo yog 35, uas piav qhia qhov me me ntawm qhov ntsuas. Rau ELK lub cav kuj raug tshem tawm. Yog li, Loki compressed cov ntaub ntawv qub los ntawm 96%, thiab ELK los ntawm 70%.

Nco noj

Yog tias peb piv tag nrho Prometheus thiab ELK pawg, ces Loki "noj" ob peb zaug tsawg dua. Nws yog qhov tseeb tias qhov kev pabcuam Go siv tsawg dua li qhov kev pabcuam Java, thiab muab piv rau qhov loj ntawm JVM Heap Elasticsearch thiab faib lub cim xeeb rau Loki yog qhov tsis raug, tab sis txawm li cas los xij nws tsim nyog sau cia tias Loki siv lub cim xeeb tsawg dua. Nws cov txiaj ntsig CPU tsis yog qhov pom tseeb, tab sis nws kuj yog tam sim no.

Ceev

Loki "devurs" cav nrawm dua. Qhov ceev yog nyob ntawm ntau yam - cov cav zoo li cas, peb zoo li cas hauv kev txheeb xyuas lawv, network, disk, thiab lwm yam - tab sis nws yog qhov tseeb siab dua ELK (hauv kuv qhov kev sim - txog ob zaug ntau dua). Qhov no tau piav qhia los ntawm qhov tseeb tias Loki muab cov ntaub ntawv tsawg dua rau hauv qhov ntsuas thiab, raws li, siv sijhawm tsawg dua rau kev ntsuas. Nrog rau kev tshawb nrhiav nrawm, qhov xwm txheej yog qhov sib txawv: Loki pom tau tias qeeb ntawm cov ntaub ntawv loj dua li ob peb gigabytes, thaum ELK qhov kev tshawb nrhiav nrawm tsis nyob ntawm qhov loj ntawm cov ntaub ntawv.

Nrhiav los ntawm cov cav

Loki yog qhov qis dua rau ELK nyob rau hauv cov nqe lus ntawm kev tshawb nrhiav cav. Grep nrog cov lus qhia tsis tu ncua yog qhov muaj zog, tab sis nws tsis zoo rau cov ntaub ntawv paub tab. Qhov tsis muaj ntau cov lus nug, kev sib sau tsuas yog los ntawm cov ntawv sau, tsis muaj peev xwm tshawb nrhiav yam tsis muaj cov ntawv sau - tag nrho cov no txwv peb hauv kev tshawb nrhiav cov ntaub ntawv txaus siab hauv Loki. Qhov no tsis tau txhais hais tias tsis muaj dab tsi tuaj yeem nrhiav tau siv Loki, tab sis nws txhais tau tias kev khiav ntawm kev ua hauj lwm nrog cov cav thaum koj xub pom ib qho teeb meem hauv Prometheus kab kos, thiab tom qab ntawd siv cov ntawv no los nrhiav qhov tshwm sim hauv cov cav.

interface

Ua ntej tshaj plaws, nws zoo nkauj (thov txim, tsis tuaj yeem tiv taus). Grafana muaj qhov zoo saib interface, tab sis Kibana muaj ntau yam khoom nplua nuj.

Qhov zoo thiab qhov tsis zoo ntawm Loki

Ib qho ntawm qhov zoo yog tias Loki koom nrog Prometheus, yog li peb tau txais kev ntsuas thiab ceeb toom tawm ntawm lub thawv. Nws yog qhov yooj yim rau kev sau cov cav thiab khaws cia los ntawm Kubernetes Pods, vim tias nws muaj kev pabcuam nrhiav tau los ntawm Prometheus thiab cia li txuas cov ntawv.

Lub downside yog cov ntaub ntawv tsis muaj zog. Qee yam, piv txwv li, cov yam ntxwv thiab lub peev xwm ntawm Promtail, Kuv pom tsuas yog hauv cov txheej txheem ntawm kev kawm cov cai, hmoov zoo nws yog qhov qhib-qhov. Lwm qhov tsis zoo yog qhov muaj peev xwm txheeb xyuas tsis muaj zog. Piv txwv li, Loki tsis tuaj yeem txheeb xyuas cov kab ntau kab. Lwm qhov tsis zoo yog tias Loki yog cov thev naus laus zis thev naus laus zis (tso tawm 1.0 yog lub Kaum Ib Hlis 2019).

xaus

Loki yog 100% kev nthuav dav thev naus laus zis uas tsim nyog rau cov haujlwm me me thiab nruab nrab, tso cai rau koj los daws ntau yam teeb meem ntawm kev sib sau ua ke, tshawb nrhiav cav, saib xyuas thiab txheeb xyuas cov cav.

Peb tsis siv Loki hauv Badoo vim tias peb muaj ELK pawg uas haum rau peb thiab uas tau dhau los nrog ntau yam kev cai daws teeb meem ntau xyoo. Rau peb, lub stumbling block yog nrhiav los ntawm cav. Nrog yuav luag 100 GB ntawm cov cav hauv ib hnub, nws yog ib qho tseem ceeb rau peb kom nrhiav tau txhua yam thiab me ntsis ntxiv thiab ua sai sai. Rau kev teeb tsa thiab kev saib xyuas, peb siv lwm cov kev daws teeb meem uas haum rau peb cov kev xav tau thiab kev sib koom ua ke. Loki pawg muaj cov txiaj ntsig zoo, tab sis nws yuav tsis muab ntau dua li peb twb muaj, thiab nws cov txiaj ntsig yeej yuav tsis ntau dua tus nqi ntawm kev tsiv teb tsaws chaw.

Thiab txawm hais tias tom qab kev tshawb fawb tau pom tseeb tias peb tsis tuaj yeem siv Loki, peb cia siab tias cov ntawv tshaj tawm no yuav pab koj hauv koj txoj kev xaiv.

Lub repository nrog cov cai siv nyob rau hauv tsab xov xwm yog nyob S, SѓS,.

Tau qhov twg los: www.hab.com