🥇 Fautuaga ma togafiti mo le suia o faʻamatalaga e leʻi faʻatulagaina mai ogalaau i le ELK Stack faʻaaoga GROK i LogStash

Faʻatulagaina o faʻamaumauga e leʻi faʻatulagaina ma GROK

Afai o loʻo e faʻaogaina le Elastic (ELK) stack ma e te manaʻo e faʻapipiʻi faʻasologa masani Logstash ogalaau i Elasticsearch, o lenei pou e mo oe.

O le fa'aputuga ELK o se fa'apuupuuga mo galuega fa'alauiloa e tolu: Elasticsearch, Logstash ma Kibana. Fa'atasi latou te fa'atūina se fa'avae fa'afoe o ogalaau.

Elasticsearch ose faiga su'esu'e ma au'ili'ili.
Lologa o se laina faʻapipiʻi faʻamaumauga o faʻamaumauga e faʻaogaina faʻamatalaga mai le tele o punaoa i le taimi e tasi, suia, ona tuʻuina atu lea i se "stash" pei o Elasticsearch.
kibana fa'atagaina tagata fa'aoga e va'ava'ai fa'amatalaga e fa'aaoga ai siata ma kalafi ile Elasticsearch.

Puti na o'o mai mulimuli ane ma ose fa'amatalaga mama mama. O le folasaga o Beats na suia ai le Elk Stack i le Elastic Stack, ae e le o le manatu lena.

O lenei tusiga e uiga ia Grok, o se vaega i Logstash e mafai ona suia au ogalaau aʻo leʻi auina atu i le faʻamau. Mo o matou faʻamoemoega, o le a ou talanoa e uiga i le gaosiga o faʻamaumauga mai Logstash i Elasticsearch.

O le Grok o se faamama i totonu o Logstash lea e faʻaaogaina e faʻasalalau ai faʻamatalaga e leʻi faʻatulagaina i se mea faʻatulagaina ma fesiligia. O loʻo nofo i luga o se faʻamatalaga masani (regex) ma faʻaoga mamanu o tusitusiga e fetaui ma manoa i faila ogalaau.

E pei ona o le a tatou vaʻaia i vaega nei, o le faʻaaogaina o Grok e faia ai se eseesega tele pe a oʻo mai i le puleaina lelei o ogalaau.

A aunoa ma Grok o au faʻamaumauga ogalaau e le faʻatulagaina

A aunoa ma Grok, pe a lafoina ogalaau mai Logstash i Elasticsearch ma tuʻuina atu i Kibana, e naʻo le faʻaalia o le feʻau tau.

O le fesiligia o fa'amatalaga anoa i lenei tulaga e faigata ona o fa'amaumauga uma o fa'amaumauga o lo'o teuina i se ki e tasi. E sili atu pe a sili atu le faʻatulagaina o feʻau ogalaau.

Fa'amatalaga le fa'atulagaina mai ogalaau

localhost GET /v2/applink/5c2f4bb3e9fda1234edc64d 400 46ms 5bc6e716b5d6cb35fc9687c0

Afai e te vaʻavaʻai totoʻa i faʻamatalaga mataʻutia, o le a e iloa ai o loʻo aofia ai vaega eseese, taʻitasi e vaeluaina e se avanoa.

Mo tagata atinaʻe sili atu le poto masani, atonu e mafai ona e mateina le uiga o vaega taʻitasi ma pe o le a le feʻau ogalaau mai le API call. O lo'o fa'amatala atu i lalo le fa'aaliga o mea ta'itasi.

Va'aiga fa'atulagaina o matou fa'amaumauga

localhost == siosiomaga
GET == auala
/v2/applink/5c2f4bb3e9fda1234edc64d == url
400 == tali_tulaga
46ms == tali_taimi
5bc6e716b5d6cb35fc9687c0 == user_id

E pei ona tatou vaʻai i faʻamaumauga faʻatulagaina, o loʻo i ai se faʻatonuga mo ogalaau e leʻi faʻatulagaina. O le isi laasaga o le faʻaogaina o masini komepiuta o faʻamatalaga mataʻutia. O le mea lea e susulu ai Grok.

Grok Templates

Faʻataʻitaʻiga Grok faʻapipiʻiina

Logstash e sau ma sili atu i le 100 faʻataʻitaʻiga faʻapipiʻiina mo le faʻatulagaina o faʻamaumauga e leʻi faʻatulagaina. E tatau ona e faʻaaogaina lenei mea i soʻo se taimi e mafai ai mo syslogs lautele e pei o apache, linux, haproxy, aws ma isi.

Ae ui i lea, o le a le mea e tupu pe a iai sau ogalaau masani e pei o le faʻataʻitaʻiga i luga? E tatau ona e fausia lau lava Grok template.

Custom Grok templates

E tatau ona e taumafai e fausia lau lava Grok template. Sa ou faaaogaina Grok Debugger и Grok Patterns.

Manatua o le Grok template syntax e faʻapea: %{SYNTAX:SEMANTIC}

O le mea muamua na ou taumafai e fai o le alu i le tab maua i le Grok debugger. Na ou manatu o le a manaia pe a mafai e lenei meafaigaluega ona otometi lava ona gaosia se mamanu Grok, ae e leʻi aoga tele talu ai na maua ai na o le lua afitusi.

I le faʻaaogaina o lenei sailiga, na amata ai ona ou fatuina laʻu lava faʻataʻitaʻiga i le Grok debugger e faʻaaoga ai le syntax o loʻo maua ile itulau Elastic Github.

Ina ua uma ona taʻalo faʻatasi ma faʻasologa eseese, na iu lava ina mafai ona ou faʻatulagaina faʻamaumauga ogalaau i le auala ou te manaʻo ai.

Grok Debugger Link https://grokdebug.herokuapp.com/

Uluai tusitusiga:

localhost GET /v2/applink/5c2f4bb3e9fda1234edc64d 400 46ms 5bc6e716b5d6cb35fc9687c0

mamanu:

%{WORD:environment} %{WORD:method} %{URIPATH:url} %{NUMBER:response_status} %{WORD:response_time} %{USERNAME:user_id}

O le a le mea na tupu i le iuga

{
  "environment": [
    [
      "localhost"
    ]
  ],
  "method": [
    [
      "GET"
    ]
  ],
  "url": [
    [
      "/v2/applink/5c2f4bb3e9fda1234edc64d"
    ]
  ],
  "response_status": [
    [
      "400"
    ]
  ],
  "BASE10NUM": [
    [
      "400"
    ]
  ],
  "response_time": [
    [
      "46ms"
    ]
  ],
  "user_id": [
    [
      "5bc6e716b5d6cb35fc9687c0"
    ]
  ]
}

Faatasi ai ma le Grok template ma faʻafanua faʻamaumauga i lima, o le laasaga mulimuli o le faʻaopoopo i Logstash.

Faʻafouina le faila faʻatulagaina Logstash.conf

I luga o le 'auʻaunaga na e faʻapipiʻi ai le ELK stack, alu i le Logstash configuration:

sudo vi /etc/logstash/conf.d/logstash.conf

Fa'amau suiga.

input { 
  file {
    path => "/your_logs/*.log"
  }
}
filter{
  grok {
    match => { "message" => "%{WORD:environment} %{WORD:method} %{URIPATH:url} %{NUMBER:response_status} %{WORD:response_time} %{USERNAME:user_id}"}
  }
}
output {
  elasticsearch {
    hosts => [ "localhost:9200" ]
  }
}

A uma ona teu au suiga, toe amata Logstash ma siaki lona tulaga ina ia mautinoa o loʻo galue pea.

sudo service logstash restart
sudo service logstash status

Mulimuli ane, ina ia mautinoa ua aoga suiga, Ia mautinoa e faʻafouina lau Elasticsearch index mo Logstash i Kibana!

Faatasi ai ma Grok, ua faʻatulagaina au faʻamatalaga ogalaau!

E pei ona mafai ona tatou vaʻai i le ata o loʻo i luga, e mafai e Grok ona faʻafetaui faʻamaumauga faʻamaumauga ma Elasticsearch. O lea e faafaigofie ai le pulea o ogalaau ma vave ona fesiligia faamatalaga. Nai lo le eliina i faila ogalaau e debug, e mafai ona e faʻamama i mea o loʻo e sailia, pei o se siosiomaga poʻo se url.

Fa'ata'ita'i fa'amatalaga Grok! Afai ei ai se isi auala e fai ai lenei mea pe iai ni faʻafitauli i faʻataʻitaʻiga o loʻo i luga, naʻo le tusi o se faʻamatalaga i lalo e faʻailoa mai ia te aʻu.

Fa'afetai mo le faitau-ma fa'amolemole mulimuli mai ia te a'u iinei i luga ole Medium mo nisi fa'amatalaga fa'ainisinia faakomepiuta manaia!

Punaoa

https://www.elastic.co/blog/do-you-grok-grok
https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns
https://grokdebug.herokuapp.com/

PS So'oga puna

Telegram alaala e Elasticsearch

puna: www.habr.com