🥇Mazano uye matipi ekushandura isina kurongeka data kubva mumatanda kuenda kuELK Stack uchishandisa GROK muLogStash

Kugadzira data isina kurongeka neGROK

Kana iwe uri kushandisa iyo Elastic (ELK) stack uye uchifarira kugadzira mepu yetsika Logstash matanda kuElasticsearch, saka iyi positi ndeyako.

Iyo ELK stack ndeye acronym yematatu akavhurika sosi mapurojekiti: Elasticsearch, Logstash uye Kibana. Pamwe chete vanogadzira dhizaini manejimendi.

Elasticsearch inzira yekutsvaga uye yekuongorora.
logstash i-server-side data processing pombi iyo inopinza data kubva kune akawanda masosi panguva imwe chete, inoishandura, uye yozoitumira kune "stash" senge Elasticsearch.
kibana inobvumira vashandisi kuona data vachishandisa machati uye magirafu muElasticsearch.

Beats yakauya gare gare uye iri lightweight data shipper. Kuunzwa kweBeats kwakashandura Elk Stack kuita Elastic Stack, asi handiyo iyo poindi.

Ichi chinyorwa chiri pamusoro peGrok, icho chiri muLogstash chinogona kushandura matanda ako asati atumirwa kune stash. Nezvinangwa zvedu, ini ndinongotaura nezve kugadzirisa data kubva kuLogstash kupinda muElasticsearch.

Grok isefa mukati meLogstash iyo inoshandiswa kupatsanura isina kurongeka data mune chimwe chinhu chakarongeka uye chinobvunzwa. Iyo inogara pamusoro peyakajairwa kutaura (regex) uye inoshandisa mameseji mapatani kuenzanisa tambo mumafaira elogi.

Sezvatichaona muzvikamu zvinotevera, kushandisa Grok kunoita mutsauko mukuru kana zvasvika pakubata zvakanaka kwelogi.

Pasina Grok data rako regi harina kurongeka

Pasina Grok, kana matanda anotumirwa kubva kuLogstash kuenda kuElasticsearch uye akashandurwa muKibana, anongoonekwa muhuwandu hwemashoko.

Kubvunza ruzivo rwakakosha mumamiriro ezvinhu aya kwakaoma nekuti data rese relogi rinochengetwa mukiyi imwechete. Zvingave nani kana mameseji elogi akarongeka zviri nani.

Unstructured data kubva mumatanda

localhost GET /v2/applink/5c2f4bb3e9fda1234edc64d 400 46ms 5bc6e716b5d6cb35fc9687c0

Kana iwe ukanyatsotarisisa iyo mbishi data, iwe uchaona kuti inonyatso ine zvikamu zvakasiyana, imwe neimwe yakaparadzaniswa nenzvimbo.

Kune vanogadzira vane ruzivo, unogona kufungidzira kuti chimwe nechimwe chezvikamu zvinorevei uye kuti irogi meseji iri kubva kuAPI call. Mharidzo yechinhu chimwe nechimwe yakarongwa pasi apa.

Yakarongeka maonero e data yedu

localhost == nharaunda
GET == nzira
/v2/applink/5c2f4bb3e9fda1234edc64d == url
400 ==mhinduro_status
46ms == mhinduro_nguva
5bc6e716b5d6cb35fc9687c0 == user_id

Sezvatinoona mune yakarongeka data, pane kurongeka kwematanda asina kuumbwa. Nhanho inotevera ndeyekugadzirisa software ye data raw. Apa ndipo panopenya Grok.

Grok Templates

Yakavakwa-mukati Grok matemplate

Logstash inouya neanopfuura zana akavakirwa-mukati matemplate ekugadzirisa isina kurongeka data. Iwe unofanirwa kutora mukana weizvi pese pazvinogoneka kune general syslogs seapache, linux, haproxy, aws uye zvichingodaro.

Nekudaro, chii chinoitika kana iwe uine tsika matanda senge mumuenzaniso uri pamusoro? Iwe unofanirwa kuvaka yako wega Grok template.

Custom Grok templates

Iwe unofanirwa kuedza kuvaka yako wega Grok template. Ndakashandisa Grok Debugger и Sravana Sameeralu Serial XNUMXth Grok Patterns.

Ziva kuti iyo Grok template syntax yakaita seiyi: %{SYNTAX:SEMANTIC}

Chinhu chekutanga chandakaedza kuita kwaive kuenda kune tab Discover muGrok debugger. Ini ndaifunga kuti zvingave zvakanaka kana chishandiso ichi chaigona kugadzira otomatiki Grok pateni, asi yanga isinganyanyi kubatsira sezvo yangowana machisi maviri.

Ndichishandisa kuwanikwa uku, ndakatanga kugadzira yangu template muGrok debugger ndichishandisa syntax inowanikwa pane Elastic Github peji.

Mushure mekutamba ndichitenderedza nemasyntaxes akasiyana, ndakazokwanisa kugadzira iyo log data nenzira yandaida.

Grok Debugger Link https://grokdebug.herokuapp.com/

Mavara ekutanga:

localhost GET /v2/applink/5c2f4bb3e9fda1234edc64d 400 46ms 5bc6e716b5d6cb35fc9687c0

Muenzaniso:

%{WORD:environment} %{WORD:method} %{URIPATH:url} %{NUMBER:response_status} %{WORD:response_time} %{USERNAME:user_id}

Chii chakaitika pakupedzisira

{
  "environment": [
    [
      "localhost"
    ]
  ],
  "method": [
    [
      "GET"
    ]
  ],
  "url": [
    [
      "/v2/applink/5c2f4bb3e9fda1234edc64d"
    ]
  ],
  "response_status": [
    [
      "400"
    ]
  ],
  "BASE10NUM": [
    [
      "400"
    ]
  ],
  "response_time": [
    [
      "46ms"
    ]
  ],
  "user_id": [
    [
      "5bc6e716b5d6cb35fc9687c0"
    ]
  ]
}

Iine Grok template uye mepu data iri muruoko, danho rekupedzisira nderekuwedzera kuLogstash.

Kuvandudza Logstash.conf configuration file

Pane sevha yawakaisa ELK stack, enda kune Logstash configuration:

sudo vi /etc/logstash/conf.d/logstash.conf

Namatidza shanduko.

input { 
  file {
    path => "/your_logs/*.log"
  }
}
filter{
  grok {
    match => { "message" => "%{WORD:environment} %{WORD:method} %{URIPATH:url} %{NUMBER:response_status} %{WORD:response_time} %{USERNAME:user_id}"}
  }
}
output {
  elasticsearch {
    hosts => [ "localhost:9200" ]
  }
}

Mushure mekuchengetedza shanduko dzako, tangazve Logstash uye tarisa chimiro chayo kuti uone kuti ichiri kushanda.

sudo service logstash restart
sudo service logstash status

Chekupedzisira, kuve nechokwadi chekuti shanduko dzaita, Ita shuwa yekuvandudza yako Elasticsearch index yeLogstash muKibana!

NeGrok, data rako regi rakagadzirwa!

Sezvatinoona mumufananidzo uri pamusoro, Grok inokwanisa kuenzanisa otomatiki data neElasticsearch. Izvi zvinoita kuti zvive nyore kubata matanda uye nekukurumidza kubvunza ruzivo. Panzvimbo pekuchera kuburikidza nefaira mafaira kuti ugadzirise, unogona kungosefa nezvauri kutsvaga, senge nharaunda kana url.

Edza matauriro echiGrok! Kana iwe uine imwe nzira yekuita izvi kana uine chero matambudziko nemienzaniso iri pamusoro, ingonyora mhinduro pazasi kuti undizivise.

Ndatenda nekuverenga-uye ndokumbira unditevere pano paMedium kune zvimwe zvinonakidza software engineering zvinyorwa!

Resources

https://www.elastic.co/blog/do-you-grok-grok
https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns
https://grokdebug.herokuapp.com/

PS Batanidza kune sosi

Telegraph channel by Elasticsearch

Source: www.habr.com

Mazano uye matipi ekushandura isina kurongeka data kubva kumatanda kuenda kuELK Stack uchishandisa GROK muLogStash.