Mu abokai ne tare da ELK da Exchange. Kashi na 2

Na ci gaba da labarina game da yadda ake yin abokai Musanya da ELK (fara a nan). Bari in tunatar da ku cewa wannan haɗin yana iya sarrafa adadi mai yawa na katako ba tare da jinkiri ba. A wannan karon za mu yi magana game da yadda ake samun Musanya aiki tare da abubuwan Logstash da Kibana.

Logstash a cikin tari na ELK ana amfani dashi don sarrafa rajistan ayyukan cikin hikima da shirya su don sanyawa a cikin Elastic a cikin nau'ikan takardu, akan abin da ya dace don gina abubuwan gani daban-daban a Kibana.

saitin

Ya ƙunshi matakai biyu:

• Shigarwa da daidaita fakitin OpenJDK.
• Shigarwa da daidaita kunshin Logstash.

Shigarwa da daidaita fakitin OpenJDK

Dole ne a zazzage fakitin OpenJDK kuma a buɗe shi cikin takamaiman kundin adireshi. Sannan dole ne a shigar da hanyar zuwa wannan directory a cikin $ env:Path da $ env: JAVA_HOME masu canji na tsarin aiki na Windows:

Bari mu duba sigar Java:

PS C:> java -version
openjdk version "13.0.1" 2019-10-15
OpenJDK Runtime Environment (build 13.0.1+9)
OpenJDK 64-Bit Server VM (build 13.0.1+9, mixed mode, sharing)

Shigarwa da daidaita kunshin Logstash

Zazzage fayil ɗin ajiya tare da rarraba Logstash daga nan. Dole ne a buɗe kayan tarihin zuwa tushen faifan. Cire kaya zuwa babban fayil C:Program Files Ba shi da daraja, Logstash zai ƙi farawa kullum. Sannan kuna buƙatar shigar da fayil ɗin jvm.options gyara da alhakin rarraba RAM don tsarin Java. Ina ba da shawarar tantance rabin RAM ɗin uwar garken. Idan yana da 16 GB na RAM akan jirgin, to maɓallan tsoho sune:

-Xms1g
-Xmx1g

dole ne a maye gurbinsu da:

-Xms8g
-Xmx8g

Bugu da ƙari, yana da kyau a yi sharhi fitar da layi -XX:+UseConcMarkSweepGC. Karin bayani akan wannan a nan. Mataki na gaba shine ƙirƙirar saitin tsoho a cikin fayil ɗin logstash.conf:

input {
 stdin{}
}
 
filter {
}
 
output {
 stdout {
 codec => "rubydebug"
 }
}

Tare da wannan saitin, Logstash yana karanta bayanai daga na'ura wasan bidiyo, ya wuce ta cikin tacewa mara kyau, kuma ya fitar da shi zuwa na'ura wasan bidiyo. Yin amfani da wannan saitin zai gwada aikin Logstash. Don yin wannan, bari mu gudanar da shi cikin yanayin hulɗa:

PS C:...bin> .logstash.bat -f .logstash.conf
...
[2019-12-19T11:15:27,769][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[2019-12-19T11:15:27,847][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-12-19T11:15:28,113][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

An ƙaddamar da Logstash cikin nasara akan tashar jiragen ruwa 9600.

Matakin shigarwa na ƙarshe: ƙaddamar da Logstash azaman sabis na Windows. Ana iya yin wannan, alal misali, ta amfani da kunshin NSSM:

PS C:...bin> .nssm.exe install logstash
Service "logstash" installed successfully!

hakuri da laifi

Ana tabbatar da amincin rajistan ayyukan lokacin da aka canjawa wuri daga uwar garken tushe ta hanyar na'urar dagewar lauyoyi.

Yadda yake aiki

Tsarin layin layi yayin sarrafa katako shine: shigarwa → layi → tace + fitarwa.

Filogin ɗin shigarwa yana karɓar bayanai daga tushen log, rubuta shi zuwa jerin gwano, kuma yana aika tabbaci cewa an karɓi bayanan zuwa tushen.

Ana sarrafa saƙon daga layin Logstash, an wuce ta cikin tacewa da kayan aikin fitarwa. Lokacin karɓar tabbaci daga fitarwa cewa an aiko da log ɗin, Logstash yana cire log ɗin da aka sarrafa daga jerin gwano. Idan Logstash ya tsaya, duk saƙonni da saƙonnin da ba a sarrafa su ba waɗanda ba a sami tabbacin su ba suna cikin jerin gwano, kuma Logstash zai ci gaba da sarrafa su a gaba lokacin da ya fara.

gyara

Daidaitacce ta maɓallai a cikin fayil ɗin C:Logstashconfiglogstash.yml:

• queue.type: (mai yiwuwa dabi'u - persisted и memory (default)).
• path.queue: (hanyar zuwa babban fayil tare da fayilolin layi, waɗanda aka adana a cikin C: Logstashqueue ta tsohuwa).
• queue.page_capacity: (mafi girman girman shafin layi, ƙimar da aka saba shine 64mb).
• queue.drain: (gaskiya / ƙarya - yana ba da damar / hana dakatar da aikin layin layi kafin rufe Logstash. Ban bayar da shawarar kunna shi ba, saboda wannan zai shafi saurin kashe uwar garken kai tsaye).
• queue.max_events: (mafi girman adadin abubuwan da suka faru a cikin jerin gwano, tsoho shine 0 (mara iyaka)).
• queue.max_bytes: (mafi girman girman layi a cikin bytes, tsoho - 1024mb (1gb)).

Idan an saita queue.max_events и queue.max_bytes, sa'an nan kuma daina karɓar saƙonni a cikin layi lokacin da darajar kowane ɗayan waɗannan saitunan ya kai. Koyi ƙarin koyo game da Dagewar Queues a nan.

Misalin ɓangaren logstash.yml da ke da alhakin saita jerin gwano:

queue.type: persisted
queue.max_bytes: 10gb

gyara

Tsarin Logstash yawanci ya ƙunshi sassa uku, masu alhakin matakai daban-daban na sarrafa rajistan ayyukan masu shigowa: karɓar (sashen shigarwa), ɓarna (sashen tacewa) da aikawa zuwa Elastic (sashen fitarwa). A ƙasa za mu yi dubi sosai ga kowannensu.

Input

Muna karɓar rafi mai shigowa tare da ɗanyen rajistan ayyukan daga wakilan filebeat. Wannan plugin ɗin ne muke nunawa a sashin shigarwa:

input {
  beats {
    port => 5044
  }
}

Bayan wannan saitin, Logstash yana fara sauraron tashar jiragen ruwa 5044, kuma lokacin karɓar rajistan ayyukan, aiwatar da su daidai da saitunan sashin tacewa. Idan ya cancanta, zaku iya kunsa tashar don karɓar rajistan ayyukan daga filebit a cikin SSL. Kara karantawa game da saitin plugins na Beats a nan.

Tace

Duk rajistan ayyukan rubutu masu ban sha'awa don sarrafa abin da Exchange ke haifarwa suna cikin tsarin csv tare da filayen da aka siffanta a cikin fayil ɗin log ɗin kanta. Don tantance bayanan csv, Logstash yana ba mu plugins uku: rarrabuwa, csv da grok. Na farko shine ya fi yawa da sauri, amma ya jimre tare da yin nazari kawai mafi sauƙi rajistan ayyukan.
Misali, zai raba rikodin da ke zuwa gida biyu (saboda kasancewar waƙafi a cikin filin), wanda shine dalilin da ya sa za a karkatar da log ɗin ba daidai ba:

…,"MDB:GUID1, Mailbox:GUID2, Event:526545791, MessageClass:IPM.Note, CreationTime:2020-05-15T12:01:56.457Z, ClientType:MOMT, SubmissionAssistant:MailboxTransportSubmissionEmailAssistant",…

Ana iya amfani da shi lokacin tantance rajistan ayyukan, misali, IIS. A wannan yanayin, sashin tacewa zai yi kama da haka:

filter {
  if "IIS" in [tags] {
    dissect {
      mapping => {
        "message" => "%{date} %{time} %{s-ip} %{cs-method} %{cs-uri-stem} %{cs-uri-query} %{s-port} %{cs-username} %{c-ip} %{cs(User-Agent)} %{cs(Referer)} %{sc-status} %{sc-substatus} %{sc-win32-status} %{time-taken}"
      }
      remove_field => ["message"]
      add_field => { "application" => "exchange" }
    }
  }
} 

Tsarin Logstash yana ba ku damar amfani maganganun sharadi, don haka kawai za mu iya aika rajistan ayyukan da aka yi wa alama tare da filebeat tag zuwa dissect plugin IIS. A cikin plugin ɗin muna daidaita ƙimar filin tare da sunayensu, share asalin filin message, wanda ya ƙunshi shigarwa daga log ɗin, kuma za mu iya ƙara filin al'ada wanda zai, alal misali, ya ƙunshi sunan aikace-aikacen da muke tattara logs daga ciki.

Game da rajistan ayyukan bin diddigin, yana da kyau a yi amfani da plugin csv; yana iya aiwatar da fagage masu rikitarwa daidai:

filter {
  if "Tracking" in [tags] {
    csv {
      columns => ["date-time","client-ip","client-hostname","server-ip","server-hostname","source-context","connector-id","source","event-id","internal-message-id","message-id","network-message-id","recipient-address","recipient-status","total-bytes","recipient-count","related-recipient-address","reference","message-subject","sender-address","return-path","message-info","directionality","tenant-id","original-client-ip","original-server-ip","custom-data","transport-traffic-type","log-id","schema-version"]
      remove_field => ["message", "tenant-id", "schema-version"]
      add_field => { "application" => "exchange" }
    }
}

A cikin plugin ɗin muna daidaita ƙimar filin tare da sunayensu, share asalin filin message (da kuma filayen tenant-id и schema-version), wanda ya ƙunshi shigarwa daga log ɗin, kuma za mu iya ƙara filin al'ada, wanda zai, alal misali, ya ƙunshi sunan aikace-aikacen da muke tattara rajistan ayyukan.

A fitowa daga matakin tacewa, za mu karɓi takardu a cikin kima na farko, a shirye don gani a Kibana. Za mu rasa abubuwa masu zuwa:

• Za a gane filayen lambobi azaman rubutu, wanda ke hana aiki akan su. Wato, filayen time-taken IIS log, kazalika da filayen recipient-count и total-bites Log Tracking.
• Daidaitaccen lokaci tambarin daftarin aiki zai ƙunshi lokacin da aka sarrafa log ɗin, ba lokacin da aka rubuta shi a gefen uwar garken ba.
• filin recipient-address zai yi kama da ginin gine-gine guda ɗaya, wanda baya ba da izinin bincike don ƙidaya masu karɓar haruffa.

Lokaci ya yi da za a ƙara ɗan sihiri kaɗan zuwa aikin sarrafa log.

Maida filayen lamba

Abubuwan plugin ɗin dissect yana da zaɓi convert_datatype, wanda za a iya amfani dashi don canza filin rubutu zuwa tsarin dijital. Misali, kamar haka:

dissect {
  …
  convert_datatype => { "time-taken" => "int" }
  …
}

Yana da kyau a tuna cewa wannan hanya ta dace ne kawai idan filin zai kasance yana ƙunshe da kirtani. Zaɓin baya aiwatar da ƙima mara kyau daga filayen kuma yana jefa keɓantacce.

Don bin diddigin rajistan ayyukan, yana da kyau kada a yi amfani da irin wannan hanyar juyawa, tunda filayen recipient-count и total-bites yana iya zama fanko. Don canza waɗannan filayen yana da kyau a yi amfani da plugin canza:

mutate {
  convert => [ "total-bytes", "integer" ]
  convert => [ "recipient-count", "integer" ]
}

Rarraba adireshin mai karɓa zuwa cikin daidaikun masu karɓa

Hakanan ana iya magance wannan matsalar ta amfani da mutate plugin:

mutate {
  split => ["recipient_address", ";"]
}

Canza tambarin lokaci

Game da rajistan ayyukan bin diddigin, ana samun sauƙin warware matsalar ta hanyar plugin ɗin date, wanda zai taimake ka ka rubuta a cikin filin timestamp kwanan wata da lokaci a tsarin da ake buƙata daga filin date-time:

date {
  match => [ "date-time", "ISO8601" ]
  timezone => "Europe/Moscow"
  remove_field => [ "date-time" ]
}

A cikin yanayin rajistan ayyukan IIS, za mu buƙaci haɗa bayanan filin date и time ta amfani da mutate plugin, yi rajistar yankin lokacin da muke buƙata kuma sanya wannan tambarin lokaci a ciki timestamp amfani da kwanan wata plugin:

mutate { 
  add_field => { "data-time" => "%{date} %{time}" }
  remove_field => [ "date", "time" ]
}
date { 
  match => [ "data-time", "YYYY-MM-dd HH:mm:ss" ]
  timezone => "UTC"
  remove_field => [ "data-time" ]
}

Output

Ana amfani da sashin fitarwa don aika rajistan ayyukan da aka sarrafa zuwa mai karɓar log ɗin. Idan akwai aika kai tsaye zuwa Elastic, ana amfani da plugin maganin roba, wanda ke ƙayyadaddun adireshin uwar garken da samfurin sunan fihirisar don aika daftarin aiki:

output {
  elasticsearch {
    hosts => ["127.0.0.1:9200", "127.0.0.2:9200"]
    manage_template => false
    index => "Exchange-%{+YYYY.MM.dd}"
  }
}

Tsarin ƙarshe

Tsarin ƙarshe zai yi kama da haka:

input {
  beats {
    port => 5044
  }
}
 
filter {
  if "IIS" in [tags] {
    dissect {
      mapping => {
        "message" => "%{date} %{time} %{s-ip} %{cs-method} %{cs-uri-stem} %{cs-uri-query} %{s-port} %{cs-username} %{c-ip} %{cs(User-Agent)} %{cs(Referer)} %{sc-status} %{sc-substatus} %{sc-win32-status} %{time-taken}"
      }
      remove_field => ["message"]
      add_field => { "application" => "exchange" }
      convert_datatype => { "time-taken" => "int" }
    }
    mutate { 
      add_field => { "data-time" => "%{date} %{time}" }
      remove_field => [ "date", "time" ]
    }
    date { 
      match => [ "data-time", "YYYY-MM-dd HH:mm:ss" ]
      timezone => "UTC"
      remove_field => [ "data-time" ]
    }
  }
  if "Tracking" in [tags] {
    csv {
      columns => ["date-time","client-ip","client-hostname","server-ip","server-hostname","source-context","connector-id","source","event-id","internal-message-id","message-id","network-message-id","recipient-address","recipient-status","total-bytes","recipient-count","related-recipient-address","reference","message-subject","sender-address","return-path","message-info","directionality","tenant-id","original-client-ip","original-server-ip","custom-data","transport-traffic-type","log-id","schema-version"]
      remove_field => ["message", "tenant-id", "schema-version"]
      add_field => { "application" => "exchange" }
    }
    mutate {
      convert => [ "total-bytes", "integer" ]
      convert => [ "recipient-count", "integer" ]
      split => ["recipient_address", ";"]
    }
    date {
      match => [ "date-time", "ISO8601" ]
      timezone => "Europe/Moscow"
      remove_field => [ "date-time" ]
    }
  }
}
 
output {
  elasticsearch {
    hosts => ["127.0.0.1:9200", "127.0.0.2:9200"]
    manage_template => false
    index => "Exchange-%{+YYYY.MM.dd}"
  }
}

Hanyoyi masu amfani:

• Yadda ake shigar OpenJDK 11 akan Windows?
• Zazzage Logstash
• Elastic yana amfani da zaɓin da aka yanke UseConcMarkSweepGC #36828
• NSSM
• Layukan dagewa
• Abubuwan shigar da Beats plugin
• Logstash Dude, ina chainsaw na? Ina bukata in rarraba gundumomi na
• Yanke kayan aikin tacewa
• Yanayi
• Mutate tace plugin
• Kwanan tace plugin
• Elasticsearch fitarwa plugin

source: www.habr.com