🥇 A jẹ ọrẹ pẹlu ELK ati Exchange. Apá 2

Mo tẹsiwaju itan mi nipa bi o ṣe le ṣe awọn ọrẹ Exchange ati ELK (ibẹrẹ nibi). Jẹ ki n leti pe apapo yii ni agbara lati ṣiṣẹ nọmba ti o tobi pupọ ti awọn akọọlẹ laisi iyemeji. Ni akoko yii a yoo sọrọ nipa bii o ṣe le gba Exchange ṣiṣẹ pẹlu awọn paati Logstash ati Kibana.

Logstash ni akopọ ELK ni a lo lati ṣe ilana awọn igbasilẹ ni oye ati mura wọn fun gbigbe ni Elastic ni irisi awọn iwe aṣẹ, lori ipilẹ eyiti o rọrun lati kọ ọpọlọpọ awọn iwoye ni Kibana.

eto

Ni awọn ipele meji:

Fifi sori ẹrọ ati tunto package OpenJDK.
Fifi sori ẹrọ ati tunto package Logstash.

Fifi sori ẹrọ ati tunto package OpenJDK

Awọn akojọpọ OpenJDK gbọdọ jẹ igbasilẹ ati ṣiṣi silẹ sinu iwe ilana kan pato. Lẹhinna ọna si itọsọna yii gbọdọ wa ni titẹ si $ env:Path ati $ env: JAVA_HOME awọn oniyipada ti ẹrọ iṣẹ Windows:

Jẹ ki a ṣayẹwo ẹya Java:

PS C:> java -version
openjdk version "13.0.1" 2019-10-15
OpenJDK Runtime Environment (build 13.0.1+9)
OpenJDK 64-Bit Server VM (build 13.0.1+9, mixed mode, sharing)

Fifi sori ẹrọ ati tunto package Logstash

Ṣe igbasilẹ faili pamosi pẹlu pinpin Logstash lati ibi. Ile-ipamọ gbọdọ jẹ ṣiṣi silẹ si gbongbo disk naa. Yọọ si folda C:Program Files Ko tọ si, Logstash yoo kọ lati bẹrẹ ni deede. Lẹhinna o nilo lati wọle si faili naa jvm.options awọn atunṣe lodidi fun ipin Ramu fun ilana Java. Mo ṣeduro pato idaji Ramu olupin naa. Ti o ba ni 16 GB ti Ramu lori ọkọ, lẹhinna awọn bọtini aiyipada ni:

-Xms1g
-Xmx1g

gbọdọ paarọ rẹ pẹlu:

-Xms8g
-Xmx8g

Ni afikun, o ni imọran lati sọ asọye laini naa -XX:+UseConcMarkSweepGC. Diẹ ẹ sii nipa eyi nibi. Igbesẹ t’okan ni lati ṣẹda iṣeto aifọwọyi ninu faili logstash.conf:

input {
 stdin{}
}
 
filter {
}
 
output {
 stdout {
 codec => "rubydebug"
 }
}

Pẹlu iṣeto ni yii, Logstash ka data lati console, kọja nipasẹ àlẹmọ ti o ṣofo, ati jade pada si console. Lilo iṣeto yii yoo ṣe idanwo iṣẹ-ṣiṣe ti Logstash. Lati ṣe eyi, jẹ ki a ṣiṣẹ ni ipo ibaraenisepo:

PS C:...bin> .logstash.bat -f .logstash.conf
...
[2019-12-19T11:15:27,769][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[2019-12-19T11:15:27,847][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-12-19T11:15:28,113][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

Logstash ṣe ifilọlẹ ni aṣeyọri lori ibudo 9600.

Igbesẹ fifi sori ẹrọ ikẹhin: ifilọlẹ Logstash bi iṣẹ Windows kan. Eyi le ṣee ṣe, fun apẹẹrẹ, lilo package NSSM:

PS C:...bin> .nssm.exe install logstash
Service "logstash" installed successfully!

ifarada ẹbi

Aabo ti awọn igbasilẹ nigbati o ba gbe lati olupin orisun jẹ iṣeduro nipasẹ ẹrọ Awọn Queues Jubẹẹlo.

Bawo ni o ṣe n ṣiṣẹ

Ifilelẹ ti awọn ila lakoko ṣiṣe log jẹ: titẹ sii → isinyi → àlẹmọ + iṣẹjade.

Ohun itanna titẹ sii gba data lati orisun log kan, kọwe si isinyi, o si fi ijẹrisi ranṣẹ pe a ti gba data naa si orisun.

Awọn ifiranṣẹ lati awọn ti isinyi ti wa ni ilọsiwaju nipasẹ Logstash, koja nipasẹ awọn àlẹmọ ati awọn ti o wu itanna. Nigbati o ba n gba ijẹrisi lati inu abajade pe a ti fi iwe ranṣẹ, Logstash yọkuro iwe ti a ṣe ilana lati isinyi. Ti Logstash ba duro, gbogbo awọn ifiranṣẹ ti ko ni ilọsiwaju ati awọn ifiranṣẹ eyiti ko si ijẹrisi ti o gba wa ninu isinyi, ati Logstash yoo tẹsiwaju lati ṣe ilana wọn nigbamii ti o ba bẹrẹ.

Ṣe akanṣe

Atunṣe nipasẹ awọn bọtini inu faili C:Logstashconfiglogstash.yml:

queue.type: (awọn iye to ṣeeṣe - persisted и memory (default)).
path.queue: (ọna si folda pẹlu awọn faili isinyi, eyiti o wa ni ipamọ ni C: Logstashqueue nipasẹ aiyipada).
queue.page_capacity: (iwọn oju-iwe isinyi ti o pọju, iye aiyipada jẹ 64mb).
queue.drain: (otitọ / eke - mu ṣiṣẹ / mu idaduro idaduro isinyi ṣaaju ki o to pa Logstash. Emi ko ṣe iṣeduro muu ṣiṣẹ, nitori eyi yoo ni ipa taara iyara ti tiipa olupin).
queue.max_events: (o pọju nọmba ti awọn iṣẹlẹ ninu awọn ti isinyi, aiyipada 0 (Kolopin)).
queue.max_bytes: (o pọju ti isinyi iwọn ni awọn baiti, aiyipada - 1024mb (1gb)).

Ti o ba tunto queue.max_events и queue.max_bytes, lẹhinna awọn ifiranṣẹ da gbigba wọle sinu isinyin nigbati iye eyikeyi ninu awọn eto wọnyi ba ti de. Kọ ẹkọ diẹ sii nipa Awọn isinyi ti o tẹsiwaju nibi.

Apeere ti apakan logstash.yml ti o ni iduro fun iṣeto ti isinyi:

queue.type: persisted
queue.max_bytes: 10gb

Ṣe akanṣe

Iṣeto Logstash nigbagbogbo ni awọn ẹya mẹta, lodidi fun awọn ipele oriṣiriṣi ti sisẹ awọn iforukọsilẹ ti nwọle: gbigba (apakan titẹ sii), sisọ (apakan àlẹmọ) ati fifiranṣẹ si Elastic (apakan abajade). Ni isalẹ a yoo ṣe akiyesi diẹ sii ni ọkọọkan wọn.

Input

A gba ṣiṣan ti nwọle pẹlu awọn akọọlẹ aise lati awọn aṣoju filebeat. Ohun itanna yii ni a tọka si ni apakan titẹ sii:

input {
  beats {
    port => 5044
  }
}

Lẹhin iṣeto yii, Logstash bẹrẹ gbigbọ ibudo 5044, ati nigbati o ba ngba awọn igbasilẹ, ṣe ilana wọn ni ibamu si awọn eto ti apakan àlẹmọ. Ti o ba jẹ dandan, o le fi ipari si ikanni fun gbigba awọn igbasilẹ lati filebit ni SSL. Ka diẹ sii nipa awọn eto itanna lu lu nibi.

Àlẹmọ

Gbogbo awọn akọọlẹ ọrọ ti o nifẹ fun sisẹ ti awọn ipilẹṣẹ paṣipaarọ wa ni ọna kika csv pẹlu awọn aaye ti a ṣalaye ninu faili log funrararẹ. Fun sisọ awọn igbasilẹ csv, Logstash fun wa ni awọn afikun mẹta: pinpin, csv ati grk. Ni igba akọkọ ti o jẹ julọ aṣiṣe, ṣugbọn copes pẹlu parsing nikan awọn alinisoro àkọọlẹ.
Fun apẹẹrẹ, yoo pin igbasilẹ atẹle si meji (nitori wiwa aami idẹsẹ kan ninu aaye), eyiti o jẹ idi ti akọọlẹ naa yoo ṣe itupalẹ ni aṣiṣe:

…,"MDB:GUID1, Mailbox:GUID2, Event:526545791, MessageClass:IPM.Note, CreationTime:2020-05-15T12:01:56.457Z, ClientType:MOMT, SubmissionAssistant:MailboxTransportSubmissionEmailAssistant",…

O le ṣee lo nigbati o ba n ṣe itupalẹ awọn akọọlẹ, fun apẹẹrẹ, IIS. Ni ọran yii, apakan àlẹmọ le dabi eyi:

filter {
  if "IIS" in [tags] {
    dissect {
      mapping => {
        "message" => "%{date} %{time} %{s-ip} %{cs-method} %{cs-uri-stem} %{cs-uri-query} %{s-port} %{cs-username} %{c-ip} %{cs(User-Agent)} %{cs(Referer)} %{sc-status} %{sc-substatus} %{sc-win32-status} %{time-taken}"
      }
      remove_field => ["message"]
      add_field => { "application" => "exchange" }
    }
  }
}

Logstash iṣeto ni faye gba o lati lo àídájú gbólóhùn, nitorinaa a le fi awọn akọọlẹ ranṣẹ nikan ti a samisi pẹlu tag filebeat si ohun itanna dissect IIS. Ninu ohun itanna a baramu awọn iye aaye pẹlu awọn orukọ wọn, paarẹ aaye atilẹba naa message, eyiti o ni titẹ sii lati inu akọọlẹ, ati pe a le ṣafikun aaye aṣa kan ti yoo, fun apẹẹrẹ, ni orukọ ohun elo ninu eyiti a gba awọn akọọlẹ.

Ni ọran ti awọn igbasilẹ ipasẹ, o dara lati lo ohun itanna csv; o le ṣe ilana awọn aaye eka ni deede:

filter {
  if "Tracking" in [tags] {
    csv {
      columns => ["date-time","client-ip","client-hostname","server-ip","server-hostname","source-context","connector-id","source","event-id","internal-message-id","message-id","network-message-id","recipient-address","recipient-status","total-bytes","recipient-count","related-recipient-address","reference","message-subject","sender-address","return-path","message-info","directionality","tenant-id","original-client-ip","original-server-ip","custom-data","transport-traffic-type","log-id","schema-version"]
      remove_field => ["message", "tenant-id", "schema-version"]
      add_field => { "application" => "exchange" }
    }
}

Ninu ohun itanna a baramu awọn iye aaye pẹlu awọn orukọ wọn, paarẹ aaye atilẹba naa message (ati awọn aaye tun tenant-id и schema-version), eyiti o ni titẹ sii lati inu akọọlẹ, ati pe a le ṣafikun aaye aṣa kan, eyiti yoo, fun apẹẹrẹ, ni orukọ ohun elo ninu eyiti a gba awọn akọọlẹ.

Ni ijade kuro ni ipele sisẹ, a yoo gba awọn iwe aṣẹ ni isunmọ akọkọ, ti ṣetan fun iworan ni Kibana. A yoo padanu nkan wọnyi:

Awọn aaye nọmba yoo jẹ idanimọ bi ọrọ, eyiti o ṣe idiwọ awọn iṣẹ ṣiṣe lori wọn. Eyun, awọn aaye time-taken IIS log, ati awọn aaye recipient-count и total-bites Wọle Àtòjọ.
Awọn akoko iwe aṣẹ boṣewa yoo ni akoko ti a ṣe ilana akọọlẹ naa, kii ṣe akoko ti a kọ si ẹgbẹ olupin naa.
Aaye recipient-address yoo dabi aaye ikole kan, eyiti ko gba laaye fun itupalẹ lati ka awọn olugba ti awọn lẹta.

O to akoko lati ṣafikun idan kekere kan si ilana ṣiṣe log.

Yiyipada nomba aaye

Awọn ohun itanna dissect ni aṣayan kan convert_datatype, eyi ti o le ṣee lo lati yi aaye ọrọ pada si ọna kika oni-nọmba kan. Fun apẹẹrẹ, bii eyi:

dissect {
  …
  convert_datatype => { "time-taken" => "int" }
  …
}

O tọ lati ranti pe ọna yii dara nikan ti aaye naa yoo ni pato okun kan. Aṣayan naa ko ṣe ilana awọn iye Null lati awọn aaye ati ju imukuro kan.

Fun awọn igbasilẹ titele, o dara ki a ma lo ọna iyipada ti o jọra, niwon awọn aaye recipient-count и total-bites le jẹ ofo. Lati yi awọn aaye wọnyi pada o dara lati lo ohun itanna kan ṣatunṣe:

mutate {
  convert => [ "total-bytes", "integer" ]
  convert => [ "recipient-count", "integer" ]
}

Pipin olugba_adirẹsi si awọn olugba kọọkan

Iṣoro yii tun le yanju nipa lilo ohun itanna mutate:

mutate {
  split => ["recipient_address", ";"]
}

Yiyipada awọn timestamp

Ninu ọran ti awọn akọọlẹ ipasẹ, iṣoro naa ni irọrun ni irọrun nipasẹ ohun itanna ọjọ, eyi ti yoo ran o kọ ni awọn aaye timestamp ọjọ ati akoko ni ọna kika ti a beere lati aaye date-time:

date {
  match => [ "date-time", "ISO8601" ]
  timezone => "Europe/Moscow"
  remove_field => [ "date-time" ]
}

Ninu ọran ti awọn akọọlẹ IIS, a yoo nilo lati darapo data aaye date и time lilo ohun itanna mutate, forukọsilẹ agbegbe aago ti a nilo ki o si fi ontẹ akoko yii sinu timestamp lilo ohun itanna ọjọ:

mutate { 
  add_field => { "data-time" => "%{date} %{time}" }
  remove_field => [ "date", "time" ]
}
date { 
  match => [ "data-time", "YYYY-MM-dd HH:mm:ss" ]
  timezone => "UTC"
  remove_field => [ "data-time" ]
}

o wu

Abala ti o wujade ni a lo lati firanṣẹ awọn igbasilẹ ti a ṣe ilana si olugba log. Ni ọran ti fifiranṣẹ taara si Rirọ, a lo ohun itanna kan rirọsi, eyiti o ṣalaye adirẹsi olupin ati awoṣe orukọ atọka fun fifiranṣẹ iwe ti ipilẹṣẹ:

output {
  elasticsearch {
    hosts => ["127.0.0.1:9200", "127.0.0.2:9200"]
    manage_template => false
    index => "Exchange-%{+YYYY.MM.dd}"
  }
}

Ik iṣeto ni

Iṣeto ikẹhin yoo dabi eyi:

input {
  beats {
    port => 5044
  }
}
 
filter {
  if "IIS" in [tags] {
    dissect {
      mapping => {
        "message" => "%{date} %{time} %{s-ip} %{cs-method} %{cs-uri-stem} %{cs-uri-query} %{s-port} %{cs-username} %{c-ip} %{cs(User-Agent)} %{cs(Referer)} %{sc-status} %{sc-substatus} %{sc-win32-status} %{time-taken}"
      }
      remove_field => ["message"]
      add_field => { "application" => "exchange" }
      convert_datatype => { "time-taken" => "int" }
    }
    mutate { 
      add_field => { "data-time" => "%{date} %{time}" }
      remove_field => [ "date", "time" ]
    }
    date { 
      match => [ "data-time", "YYYY-MM-dd HH:mm:ss" ]
      timezone => "UTC"
      remove_field => [ "data-time" ]
    }
  }
  if "Tracking" in [tags] {
    csv {
      columns => ["date-time","client-ip","client-hostname","server-ip","server-hostname","source-context","connector-id","source","event-id","internal-message-id","message-id","network-message-id","recipient-address","recipient-status","total-bytes","recipient-count","related-recipient-address","reference","message-subject","sender-address","return-path","message-info","directionality","tenant-id","original-client-ip","original-server-ip","custom-data","transport-traffic-type","log-id","schema-version"]
      remove_field => ["message", "tenant-id", "schema-version"]
      add_field => { "application" => "exchange" }
    }
    mutate {
      convert => [ "total-bytes", "integer" ]
      convert => [ "recipient-count", "integer" ]
      split => ["recipient_address", ";"]
    }
    date {
      match => [ "date-time", "ISO8601" ]
      timezone => "Europe/Moscow"
      remove_field => [ "date-time" ]
    }
  }
}
 
output {
  elasticsearch {
    hosts => ["127.0.0.1:9200", "127.0.0.2:9200"]
    manage_template => false
    index => "Exchange-%{+YYYY.MM.dd}"
  }
}

Awọn ọna asopọ to wulo:

orisun: www.habr.com

A jẹ ọrẹ pẹlu ELK ati Exchange. Apa keji

eto

Fifi sori ẹrọ ati tunto package Logstash

ifarada ẹbi

Bawo ni o ṣe n ṣiṣẹ

Ṣe akanṣe

Ṣe akanṣe

Input

Àlẹmọ

Yiyipada nomba aaye

Pipin olugba_adirẹsi si awọn olugba kọọkan

Yiyipada awọn timestamp

o wu

Ik iṣeto ni

Fi ọrọìwòye kun Fagilee esi