Isu tiri shamwari neELK uye Exchange. Chikamu 2

Ndinoenderera mberi nenyaya yangu pamusoro pekuita shamwari Exchange uye ELK (kutanga pano) Rega ndikuyeuchidze kuti musanganiswa uyu unokwanisa kugadzirisa nhamba yakakura kwazvo yematanda pasina kuzeza. Panguva ino tichataura nezve maitiro ekuita kuti Exchange ishande neLogstash uye Kibana zvikamu.

Logstash mune ELK stack inoshandiswa nehungwaru kugadzirisa matanda uye kuagadzirira kuiswa muElastic muchimiro chemagwaro, pahwaro hwayo hwakanakira kuvaka maratidziro akasiyana muKibana.

Kuiswa

Rine zvikamu zviviri:

• Kuisa uye kugadzirisa iyo OpenJDK package.
• Kuisa uye kugadzirisa iyo Logstash package.

Kuisa uye kugadzirisa iyo OpenJDK package

Iyo OpenJDK package inofanirwa kudhaunirodwa uye kuburitswa mune chaiyo dhairekitori. Zvino nzira inoenda kudhairekitori iyi inofanirwa kuiswa mu $env:Path uye $env:JAVA_HOME zvinosiyana zveWindows operating system:

Ngatitarisei shanduro yeJava:

PS C:> java -version
openjdk version "13.0.1" 2019-10-15
OpenJDK Runtime Environment (build 13.0.1+9)
OpenJDK 64-Bit Server VM (build 13.0.1+9, mixed mode, sharing)

Kuisa uye kugadzirisa iyo Logstash package

Dhawunirodha iyo archive faira ine Logstash kugovera kubva pano. Iyo archive inofanirwa kuburitswa kumudzi we diski. Bvisa kune folda C:Program Files Izvo hazvina kukosha, Logstash inoramba kutanga zvakajairika. Ipapo iwe unofanirwa kupinda mufaira jvm.options inogadzirisa basa rekugovera RAM yeJava maitiro. Ini ndinokurudzira kutsanangura hafu ye server's RAM. Kana iine 16 GB ye RAM pabhodhi, saka makiyi akasarudzika ndeaya:

-Xms1g
-Xmx1g

inofanira kutsiviwa ne:

-Xms8g
-Xmx8g

Mukuwedzera, zvinokurudzirwa kutaura pamusoro pemutsara -XX:+UseConcMarkSweepGC. Zvimwe pamusoro peizvi pano. Nhanho inotevera ndeyekugadzira dhizaini yekumisikidza mulogstash.conf faira:

input {
 stdin{}
}
 
filter {
}
 
output {
 stdout {
 codec => "rubydebug"
 }
}

Nekugadziriswa uku, Logstash inoverenga data kubva kune koni, inoipfuudza kuburikidza nefirita isina chinhu, uye inoiburitsa ichidzosera kune koni. Kushandisa iyi gadziriro kunoedza kushanda kweLogstash. Kuti tiite izvi, ngatimhanyei inopindirana:

PS C:...bin> .logstash.bat -f .logstash.conf
...
[2019-12-19T11:15:27,769][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[2019-12-19T11:15:27,847][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-12-19T11:15:28,113][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

Logstash yakatangwa zvinobudirira pachiteshi 9600.

Yekupedzisira yekuisa nhanho: vhura Logstash seWindows sevhisi. Izvi zvinogona kuitwa, semuenzaniso, uchishandisa package NSSM:

PS C:...bin> .nssm.exe install logstash
Service "logstash" installed successfully!

kukanganisa kushivirira

Kuchengetedzeka kwematanda kana kuendeswa kubva kune sosi server kunovimbiswa neiyo Persistent Queues mechanism.

Iyo inoshanda sei

Kurongeka kwemitsara panguva yekugadziriswa kwelogi ndeiyi: kuisa → mutsara → sefa + kubuda.

Iyo yekuisa plugin inogamuchira data kubva kune regi sosi, inoinyora kumutsetse, uye inotumira simbiso kuti data ragamuchirwa kune kwakabva.

Mharidzo kubva pamutsetse inogadziriswa neLogstash, yakapfuura nepasefa uye inobuda plugin. Paunenge uchigamuchira simbiso kubva kune yakabuda kuti irogi ratumirwa, Logstash inobvisa iyo yakagadziriswa logi kubva pamutsetse. Kana Logstash ikamira, mameseji ese asina kugadziridzwa uye mameseji ayo asina simbiso yakagamuchirwa anoramba ari mumutsara, uye Logstash icharamba ichizvigadzirisa nguva inotevera paichatanga.

kuchinja

Inogadziriswa nemakiyi mufaira C:Logstashconfiglogstash.yml:

• queue.type: (zvinogoneka kukosha - persisted и memory (default)).
• path.queue: (nzira inoenda kune folda ine mafaera emutsetse, ayo anochengetwa muC: Logstashqueue nekusarudzika).
• queue.page_capacity: (yakanyanya mutsara peji saizi, default kukosha ndeye 64mb).
• queue.drain: (chokwadi / nhema - inogonesa / inomisa kumisa mitsara yekugadzirisa isati yavhara Logstash. Handikurudziri kuigonesa, nokuti izvi zvichakanganisa zvakananga kukurumidza kwevhavha yevhavha).
• queue.max_events: (yakawanda nhamba yezviitiko mumutsara, default ndeye 0 (isina muganho)).
• queue.max_bytes: (hukuru hwemutsetse saizi mumabhaiti, default - 1024mb (1gb)).

Kana configured queue.max_events и queue.max_bytes, ipapo mameseji anomira kugamuchirwa mumutsara kana kukosha kwechero ipi yezvirongwa izvi kwasvikwa. Dzidza zvakawanda nezve Persistent Queues pano.

Muenzaniso wechikamu che logstash.yml ine basa rekumisikidza mutsara:

queue.type: persisted
queue.max_bytes: 10gb

kuchinja

Kugadziriswa kweLogstash kunowanzo kuve nezvikamu zvitatu, zvinotarisana nezvikamu zvakasiyana zvekugadzirisa matanda anouya: kugamuchira (chikamu chekupinza), parsing (sefa chikamu) uye kutumira kuElastic (chikamu chekubuda). Pazasi isu tichanyatso tarisa kune mumwe nemumwe wavo.

chiyamuro

Isu tinogashira rukova runouya nematanda akaomeswa kubva kune filebeat agents. Ndiyo plugin iyi yatinoratidza muchikamu chekuisa:

input {
  beats {
    port => 5044
  }
}

Mushure mekugadzirisa uku, Logstash inotanga kuteerera kuchiteshi 5044, uye kana ichigamuchira matanda, inoagadzirisa maererano nekugadzirisa kwechikamu chesefa. Kana zvichidikanwa, unogona kuputira chiteshi chekugamuchira matanda kubva filebit muSSL. Verenga zvakawanda nezve beats plugin marongero pano.

firita

Ese mameseji matanda anogadzirwa neShanduro anonakidza kugadzirisa ari mu csv fomati ine minda inotsanangurwa mugwaro faira pacharo. Nekuparadzanisa marekodhi ecsv, Logstash inotipa maplugins matatu: kubvarura, csv uye grok. Yekutanga ndiyo yakanyanya быстрый, asi zvinokwanisa kutara chete matanda akareruka.
Semuenzaniso, ichatsemura rekodhi inotevera kuita maviri (nekuda kwekuvapo kwecomma mukati memunda), ndosaka irogi richizopatsanurwa zvisizvo:

…,"MDB:GUID1, Mailbox:GUID2, Event:526545791, MessageClass:IPM.Note, CreationTime:2020-05-15T12:01:56.457Z, ClientType:MOMT, SubmissionAssistant:MailboxTransportSubmissionEmailAssistant",…

Inogona kushandiswa pakudhirowa matanda, semuenzaniso, IIS. Muchiitiko ichi, chikamu chesefa chinogona kutaridzika seizvi:

filter {
  if "IIS" in [tags] {
    dissect {
      mapping => {
        "message" => "%{date} %{time} %{s-ip} %{cs-method} %{cs-uri-stem} %{cs-uri-query} %{s-port} %{cs-username} %{c-ip} %{cs(User-Agent)} %{cs(Referer)} %{sc-status} %{sc-substatus} %{sc-win32-status} %{time-taken}"
      }
      remove_field => ["message"]
      add_field => { "application" => "exchange" }
    }
  }
} 

Logstash gadziriso inobvumidza iwe kushandisa zvirevo zvezvimiso, saka isu tinogona chete kutumira matanda aive akaiswa neiyo filebeat tag kune dissect plugin. IIS. Mukati me plugin tinofananidza kukosha kwemunda nemazita avo, bvisa iyo yekutanga munda message, iyo yaive nekupinda kubva kurogi, uye isu tinogona kuwedzera tsika yemunda iyo, semuenzaniso, ine zita rekushandisa kwatinounganidza matanda.

Panyaya yekutevera matanda, zviri nani kushandisa csv plugin; inogona kunyatso gadzira minda yakaoma:

filter {
  if "Tracking" in [tags] {
    csv {
      columns => ["date-time","client-ip","client-hostname","server-ip","server-hostname","source-context","connector-id","source","event-id","internal-message-id","message-id","network-message-id","recipient-address","recipient-status","total-bytes","recipient-count","related-recipient-address","reference","message-subject","sender-address","return-path","message-info","directionality","tenant-id","original-client-ip","original-server-ip","custom-data","transport-traffic-type","log-id","schema-version"]
      remove_field => ["message", "tenant-id", "schema-version"]
      add_field => { "application" => "exchange" }
    }
}

Mukati me plugin tinofananidza kukosha kwemunda nemazita avo, bvisa iyo yekutanga munda message (uyewo minda tenant-id и schema-version), iyo yaive nekupinda kubva kurogi, uye isu tinogona kuwedzera tsika yemunda, iyo, semuenzaniso, ine zita rekushandisa kwatinounganidza matanda.

Pakubuda kubva padanho rekusefa, tinogashira magwaro mukufungidzira kwekutanga, akagadzirira kuoneswa muKibana. Tichange tichishaya zvinotevera:

• Numeric minda ichaonekwa semavara, izvo zvinodzivirira kushanda pazviri. Kureva, minda time-taken IIS log, pamwe neminda recipient-count и total-bites Log Tracking.
• Iyo yakajairwa gwaro timestamp ichava nenguva iyo log yakagadziriswa, kwete nguva yayakanyorwa padivi reseva.
• munda recipient-address ichaita seimwe nzvimbo yekuvaka, iyo isingabvumiri kuongororwa kuverenga vagamuchiri vetsamba.

Inguva yekuwedzera mashiripiti mashoma kune iyo log processing process.

Kushandura nhamba dzezvikamu

Iyo dissect plugin ine sarudzo convert_datatype, iyo inogona kushandiswa kushandura ndima yemavara kune chimiro chedhijitari. Semuenzaniso, seizvi:

dissect {
  …
  convert_datatype => { "time-taken" => "int" }
  …
}

Zvakakosha kuyeuka kuti nzira iyi yakakodzera chete kana munda uchange uine tambo. Iyo sarudzo haigadzirise Null kukosha kubva kuminda uye inokanda mutsauko.

Kune matanda ekutevera, zviri nani kusashandisa nzira yekushandura yakafanana, kubva kuminda recipient-count и total-bites inogona kunge isina chinhu. Kushandura minda iyi zviri nani kushandisa plugin shanduka:

mutate {
  convert => [ "total-bytes", "integer" ]
  convert => [ "recipient-count", "integer" ]
}

Kupatsanura recipient_address mumunhu anenge agamuchira

Dambudziko iri rinogonawo kugadziriswa uchishandisa mutate plugin:

mutate {
  split => ["recipient_address", ";"]
}

Kuchinja timetamp

Munyaya yekutevera matanda, dambudziko rinogadziriswa nyore nyore ne plugin zuva, izvo zvichakubatsira kunyora mumunda timestamp zuva uye nguva mune inodiwa fomati kubva kumunda date-time:

date {
  match => [ "date-time", "ISO8601" ]
  timezone => "Europe/Moscow"
  remove_field => [ "date-time" ]
}

Panyaya yeIIS matanda, isu tichada kubatanidza data remunda date и time uchishandisa mutate plugin, nyoresa nguva yenguva yatinoda uye isa iyi nguva chitambi mukati timestamp uchishandisa date plugin:

mutate { 
  add_field => { "data-time" => "%{date} %{time}" }
  remove_field => [ "date", "time" ]
}
date { 
  match => [ "data-time", "YYYY-MM-dd HH:mm:ss" ]
  timezone => "UTC"
  remove_field => [ "data-time" ]
}

goho

Chikamu chinobuda chinoshandiswa kutumira matanda akagadziriswa kune anogamuchira log. Muchiitiko chekutumira zvakananga kuElastic, plugin inoshandiswa elasticsearch, iyo inotsanangura iyo server kero uye index zita template yekutumira iyo yakagadzirwa gwaro:

output {
  elasticsearch {
    hosts => ["127.0.0.1:9200", "127.0.0.2:9200"]
    manage_template => false
    index => "Exchange-%{+YYYY.MM.dd}"
  }
}

Final configuration

Iyo yekupedzisira gadziriso ichaita seizvi:

input {
  beats {
    port => 5044
  }
}
 
filter {
  if "IIS" in [tags] {
    dissect {
      mapping => {
        "message" => "%{date} %{time} %{s-ip} %{cs-method} %{cs-uri-stem} %{cs-uri-query} %{s-port} %{cs-username} %{c-ip} %{cs(User-Agent)} %{cs(Referer)} %{sc-status} %{sc-substatus} %{sc-win32-status} %{time-taken}"
      }
      remove_field => ["message"]
      add_field => { "application" => "exchange" }
      convert_datatype => { "time-taken" => "int" }
    }
    mutate { 
      add_field => { "data-time" => "%{date} %{time}" }
      remove_field => [ "date", "time" ]
    }
    date { 
      match => [ "data-time", "YYYY-MM-dd HH:mm:ss" ]
      timezone => "UTC"
      remove_field => [ "data-time" ]
    }
  }
  if "Tracking" in [tags] {
    csv {
      columns => ["date-time","client-ip","client-hostname","server-ip","server-hostname","source-context","connector-id","source","event-id","internal-message-id","message-id","network-message-id","recipient-address","recipient-status","total-bytes","recipient-count","related-recipient-address","reference","message-subject","sender-address","return-path","message-info","directionality","tenant-id","original-client-ip","original-server-ip","custom-data","transport-traffic-type","log-id","schema-version"]
      remove_field => ["message", "tenant-id", "schema-version"]
      add_field => { "application" => "exchange" }
    }
    mutate {
      convert => [ "total-bytes", "integer" ]
      convert => [ "recipient-count", "integer" ]
      split => ["recipient_address", ";"]
    }
    date {
      match => [ "date-time", "ISO8601" ]
      timezone => "Europe/Moscow"
      remove_field => [ "date-time" ]
    }
  }
}
 
output {
  elasticsearch {
    hosts => ["127.0.0.1:9200", "127.0.0.2:9200"]
    manage_template => false
    index => "Exchange-%{+YYYY.MM.dd}"
  }
}

Useful links:

• Maitiro ekuisa OpenJDK 11 paWindows?
• Download Logstash
• Elastic inoshandisa yakanyimwa sarudzo UseConcMarkSweepGC #36828
• NSSM
• Persistent Queues
• Inorova plugin yekupinza
• Logstash Dude, chainsaw yangu iripi? Ndinofanira kucheka matanda angu
• Dissect filter plugin
• Mamiriro ezvinhu
• Mutate filter plugin
• Date filter plugin
• Elasticsearch yakabuda plugin

Source: www.habr.com