He hoa matou ki a ELK me Exchange. Wāhanga 2

Ka haere tonu taku korero me pehea te whakahoa a Exchange me ELK (timata konei). Me whakamahara ahau ki a koe ka taea e tenei huinga te tukatuka i te maha rawa o nga rakau me te kore e ruarua. I tenei wa ka korero maatau me pehea te mahi a Exchange me nga waahanga Logstash me Kibana.

Ko te Logstash i roto i te puranga ELK e whakamahia ana ki te whakahaere maamaa i nga raarangi me te whakarite mo te tuunga ki Elastic i roto i te ahua o nga tuhinga, i runga i te mea he pai ki te hanga i nga momo tirohanga ki Kibana.

tāutanga

E rua nga waahanga:

• Te whakauru me te whirihora i te kete OpenJDK.
• Te whakauru me te whirihora i te kete Logstash.

Te whakauru me te whirihora i te kete OpenJDK

Me tango te kete OpenJDK me te wetewete ki tetahi raarangi motuhake. Na me whakauru te ara ki tenei whaiaronga ki nga taurangi $env:Path me $env:JAVA_HOME o te punaha whakahaere Windows:

Kia tirohia te putanga Java:

PS C:> java -version
openjdk version "13.0.1" 2019-10-15
OpenJDK Runtime Environment (build 13.0.1+9)
OpenJDK 64-Bit Server VM (build 13.0.1+9, mixed mode, sharing)

Te whakauru me te whirihora i te kete Logstash

Tikiake i te kōnae pūranga me te tohatoha Logstash mai i konei. Me wetewete te puranga ki te putake o te kōpae. Wewetehia ki te kōpaki C:Program Files Kaore he utu, ka kore a Logstash e pai ki te tiimata. Na me uru koe ki te konae jvm.options whakatika te kawenga mo te tohatoha RAM mo te tukanga Java. Ka tūtohu ahau kia tohua te haurua o te RAM o te tūmau. Mena he 16 GB o RAM kei runga, ko nga taviri taunoa ko:

-Xms1g
-Xmx1g

me whakakapi ki:

-Xms8g
-Xmx8g

I tua atu, he mea tika ki te korero i te rarangi -XX:+UseConcMarkSweepGC. Ētahi atu kōrero mō tēnei konei. Ko te mahi e whai ake nei ko te hanga whirihoranga taunoa ki te konae logstash.conf:

input {
 stdin{}
}
 
filter {
}
 
output {
 stdout {
 codec => "rubydebug"
 }
}

Ma tenei whirihoranga, ka panuihia e Logstash nga raraunga mai i te papatohu, ka tukuna ma te tātari kore, ka whakahoki ano ki te papatohu. Ma te whakamahi i tenei whirihoranga ka whakamatau i te mahi a Logstash. Hei mahi i tenei, me whakahaere i roto i te aratau tauwhitiwhiti:

PS C:...bin> .logstash.bat -f .logstash.conf
...
[2019-12-19T11:15:27,769][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[2019-12-19T11:15:27,847][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-12-19T11:15:28,113][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

I angitu te whakarewanga o Logstash ki te tauranga 9600.

Ko te taahiraa whakauru whakamutunga: whakarewahia a Logstash hei ratonga Windows. Ka taea tenei, hei tauira, ma te whakamahi i te kete NSSM:

PS C:...bin> .nssm.exe install logstash
Service "logstash" installed successfully!

te manawanui ki te he

Ko te haumarutanga o nga pororakau ina whakawhitia mai i te tuumau puna ka whakapumauhia e te tikanga Tutira Tonu.

Pehea te mahi

Ko te whakatakotoranga o nga rarangi i te wa e tukatuka ana i te raarangi ko: whakauru → tūtira → tātari + putanga.

Ka whiwhi raraunga te mono whakauru mai i te puna rangitaki, ka tuhia ki te rarangi, ka tukuna he whakapumautanga kua tae mai nga raraunga ki te puna.

Ko nga karere mai i te rarangi ka tukatukahia e Logstash, i tukuna i roto i te tātari me te mono putanga. I te wa e whiwhi ana i te whakapumautanga mai i te whakaputanga kua tukuna te pororakau, ka tangohia e Logstash te raarangi mahi mai i te rarangi. Mena ka mutu a Logstash, ka noho tonu nga karere me nga karere kaore ano kia tukaia, karekau he whakapumautanga i tae mai ki te rarangi, ka mahi tonu a Logstash i te wa e timata ai.

whakatikatikanga

Ka taea te whakarite ma nga ki o te konae C:Logstashconfiglogstash.yml:

• queue.type: (nga uara pea- persisted и memory (default)).
• path.queue: (ara ki te kōpaki me nga konae tūtira, ka penapena ki C:Logstashqueue ma te taunoa).
• queue.page_capacity: (te rahi o te wharangi rarangi, ko te uara taunoa he 64mb).
• queue.drain: (tika/teka - ka taea/whakakorea te whakamutu i te tukatuka tūtira i mua i te katinga i te Logstash. Kaore au e kii kia whakahohea, na te mea ka pa te tere o te katinga o te tūmau).
• queue.max_events: (te maha o nga huihuinga i roto i te rarangi, ko te taunoa he 0 (kore mutunga)).
• queue.max_bytes: (te rahi o te rarangi i roto i nga paita, taunoa - 1024mb (1gb)).

Mena kua whirihora queue.max_events и queue.max_bytes, ka mutu te urunga o nga karere ki te rarangi ina tae ki te uara o enei tautuhinga. Ako atu mo nga Tutira Tonu konei.

He tauira o te wahanga o logstash.yml te kawenga mo te whakatu i te rarangi:

queue.type: persisted
queue.max_bytes: 10gb

whakatikatikanga

Ko te whirihoranga Logstash e toru nga waahanga, he kawenga mo nga waahanga rereke o te tukatuka i nga raarangi taumai: te tango (waahanga whakauru), te parsing (waahanga tātari) me te tuku ki te Elastic (waahanga putanga). Kei raro nei ka ata titiro tatou ki ia o ratou.

tāuru

Ka whiwhi matou i te awa taumai me nga poro rakau mai i nga kaihoko filebeat. Ko tenei mono ka tohuhia e matou i te waahanga whakauru:

input {
  beats {
    port => 5044
  }
}

I muri i tenei whirihoranga, ka timata a Logstash ki te whakarongo ki te tauranga 5044, a, ka whiwhi i nga raarangi, ka tukatukahia kia rite ki nga tautuhinga o te waahanga tātari. Mena e tika ana, ka taea e koe te takai i te hongere mo te tango i nga raarangi mai i te filebit i roto i te SSL. Pānuitia atu mo nga tautuhinga mono beats konei.

Tātari

Ko nga rangitaki kuputuhi katoa e pai ana mo te tukatuka i hangaia e Exchange kei te whakatakotoranga csv me nga mara e whakaahuatia ana i roto i te konae rangitaki. Mo te tarai i nga rekoata csv, ka tukuna e Logstash nga taputapu e toru: wehe, csv me te grok. Ko te tuatahi te mea tino nui быстрый, engari ka taea e koe te tarai i nga raarangi ngawari noa iho.
Hei tauira, ka wehewehea te rekoata e whai ake nei kia rua (na te mea he piko kei roto i te mara), na reira ka he te poroporoaki i te pororaka:

…,"MDB:GUID1, Mailbox:GUID2, Event:526545791, MessageClass:IPM.Note, CreationTime:2020-05-15T12:01:56.457Z, ClientType:MOMT, SubmissionAssistant:MailboxTransportSubmissionEmailAssistant",…

Ka taea te whakamahi i te wa e tarai ana i nga raarangi, hei tauira, IIS. I tenei take, penei pea te ahua o te waahanga tātari:

filter {
  if "IIS" in [tags] {
    dissect {
      mapping => {
        "message" => "%{date} %{time} %{s-ip} %{cs-method} %{cs-uri-stem} %{cs-uri-query} %{s-port} %{cs-username} %{c-ip} %{cs(User-Agent)} %{cs(Referer)} %{sc-status} %{sc-substatus} %{sc-win32-status} %{time-taken}"
      }
      remove_field => ["message"]
      add_field => { "application" => "exchange" }
    }
  }
} 

Ko te whirihoranga Logstash ka taea e koe te whakamahi nga korero here, no reira ka taea noa e matou te tuku i nga raarangi kua tohua me te tohu filebeat ki te mono dissect IIS. I roto i te mono ka rite taatau ki nga uara mara me o raatau ingoa, mukua te mara taketake message, kei roto he urunga mai i te raarangi, a ka taea e matou te taapiri i tetahi mara ritenga ka mau, hei tauira, te ingoa o te tono ka kohia e matou nga raarangi.

Mo te whai i nga raarangi, he pai ake te whakamahi i te mono csv; ka taea e ia te whakahaere tika i nga mara uaua:

filter {
  if "Tracking" in [tags] {
    csv {
      columns => ["date-time","client-ip","client-hostname","server-ip","server-hostname","source-context","connector-id","source","event-id","internal-message-id","message-id","network-message-id","recipient-address","recipient-status","total-bytes","recipient-count","related-recipient-address","reference","message-subject","sender-address","return-path","message-info","directionality","tenant-id","original-client-ip","original-server-ip","custom-data","transport-traffic-type","log-id","schema-version"]
      remove_field => ["message", "tenant-id", "schema-version"]
      add_field => { "application" => "exchange" }
    }
}

I roto i te mono ka rite taatau ki nga uara mara me o raatau ingoa, mukua te mara taketake message (me nga mara hoki tenant-id и schema-version), kei roto he urunga mai i te raarangi, a ka taea e matou te taapiri i tetahi mara ritenga, hei tauira, kei roto te ingoa o te tono ka kohia e matou nga raarangi.

I te putanga mai i te atamira tātari, ka whiwhi tatou i nga tuhinga i roto i te tatai tuatahi, kua rite mo te tirohanga ki Kibana. Ka ngaro tatou i nga mea e whai ake nei:

• Ko nga mara tau ka mohiotia hei kuputuhi, ka aukati i nga mahi ki runga. Ara, ko nga mara time-taken IIS rangitaki, me nga mara recipient-count и total-bites Aroturuki Rangitaki.
• Kei roto i te waitohu wa tuhinga paerewa te wa i tukatukahia te raarangi, kaua ko te waa i tuhia ki te taha tūmau.
• mara recipient-address ka rite ki te waahi hanga kotahi, kaore e taea e te tātari te tatau i nga kaiwhiwhi o nga reta.

Kua tae ki te wa ki te taapiri i tetahi makutu iti ki te tukanga tukatuka rangitaki.

Te huri i nga mara tau

He kōwhiringa te mono dissect convert_datatype, ka taea te whakamahi hei huri i te mara kuputuhi ki te whakatakotoranga mamati. Hei tauira, penei:

dissect {
  …
  convert_datatype => { "time-taken" => "int" }
  …
}

He mea tika kia maumahara he pai noa tenei tikanga mena ka mau te mara he aho. Ko te whiringa kaore e tukatuka i nga uara Null mai i nga mara ka maka he tuunga.

Mo nga rangitaki aroturuki, he pai ake kia kaua e whakamahi i te tikanga tahuri rite, mai i nga mara recipient-count и total-bites karekau pea. Hei huri i enei mara he pai ake te whakamahi i te mono hurihuri:

mutate {
  convert => [ "total-bytes", "integer" ]
  convert => [ "recipient-count", "integer" ]
}

Te wehewehe i te wahitau kaiwhiwhi ki nga kaiwhiwhi takitahi

Ka taea hoki te whakatau i tenei raru ma te whakamahi i te mono mutate:

mutate {
  split => ["recipient_address", ";"]
}

Te huri i te tohu wa

I roto i te take o te aroturuki rangitaki, he tino ngawari te whakaoti i te raruraru e te mono rā, ka awhina koe ki te tuhi i te mara timestamp te ra me te wa i roto i te whakatakotoranga e hiahiatia ana mai i te mara date-time:

date {
  match => [ "date-time", "ISO8601" ]
  timezone => "Europe/Moscow"
  remove_field => [ "date-time" ]
}

Mo nga raarangi IIS, me whakakotahi tatou i nga raraunga mara date и time ma te whakamahi i te mono mutate, rehitatia te rohe wa e hiahia ana tatou ka tuu tenei tohu wa ki roto timestamp te whakamahi i te mono ra:

mutate { 
  add_field => { "data-time" => "%{date} %{time}" }
  remove_field => [ "date", "time" ]
}
date { 
  match => [ "data-time", "YYYY-MM-dd HH:mm:ss" ]
  timezone => "UTC"
  remove_field => [ "data-time" ]
}

huaputa

Ka whakamahia te waahanga whakaputa ki te tuku i nga raarangi kua tukatukahia ki te kaikawe rangitaki. Ki te tuku tika ki te Elastic, ka whakamahia he mono kirihipi, ka whakapūtā te wāhitau tūmau me te tātauira ingoa taupū mō te tuku i te tuhinga i hangaia:

output {
  elasticsearch {
    hosts => ["127.0.0.1:9200", "127.0.0.2:9200"]
    manage_template => false
    index => "Exchange-%{+YYYY.MM.dd}"
  }
}

whirihoranga whakamutunga

Ko te whirihoranga whakamutunga ka penei te ahua:

input {
  beats {
    port => 5044
  }
}
 
filter {
  if "IIS" in [tags] {
    dissect {
      mapping => {
        "message" => "%{date} %{time} %{s-ip} %{cs-method} %{cs-uri-stem} %{cs-uri-query} %{s-port} %{cs-username} %{c-ip} %{cs(User-Agent)} %{cs(Referer)} %{sc-status} %{sc-substatus} %{sc-win32-status} %{time-taken}"
      }
      remove_field => ["message"]
      add_field => { "application" => "exchange" }
      convert_datatype => { "time-taken" => "int" }
    }
    mutate { 
      add_field => { "data-time" => "%{date} %{time}" }
      remove_field => [ "date", "time" ]
    }
    date { 
      match => [ "data-time", "YYYY-MM-dd HH:mm:ss" ]
      timezone => "UTC"
      remove_field => [ "data-time" ]
    }
  }
  if "Tracking" in [tags] {
    csv {
      columns => ["date-time","client-ip","client-hostname","server-ip","server-hostname","source-context","connector-id","source","event-id","internal-message-id","message-id","network-message-id","recipient-address","recipient-status","total-bytes","recipient-count","related-recipient-address","reference","message-subject","sender-address","return-path","message-info","directionality","tenant-id","original-client-ip","original-server-ip","custom-data","transport-traffic-type","log-id","schema-version"]
      remove_field => ["message", "tenant-id", "schema-version"]
      add_field => { "application" => "exchange" }
    }
    mutate {
      convert => [ "total-bytes", "integer" ]
      convert => [ "recipient-count", "integer" ]
      split => ["recipient_address", ";"]
    }
    date {
      match => [ "date-time", "ISO8601" ]
      timezone => "Europe/Moscow"
      remove_field => [ "date-time" ]
    }
  }
}
 
output {
  elasticsearch {
    hosts => ["127.0.0.1:9200", "127.0.0.2:9200"]
    manage_template => false
    index => "Exchange-%{+YYYY.MM.dd}"
  }
}

Hononga whaihua:

• Me pehea te whakauru i te OpenJDK 11 ki Windows?
• Tikiake Logstash
• Ka whakamahia elastic te kowhiringa koretake UseConcMarkSweepGC #36828
• NSSM
• Tūtira Tonu
• Patua te mono whakauru
• Logstash E hoa, kei hea taku hiri? Me wetewete au i aku rakau
• Wehea te mono tātari
• Nga Tikanga
• Whakakorehia te mono tātari
• mono tātari rā
• Mono putanga Elasticsearch

Source: will.com