Singabangane ne-ELK kanye ne-Exchange. Ingxenye 2

Ngiqhubeka nendaba yami mayelana nendlela yokwenza abangane I-Exchange kanye ne-ELK (ukuqala lapha). Ake ngikukhumbuze ukuthi le nhlanganisela iyakwazi ukucubungula inani elikhulu kakhulu lamalogi ngaphandle kokungabaza. Kulokhu sizokhuluma ngendlela yokwenza i-Exchange isebenze nezingxenye ze-Logstash ne-Kibana.

I-Logstash kusitaki se-ELK isetshenziselwa ukucubungula izingodo ngobuhlakani futhi izilungiselele ukubekwa ku-Elastic ngendlela yemibhalo, ngesisekelo sokuthi kulula ukwakha ukubonwa okuhlukahlukene e-Kibana.

setting

Iqukethe izigaba ezimbili:

• Ukufaka nokumisa iphakheji ye-OpenJDK.
• Ukufaka nokumisa iphakheji ye-Logstash.

Ukufaka nokumisa iphakheji ye-OpenJDK

Iphakheji ye-OpenJDK kufanele ilandwe futhi ikhishwe kuhla lwemibhalo oluthile. Bese indlela eya kulolu hlu lwemibhalo kufanele ifakwe ku-$env:Path kanye neziguquguqukayo ze-$env:JAVA_HOME zesistimu yokusebenza ye-Windows:

Ake sihlole inguqulo ye-Java:

PS C:> java -version
openjdk version "13.0.1" 2019-10-15
OpenJDK Runtime Environment (build 13.0.1+9)
OpenJDK 64-Bit Server VM (build 13.0.1+9, mixed mode, sharing)

Ukufaka nokumisa iphakheji ye-Logstash

Landa ifayela lengobo yomlando ngokusatshalaliswa kwe-Logstash kusuka lapha. Ingobo yomlando kufanele ikhishwe impande yediski. Khipha kufolda C:Program Files Akufanele, i-Logstash izokwenqaba ukuqala ngokujwayelekile. Bese udinga ukungena kufayela jvm.options ukulungiswa okunesibopho sokwaba i-RAM yenqubo ye-Java. Ngincoma ukucacisa uhhafu we-RAM yeseva. Uma ino-16 GB we-RAM ebhodini, okhiye abazenzakalelayo yilaba:

-Xms1g
-Xmx1g

kufanele kushintshwe:

-Xms8g
-Xmx8g

Ngaphezu kwalokho, kuhle ukuphawula umugqa -XX:+UseConcMarkSweepGC. Okuningi mayelana nalokhu lapha. Isinyathelo esilandelayo ukudala ukucushwa okuzenzakalelayo kufayela le-logstash.conf:

input {
 stdin{}
}
 
filter {
}
 
output {
 stdout {
 codec => "rubydebug"
 }
}

Ngalokhu kulungiselelwa, i-Logstash ifunda idatha kusuka kukhonsoli, iyidlulise ngesihlungi esingenalutho, bese iyikhipha iyibuyisela kukhonsoli. Ukusebenzisa lokhu kulungiselelwa kuzohlola ukusebenza kwe-Logstash. Ukuze senze lokhu, masiyiqalise ngemodi yokusebenzisana:

PS C:...bin> .logstash.bat -f .logstash.conf
...
[2019-12-19T11:15:27,769][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[2019-12-19T11:15:27,847][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-12-19T11:15:28,113][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

I-Logstash yethulwe ngempumelelo ku-port 9600.

Isinyathelo sokugcina sokufaka: qalisa i-Logstash njengesevisi ye-Windows. Lokhu kungenziwa, isibonelo, ngokusebenzisa iphakheji I-NSSM:

PS C:...bin> .nssm.exe install logstash
Service "logstash" installed successfully!

ukubekezelelana kwamaphutha

Ukuphepha kwamalogi lapho edluliswa esuka kuseva yomthombo kuqinisekiswa indlela Yemigqa Eqhubekayo.

Isebenza kanjani

Isakhiwo solayini ngesikhathi sokucutshungulwa kwelogi sithi: okokufaka → umugqa → isihlungi + okukhiphayo.

I-plugin yokufaka ithola idatha esuka kumthombo welogi, iyibhale kulayini, futhi ithumele isiqinisekiso sokuthi idatha itholiwe kumthombo.

Imilayezo esuka kulayini icutshungulwa yi-Logstash, idlule kusihlungi kanye ne-plugin yokuphumayo. Lapho ithola ukuqinisekiswa kokuphumayo kokuthi ilogu ithunyelwe, i-Logstash isusa ilogu ecutshunguliwe emgqeni. Uma i-Logstash ima, yonke imilayezo engacutshungulwanga nemilayezo engatholi siqinisekiso esamukelwe ihlala kumugqa, futhi i-Logstash izoqhubeka nokuyicubungula ngesikhathi esilandelayo lapho iqala.

Yenza ngokwezifiso

Ilungiseka ngokhiye abakufayela C:Logstashconfiglogstash.yml:

• queue.type: (amanani angenzeka - persisted и memory (default)).
• path.queue: (indlela eya kufolda enamafayela omugqa, agcinwa ku-C:Logstashqueue ngokuzenzakalelayo).
• queue.page_capacity: (ubukhulu bosayizi wekhasi lomugqa, inani elizenzakalelayo ngu-64mb).
• queue.drain: (iqiniso/amanga - inika amandla/ikhubaza ukumisa ukucutshungulwa komugqa ngaphambi kokuvala i-Logstash. Angincomi ukuyinika amandla, ngoba lokhu kuzothinta ngqo isivinini sokuvala iseva).
• queue.max_events: (inombolo enkulu yemicimbi kulayini, okuzenzakalelayo ngu-0 (akunamkhawulo)).
• queue.max_bytes: (ubukhulu bosayizi womugqa ngamabhayithi, okuzenzakalelayo - 1024mb (1gb)).

Uma imisiwe queue.max_events и queue.max_bytes, bese imilayezo iyayeka ukwamukelwa kulayini uma inani lanoma yiziphi lezi zilungiselelo selifinyelelwe. Funda kabanzi mayelana ne-Persistent Queues lapha.

Isibonelo sengxenye ye-logstash.yml enesibopho sokumisa ulayini:

queue.type: persisted
queue.max_bytes: 10gb

Yenza ngokwezifiso

Ukucushwa kwe-Logstash kuvame ukuba nezingxenye ezintathu, ezinesibopho sezigaba ezihlukene zokucubungula amalogi angenayo: ukwamukela (isigaba sokufaka), ukuhlukanisa (isigaba sesihlungi) nokuthumela ku-Elastic (isigaba sokuphumayo). Ngezansi sizobheka ngokucophelela ngayinye yazo.

Ukufaka

Sithola ukusakaza okungenayo ngamalogi aluhlaza avela kubasebenzeli be-filebeat. Yile plugin esiyikhombisa esigabeni sokufaka:

input {
  beats {
    port => 5044
  }
}

Ngemuva kwalokhu kulungiselelwa, i-Logstash iqala ukulalela i-port 5044, futhi lapho ithola izingodo, iwacubungule ngokuvumelana nezilungiselelo zesigaba sokuhlunga. Uma kunesidingo, ungasonga isiteshi ukuze uthole amalogi ku-filebit nge-SSL. Funda kabanzi mayelana nezilungiselelo ze-plugin ye-beats lapha.

Hlunga

Wonke amalogi ombhalo anentshisekelo ekucutshungulweni akhiqizwa yi-Exchange akufomethi ye-csv nezinkambu ezichazwe efayeleni lokungena ngokwalo. Ukuhlaziya amarekhodi e-csv, i-Logstash isinika ama-plugin amathathu: hlinza, csv futhi grok. Eyokuqala iyona eminingi быстрый, kodwa ibhekana nokwehlukanisa izingodo ezilula kuphela.
Isibonelo, izohlukanisa irekhodi elilandelayo libe kabili (ngenxa yokuba khona kokhefana ngaphakathi kwenkundla), yingakho ilogu izocutshungulwa ngokungalungile:

…,"MDB:GUID1, Mailbox:GUID2, Event:526545791, MessageClass:IPM.Note, CreationTime:2020-05-15T12:01:56.457Z, ClientType:MOMT, SubmissionAssistant:MailboxTransportSubmissionEmailAssistant",…

Ingasetshenziswa uma uhlaziya izingodo, isibonelo, i-IIS. Kulokhu, isigaba sokuhlunga singase sibukeke kanje:

filter {
  if "IIS" in [tags] {
    dissect {
      mapping => {
        "message" => "%{date} %{time} %{s-ip} %{cs-method} %{cs-uri-stem} %{cs-uri-query} %{s-port} %{cs-username} %{c-ip} %{cs(User-Agent)} %{cs(Referer)} %{sc-status} %{sc-substatus} %{sc-win32-status} %{time-taken}"
      }
      remove_field => ["message"]
      add_field => { "application" => "exchange" }
    }
  }
} 

Ukucushwa kwe-Logstash kukuvumela ukuthi usebenzise izitatimende ezinemibandela, ukuze sikwazi ukuthumela kuphela izingodo ezimakwe umaka we-filebeat ku-plugin ye-dissect IIS. Ngaphakathi kwe-plugin sifanisa amanani enkambu namagama awo, susa inkambu yoqobo message, equkethe okufakiwe okuvela kulogi, futhi singangeza inkambu yangokwezifiso, ngokwesibonelo, ezoqukatha igama lohlelo lokusebenza esiqoqa kulo amalogi.

Endabeni yokulandelela izingodo, kungcono ukusebenzisa i-plugin ye-csv; ingakwazi ukucubungula kahle izinkambu eziyinkimbinkimbi:

filter {
  if "Tracking" in [tags] {
    csv {
      columns => ["date-time","client-ip","client-hostname","server-ip","server-hostname","source-context","connector-id","source","event-id","internal-message-id","message-id","network-message-id","recipient-address","recipient-status","total-bytes","recipient-count","related-recipient-address","reference","message-subject","sender-address","return-path","message-info","directionality","tenant-id","original-client-ip","original-server-ip","custom-data","transport-traffic-type","log-id","schema-version"]
      remove_field => ["message", "tenant-id", "schema-version"]
      add_field => { "application" => "exchange" }
    }
}

Ngaphakathi kwe-plugin sifanisa amanani enkambu namagama awo, susa inkambu yoqobo message (kanye nezinkambu tenant-id и schema-version), equkethe okufakiwe okuvela kulogi, futhi singakwazi ukwengeza inkambu yangokwezifiso, okuzothi, isibonelo, iqukathe igama lohlelo lokusebenza esiqoqa kulo izingodo.

Ekuphumeni kwesigaba sokuhlunga, sizothola amadokhumenti ngokulinganisa kokuqala, alungele ukubonwa ngeso lengqondo e-Kibana. Sizobe siphuthelwa okulandelayo:

• Izinkambu zezinombolo zizobonwa njengombhalo, ovimbela ukusebenza kuzo. Okungukuthi, amasimu time-taken Ilogi ye-IIS, kanye nezinkambu recipient-count и total-bites Ukulandelela Ilogi.
• Isitembu sesikhathi sedokhumenti esijwayelekile sizoqukatha isikhathi ilogi icutshungulwe ngaso, hhayi isikhathi ebhalwe ngaso ohlangothini lweseva.
• Insimu recipient-address izobukeka njengendawo eyodwa yokwakha, engavumeli ukuhlaziya ukubala abamukeli bezinhlamvu.

Isikhathi sokwengeza umlingo omncane enqubweni yokucubungula ilogu.

Iguqula izinkambu zezinombolo

I-plugin ye-dissect inenketho convert_datatype, engasetshenziswa ukuguqula inkambu yombhalo ibe ifomethi yedijithali. Ngokwesibonelo, kanje:

dissect {
  …
  convert_datatype => { "time-taken" => "int" }
  …
}

Kuyafaneleka ukukhumbula ukuthi le ndlela ifaneleka kuphela uma insimu izoqukatha intambo. Inketho ayicubunguli amanani angenalutho kusuka ezinkambu futhi iphonsa okuhlukile.

Ngokulandelela izingodo, kungcono ukungasebenzisi indlela yokuguqula efanayo, kusukela emasimini recipient-count и total-bites ingase ingabi nalutho. Ukuze uguqule lezi zinkambu kungcono ukusebenzisa i-plugin shintsha:

mutate {
  convert => [ "total-bytes", "integer" ]
  convert => [ "recipient-count", "integer" ]
}

Ukuhlukanisa i-recipient_address kube abamukeli ngabanye

Le nkinga ingaxazululwa kusetshenziswa i-plugin ye-mutate:

mutate {
  split => ["recipient_address", ";"]
}

Ukushintsha isitembu sesikhathi

Endabeni yokulandelela izingodo, inkinga ixazululwa kalula yi-plugin Usuku, okuzokusiza ukuthi ubhale ensimini timestamp idethi nesikhathi ngefomethi edingekayo evela ensimini date-time:

date {
  match => [ "date-time", "ISO8601" ]
  timezone => "Europe/Moscow"
  remove_field => [ "date-time" ]
}

Endabeni yamalogi e-IIS, sizodinga ukuhlanganisa idatha yasensimini date и time usebenzisa i-plugin ye-mutate, bhalisa indawo yesikhathi esiyidingayo futhi ufake lesi sitembu sesikhathi timestamp usebenzisa i-plugin yedethi:

mutate { 
  add_field => { "data-time" => "%{date} %{time}" }
  remove_field => [ "date", "time" ]
}
date { 
  match => [ "data-time", "YYYY-MM-dd HH:mm:ss" ]
  timezone => "UTC"
  remove_field => [ "data-time" ]
}

Okukhiphayo

Ingxenye yokukhiphayo isetshenziselwa ukuthumela amalogi acutshunguliwe kumamukeli welogi. Esimeni sokuthumela ngqo ku-Elastic, kusetshenziswa i-plugin umabhebhana, ecacisa ikheli leseva kanye nesifanekiso segama lenkomba sokuthumela idokhumenti ekhiqiziwe:

output {
  elasticsearch {
    hosts => ["127.0.0.1:9200", "127.0.0.2:9200"]
    manage_template => false
    index => "Exchange-%{+YYYY.MM.dd}"
  }
}

Ukucushwa kokugcina

Ukucushwa kokugcina kuzobukeka kanje:

input {
  beats {
    port => 5044
  }
}
 
filter {
  if "IIS" in [tags] {
    dissect {
      mapping => {
        "message" => "%{date} %{time} %{s-ip} %{cs-method} %{cs-uri-stem} %{cs-uri-query} %{s-port} %{cs-username} %{c-ip} %{cs(User-Agent)} %{cs(Referer)} %{sc-status} %{sc-substatus} %{sc-win32-status} %{time-taken}"
      }
      remove_field => ["message"]
      add_field => { "application" => "exchange" }
      convert_datatype => { "time-taken" => "int" }
    }
    mutate { 
      add_field => { "data-time" => "%{date} %{time}" }
      remove_field => [ "date", "time" ]
    }
    date { 
      match => [ "data-time", "YYYY-MM-dd HH:mm:ss" ]
      timezone => "UTC"
      remove_field => [ "data-time" ]
    }
  }
  if "Tracking" in [tags] {
    csv {
      columns => ["date-time","client-ip","client-hostname","server-ip","server-hostname","source-context","connector-id","source","event-id","internal-message-id","message-id","network-message-id","recipient-address","recipient-status","total-bytes","recipient-count","related-recipient-address","reference","message-subject","sender-address","return-path","message-info","directionality","tenant-id","original-client-ip","original-server-ip","custom-data","transport-traffic-type","log-id","schema-version"]
      remove_field => ["message", "tenant-id", "schema-version"]
      add_field => { "application" => "exchange" }
    }
    mutate {
      convert => [ "total-bytes", "integer" ]
      convert => [ "recipient-count", "integer" ]
      split => ["recipient_address", ";"]
    }
    date {
      match => [ "date-time", "ISO8601" ]
      timezone => "Europe/Moscow"
      remove_field => [ "date-time" ]
    }
  }
}
 
output {
  elasticsearch {
    hosts => ["127.0.0.1:9200", "127.0.0.2:9200"]
    manage_template => false
    index => "Exchange-%{+YYYY.MM.dd}"
  }
}

Izixhumanisi eziwusizo:

• Ungayifaka kanjani i-OpenJDK 11 ku-Windows?
• Landa i-logstash
• I-Elastic isebenzisa inketho encishiwe UseConcMarkSweepGC #36828
• I-NSSM
• Imigqa Ephikelelayo
• Ishaya i-plugin yokufaka
• Logstash Dude, iphi i-chainsaw yami? Ngidinga ukuhlinza izingodo zami
• Hlukanisa i-plugin yesihlungi
• Izimo
• Guqula i-plugin yesihlungi
• I-plugin yesihlungi sedethi
• I-plugin ephumayo ye-Elasticsearch

Source: www.habr.com