Ke tsoela pele pale ea ka mabapi le mokhoa oa ho etsa metsoalle Exchange le ELK (ho qala ). E-re ke u hopotse hore motsoako ona o khona ho sebetsana le palo e kholo haholo ea lifate ntle le ho tsilatsila. Lekhetlong lena re tla bua ka mokhoa oa ho etsa hore Exchange e sebetse le likarolo tsa Logstash le Kibana.
Logstash ka har'a stack ea ELK e sebelisetsoa ho sebetsana le li-logs ka bohlale le ho li lokisetsa ho behoa ka Elastic ka mokhoa oa litokomane, motheong oa eona ho leng bonolo ho haha lipono tse sa tšoaneng tsa Kibana.
bophirima
E na le mekhahlelo e 'meli:
- Ho kenya le ho hlophisa sephutheloana sa OpenJDK.
- Ho kenya le ho lokisa sephutheloana sa Logstash.
Ho kenya le ho hlophisa sephutheloana sa OpenJDK
Sephutheloana sa OpenJDK se tlameha ho kopitsoa le ho phutholloa bukeng e itseng. Ebe tsela e eang bukeng ena e tlameha ho kenngoa ho $env:Path le $env:JAVA_HOME mefuta e fapaneng ea sistimi e sebetsang ea Windows:
Ha re hlahlobe mofuta oa Java:
PS C:> java -version
openjdk version "13.0.1" 2019-10-15
OpenJDK Runtime Environment (build 13.0.1+9)
OpenJDK 64-Bit Server VM (build 13.0.1+9, mixed mode, sharing)
Ho kenya le ho lokisa sephutheloana sa Logstash
Khoasolla faele ea polokelo ka phano ea Logstash . Sebaka sa polokelo se tlameha ho phutholloa ho motso oa disk. Tlosa ho sephutheli C:Program Files Ha e na thuso, Logstash e tla hana ho qala ka tloaelo. Ebe o hloka ho kenya file jvm.options Litokiso tse ikarabellang bakeng sa ho fana ka RAM bakeng sa ts'ebetso ea Java. Ke khothaletsa ho hlakisa halofo ea RAM ea seva. Haeba e na le 16 GB ea RAM ka sekepeng, joale linotlolo tsa kamehla ke:
-Xms1g
-Xmx1g
e tlameha ho nkeloa sebaka ke:
-Xms8g
-Xmx8g
Ho feta moo, ho eletsoa ho fana ka maikutlo ho tsoa mola -XX:+UseConcMarkSweepGC. Tse ling ka sena . Mohato o latelang ke ho theha tlhophiso ea kamehla faeleng ea logstash.conf:
input {
stdin{}
}
filter {
}
output {
stdout {
codec => "rubydebug"
}
}
Ka tlhophiso ena, Logstash e bala data ho tsoa ho khomphutha, e e fetisa ka sefahla se se nang letho, ebe e e khutlisetsa ho console. Ho sebelisa tlhophiso ena ho tla leka ts'ebetso ea Logstash. Ho etsa sena, a re e tsamaise ka mokhoa oa ho sebelisana:
PS C:...bin> .logstash.bat -f .logstash.conf
...
[2019-12-19T11:15:27,769][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[2019-12-19T11:15:27,847][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-12-19T11:15:28,113][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
Logstash e qalile ka katleho ho port 9600.
Mohato oa ho qetela oa ho kenya: qala Logstash e le tšebeletso ea Windows. Sena se ka etsoa, ka mohlala, ho sebelisa sephutheloana :
PS C:...bin> .nssm.exe install logstash
Service "logstash" installed successfully!
mamello ea liphoso
Tšireletseho ea li-log ha e fetisoa ho tloha mohloling oa mohloli e netefatsoa ke mokhoa oa Persistent Queues.
E sebetsa joang
Popeho ea mela nakong ea ts'ebetso ea li-log ke: ho kenya → queue → sefe + tlhahiso.
Plugin e kentsoeng e fumana data ho tsoa mohloling oa log, e e ngolla moleng, ebe e romela netefatso ea hore data e amohetsoe mohloling.
Melaetsa e tsoang ho queue e sebetsoa ke Logstash, e fetisoa ka sefeng le plugin ea tlhahiso. Ha Logstash e fumana netefatso ho tsoa ho tlhahiso ea hore log e rometsoe, e tlosa logi e sebetsitsoeng ho tloha moleng. Haeba Logstash e emisa, melaetsa eohle e sa sebetsoang le melaetsa eo ho sa amoheloeng netefatso ea eona e lula e le moleng, 'me Logstash e tla tsoela pele ho e sebetsa nakong e tlang ha e qala.
phetoho
E ka feto-fetoha ka linotlolo faeleng C:Logstashconfiglogstash.yml:
queue.type: (litekanyetso tse ka khonehang -persistedиmemory (default)).path.queue: (tsela e eang foldareng e nang le lifaele tsa queue, tse bolokiloeng ho C: Logstashqueue ka ho sa feleng).queue.page_capacity: (boholo ba boholo ba leqephe la queue, boleng ba kamehla ke 64mb).queue.drain: ('nete / bohata - e nolofalletsa / e thibela ho emisa ho sebetsa ha mela pele e koala Logstash. Ha ke khothaletse ho e nolofalletsa, hobane sena se tla ama ka ho toba lebelo la ho koala seva).queue.max_events: (palo e kahodimodimo ya diketsahalo moleng, ya kamehla ke 0 (e se nang moeli)).queue.max_bytes: (boholo ba boholo ba mela ka li-byte, kamehla - 1024mb (1gb)).
Haeba e hlophisitsoe queue.max_events и queue.max_bytes, ebe melaetsa e khaotsa ho amoheloa moleng ha boleng ba efe kapa efe ea litlhophiso tsena bo fihletsoe. Ithute haholoanyane ka Mela e Phehellang .
Mohlala oa karolo ea logstash.yml e ikarabellang bakeng sa ho theha queue:
queue.type: persisted
queue.max_bytes: 10gb
phetoho
Sebopeho sa Logstash hangata se na le likarolo tse tharo, tse ikarabellang bakeng sa mekhahlelo e fapaneng ea ho sebetsana le li-logs tse kenang: ho amohela (karolo ea ho kenya letsoho), ho arola (karolo ea sefe) le ho romela ho Elastic (karolo ea lihlahisoa). Ka tlase re tla shebisisa e 'ngoe le e' ngoe ea tsona.
input
Re amohela molapo o kenang ka li-log tse tala ho tsoa ho baemeli ba filebeat. Ke plugin ena eo re e bonts'ang karolong ea ho kenya:
input {
beats {
port => 5044
}
}
Kamora tlhophiso ena, Logstash e qala ho mamela port 5044, 'me ha e amohela lits'oants'o, e li sebetsa ho latela litlhophiso tsa karolo ea filthara. Haeba ho hlokahala, o ka phuthela mocha bakeng sa ho amohela li-log ho tsoa ho filebit ho SSL. Bala haholoanyane ka litlhophiso tsa plugin tsa beats .
filthara
Lingoliloeng tsohle tsa mongolo tse khahlisang bakeng sa ts'ebetso e hlahisoang ke Exchange li ka sebopeho sa csv ka likarolo tse hlalositsoeng faeleng ea log ka boeona. Bakeng sa ho hlophisa lirekoto tsa csv, Logstash e re fa li-plugins tse tharo: , csv le grok. Ea pele ke eona e fetisisang , empa e sebetsana le ho arola likutu tse bonolo feela.
Mohlala, e tla arola rekoto e latelang ho etsa tse peli (ka lebaka la ho ba teng ha pheelo ka har'a lebala), ke ka lebaka leo log e tla aroloa ka phoso:
…,"MDB:GUID1, Mailbox:GUID2, Event:526545791, MessageClass:IPM.Note, CreationTime:2020-05-15T12:01:56.457Z, ClientType:MOMT, SubmissionAssistant:MailboxTransportSubmissionEmailAssistant",…
E ka sebelisoa ha ho arola lifate, mohlala, IIS. Tabeng ena, karolo ea filter e ka shebahala tjena:
filter {
if "IIS" in [tags] {
dissect {
mapping => {
"message" => "%{date} %{time} %{s-ip} %{cs-method} %{cs-uri-stem} %{cs-uri-query} %{s-port} %{cs-username} %{c-ip} %{cs(User-Agent)} %{cs(Referer)} %{sc-status} %{sc-substatus} %{sc-win32-status} %{time-taken}"
}
remove_field => ["message"]
add_field => { "application" => "exchange" }
}
}
}
Logstash tlhophiso e u lumella ho e sebelisa , kahoo re ka romella li-log tse neng li tšoailoe ka tag ea filebeat ho plugin ea dissect IIS. Ka har'a plugin re bapisa boleng ba tšimo le mabitso a bona, hlakola tšimo ea mantlha message, e neng e e-na le ho kena ho tsoa ho log, 'me re ka eketsa tšimo e tloaelehileng eo, ka mohlala, e tla ba le lebitso la kopo eo re bokellang lifate ho eona.
Tabeng ea lits'oants'o tsa ho latela, ho molemo ho sebelisa plugin ea csv; e ka sebetsana ka nepo masimo a rarahaneng:
filter {
if "Tracking" in [tags] {
csv {
columns => ["date-time","client-ip","client-hostname","server-ip","server-hostname","source-context","connector-id","source","event-id","internal-message-id","message-id","network-message-id","recipient-address","recipient-status","total-bytes","recipient-count","related-recipient-address","reference","message-subject","sender-address","return-path","message-info","directionality","tenant-id","original-client-ip","original-server-ip","custom-data","transport-traffic-type","log-id","schema-version"]
remove_field => ["message", "tenant-id", "schema-version"]
add_field => { "application" => "exchange" }
}
}
Ka har'a plugin re bapisa boleng ba tšimo le mabitso a bona, hlakola tšimo ea mantlha message (le masimo tenant-id и schema-version), e neng e e-na le ho kena ho tsoa ho log, 'me re ka eketsa tšimo e tloaelehileng, eo ka mohlala, e tla ba le lebitso la kopo eo re bokellang lifate ho eona.
Ha re tsoa sethaleng sa ho sefa, re tla fumana litokomane ka tekanyo ea pele, e lokiselitsoeng ho bonoa ka Kibana. Re tla lahleheloa ke tse latelang:
- Likarolo tsa linomoro li tla tsejoa e le mongolo, e leng se thibelang ts'ebetso ho tsona. E leng, masimo
time-takenIIS log, hammoho le masimorecipient-countиtotal-bitesLog Tracking. - Setempe se tloaelehileng sa tokomane se tla ba le nako eo log e sebelitsoeng ka eona, eseng nako eo e ngotsoeng ka eona lehlakoreng la seva.
- tšimo
recipient-addresse tla shebahala joaloka sebaka se le seng sa kaho, se sa lumelleng tlhahlobo ho bala ba amohelang mangolo.
Ke nako ea ho kenya boselamose bo fokolang ts'ebetsong ea ts'ebetso ea log.
Ho fetola likarolo tsa linomoro
Sesebelisoa sa dissect se na le khetho convert_datatype, e ka sebelisoang ho fetolela sebaka sa mongolo ho sebopeho sa dijithale. Ka mohlala, joalo ka:
dissect {
…
convert_datatype => { "time-taken" => "int" }
…
}
Ke habohlokoa ho hopola hore mokhoa ona o loketse feela haeba tšimo e tla ba le khoele. Khetho ha e sebetse boleng ba Null ho tsoa masimong mme e etsa mokhelo.
Bakeng sa lintlha tsa morao-rao, ho molemo hore u se ke ua sebelisa mokhoa o tšoanang oa ho fetola, kaha masimo recipient-count и total-bites e ka ba e se nang letho. Ho fetolela masimo ana ho molemo ho sebelisa plugin :
mutate {
convert => [ "total-bytes", "integer" ]
convert => [ "recipient-count", "integer" ]
}
Arola moamoheli_aterese ho batho ka bomong
Bothata bona bo ka rarolloa ka ho sebelisa plugin ea mutate:
mutate {
split => ["recipient_address", ";"]
}
Ho fetola setempe sa nako
Tabeng ea ho latela li-log, bothata bo rarolloa habonolo ke plugin , e tla u thusa ho ngola tšimong timestamp letsatsi le nako ka sebopeho se hlokahalang ho tsoa tšimong date-time:
date {
match => [ "date-time", "ISO8601" ]
timezone => "Europe/Moscow"
remove_field => [ "date-time" ]
}
Tabeng ea li-logs tsa IIS, re tla hloka ho kopanya lintlha tsa tšimo date и time u sebelisa plugin ea mutate, ngolisa sebaka sa nako seo re se hlokang 'me u behe setempe sa nako ena timestamp ho sebelisa plugin ea letsatsi:
mutate {
add_field => { "data-time" => "%{date} %{time}" }
remove_field => [ "date", "time" ]
}
date {
match => [ "data-time", "YYYY-MM-dd HH:mm:ss" ]
timezone => "UTC"
remove_field => [ "data-time" ]
}
khumo
Karolo e hlahisoang e sebelisoa ho romela li-log tse sebetsitsoeng ho moamoheli oa log. Tabeng ea ho romela ka ho toba ho Elastic, ho sebelisoa plugin , e hlalosang aterese ea seva le template ea lebitso la index bakeng sa ho romela tokomane e hlahisitsoeng:
output {
elasticsearch {
hosts => ["127.0.0.1:9200", "127.0.0.2:9200"]
manage_template => false
index => "Exchange-%{+YYYY.MM.dd}"
}
}
Phetoho ea ho qetela
Sebopeho sa ho qetela se tla shebahala tjena:
input {
beats {
port => 5044
}
}
filter {
if "IIS" in [tags] {
dissect {
mapping => {
"message" => "%{date} %{time} %{s-ip} %{cs-method} %{cs-uri-stem} %{cs-uri-query} %{s-port} %{cs-username} %{c-ip} %{cs(User-Agent)} %{cs(Referer)} %{sc-status} %{sc-substatus} %{sc-win32-status} %{time-taken}"
}
remove_field => ["message"]
add_field => { "application" => "exchange" }
convert_datatype => { "time-taken" => "int" }
}
mutate {
add_field => { "data-time" => "%{date} %{time}" }
remove_field => [ "date", "time" ]
}
date {
match => [ "data-time", "YYYY-MM-dd HH:mm:ss" ]
timezone => "UTC"
remove_field => [ "data-time" ]
}
}
if "Tracking" in [tags] {
csv {
columns => ["date-time","client-ip","client-hostname","server-ip","server-hostname","source-context","connector-id","source","event-id","internal-message-id","message-id","network-message-id","recipient-address","recipient-status","total-bytes","recipient-count","related-recipient-address","reference","message-subject","sender-address","return-path","message-info","directionality","tenant-id","original-client-ip","original-server-ip","custom-data","transport-traffic-type","log-id","schema-version"]
remove_field => ["message", "tenant-id", "schema-version"]
add_field => { "application" => "exchange" }
}
mutate {
convert => [ "total-bytes", "integer" ]
convert => [ "recipient-count", "integer" ]
split => ["recipient_address", ";"]
}
date {
match => [ "date-time", "ISO8601" ]
timezone => "Europe/Moscow"
remove_field => [ "date-time" ]
}
}
}
output {
elasticsearch {
hosts => ["127.0.0.1:9200", "127.0.0.2:9200"]
manage_template => false
index => "Exchange-%{+YYYY.MM.dd}"
}
}
Lihokela tse sebetsang:
Source: www.habr.com