Ke hoʻomau nei au i kaʻu moʻolelo e pili ana i ka hana ʻana i nā hoaaloha Exchange a me ELK (hoʻomaka ). E hoʻomanaʻo wau iā ʻoe e hiki i kēia hui ke hana i kahi helu nui loa o nā lāʻau me ke kānalua ʻole. I kēia manawa e kamaʻilio mākou e pili ana i ka hana ʻana o Exchange me nā ʻāpana Logstash a me Kibana.
Hoʻohana ʻia ʻo Logstash i ka waihona ELK e hoʻoponopono naʻauao i nā lāʻau a hoʻomākaukau iā lākou no ka hoʻokomo ʻana ma Elastic ma ke ʻano o nā palapala, ma ke kumu e kūpono ai ke kūkulu ʻana i nā hiʻohiʻona like ʻole ma Kibana.
Kāu Mau Koho Paʻamau
Aia i ʻelua mau pae:
- Ke kau ʻana a me ka hoʻonohonoho ʻana i ka pūʻolo OpenJDK.
- Hoʻokomo a hoʻonohonoho i ka pūʻolo Logstash.
Hoʻokomo a hoʻonohonoho i ka pūʻolo OpenJDK
Pono e hoʻoiho a wehe ʻia ka pūʻolo OpenJDK i kahi papa kuhikuhi kikoʻī. A laila pono e hoʻokomo ʻia ke ala i kēia papa kuhikuhi i loko o ka $env:Path a me $env:JAVA_HOME mau ʻano o ka ʻōnaehana hana Windows:
E nānā kākou i ka mana Java:
PS C:> java -version
openjdk version "13.0.1" 2019-10-15
OpenJDK Runtime Environment (build 13.0.1+9)
OpenJDK 64-Bit Server VM (build 13.0.1+9, mixed mode, sharing)
Hoʻokomo a hoʻonohonoho i ka pūʻolo Logstash
Hoʻoiho i ka waihona waihona me ka hoʻohele Logstash . Pono e wehe ʻia ka waihona i ke kumu o ka disk. Wehe i ka waihona C:Program Files ʻAʻole pono ia, e hōʻole ʻo Logstash e hoʻomaka maʻamau. A laila pono ʻoe e komo i ka faila jvm.options hoʻoponopono i ke kuleana no ka hoʻokaʻawale ʻana i ka RAM no ke kaʻina Java. Manaʻo wau e wehewehe i ka hapalua o ka RAM o ke kikowaena. Inā loaʻa iā 16 GB o RAM ma luna o ka moku, a laila ʻo nā kī paʻamau:
-Xms1g
-Xmx1g
pono e pani ia me:
-Xms8g
-Xmx8g
Eia kekahi, pono e hoʻopuka i ka laina -XX:+UseConcMarkSweepGC. Nā mea hou aku e pili ana i kēia . ʻO ka hana aʻe e hana i kahi hoʻonohonoho paʻamau i ka faila logstash.conf:
input {
stdin{}
}
filter {
}
output {
stdout {
codec => "rubydebug"
}
}
Me kēia hoʻonohonoho, heluhelu ʻo Logstash i ka ʻikepili mai ka console, hoʻohele iā ia ma kahi kānana kaʻawale, a hoʻihoʻi iā ia i ka console. Ke hoʻohana nei i kēia hoʻonohonoho e hoʻāʻo i ka hana o Logstash. No ka hana ʻana i kēia, e holo kākou ma ke ʻano pāʻani:
PS C:...bin> .logstash.bat -f .logstash.conf
...
[2019-12-19T11:15:27,769][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[2019-12-19T11:15:27,847][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-12-19T11:15:28,113][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
Hoʻomaka maikaʻi ʻo Logstash ma ke awa 9600.
Kaʻina hana hope: e holo ana iā Logstash ma ke ʻano he lawelawe Windows. Hiki ke hana i kēia, no ka laʻana, me ka hoʻohana ʻana i ka pā :
PS C:...bin> .nssm.exe install logstash
Service "logstash" installed successfully!
hoʻomanawanui hewa
Hoʻopaʻa ʻia ka palekana o nā lāʻau i ka wā i hoʻoili ʻia mai ke kikowaena kumu e ka Persistent Queues mechanism.
Pehea e hana ai
ʻO ka hoʻonohonoho ʻana o nā pila i ka wā e hoʻoili ai i ka lāʻau: hoʻokomo → queue → kānana + puka.
Loaʻa i ka plugin input ka ʻikepili mai kahi kumu log, kākau iā ia i kahi queue, a hoʻouna i ka hōʻoia ua loaʻa ka ʻikepili i ke kumu.
Hoʻopili ʻia nā memo mai ka queue e Logstash, hele i ka kānana a me ka plugin output. I ka loaʻa ʻana o ka hōʻoia mai ka hoʻopuka ua hoʻouna ʻia ka log, wehe ʻo Logstash i ka log i hana ʻia mai ka pila. Inā kū ʻo Logstash, e noho mau nā memo a me nā memo a pau ʻaʻole i hōʻoia ʻia i ka pila, a e hoʻomau ʻo Logstash i ka hana ʻana iā lākou i ka manawa aʻe e hoʻomaka ai.
hoʻoponopono
Hiki ke hoʻololi ʻia e nā kī ma ka faila C:Logstashconfiglogstash.yml:
queue.type: (mau waiwai hiki -persistedиmemory (default)).path.queue: (ke ala i ka waihona me nā faila queue, i mālama ʻia ma C: Logstashqueue ma ka paʻamau).queue.page_capacity: (ʻO ka nui o ka ʻaoʻao queue, ʻo 64mb ka waiwai paʻamau).queue.drain: (ʻoiaʻiʻo / wahaheʻe - hiki / hoʻopau i ka hoʻopau ʻana i ka hana queue ma mua o ka pani ʻana i ka Logstash.queue.max_events: (ka helu kiʻekiʻe o nā hanana i ka pila, ʻo 0 (palena ʻole)).queue.max_bytes: (ka nui o ka queue nui ma nā bytes, paʻamau - 1024mb (1gb)).
Inā hoʻonohonoho ʻia queue.max_events и queue.max_bytes, a laila pau ka ʻae ʻia ʻana o nā memo i ka pila ke hiki i ka waiwai o kekahi o kēia mau hoʻonohonoho. E aʻo hou aʻe e pili ana i nā queues mau loa .
He laʻana o ka ʻāpana o logstash.yml kuleana no ka hoʻonohonoho ʻana i ka pila:
queue.type: persisted
queue.max_bytes: 10gb
hoʻoponopono
ʻO ka hoʻonohonoho Logstash ka mea maʻamau i ʻekolu ʻāpana, kuleana no nā ʻano hana like ʻole o ka hoʻoili ʻana i nā log e komo mai ana: ka loaʻa ʻana (ʻāpana hoʻokomo), parsing (ʻāpana kānana) a me ka hoʻouna ʻana iā Elastic (ʻāpana puka). Ma lalo nei mākou e nānā pono i kēlā me kēia o lākou.
hoʻokomo o
Loaʻa iā mākou ke kahawai e komo mai ana me nā lāʻau maka mai nā ʻelele filebeat. ʻO kēia plugin kā mākou e hōʻike ai i ka ʻāpana komo:
input {
beats {
port => 5044
}
}
Ma hope o kēia hoʻonohonoho, hoʻomaka ʻo Logstash e hoʻolohe i ke awa 5044, a i ka loaʻa ʻana o nā lāʻau, hoʻoponopono iā lākou e like me nā hoʻonohonoho o ka ʻāpana kānana. Inā pono, hiki iā ʻoe ke kāʻei i ke kahawai no ka loaʻa ʻana o nā lāʻau mai filebit ma SSL. E heluhelu hou e pili ana i nā hoʻonohonoho plugin beats .
Kānana
ʻO nā moʻolelo kikokikona hoihoi no ka hoʻoponopono ʻana i hoʻopuka ʻia e Exchange ma ke ʻano csv me nā kahua i wehewehe ʻia i loko o ka faila log ponoʻī. No ka hoʻopau ʻana i nā moʻolelo csv, hāʻawi ʻo Logstash iā mākou i ʻekolu plugins: , csv a me grok. ʻO ka mua ka mea nui loa , akā, hoʻopaʻa ʻia me ka pau ʻana i nā lāʻau maʻalahi loa.
No ka laʻana, e hoʻokaʻawale i kēia moʻolelo i ʻelua (no ka loaʻa ʻana o kahi koma i loko o ke kahua), ʻo ia ke kumu e paʻi hewa ʻia ai ka log:
…,"MDB:GUID1, Mailbox:GUID2, Event:526545791, MessageClass:IPM.Note, CreationTime:2020-05-15T12:01:56.457Z, ClientType:MOMT, SubmissionAssistant:MailboxTransportSubmissionEmailAssistant",…
Hiki ke hoʻohana ʻia i ka wā e hoʻopau ai i nā lāʻau, no ka laʻana, IIS. I kēia hihia, penei paha ka ʻāpana kānana:
filter {
if "IIS" in [tags] {
dissect {
mapping => {
"message" => "%{date} %{time} %{s-ip} %{cs-method} %{cs-uri-stem} %{cs-uri-query} %{s-port} %{cs-username} %{c-ip} %{cs(User-Agent)} %{cs(Referer)} %{sc-status} %{sc-substatus} %{sc-win32-status} %{time-taken}"
}
remove_field => ["message"]
add_field => { "application" => "exchange" }
}
}
}
ʻO ka hoʻonohonoho Logstash hiki iā ʻoe ke hoʻohana , no laila hiki iā mākou ke hoʻouna wale i nā lāʻau i kau ʻia me ka filebeat tag i ka dissect plugin IIS. I loko o ka plugin hoʻohālikelike mākou i nā koina kahua me ko lākou mau inoa, kāpae i ka kahua kumu message, i loaʻa kahi komo mai ka log, a hiki iā mākou ke hoʻohui i kahi kahua maʻamau e loaʻa, no ka laʻana, ka inoa o ka noi kahi a mākou e hōʻiliʻili ai i nā lāʻau.
I ka hihia o ka nānā ʻana i nā lāʻau, ʻoi aku ka maikaʻi o ka hoʻohana ʻana i ka plugin csv; hiki iā ia ke hana pololei i nā māla paʻakikī:
filter {
if "Tracking" in [tags] {
csv {
columns => ["date-time","client-ip","client-hostname","server-ip","server-hostname","source-context","connector-id","source","event-id","internal-message-id","message-id","network-message-id","recipient-address","recipient-status","total-bytes","recipient-count","related-recipient-address","reference","message-subject","sender-address","return-path","message-info","directionality","tenant-id","original-client-ip","original-server-ip","custom-data","transport-traffic-type","log-id","schema-version"]
remove_field => ["message", "tenant-id", "schema-version"]
add_field => { "application" => "exchange" }
}
}
I loko o ka plugin hoʻohālikelike mākou i nā koina kahua me ko lākou mau inoa, kāpae i ka kahua kumu message (a me na kahua tenant-id и schema-version), i loaʻa kahi komo mai ka log, a hiki iā mākou ke hoʻohui i kahi kahua maʻamau, e like me ka inoa o ka noi kahi a mākou e hōʻiliʻili ai i nā lāʻau.
Ma ka puka ʻana mai ke kahua kānana, e loaʻa iā mākou nā palapala ma kahi kokoke mua, mākaukau no ka nānā ʻana ma Kibana. E nele ana mākou i kēia mau mea:
- E ʻike ʻia nā māla helu ma ke ʻano he kikokikona, kahi e pale ai i nā hana ma luna o lākou. ʻO ia hoʻi, nā māla
time-takenIIS log, a me nā kahuarecipient-countиtotal-bitesHoʻopaʻa moʻolelo. - Aia i loko o ka timestamp palapala maʻamau ka manawa i hana ʻia ai ka log, ʻaʻole ka manawa i kākau ʻia ma ka ʻaoʻao kikowaena.
- kahua
recipient-addresse like me kahi kahua kūkulu, ʻaʻole e ʻae i ka nānā ʻana e helu i ka poʻe i loaʻa i nā leka.
ʻO ka manawa kēia e hoʻohui i kekahi mea kilokilo i ke kaʻina logging.
Ke hoʻololi nei i nā kahua helu
He koho ka plugin dissect convert_datatype, hiki ke hoʻohana ʻia no ka hoʻololi ʻana i kahi kahua kikokikona i kahi ʻano kikohoʻe. No ka laʻana, e like me kēia:
dissect {
…
convert_datatype => { "time-taken" => "int" }
…
}
Pono e hoʻomanaʻo he kūpono wale kēia ʻano inā loaʻa i ke kahua kahi kaula. ʻAʻole hana ʻia nā waiwai null mai nā māla e ke koho a hoʻolei ʻia i kahi ʻokoʻa.
No ka hoʻopaʻa ʻana i nā lāʻau, ʻoi aku ka maikaʻi o ka hoʻohana ʻole ʻana i kahi ʻano hoʻololi like, ʻoiai nā māla recipient-count и total-bites nele paha. No ka hoʻololi ʻana i kēia mau kahua ʻoi aku ka maikaʻi o ka hoʻohana ʻana i kahi plugin :
mutate {
convert => [ "total-bytes", "integer" ]
convert => [ "recipient-count", "integer" ]
}
E hoʻokaʻawale ana i ka recipient_address i nā mea loaʻa
Hiki ke hoʻoponopono ʻia kēia pilikia me ka hoʻohana ʻana i ka plugin mutate:
mutate {
split => ["recipient_address", ";"]
}
Ke hoʻololi nei i ka hōʻailona manawa
I ka hihia o ka nānā ʻana i nā lāʻau, hoʻopau maʻalahi ka hana e ka plugin , e kōkua iā ʻoe e kākau ma ke kahua timestamp lā a me ka manawa ma ke ʻano i koi ʻia mai ke kahua date-time:
date {
match => [ "date-time", "ISO8601" ]
timezone => "Europe/Moscow"
remove_field => [ "date-time" ]
}
I ka hihia o nā lāʻau IIS, pono mākou e hoʻohui i ka ʻikepili kahua date и time me ka hoʻohana ʻana i ka plugin mutate, e hoʻopaʻa inoa i ka palena manawa a mākou e pono ai a kau i kēia manawa i loko timestamp me ka hoʻohana ʻana i ka plugin date:
mutate {
add_field => { "data-time" => "%{date} %{time}" }
remove_field => [ "date", "time" ]
}
date {
match => [ "data-time", "YYYY-MM-dd HH:mm:ss" ]
timezone => "UTC"
remove_field => [ "data-time" ]
}
ia auoiaea
Hoʻohana ʻia ka ʻāpana hoʻopuka no ka hoʻouna ʻana i nā lāʻau i hana ʻia i ka loaʻa lāʻau. I ka hoʻouna pololei ʻana iā Elastic, hoʻohana ʻia kahi plugin , e kuhikuhi ana i ka helu kikowaena a me ka papa kuhikuhi inoa no ka hoʻouna ʻana i ka palapala i hana ʻia:
output {
elasticsearch {
hosts => ["127.0.0.1:9200", "127.0.0.2:9200"]
manage_template => false
index => "Exchange-%{+YYYY.MM.dd}"
}
}
hoʻonohonoho hope
ʻO ka hoʻonohonoho hope e like me kēia:
input {
beats {
port => 5044
}
}
filter {
if "IIS" in [tags] {
dissect {
mapping => {
"message" => "%{date} %{time} %{s-ip} %{cs-method} %{cs-uri-stem} %{cs-uri-query} %{s-port} %{cs-username} %{c-ip} %{cs(User-Agent)} %{cs(Referer)} %{sc-status} %{sc-substatus} %{sc-win32-status} %{time-taken}"
}
remove_field => ["message"]
add_field => { "application" => "exchange" }
convert_datatype => { "time-taken" => "int" }
}
mutate {
add_field => { "data-time" => "%{date} %{time}" }
remove_field => [ "date", "time" ]
}
date {
match => [ "data-time", "YYYY-MM-dd HH:mm:ss" ]
timezone => "UTC"
remove_field => [ "data-time" ]
}
}
if "Tracking" in [tags] {
csv {
columns => ["date-time","client-ip","client-hostname","server-ip","server-hostname","source-context","connector-id","source","event-id","internal-message-id","message-id","network-message-id","recipient-address","recipient-status","total-bytes","recipient-count","related-recipient-address","reference","message-subject","sender-address","return-path","message-info","directionality","tenant-id","original-client-ip","original-server-ip","custom-data","transport-traffic-type","log-id","schema-version"]
remove_field => ["message", "tenant-id", "schema-version"]
add_field => { "application" => "exchange" }
}
mutate {
convert => [ "total-bytes", "integer" ]
convert => [ "recipient-count", "integer" ]
split => ["recipient_address", ";"]
}
date {
match => [ "date-time", "ISO8601" ]
timezone => "Europe/Moscow"
remove_field => [ "date-time" ]
}
}
}
output {
elasticsearch {
hosts => ["127.0.0.1:9200", "127.0.0.2:9200"]
manage_template => false
index => "Exchange-%{+YYYY.MM.dd}"
}
}
Hoʻohui pono:
Source: www.habr.com