Kuv txuas ntxiv kuv zaj dab neeg hais txog yuav ua li cas ua phooj ywg Txauv thiab ELK (pib ). Cia kuv ceeb toom rau koj tias qhov kev sib xyaw ua ke no muaj peev xwm ua tiav cov cav ntau ntau yam tsis muaj kev cuam tshuam. Lub sijhawm no peb yuav tham txog yuav ua li cas kom tau txais Kev Sib Pauv ua haujlwm nrog Logstash thiab Kibana Cheebtsam.
Logstash nyob rau hauv lub ELK pawg yog siv los ua kom txawj ntse txheej txheem cav thiab npaj lawv rau kev tso kawm nyob rau hauv Elastic nyob rau hauv daim ntawv ntawm cov ntaub ntawv, nyob rau hauv lub hauv paus ntawm nws yog yooj yim los tsim ntau yam visualizations nyob rau hauv Kibana.
chaw
Nws muaj ob theem:
- Txhim kho thiab teeb tsa OpenJDK pob.
- Txhim kho thiab teeb tsa Logstash pob.
Txhim kho thiab teeb tsa OpenJDK pob
Lub pob OpenJDK yuav tsum tau rub tawm thiab muab tso rau hauv ib qho kev qhia tshwj xeeb. Tom qab ntawd txoj kev mus rau phau ntawv teev npe no yuav tsum tau nkag mus rau hauv $env: Path thiab $env:JAVA_HOME hloov pauv ntawm Windows operating system:
Cia peb tshawb xyuas Java version:
PS C:> java -version
openjdk version "13.0.1" 2019-10-15
OpenJDK Runtime Environment (build 13.0.1+9)
OpenJDK 64-Bit Server VM (build 13.0.1+9, mixed mode, sharing)
Txhim kho thiab teeb tsa Logstash pob
Download tau cov ntaub ntawv archive nrog Logstash faib . Lub archive yuav tsum tau unpacked rau hauv paus ntawm lub disk. Unpack rau folder C:Program Files Nws tsis tsim nyog, Logstash yuav tsis kam pib ib txwm. Tom qab ntawd koj yuav tsum nkag mus rau hauv cov ntaub ntawv jvm.options kho lub luag haujlwm rau faib RAM rau Java txheej txheem. Kuv pom zoo kom qhia meej ib nrab ntawm lub server lub RAM. Yog tias nws muaj 16 GB ntawm RAM ntawm lub nkoj, ces cov yuam sij ua ntej yog:
-Xms1g
-Xmx1g
yuav tsum tau hloov nrog:
-Xms8g
-Xmx8g
Tsis tas li ntawd, nws raug nquahu kom tawm tswv yim tawm ntawm kab -XX:+UseConcMarkSweepGC. Xav paub ntxiv txog qhov no . Cov kauj ruam tom ntej yog los tsim ib qho kev teeb tsa nyob rau hauv cov ntaub ntawv logstash.conf:
input {
stdin{}
}
filter {
}
output {
stdout {
codec => "rubydebug"
}
}
Nrog rau qhov kev teeb tsa no, Logstash nyeem cov ntaub ntawv los ntawm lub console, hla nws los ntawm cov lim khoob, thiab tso tawm nws rov qab mus rau lub console. Siv qhov kev teeb tsa no yuav sim ua haujlwm ntawm Logstash. Txhawm rau ua qhov no, cia peb khiav nws hauv kev sib tham sib hom:
PS C:...bin> .logstash.bat -f .logstash.conf
...
[2019-12-19T11:15:27,769][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[2019-12-19T11:15:27,847][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-12-19T11:15:28,113][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
Logstash tau pib ua tiav ntawm qhov chaw nres nkoj 9600.
Cov kauj ruam kawg ntawm kev teeb tsa: tso Logstash ua qhov kev pabcuam Windows. Qhov no tuaj yeem ua tau, piv txwv li, siv lub pob :
PS C:...bin> .nssm.exe install logstash
Service "logstash" installed successfully!
txhaum cai
Kev nyab xeeb ntawm cov cav thaum xa tawm los ntawm cov neeg rau zaub mov tau lees paub los ntawm Persistent Queues mechanism.
Nws ua haujlwm li cas
Cov txheej txheem ntawm queues thaum lub cav ua haujlwm yog: input β queue β filter + output.
Lub tswv yim plugin tau txais cov ntaub ntawv los ntawm lub cav qhov chaw, sau nws mus rau ib kab, thiab xa cov ntaub ntawv lees paub tias cov ntaub ntawv tau txais mus rau qhov chaw.
Cov lus los ntawm cov kab yog ua tiav los ntawm Logstash, dhau los ntawm cov lim thiab cov zis plugin. Thaum tau txais kev pom zoo los ntawm cov zis uas lub cav tau raug xa mus, Logstash tshem tawm cov txheej txheem txheej txheem los ntawm cov kab. Yog tias Logstash nres, tag nrho cov lus tsis tau ua tiav thiab cov lus uas tsis tau txais kev lees paub tseem nyob hauv kab, thiab Logstash yuav txuas ntxiv ua lawv lub sijhawm tom ntej nws pib.
hloov
Kho tau los ntawm cov yuam sij hauv cov ntaub ntawv C:Logstashconfiglogstash.yml:
queue.type: (qhov muaj nqis -persistedΠΈmemory (default)).path.queue: (txoj kev mus rau lub nplaub tshev nrog cov ntaub ntawv queue, uas tau khaws cia hauv C: Logstashqueue los ntawm lub neej ntawd).queue.page_capacity: (qhov siab tshaj plaws kab ntawv loj, tus nqi pib yog 64mb).queue.drain: ( tseeb / cuav - enables / disables nres queue ua ua ntej kaw Logstash. Kuv tsis pom zoo kom qhib nws, vim hais tias qhov no yuav ncaj qha cuam tshuam qhov ceev ntawm lub server kaw).queue.max_events: (cov xwm txheej ntau tshaj plaws hauv kab, lub neej ntawd yog 0 (unlimited)).queue.max_bytes: (qhov loj tshaj kab nyob hauv bytes, default - 1024mb (1gb)).
Yog configured queue.max_events ΠΈ queue.max_bytes, ces cov lus nres raug lees txais rau hauv kab thaum tus nqi ntawm ib qho ntawm cov kev teeb tsa no tau mus txog. Kawm ntxiv txog Persistent Queues .
Ib qho piv txwv ntawm ib feem ntawm logstash.yml lub luag haujlwm rau kev teeb tsa kab:
queue.type: persisted
queue.max_bytes: 10gb
hloov
Logstash configuration feem ntau muaj peb ntu, lub luag haujlwm rau ntau theem ntawm kev ua cov ntaub ntawv nkag: tau txais (cov lus nkag), parsing (filter section) thiab xa mus rau Elastic (tso zis seem). Hauv qab no peb yuav saib ze dua ntawm lawv txhua tus.
Tswv yim
Peb tau txais cov kwj tuaj nrog cov ntaub ntawv nyoos los ntawm cov neeg ua haujlwm filebeat. Nws yog qhov plugin no uas peb qhia hauv nqe lus nkag:
input {
beats {
port => 5044
}
}
Tom qab qhov kev teeb tsa no, Logstash pib mloog qhov chaw nres nkoj 5044, thiab thaum tau txais cov ntawv teev tseg, ua haujlwm raws li qhov chaw ntawm ntu lim. Yog tias tsim nyog, koj tuaj yeem qhwv cov channel kom tau txais cov cav los ntawm filebit hauv SSL. Nyeem ntxiv txog beats plugin nqis .
Lim
Tag nrho cov ntawv teev lus uas nthuav dav rau kev ua haujlwm uas Txauv tsim muaj nyob rau hauv csv hom nrog cov teb tau piav qhia hauv cov ntaub ntawv teev cia nws tus kheej. Rau kev txheeb xyuas csv cov ntaub ntawv, Logstash muab peb lub plugins: , csv and grok. Thawj tus yog tshaj plaws , tab sis copes nrog parsing tsuas yog cov yooj yim cav.
Piv txwv li, nws yuav faib cov ntaub ntawv hauv qab no ua ob (vim muaj lub cim cim hauv daim teb), uas yog vim li cas lub cav yuav raug parsed tsis raug:
β¦,"MDB:GUID1, Mailbox:GUID2, Event:526545791, MessageClass:IPM.Note, CreationTime:2020-05-15T12:01:56.457Z, ClientType:MOMT, SubmissionAssistant:MailboxTransportSubmissionEmailAssistant",β¦
Nws tuaj yeem siv thaum txheeb xyuas cov cav, piv txwv li, IIS. Hauv qhov no, ntu lim yuav zoo li no:
filter {
if "IIS" in [tags] {
dissect {
mapping => {
"message" => "%{date} %{time} %{s-ip} %{cs-method} %{cs-uri-stem} %{cs-uri-query} %{s-port} %{cs-username} %{c-ip} %{cs(User-Agent)} %{cs(Referer)} %{sc-status} %{sc-substatus} %{sc-win32-status} %{time-taken}"
}
remove_field => ["message"]
add_field => { "application" => "exchange" }
}
}
}
Logstash configuration tso cai rau koj siv , yog li peb tsuas tuaj yeem xa cov cav uas tau tag nrog filebeat tag mus rau qhov dissect plugin IIS. Sab hauv lub plugin peb phim cov nqi teb nrog lawv cov npe, tshem tawm thawj daim teb message, uas muaj ib qho kev nkag los ntawm lub cav, thiab peb tuaj yeem ntxiv ib qho kev cai teb uas yuav, piv txwv li, muaj lub npe ntawm daim ntawv thov uas peb sau cov cav.
Nyob rau hauv cov ntaub ntawv ntawm kev taug qab cov cav, nws yog qhov zoo dua los siv lub csv plugin; nws tuaj yeem ua cov txheej txheem nyuaj:
filter {
if "Tracking" in [tags] {
csv {
columns => ["date-time","client-ip","client-hostname","server-ip","server-hostname","source-context","connector-id","source","event-id","internal-message-id","message-id","network-message-id","recipient-address","recipient-status","total-bytes","recipient-count","related-recipient-address","reference","message-subject","sender-address","return-path","message-info","directionality","tenant-id","original-client-ip","original-server-ip","custom-data","transport-traffic-type","log-id","schema-version"]
remove_field => ["message", "tenant-id", "schema-version"]
add_field => { "application" => "exchange" }
}
}
Sab hauv lub plugin peb phim cov nqi teb nrog lawv cov npe, tshem tawm thawj daim teb message (thiab cov teb tenant-id ΠΈ schema-version), uas muaj ib qho kev nkag los ntawm lub cav, thiab peb tuaj yeem ntxiv ib qho kev cai teb, uas yuav, piv txwv li, muaj lub npe ntawm daim ntawv thov uas peb sau cov cav.
Ntawm qhov tawm ntawm lub lim tiam, peb yuav tau txais cov ntaub ntawv hauv thawj qhov kwv yees, npaj rau kev pom hauv Kibana. Peb yuav tsis nco qab cov hauv qab no:
- Cov lej teb yuav raug lees paub tias yog cov ntawv nyeem, uas tiv thaiv kev ua haujlwm ntawm lawv. Namely, cov teb
time-takenIIS log, as well as fieldsrecipient-countΠΈtotal-bitesNkag mus. - Daim ntawv teev sij hawm tus txheej txheem yuav muaj lub sij hawm lub cav tau ua tiav, tsis yog lub sijhawm nws tau sau rau ntawm server sab.
- teb
recipient-addressyuav zoo li ib qhov chaw tsim kho, uas tsis tso cai rau kev txheeb xyuas suav cov neeg tau txais cov ntawv.
Nws yog lub sij hawm los ntxiv me ntsis khawv koob rau lub cav ua cov txheej txheem.
Hloov cov zauv teb
Lub dissect plugin muaj kev xaiv convert_datatype, uas tuaj yeem siv los hloov cov ntawv sau rau hauv hom digital. Piv txwv li, zoo li no:
dissect {
β¦
convert_datatype => { "time-taken" => "int" }
β¦
}
Nws yog tsim nyog nco ntsoov tias txoj kev no tsuas yog tsim nyog yog tias daim teb yuav twv yuav raug hu muaj ib txoj hlua. Qhov kev xaiv tsis ua Null qhov tseem ceeb ntawm cov teb thiab cuam tshuam qhov kev zam.
Rau kev taug qab cov cav, nws yog qhov zoo dua tsis siv txoj kev hloov pauv zoo sib xws, txij li cov teb recipient-count ΠΈ total-bites tej zaum yuav npliag. Txhawm rau hloov cov teb no nws yog qhov zoo dua los siv lub plugin :
mutate {
convert => [ "total-bytes", "integer" ]
convert => [ "recipient-count", "integer" ]
}
Spliting recipient_address rau ib tus neeg tau txais
Qhov teeb meem no tseem tuaj yeem daws tau los ntawm kev hloov pauv plugin:
mutate {
split => ["recipient_address", ";"]
}
Hloov daim ntawv teev sijhawm
Nyob rau hauv cov ntaub ntawv ntawm taug qab cav, qhov teeb meem yog heev yooj yim daws tau los ntawm lub plugin , uas yuav pab tau koj sau hauv daim teb timestamp hnub thiab sijhawm nyob rau hauv hom ntawv xav tau los ntawm daim teb date-time:
date {
match => [ "date-time", "ISO8601" ]
timezone => "Europe/Moscow"
remove_field => [ "date-time" ]
}
Nyob rau hauv cov ntaub ntawv ntawm IIS cav, peb yuav tsum tau muab cov ntaub ntawv teb date ΠΈ time siv lub mutate plugin, sau npe lub sij hawm cheeb tsam peb xav tau thiab tso lub sij hawm no thwj rau hauv timestamp siv lub hnub plugin:
mutate {
add_field => { "data-time" => "%{date} %{time}" }
remove_field => [ "date", "time" ]
}
date {
match => [ "data-time", "YYYY-MM-dd HH:mm:ss" ]
timezone => "UTC"
remove_field => [ "data-time" ]
}
tso zis
Cov zis seem yog siv los xa cov txheej txheem ua tiav rau lub cav txais. Nyob rau hauv cov ntaub ntawv ntawm xa ncaj qha mus rau Elastic, ib tug plugin yog siv , uas qhia qhov chaw nyob server thiab lub npe index template rau xa cov ntaub ntawv generated:
output {
elasticsearch {
hosts => ["127.0.0.1:9200", "127.0.0.2:9200"]
manage_template => false
index => "Exchange-%{+YYYY.MM.dd}"
}
}
Qhov kawg configuration
Qhov kawg configuration yuav zoo li no:
input {
beats {
port => 5044
}
}
filter {
if "IIS" in [tags] {
dissect {
mapping => {
"message" => "%{date} %{time} %{s-ip} %{cs-method} %{cs-uri-stem} %{cs-uri-query} %{s-port} %{cs-username} %{c-ip} %{cs(User-Agent)} %{cs(Referer)} %{sc-status} %{sc-substatus} %{sc-win32-status} %{time-taken}"
}
remove_field => ["message"]
add_field => { "application" => "exchange" }
convert_datatype => { "time-taken" => "int" }
}
mutate {
add_field => { "data-time" => "%{date} %{time}" }
remove_field => [ "date", "time" ]
}
date {
match => [ "data-time", "YYYY-MM-dd HH:mm:ss" ]
timezone => "UTC"
remove_field => [ "data-time" ]
}
}
if "Tracking" in [tags] {
csv {
columns => ["date-time","client-ip","client-hostname","server-ip","server-hostname","source-context","connector-id","source","event-id","internal-message-id","message-id","network-message-id","recipient-address","recipient-status","total-bytes","recipient-count","related-recipient-address","reference","message-subject","sender-address","return-path","message-info","directionality","tenant-id","original-client-ip","original-server-ip","custom-data","transport-traffic-type","log-id","schema-version"]
remove_field => ["message", "tenant-id", "schema-version"]
add_field => { "application" => "exchange" }
}
mutate {
convert => [ "total-bytes", "integer" ]
convert => [ "recipient-count", "integer" ]
split => ["recipient_address", ";"]
}
date {
match => [ "date-time", "ISO8601" ]
timezone => "Europe/Moscow"
remove_field => [ "date-time" ]
}
}
}
output {
elasticsearch {
hosts => ["127.0.0.1:9200", "127.0.0.2:9200"]
manage_template => false
index => "Exchange-%{+YYYY.MM.dd}"
}
}
Cov kev sib txuas tseem ceeb:
Tau qhov twg los: www.hab.com