🥇Waxaan saaxiib la nahay ELK iyo Exchange. Qaybtii 2aad

Waxaan sii wadaa sheekadayda ku saabsan sida loo sameeyo saaxiibo Isweydaarsiga iyo ELK (bilawga halkan). Aan ku xasuusiyo in isku-dhafkan uu awood u leeyahay inuu farsameeyo tiro aad u badan oo loox ah iyada oo aan ka labalabayn. Markan waxaan ka hadli doonaa sida loo helo beddelka oo la shaqeeya Logstash iyo qaybaha Kibana.

Logstash ee ku jira xirmada ELK waxaa loo isticmaalaa in si caqli-gal ah loo farsameeyo diiwaannada loona diyaariyo meelaynta Elastic qaab dukumeenti ah, taas oo ku saleysan taas oo ay ku habboon tahay in lagu dhiso muuqaallo kala duwan Kibana.

Ku rakibida

Waxay ka kooban tahay laba marxaladood:

Ku rakibida iyo habaynta xirmada OpenJDK
Ku rakibida iyo habaynta xirmada Logstash.

Ku rakibida iyo habaynta xirmada OpenJDK

Xirmada OpenJDK waa in la soo dejiyaa oo laga soo dejiyaa hage gaar ah. Markaa dariiqa loo maro hagahan waa in la geliyaa $env:Path iyo $env:JAVA_HOME doorsoomayaasha nidaamka Windows-ka:

Aynu eegno nooca Java:

PS C:> java -version
openjdk version "13.0.1" 2019-10-15
OpenJDK Runtime Environment (build 13.0.1+9)
OpenJDK 64-Bit Server VM (build 13.0.1+9, mixed mode, sharing)

Ku rakibida iyo habaynta xirmada Logstash

Soo deji faylka kaydka leh qaybinta Logstash halkan. Kaydka waa in laga furo xididka saxanka. Ka soo saar galka C:Program Files Uma qalmo, Logstash wuxuu diidi doonaa inuu si caadi ah u bilaabo. Markaa waxaad u baahan tahay inaad geliso faylka jvm.options hagaajinta mas'uulka ka ah u qoondaynta RAM ee habka Java. Waxaan ku talinayaa in la qeexo kala bar RAM ee server-ka. Haddii ay leedahay 16 GB oo RAM ah dusha sare, markaa furayaasha caadiga ah waa:

-Xms1g
-Xmx1g

waa in lagu badalaa:

-Xms8g
-Xmx8g

Intaa waxaa dheer, waxaa habboon in laga faalloodo khadka -XX:+UseConcMarkSweepGC. Wax badan oo arrintan ku saabsan halkan. Talaabada xigta waa in la abuuro qaabayn caadi ah faylka logstash.conf:

input {
 stdin{}
}
 
filter {
}
 
output {
 stdout {
 codec => "rubydebug"
 }
}

Qaabeyntan, Logstash wuxuu akhriyaa xogta console-ka, wuxuu dhex maraa shaandheyn madhan, wuxuuna dib ugu soo saaraa console-ka. Isticmaalka qaabeyntan waxay tijaabin doontaa shaqeynta Logstash. Si tan loo sameeyo, aan ku socodsiino habka is-dhexgalka:

PS C:...bin> .logstash.bat -f .logstash.conf
...
[2019-12-19T11:15:27,769][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[2019-12-19T11:15:27,847][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-12-19T11:15:28,113][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

Logstash ayaa si guul leh uga bilaabay dekedda 9600.

Tallaabada ugu dambeysa ee rakibidda: u bilow Logstash adeeg Windows ahaan. Tan waxaa la samayn karaa, tusaale ahaan, iyadoo la isticmaalayo xirmada NSSM:

PS C:...bin> .nssm.exe install logstash
Service "logstash" installed successfully!

dulqaadka qaladka

Badbaadada diiwaannada marka laga soo wareejiyo server-ka isha waxaa lagu hubiyaa habka safafka joogtada ah.

Sida ay u shaqeyso

Qaabka safafka inta lagu jiro habaynta log waa: gelida → safka → filter + wax soo saarka.

Plugin-ka wax gelinta waxa uu xogta ka helayaa isha log, waxa uu u qoraa saf, oo waxa uu u soo dirayaa xaqiijinta in xogta la helay.

Farriimaha safka ka imaanaya waxaa habeeya Logstash, waxaana la sii maraa shaandhada iyo plugin-ka wax soo saarka. Markaad hesho xaqiijinta soo-saarka in log-ka la diray, Logstash waxay ka saartaa log-ga la shaqeeyay safka. Haddii Logstash joogsado, dhammaan fariimaha iyo fariimaha aan la socodsiin ee aan xaqiijin la helin ayaa ku jira safka, Logstash waxay sii wadi doontaa socodsiinta iyaga marka xigta ee ay bilowdo.

sixitaanka

Lagu hagaajin karo furayaasha faylka C:Logstashconfiglogstash.yml:

queue.type: (qiimaha macquulka ah - persisted и memory (default)).
path.queue: (dariiqa loo maro galka leh faylalka safka, kuwaas oo lagu kaydiyay C: Logstashqueue default).
queue.page_capacity: ( cabbirka bogga safka ugu badan, qiimaha caadiga ah waa 64mb).
queue.drain: (run/been ah - waxay awood u siinaysaa / joojisaa joojinta habka safka ka hor inta aan la xidhin Logstash. Kuma talinayo in la awoodsiiyo, sababtoo ah tani waxay si toos ah u saameyn doontaa xawaaraha xiritaanka serverka).
queue.max_events: (tirada ugu badan ee dhacdooyinka safka, default waa 0 (aan xadidnayn)).
queue.max_bytes: (cabirka safka ugu badan ee bytes, default - 1024mb (1gb)).

Haddii la habeeyey queue.max_events и queue.max_bytes, dabadeed fariimaha la joojiyo in lagu aqbalo safka marka qiimaha mid ka mid ah goobahan la gaaro. Wax badan ka baro safafka joogtada ah halkan.

Tusaale qaybta logstash.yml ee masuulka ka ah dejinta safka:

queue.type: persisted
queue.max_bytes: 10gb

sixitaanka

Qaabeynta Logstash waxay badanaa ka kooban tahay saddex qaybood, oo mas'uul ka ah wejiyada kala duwan ee habaynta diiwaannada soo socda: helitaanka (qaybta gelinta), kala-soocidda (qaybta shaandhaynta) iyo u diridda Elastic (qaybta wax soo saarka). Hoos waxaan si dhow u eegi doonaa mid kasta oo iyaga ka mid ah.

Input

Waxaan ka helnaa qulqulka soo socda oo wata dogob ceeriin ah wakiilada filebeat Waa plugin-kan aan ku muujinayno qaybta gelinta:

input {
  beats {
    port => 5044
  }
}

Qaabeyntan ka dib, Logstash wuxuu bilaabaa dhageysiga dekedda 5044, iyo marka la helo diiwaannada, wuxuu u shaqeeyaa si waafaqsan jaangooyooyinka qaybta shaandhada. Haddii loo baahdo, waxaad ku duubi kartaa kanaalka si aad uga hesho diiwaannada filebit ee SSL. Ka akhriso wax badan oo ku saabsan jaangooyooyinka plugin halkan.

Filter

Dhammaan qoraallada qoraalka ah ee xiiseeya habaynta ee Sarrifku soo saaro waxay u qaabaysan yihiin qaabka csv oo leh meelaha lagu sifeeyay faylka log laftiisa. Si loo kala saaro diiwaannada csv, Logstash waxay na siisaa saddex plugin: kala tag, csv iyo grok. Midka hore waa kan ugu badan быстрый, laakiin waxa ay la tacaalaysaa falanqaynta kaliya qoraallada ugu fudud.
Tusaale ahaan, waxa ay u kala qaybin doontaa rikoodhada soo socda laba (sabato ah joogitaanka comma gudaha garoonka), taas oo ah sababta logu si khaldan loo kala saari doono:

…,"MDB:GUID1, Mailbox:GUID2, Event:526545791, MessageClass:IPM.Note, CreationTime:2020-05-15T12:01:56.457Z, ClientType:MOMT, SubmissionAssistant:MailboxTransportSubmissionEmailAssistant",…

Waxaa loo isticmaali karaa marka la kala saarayo diiwaannada, tusaale ahaan, IIS. Xaaladdan oo kale, qaybta filter waxay u ekaan kartaa sidan:

filter {
  if "IIS" in [tags] {
    dissect {
      mapping => {
        "message" => "%{date} %{time} %{s-ip} %{cs-method} %{cs-uri-stem} %{cs-uri-query} %{s-port} %{cs-username} %{c-ip} %{cs(User-Agent)} %{cs(Referer)} %{sc-status} %{sc-substatus} %{sc-win32-status} %{time-taken}"
      }
      remove_field => ["message"]
      add_field => { "application" => "exchange" }
    }
  }
}

Qaabeynta Logstash waxay kuu ogolaaneysaa inaad isticmaasho hadallo shuruud ah, marka waxaan kaliya u soo diri karnaa diiwaannada lagu calaamadeeyay sumadda filebeat plugin-ka kala-goynta IIS. Gudaha plugin waxaan ku dhignaa qiyamka goobta magacyadooda, tirtir goobta asalka ah message, kaas oo ka kooban gelitaanka log-ga, waxaanan ku dari karnaa goob gaar ah oo, tusaale ahaan, ka koobnaan doonta magaca codsiga aan ka soo ururinno logyada.

Marka laga hadlayo diiwaannada raadraaca, way fiican tahay in la isticmaalo csv plugin; waxay si sax ah u socodsiin kartaa meelaha adag:

filter {
  if "Tracking" in [tags] {
    csv {
      columns => ["date-time","client-ip","client-hostname","server-ip","server-hostname","source-context","connector-id","source","event-id","internal-message-id","message-id","network-message-id","recipient-address","recipient-status","total-bytes","recipient-count","related-recipient-address","reference","message-subject","sender-address","return-path","message-info","directionality","tenant-id","original-client-ip","original-server-ip","custom-data","transport-traffic-type","log-id","schema-version"]
      remove_field => ["message", "tenant-id", "schema-version"]
      add_field => { "application" => "exchange" }
    }
}

Gudaha plugin waxaan ku dhignaa qiyamka goobta magacyadooda, tirtir goobta asalka ah message (iyo sidoo kale beeraha tenant-id и schema-version), kaas oo ka kooban gelitaanka log-ga, oo aanu ku dari karnaa goob gaar ah, kaas oo, tusaale ahaan, ka koobnaan doona magaca codsiga kaas oo aan ka soo ururinay log.

Marka laga baxo heerka shaandhaynta, waxaan heli doonaa dukumeenti qiyaas ahaan marka hore, oo u diyaarsan muuqaal ahaan gudaha Kibana. Waxaa naga maqnaan doona kuwa soo socda:

Goobaha nambarada waxaa loo aqoonsan doonaa qoraal ahaan, taas oo ka hortagaysa hawlgallada iyaga. Kuwaas oo kala ah, beeraha time-taken IIS log, iyo sidoo kale beeraha recipient-count и total-bites Log Tracking.
Jadwalka wakhtiga dukumeentiga caadiga ahi waxa uu ka koobnaan doona wakhtiga logu shaqaynayo, ma aha wakhtiga lagu qoray dhinaca server-ka.
field recipient-address waxay u ekaan doontaa hal goob dhisme, taas oo aan ogolayn falanqaynta in la tiriyo dadka waraaqaha qaata.

Waa markii lagu dari lahaa sixir yar habka habaynta log.

Beddelida meelaha tirada

Plugin dissect ayaa leh ikhtiyaar convert_datatype, kaas oo loo isticmaali karo in goobta qoraalka loogu beddelo qaab dhijitaal ah. Tusaale ahaan, sida tan:

dissect {
  …
  convert_datatype => { "time-taken" => "int" }
  …
}

Waxaa habboon in la xasuusto in habkani uu ku habboon yahay oo kaliya haddii garoonku uu hubaal ahaan ku jiri doono xadhig. Doorashadu kama baaraandegto qiyamka Null ee beeraha waxayna tuurtaa ka reeban.

Si loo raad raaco diiwaannada, way fiicantahay inaadan isticmaalin habka beddelka la mid ah, tan iyo beeraha recipient-count и total-bites waxaa laga yaabaa inay madhan tahay. Si loo rogo meelahan waxaa fiican inaad isticmaasho plugin beddelaad:

mutate {
  convert => [ "total-bytes", "integer" ]
  convert => [ "recipient-count", "integer" ]
}

U kala qaybinta ciwaanka qaataha ee qaataha gaarka ah

Dhibaatadan waxa kale oo lagu xalin karaa iyadoo la isticmaalayo mutate plugin:

mutate {
  split => ["recipient_address", ";"]
}

Beddelka shaambada wakhtiga

Marka la eego diiwaannada raadraaca, dhibaatada si fudud ayaa loo xalliyaa plugin taariikhda, kaas oo kaa caawin doona inaad wax ku qorto goobta timestamp taariikhda iyo waqtiga qaabka looga baahan yahay goobta date-time:

date {
  match => [ "date-time", "ISO8601" ]
  timezone => "Europe/Moscow"
  remove_field => [ "date-time" ]
}

Marka laga hadlayo diiwaannada IIS, waxaan u baahan doonaa inaan isku darno xogta goobta date и time Adigoo isticmaalaya plugin mutate ah, diwaangeli aagga wakhtiga aan u baahanahay oo dhig shaambada wakhtigan timestamp adigoo isticmaalaya taariikhda plugin:

mutate { 
  add_field => { "data-time" => "%{date} %{time}" }
  remove_field => [ "date", "time" ]
}
date { 
  match => [ "data-time", "YYYY-MM-dd HH:mm:ss" ]
  timezone => "UTC"
  remove_field => [ "data-time" ]
}

Output

Qaybta wax-soo-saarka waxaa loo isticmaalaa in loogu diro diiwaannada habaysan ee qaata-gacanta. Haddii si toos ah loogu diro Elastic, plugin ayaa la isticmaalaa laascaanood, kaas oo qeexaya ciwaanka serverka iyo qaabka tusaha magaca ee dirida dukumeentiga la sameeyay:

output {
  elasticsearch {
    hosts => ["127.0.0.1:9200", "127.0.0.2:9200"]
    manage_template => false
    index => "Exchange-%{+YYYY.MM.dd}"
  }
}

Qaabeynta kama dambaysta ah

Qaabaynta kama dambaysta ahi waxay u ekaan doontaa sidan:

input {
  beats {
    port => 5044
  }
}
 
filter {
  if "IIS" in [tags] {
    dissect {
      mapping => {
        "message" => "%{date} %{time} %{s-ip} %{cs-method} %{cs-uri-stem} %{cs-uri-query} %{s-port} %{cs-username} %{c-ip} %{cs(User-Agent)} %{cs(Referer)} %{sc-status} %{sc-substatus} %{sc-win32-status} %{time-taken}"
      }
      remove_field => ["message"]
      add_field => { "application" => "exchange" }
      convert_datatype => { "time-taken" => "int" }
    }
    mutate { 
      add_field => { "data-time" => "%{date} %{time}" }
      remove_field => [ "date", "time" ]
    }
    date { 
      match => [ "data-time", "YYYY-MM-dd HH:mm:ss" ]
      timezone => "UTC"
      remove_field => [ "data-time" ]
    }
  }
  if "Tracking" in [tags] {
    csv {
      columns => ["date-time","client-ip","client-hostname","server-ip","server-hostname","source-context","connector-id","source","event-id","internal-message-id","message-id","network-message-id","recipient-address","recipient-status","total-bytes","recipient-count","related-recipient-address","reference","message-subject","sender-address","return-path","message-info","directionality","tenant-id","original-client-ip","original-server-ip","custom-data","transport-traffic-type","log-id","schema-version"]
      remove_field => ["message", "tenant-id", "schema-version"]
      add_field => { "application" => "exchange" }
    }
    mutate {
      convert => [ "total-bytes", "integer" ]
      convert => [ "recipient-count", "integer" ]
      split => ["recipient_address", ";"]
    }
    date {
      match => [ "date-time", "ISO8601" ]
      timezone => "Europe/Moscow"
      remove_field => [ "date-time" ]
    }
  }
}
 
output {
  elasticsearch {
    hosts => ["127.0.0.1:9200", "127.0.0.2:9200"]
    manage_template => false
    index => "Exchange-%{+YYYY.MM.dd}"
  }
}

Xiriiro waxtar leh:

Source: www.habr.com

Waxaan saaxiib la nahay ELK iyo Exchange. Qaybta 2

Ku rakibida

Ku rakibida iyo habaynta xirmada Logstash

dulqaadka qaladka

Sida ay u shaqeyso

sixitaanka

sixitaanka

Input

Filter

Beddelida meelaha tirada

U kala qaybinta ciwaanka qaataha ee qaataha gaarka ah

Beddelka shaambada wakhtiga

Output

Qaabeynta kama dambaysta ah

Add a comment cancel reply