Ndife abwenzi ndi ELK ndi Kusinthana. Gawo 2

Ndikupitiriza nkhani yanga ya momwe mungapangire abwenzi Kusinthana ndi ELK (poyamba apa). Ndiroleni ndikukumbutseni kuti kuphatikiza uku kumatha kukonza zipika zambiri popanda kukayikira. Nthawi ino tikambirana momwe tingapangire Kusinthana kugwira ntchito ndi zigawo za Logstash ndi Kibana.

Logstash mu ELK stack imagwiritsidwa ntchito pokonza zipika mwanzeru ndikuzikonzekeretsa kuti zikhazikike mu Elastic mwa mawonekedwe a zikalata, pamaziko omwe ndizosavuta kupanga zowonera zosiyanasiyana ku Kibana.

kolowera

Zili ndi magawo awiri:

• Kuyika ndi kukonza phukusi la OpenJDK.
• Kuyika ndi kukonza phukusi la Logstash.

Kuyika ndi kukonza phukusi la OpenJDK

Phukusi la OpenJDK liyenera kutsitsidwa ndikumasulidwa mu bukhu linalake. Kenako njira yopita ku bukhuli iyenera kulowetsedwa mu $env:Path ndi $env:JAVA_HOME zosintha zamakina opangira Windows:

Tiyeni tiwone mtundu wa Java:

PS C:> java -version
openjdk version "13.0.1" 2019-10-15
OpenJDK Runtime Environment (build 13.0.1+9)
OpenJDK 64-Bit Server VM (build 13.0.1+9, mixed mode, sharing)

Kuyika ndi kukonza phukusi la Logstash

Tsitsani fayilo yosungidwa ndi kugawa kwa Logstash kuchokera pano. Zosungirako ziyenera kutsegulidwa mpaka muzu wa disk. Chotsani ku foda C:Program Files Ndizosafunikira, Logstash ikana kuyamba bwino. Kenako muyenera kulowa mu fayilo jvm.options kukonza komwe kuli ndi udindo wogawa RAM panjira ya Java. Ndikupangira kufotokoza theka la RAM ya seva. Ngati ili ndi 16 GB ya RAM pa bolodi, ndiye kuti makiyi osasintha ndi awa:

-Xms1g
-Xmx1g

iyenera kusinthidwa ndi:

-Xms8g
-Xmx8g

Kuphatikiza apo, ndikofunikira kuti mupereke ndemanga pamzerewu -XX:+UseConcMarkSweepGC. Zambiri za izi apa. Chotsatira ndichopanga kusintha kosasintha mu fayilo ya logstash.conf:

input {
 stdin{}
}
 
filter {
}
 
output {
 stdout {
 codec => "rubydebug"
 }
}

Ndi kasinthidwe uku, Logstash amawerenga deta kuchokera ku kontrakitala, amadutsa muzosefera zopanda kanthu, ndikuzitulutsanso ku console. Kugwiritsa ntchito kasinthidwe kumeneku kudzayesa magwiridwe antchito a Logstash. Kuti tichite izi, tiyeni tiyendetse munjira yolumikizirana:

PS C:...bin> .logstash.bat -f .logstash.conf
...
[2019-12-19T11:15:27,769][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[2019-12-19T11:15:27,847][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-12-19T11:15:28,113][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

Logstash idakhazikitsidwa bwino padoko 9600.

Gawo lomaliza la kukhazikitsa: yambitsani Logstash ngati ntchito ya Windows. Izi zikhoza kuchitika, mwachitsanzo, pogwiritsa ntchito phukusi NSSM:

PS C:...bin> .nssm.exe install logstash
Service "logstash" installed successfully!

kulekerera zolakwika

Chitetezo cha zipika zikasamutsidwa kuchokera ku seva yoyambira zimatsimikiziridwa ndi makina a Persistent Queues.

Momwe ikugwirira ntchito

Kapangidwe ka mizere pakukonza chipika ndi: kulowetsa → mzere → fyuluta + zotuluka.

Pulagi yolowetsa imalandira deta kuchokera ku gwero la chipika, kuilembera pamzere, ndikutumiza chitsimikiziro kuti deta yalandilidwa kugwero.

Mauthenga ochokera pamzere amakonzedwa ndi Logstash, kudutsa fyuluta ndi pulogalamu yowonjezera. Mukalandira chitsimikiziro kuchokera pazotuluka kuti chipikacho chatumizidwa, Logstash imachotsa chipika chokonzedwa pamzere. Ngati Logstash iyima, mauthenga onse osasinthidwa ndi mauthenga omwe palibe chitsimikizo chalandiridwa amakhalabe pamzere, ndipo Logstash idzapitiriza kuwakonza nthawi ina ikadzayamba.

kusintha

Kusintha ndi makiyi mu fayilo C:Logstashconfiglogstash.yml:

• queue.type: (zomwe zingatheke - persisted и memory (default)).
• path.queue: (njira yopita ku foda yokhala ndi mafayilo amzere, omwe amasungidwa mu C: Logstashqueue mwachisawawa).
• queue.page_capacity: (kuchuluka kwa tsamba la mzere, mtengo wosasinthika ndi 64mb).
• queue.drain: (zowona / zabodza - zimathandiza / kulepheretsa kuyimitsa mizere musanayambe kutseka Logstash. Sindikulimbikitsani kuti muzitha, chifukwa izi zidzakhudza mwachindunji liwiro la kutseka kwa seva).
• queue.max_events: (kuchuluka kwa zochitika pamzere, kusakhulupirika ndi 0 (zopanda malire)).
• queue.max_bytes: (kuchuluka kwa mzere wama byte, kusakhazikika - 1024mb (1gb)).

Ngati kusinthidwa queue.max_events и queue.max_bytes, ndiye kuti mauthenga amasiya kulandiridwa pamzere pamene mtengo wa zoikamo zonsezi wafika. Dziwani zambiri za Mizere Yotsatizana apa.

Chitsanzo cha gawo la logstash.yml lomwe lili ndi udindo wokhazikitsa mzere:

queue.type: persisted
queue.max_bytes: 10gb

kusintha

Kusintha kwa Logstash nthawi zambiri kumakhala ndi magawo atatu, omwe amayang'anira magawo osiyanasiyana opangira zipika zomwe zikubwera: kulandira (gawo lolowetsa), kuyika (gawo losefera) ndikutumiza ku Elastic (gawo lotulutsa). Pansipa tiwona mwatsatanetsatane aliyense wa iwo.

Lowetsani

Timalandila mtsinje womwe ukubwera ndi zipika zosaphika kuchokera kwa othandizira ma filebeat. Ndi pulogalamu yowonjezera iyi yomwe tikuwonetsa mugawo lolowetsa:

input {
  beats {
    port => 5044
  }
}

Pambuyo pakusintha uku, Logstash imayamba kumvera doko 5044, ndipo ikalandira zipika, imazikonza molingana ndi makonda a gawo la fyuluta. Ngati ndi kotheka, mutha kukulunga njira yolandirira zipika kuchokera ku filebit mu SSL. Werengani zambiri za ma beats plugin makonda apa.

fyuluta

Malemba onse omwe ali osangalatsa kusinthidwa omwe Kusinthana kumapanga ali mumtundu wa csv ndi magawo omwe akufotokozedwa mu fayilo ya chipikayo. Polemba zolemba za csv, Logstash imatipatsa mapulagini atatu: kugawa, csv ndi grok. Yoyamba ndi yambiri wothamanga, koma amalimbana ndi kudula mitengo yosavuta.
Mwachitsanzo, igawa zolemba zotsatirazi kukhala ziwiri (chifukwa cha kukhalapo kwa koma mkati mwamunda), chifukwa chake chipikacho chidzagawidwa molakwika:

…,"MDB:GUID1, Mailbox:GUID2, Event:526545791, MessageClass:IPM.Note, CreationTime:2020-05-15T12:01:56.457Z, ClientType:MOMT, SubmissionAssistant:MailboxTransportSubmissionEmailAssistant",…

Itha kugwiritsidwa ntchito pogawa zipika, mwachitsanzo, IIS. Pankhaniyi, gawo losefera likhoza kuwoneka motere:

filter {
  if "IIS" in [tags] {
    dissect {
      mapping => {
        "message" => "%{date} %{time} %{s-ip} %{cs-method} %{cs-uri-stem} %{cs-uri-query} %{s-port} %{cs-username} %{c-ip} %{cs(User-Agent)} %{cs(Referer)} %{sc-status} %{sc-substatus} %{sc-win32-status} %{time-taken}"
      }
      remove_field => ["message"]
      add_field => { "application" => "exchange" }
    }
  }
} 

Kusintha kwa Logstash kumakupatsani mwayi wogwiritsa ntchito mawu okhazikika, kotero titha kutumiza zipika zomwe zidayikidwa ndi fayilo ya filebeat ku pulogalamu yowonjezera ya dissect IIS. Mkati mwa pulogalamu yowonjezera timafanana ndi zomwe zili m'munda ndi mayina awo, chotsani gawo loyambirira message, yomwe ili ndi zolembera kuchokera pa chipikacho, ndipo tikhoza kuwonjezera gawo lachizoloŵezi lomwe, mwachitsanzo, lidzakhala ndi dzina la ntchito yomwe timasonkhanitsa matabwa.

Pankhani yotsata zipika, ndibwino kugwiritsa ntchito plugin ya csv; imatha kukonza magawo ovuta:

filter {
  if "Tracking" in [tags] {
    csv {
      columns => ["date-time","client-ip","client-hostname","server-ip","server-hostname","source-context","connector-id","source","event-id","internal-message-id","message-id","network-message-id","recipient-address","recipient-status","total-bytes","recipient-count","related-recipient-address","reference","message-subject","sender-address","return-path","message-info","directionality","tenant-id","original-client-ip","original-server-ip","custom-data","transport-traffic-type","log-id","schema-version"]
      remove_field => ["message", "tenant-id", "schema-version"]
      add_field => { "application" => "exchange" }
    }
}

Mkati mwa pulogalamu yowonjezera timafanana ndi zomwe zili m'munda ndi mayina awo, chotsani gawo loyambirira message (komanso minda tenant-id и schema-version), yomwe inali ndi cholembera kuchokera pa chipikacho, ndipo tikhoza kuwonjezera gawo lachizoloŵezi, lomwe, mwachitsanzo, lidzakhala ndi dzina la ntchito yomwe timasonkhanitsa zipika.

Potuluka pagawo losefera, tilandila zikalata pakuyerekeza koyamba, zokonzeka kuwonedwa ku Kibana. Tidzakhala tikusowa zotsatirazi:

• Magawo a manambala adzazindikiridwa ngati malemba, zomwe zimalepheretsa kugwira ntchito pa iwo. Ndiko kuti, minda time-taken IIS log, komanso minda recipient-count и total-bites Log Tracking.
• Chidindo chanthawi zonse chizikhala ndi nthawi yomwe chipikacho chinakonzedwa, osati nthawi yomwe idalembedwa kumbali ya seva.
• m'munda recipient-address idzawoneka ngati malo amodzi omangira, omwe salola kusanthula kuwerengera olandira makalata.

Yakwana nthawi yoti muwonjezere matsenga pang'ono pakukonza chipika.

Kutembenuza magawo manambala

Pulagi ya dissect ili ndi mwayi convert_datatype, yomwe ingagwiritsidwe ntchito kutembenuza gawo la malemba kukhala mawonekedwe a digito. Mwachitsanzo, monga chonchi:

dissect {
  …
  convert_datatype => { "time-taken" => "int" }
  …
}

Ndikoyenera kukumbukira kuti njirayi ndi yoyenera ngati munda udzakhaladi ndi chingwe. Chisankhocho sichimakonza ma Null values ​​kuchokera m'magawo ndikusiya chosiyana.

Potsata zipika, ndibwino kuti musagwiritse ntchito njira yosinthira yofananira, popeza m'minda recipient-count и total-bites ikhoza kukhala yopanda kanthu. Kuti musinthe magawowa ndi bwino kugwiritsa ntchito pulogalamu yowonjezera kusintha:

mutate {
  convert => [ "total-bytes", "integer" ]
  convert => [ "recipient-count", "integer" ]
}

Kugawaniza recipient_address mu olandira aliyense payekha

Vutoli litha kuthetsedwanso pogwiritsa ntchito mutate plugin:

mutate {
  split => ["recipient_address", ";"]
}

Kusintha timestamp

Pankhani yotsata zipika, vutoli limathetsedwa mosavuta ndi pulogalamu yowonjezera tsiku, zomwe zingakuthandizeni kulemba m'munda timestamp tsiku ndi nthawi mu mawonekedwe ofunikira kuchokera kumunda date-time:

date {
  match => [ "date-time", "ISO8601" ]
  timezone => "Europe/Moscow"
  remove_field => [ "date-time" ]
}

Pankhani ya zipika za IIS, tidzafunika kuphatikiza deta yam'munda date и time pogwiritsa ntchito pulogalamu yowonjezera ya mutate, lembani nthawi yomwe tikufuna ndikuyikapo sitampu nthawi ino timestamp kugwiritsa ntchito pulogalamu yowonjezera:

mutate { 
  add_field => { "data-time" => "%{date} %{time}" }
  remove_field => [ "date", "time" ]
}
date { 
  match => [ "data-time", "YYYY-MM-dd HH:mm:ss" ]
  timezone => "UTC"
  remove_field => [ "data-time" ]
}

linanena bungwe

Gawo lotulutsa limagwiritsidwa ntchito kutumiza zipika zokonzedwa kwa wolandila log. Potumiza mwachindunji ku Elastic, pulogalamu yowonjezera imagwiritsidwa ntchito elasticsearch, yomwe imatchula adilesi ya seva ndi template ya dzina lolozera potumiza chikalata chopangidwa:

output {
  elasticsearch {
    hosts => ["127.0.0.1:9200", "127.0.0.2:9200"]
    manage_template => false
    index => "Exchange-%{+YYYY.MM.dd}"
  }
}

Kusintha komaliza

Kukonzekera komaliza kudzawoneka motere:

input {
  beats {
    port => 5044
  }
}
 
filter {
  if "IIS" in [tags] {
    dissect {
      mapping => {
        "message" => "%{date} %{time} %{s-ip} %{cs-method} %{cs-uri-stem} %{cs-uri-query} %{s-port} %{cs-username} %{c-ip} %{cs(User-Agent)} %{cs(Referer)} %{sc-status} %{sc-substatus} %{sc-win32-status} %{time-taken}"
      }
      remove_field => ["message"]
      add_field => { "application" => "exchange" }
      convert_datatype => { "time-taken" => "int" }
    }
    mutate { 
      add_field => { "data-time" => "%{date} %{time}" }
      remove_field => [ "date", "time" ]
    }
    date { 
      match => [ "data-time", "YYYY-MM-dd HH:mm:ss" ]
      timezone => "UTC"
      remove_field => [ "data-time" ]
    }
  }
  if "Tracking" in [tags] {
    csv {
      columns => ["date-time","client-ip","client-hostname","server-ip","server-hostname","source-context","connector-id","source","event-id","internal-message-id","message-id","network-message-id","recipient-address","recipient-status","total-bytes","recipient-count","related-recipient-address","reference","message-subject","sender-address","return-path","message-info","directionality","tenant-id","original-client-ip","original-server-ip","custom-data","transport-traffic-type","log-id","schema-version"]
      remove_field => ["message", "tenant-id", "schema-version"]
      add_field => { "application" => "exchange" }
    }
    mutate {
      convert => [ "total-bytes", "integer" ]
      convert => [ "recipient-count", "integer" ]
      split => ["recipient_address", ";"]
    }
    date {
      match => [ "date-time", "ISO8601" ]
      timezone => "Europe/Moscow"
      remove_field => [ "date-time" ]
    }
  }
}
 
output {
  elasticsearch {
    hosts => ["127.0.0.1:9200", "127.0.0.2:9200"]
    manage_template => false
    index => "Exchange-%{+YYYY.MM.dd}"
  }
}

Maulalo othandiza:

• Momwe mungakhalire OpenJDK 11 pa Windows?
• Tsitsani Logstash
• Zosangalatsa zimagwiritsa ntchito njira yotsitsidwa UseConcMarkSweepGC #36828
• NSSM
• Mizere yosalekeza
• Beats input plugin
• Logstash Dude, tcheni changa chili kuti? Ndiyenera kuthyola zipika zanga
• Dulani pulogalamu yowonjezera yosefera
• Zikhalidwe
• Sinthani zosefera pulogalamu yowonjezera
• Date sefa pulogalamu yowonjezera
• Pulogalamu yowonjezera ya Elasticsearch

Source: www.habr.com