🥇 Peb yog phooj ywg nrog ELK thiab Txauv. Part 1

Kuv tab tom pib cov kab lus uas kuv xav qhia kuv qhov kev paub txuas nrog Kev Sib Pauv thiab ELK. Cov pawg no yuav pab peb ua cov ntaub ntawv loj loj thiab tsis tas yuav xav txog qhov loj npaum li cas cov cuab yeej txiav ntoo niaj hnub yuav tsis kam pab peb. Wb tau paub nrog lub cav fighter tshiab.

Txauv muaj qhov kev txiav txim siab dav dav. Cov cav nrov tshaj plaws yog taug qab cov cav, uas taug qab cov kauj ruam dhau los ntawm ib tsab ntawv tshwj xeeb hauv ib lub koom haum xa ntawv; web server cav, uas taug qab txhua tus neeg siv kev sib kho tshiab hauv qhov system, thiab cov cav ntawm cov ntawv thov web tshwj xeeb nrog ntau qib ntawm kev sib tham nthuav dav. Txauv kuj tuaj yeem khaws cov ntaub ntawv nyoos ntawm smtp, imap thiab pop3 raws tu qauv.

Cov cuab yeej twg peb tuaj yeem siv los ua haujlwm nrog cov cav:

Txheem cmdlet Tau-MessageTrackingLog: yooj yim txheej txheem taug qab cav;
Cov cuab yeej siv logparser: rau kev txiav, siv cov lus tshawb nrhiav pseudo-SQL thiab ua haujlwm sai heev;
Sab nraud SQL neeg rau zaub mov: rau cov xwm txheej tshwj xeeb (piv txwv li, tshuaj xyuas cov ntaub ntawv nyob rau lub sijhawm ntev).

Tag nrho cov no ua haujlwm zoo thaum peb muaj ob peb lub servers thiab qhov ntim ntawm cov txheej txheem ua tiav yog ntsuas hauv kaum lossis pua pua ntawm gigabytes. Tab sis yuav ua li cas yog tias tus naj npawb ntawm cov servers nyob hauv kaum ob, thiab qhov loj ntawm cov cav ntau tshaj ib terabyte? Cov txheej txheem no feem ntau yuav pib crumble.

Thiab qhov no yog qhov tshwm sim: Tau-MessageTrackingLog pib lub sijhawm tawm, logparser tsoo lub qab nthab ntawm 32-ntsis architecture, thiab xa mus rau SQL neeg rau zaub mov tawg nyob rau lub sijhawm tsis tsim nyog tshaj plaws, yam tsis tau zom cov kab ntau tshwj xeeb los ntawm kev pabcuam.

Ntawm no tus neeg ua si tshiab nkag mus rau qhov chaw - ELK pawg, uas yog tshwj xeeb tsim los rau juggling loj loj ntawm cov cav nyob rau hauv lub sij hawm tsim nyog thiab nrog rau kev siv peev txheej.

Hauv thawj ntu kuv yuav qhia rau koj paub meej, Yuav ua li cas txuas filebeat uas yog ib feem ntawm ELK pawg - yog lub luag haujlwm rau kev nyeem ntawv thiab xa cov ntaub ntawv yooj yim rau hauv cov ntawv thov sib txawv sau lawv cov cav. Hauv cov kab lus hauv qab no peb yuav ua tib zoo saib ntawm Logstash thiab Kibana Cheebtsam.

chaw

Yog li, filebeat tus neeg saib xyuas archive file tuaj yeem rub tawm los ntawm lub xaib no.

Peb yuav ua kom tiav lub installation los ntawm tsuas yog rho tawm cov ntsiab lus ntawm zip file. Piv txwv li, hauv c:Program Filesfilebeat. Tom qab ntawd koj yuav tsum khiav PowerShell tsab ntawv install-service-filebeat.ps1, uas los nrog cov khoom siv, txhawm rau nruab qhov kev pabcuam filebeat.

Tam sim no peb npaj txhij pib teeb tsa cov ntaub ntawv teeb tsa.

txhaum cai

Filebeat lav kev xa cov cav mus rau lub cav sau. Qhov no ua tiav los ntawm kev tswj xyuas cov ntawv sau npe hauv cov ntaub ntawv teev cia. Cov ntawv sau npe khaws cov ntaub ntawv hais txog cov ntaub ntawv uas tau nyeem los ntawm cov ntaub ntawv teev tseg, thiab cim cov ntaub ntawv tshwj xeeb uas tuaj yeem xa mus rau qhov chaw.

Yog tias tsis tuaj yeem xa cov ntaub ntawv, filebeat yuav sim rov xa nws mus txog thaum nws tau txais kev pom zoo los ntawm qhov tau txais los yog cov ntaub ntawv pov thawj thawj raug muab tshem tawm thaum lub sijhawm sib hloov.

Thaum qhov kev pabcuam rov pib dua, filebeat yuav nyeem cov ntaub ntawv los ntawm kev sau npe txog cov ntaub ntawv kawg tau nyeem thiab xa tawm, thiab yuav nyeem cov ntaub ntawv hauv cov ntaub ntawv teev npe raws li cov ntaub ntawv hauv kev sau npe.

Qhov no tso cai rau koj kom txo tau qhov kev pheej hmoo ntawm kev poob cov ntaub ntawv uas yuav tsum tau xa mus rau elasticlogstash servers thaum lub sij hawm npaj txhij txog kev ua tsis tiav thiab kev ua haujlwm tu neeg rau zaub mov.

Koj tuaj yeem kawm ntxiv txog qhov no nyeem cov ntaub ntawv hauv kab lus: Yuav ua li cas Filebeat khaws cov ntaub ntawv hauv xeev thiab Filebeat ua li cas thiaj li ua kom tsawg kawg ib zaug xa khoom?

hloov

Tag nrho cov configuration yog ua nyob rau hauv cov ntaub ntawv configuration yml, uas tau muab faib ua ob peb ntu. Cia peb saib qee qhov ntawm lawv uas koom nrog hauv cov txheej txheem ntawm kev sau cov cav los ntawm Exchange servers.

Log ua thaiv

Lub log ua block pib nrog lub teb:

filebeat.inputs:

Peb yuav siv cov cuab yeej sib sau ua ke:

- type: log

Tom ntej no, qhia cov xwm txheej (enabled) thiab txoj hauv kev mus rau lub nplaub tshev nrog cov cav. Piv txwv li, nyob rau hauv cov ntaub ntawv ntawm IIS cav, cov chaw yuav ua tau raws li nram no:

    enabled: true
    paths:
	- C:inetpublogsLogFilesW3SVC1*.log
	- C:inetpublogsLogFilesW3SVC2*.log

Lwm qhov chaw tseem ceeb yog li cas filebeat yuav tsum nyeem cov ntaub ntawv ntau kab. Los ntawm lub neej ntawd, filebeat txiav txim siab ib kab ntawm cov ntaub ntawv teev cia ua ib qho kev nkag. Qhov no ua haujlwm tau zoo txog thaum peb pib tau txais kev zam hauv peb lub cav cuam tshuam txog kev ua haujlwm tsis raug ntawm kev pabcuam. Hauv qhov no, kev zam tuaj yeem muaj ob peb kab. Yog li filebeat yuav tsum suav ntau kab nkag ua ib qho yog tias kab tom ntej pib nrog hnub. Cov hom ntawv teev cov cav hauv Exchange yog raws li hauv qab no: txhua qhov kev nkag tshiab hauv cov ntaub ntawv teev npe pib nrog hnub tim. Hauv kev teeb tsa, qhov xwm txheej zoo li no:

multiline:
	pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
	negate: true
	match: after

Nws ua rau kev nkag siab ntxiv cov cim npe rau cov ntawv tshaj tawm koj xa, piv txwv li:

  tags: ['IIS', 'ex-srv1']

Thiab tsis txhob hnov qab tshem tawm los ntawm kev ua cov kab pib nrog tus cim hash:

  exclude_lines: ['^#']

Yog li, lub log nyeem ntawv thaiv yuav zoo li no:

filebeat.inputs:
- type: log
  enabled: true
  paths:
	- C:inetpublogsLogFilesW3SVC1*.log
	- C:inetpublogsLogFilesW3SVC2*.log
  multiline:
	pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
	negate: true
	match: after
  tags: ['IIS', 'ex-srv1']
  exclude_lines: ['^#']

Log xa thaiv

Filebeat xa ib tus neeg nkag rau hauv cov ntaub ntawv teev npe raws li cov khoom json, uas ib qho kev nkag los ntawm lub cav muaj nyob rau hauv ib qho lus teb. Yog tias peb xav ua haujlwm nrog cov ntaub ntawv no, peb yuav tsum xub txheeb xyuas daim teb no rau hauv cov chaw sib cais. Qhov no tuaj yeem ua tau, piv txwv li, hauv logstash. Nws yuav yog tus tau txais cov ntaub ntawv los ntawm filebeat. Nov yog qhov nws yuav zoo li hauv filebeat configuration file:

output.logstash:
  hosts: ["logstash1.domain.com:5044"]

Yog tias muaj ntau lub servers, ces koj tuaj yeem ua kom sib npaug rau lawv: tom qab ntawd filebeat yuav tsis xa cov cav mus rau thawj tus neeg rau zaub mov muaj los ntawm daim ntawv teev npe, tab sis yuav faib cov ntawv xa tawm ntawm ntau lub servers:

hosts: ["logstash1.domain.com:5044", "logstash2.domain.com:5044"]
  loadbalance: true

Filebeat, thaum ua cov ntaub ntawv nkag mus rau hauv json xa, ntxiv rau cov ntaub ntawv nkag uas muaj nyob rau hauv cov lus teb, ntxiv ib qho ntawm cov metadata, uas cuam tshuam rau qhov loj ntawm cov ntaub ntawv uas xaus rau hauv elastic. Cov metadata no tuaj yeem raug xaiv tshem tawm los ntawm kev xa tawm. Qhov no yog ua nyob rau hauv lub processor thaiv siv lub processor drop_fields. Piv txwv li, koj tuaj yeem cais cov teb hauv qab no:

processors:
- drop_fields:
	fields: ["agent.ephemeral_id", "agent.hostname", "agent.id", "agent.type", "agent.version", "agent", "ecs.version", "ecs", "input.type", "input", "log.offset", "version"]

Koj yuav tsum ua tib zoo xaiv cov teb uas tsis suav nrog, vim tias qee qhov ntawm lawv tuaj yeem siv rau sab elastic los tsim cov indexes.

Yog li, lub log xa thaiv yuav zoo li no:

output.logstash:
  hosts: ["logstash1.domain.com:5044", "logstash2.domain.com:5044"]
  loadbalance: true
 
processors:
- drop_fields:
	fields: ["agent.ephemeral_id", "agent.hostname", "agent.id", "agent.type", "agent.version", "agent", "ecs.version", "ecs", "input.type", "input", "log.offset", "version"]

filebeat logging nqis

Nws ua rau kev txiav txim siab los teeb tsa cov ntawv teev npe hauv qab no:

Cov ntaub ntawv nkag qib;
Peb sau cov cav rau cov ntaub ntawv nyob ntawm lub neej ntawd (cov ntawv teev npe, hauv cov ntawv teev npe filebeat);
Daim ntawv teev npe - filebeat;
Khaws cov ntaub ntawv teev tseg 10 kawg;
Pib tig thaum qhov loj txog 1MB.

Qhov kawg logging configuration thaiv yuav zoo li no:

logging.level: info
logging.to_files: true
logging.files:
  name: filebeat
  keepfiles: 10
  rotateeverybytes: 1048576

Qhov kawg configuration

Peb tau sau cov configuration thiab tam sim no nws zoo li no:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - C:inetpublogsLogFilesW3SVC1*.log
    - C:inetpublogsLogFilesW3SVC2*.log
  multiline:
    pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
    negate: true
    match: after
  tags: ['IIS', 'ex-srv1']
  exclude_lines: ['^#']
 
output.logstash:
  hosts: ["logstash1.domain.com:5044", "logstash2.domain.com:5044"]
  loadbalance: true
 
processors:
- drop_fields:
    fields: ["agent.ephemeral_id", "agent.hostname", "agent.id", "agent.type", "agent.version", "agent", "ecs.version", "ecs", "input.type", "input", "log.offset", "version"]
 
logging.level: info
logging.to_files: true
logging.files:
  name: filebeat
  keepfiles: 10
  rotateeverybytes: 1048576

Nws yog ib qho tseem ceeb kom nkag siab tias cov ntaub ntawv configuration yog yml. Yog li ntawd, nws yog ib qho tseem ceeb uas yuav tsum tau muab qhov chaw thiab cov cim rho tawm kom raug.

Filebeat tuaj yeem tshawb xyuas cov ntaub ntawv teeb tsa thiab, yog tias cov syntax muaj qhov tsis raug, nws yuav qhia tias kab twg thiab qhov twg hauv kab cov syntax tsis raug. Kev kuaj xyuas yog ua raws li hauv qab no: