🥇Singabangane ne-ELK kanye ne-Exchange. Isiqephu sesi-1

Ngiqala uchungechunge lwezihloko lapho ngifuna ukwabelana ngolwazi lwami lokuxhuma i-Exchange ne-ELK. Lesi sitaki sizosisiza ukuthi sicubungule amavolumu amakhulu amalogi futhi singamangali ukuthi amathuluzi avamile okugawula angeke avume ukusisiza. Ake sijwayelane ne-log fighter entsha.

I-Exchange inesistimu yokugawulwa kwemithi ebanzi. Amalogi adume kakhulu amalogi okulandelela, alandelela isinyathelo ngesinyathelo sencwadi ethile ngaphakathi kwenhlangano yeposi; amalogi eseva yewebhu, alandelela iseshini ngayinye yomsebenzisi omusha ohlelweni, namalogu ezinhlelo zokusebenza ezithile zewebhu ezinamazinga ahlukahlukene emininingwane yeseshini. I-Exchange ingase futhi igcine amalogu angahluziwe ephrothokholi ye-smtp, imap ne-pop3.

Imaphi amathuluzi esingawasebenzisa ukuze sisebenze ngamalogi:

I-cmdlet ejwayelekile Get-MessageTrackingLog: cubungula kalula izingodo zokulandelela;
Insiza ye-logparser: ekugawulweni kwemithi, isebenzisa ulimi lokusesha i-pseudo-SQL futhi isebenza ngokushesha okukhulu;
Iseva ye-SQL yangaphandle: yezimo eziqondile (isibonelo, ukuhlaziya idatha phakathi nesikhathi eside).

Konke lokhu kusebenza kahle uma sinamaseva ambalwa futhi umthamo wamalogi acutshunguliwe ulinganiswa ngamashumi noma amakhulu amagigabhayithi. Kodwa kuthiwani uma inani lamaseva likunqwaba, futhi usayizi wamalogi udlula i-terabyte? Lolu hlelo cishe luqala ukuwohloka.

Futhi nakhu okwenzekayo: I-Get-MessageTrackingLog iqala ukuphela kwesikhathi, i-logparser ishaya usilingi we-architecture ye-32-bit, futhi ukulayisha kuseva ye-SQL kuphuka ngesikhathi esingasifaneli kakhulu, ngaphandle kokugaya okuhlukile kwemigqa eminingi kusevisi.

Lapha kungena umdlali omusha esigcawini - isitaki se-ELK, esiklanyelwe ngokukhethekile ukuhlanganisa amavolumu amakhulu amalogi ngesikhathi esifanele nangokusetshenziswa kwezinsiza okubekezeleleka.

Engxenyeni yokuqala ngizokutshela kabanzi, indlela yokuxhuma i-filebeat eyingxenye yesitaki se-ELK - inesibopho sokufunda nokuthumela amafayela ombhalo alula lapho izinhlelo zokusebenza ezihlukene zibhala khona izingodo zazo. Ezihlokweni ezilandelayo sizobheka ngokucophelela izingxenye ze-Logstash ne-Kibana.

setting

Ngakho, ifayela lefayela le-ejenti yefayela le-archive ingalandwa kule sayithi.

Sizoqedela ukufaka ngokumane sikhiphe okuqukethwe kwefayela le-zip. Ngokwesibonelo, ku c:Program Filesfilebeat. Bese udinga ukusebenzisa iskripthi se-PowerShell install-service-filebeat.ps1, eza nekhithi, ukufaka isevisi ye-filebeat.

Manje sesilungele ukuqala ukusetha ifayela lokucushwa.

ukubekezelelana kwamaphutha

I-Filebeat iqinisekisa ukulethwa kwamalogi ohlelweni lokuqoqa amalogi. Lokhu kufezwa ngokugcina irejista yokufakiwe kumafayela okungena. Irejista igcina ulwazi olumayelana nalawo marekhodi afundwe kumafayela okungena, futhi imaka amarekhodi athile akwazile ukulethwa endaweni.

Uma irekhodi lingakwazi ukulethwa, ibhithi yefayela izozama ukuyithumela kabusha ize ithole isiqinisekiso sokulethwa ohlelweni lokwamukela noma ifayela lokungena langempela liyasuswa phakathi nenqubo yokuzungezisa.

Lapho isevisi iqalwa kabusha, i-filebeat izofunda ulwazi oluvela kurejista mayelana namarekhodi okugcina afundiwe futhi alethwe, futhi izofunda amarekhodi kumafayela okungena asekelwe olwazini olukurejista.

Lokhu kukuvumela ukuba unciphise ingozi yokulahlekelwa ulwazi lwelogi oludinga ukuthunyelwa kumaseva e-elasticlogstash ngesikhathi sokuhluleka okungalindelekile kanye nemisebenzi yokugcinwa kweseva.

Ungafunda okwengeziwe ngalokhu funda imibhalo ezigabeni: I-Filebeat isigcina kanjani isimo samafayela futhi I-Filebeat iqinisekisa kanjani ukulethwa okungenani kanye?

Yenza ngokwezifiso

Konke ukulungiselelwa kwenziwa kufayela lokumisa ifomethi yml, ehlukaniswe yaba izigaba eziningana. Ake sibheke ezinye zazo ezihileleke ohlelweni lokuqoqa izingodo kumaseva e-Exchange.

Ibhulokhi yokucubungula ilogu

Ibhulokhi yokucubungula ilogu iqala ngenkambu:

filebeat.inputs:

Sizosebenzisa ithuluzi elivamile lokuqoqa amalogi:

- type: log

Okulandelayo, khombisa isimo (kunikwe amandla) kanye nendlela eya kufolda enamalogi. Isibonelo, endabeni yamalogi e-IIS, izilungiselelo zingaba kanje:

    enabled: true
    paths:
	- C:inetpublogsLogFilesW3SVC1*.log
	- C:inetpublogsLogFilesW3SVC2*.log

Esinye isilungiselelo esibalulekile ukuthi i-filebeat kufanele ifunde kanjani amarekhodi anemigqa eminingi. Ngokuzenzakalelayo, i-filebeat ibheka umugqa owodwa wefayela lokungena njengokufakwayo okukodwa. Lokhu kusebenza kahle kuze kube yilapho siqala ukuthola okuhlukile kulogi yethu ehlobene nokusebenza okungalungile kwesevisi. Kulokhu, okuhlukile kungase kuhlanganise imigqa eminingana. Ngakho-ke ibhithi yefayela kufanele ibale ukufakwa kwemigqa eminingi njengokukodwa uma umugqa olandelayo uqala ngedethi. Ifomethi yokurekhoda amalogi ku-Exchange imi kanje: into ngayinye entsha efayeleni lokungena iqala ngedethi. Ekucushweni, lesi simo sibukeka kanje:

multiline:
	pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
	negate: true
	match: after

Kunengqondo ukungeza omaka kokuthunyelwe okuthumelayo, isibonelo:

  tags: ['IIS', 'ex-srv1']

Futhi ungakhohlwa ukukhipha emigqeni yokucubungula eqala ngohlamvu lwe-hashi:

  exclude_lines: ['^#']

Ngakho-ke, ibhulokhi yokufunda yelogi izobukeka kanje:

filebeat.inputs:
- type: log
  enabled: true
  paths:
	- C:inetpublogsLogFilesW3SVC1*.log
	- C:inetpublogsLogFilesW3SVC2*.log
  multiline:
	pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
	negate: true
	match: after
  tags: ['IIS', 'ex-srv1']
  exclude_lines: ['^#']

Ibhulokhi yokuthumela ilogu

I-Filebeat ithumela okufakiwe ngakunye kufayela lokungena njengento ye-json, lapho okufakiwe okuthile okuvela kulogi kuqukethwe endaweni yomlayezo owodwa. Uma sifuna ukusebenza ngalolu lwazi ngandlela thize, sidinga kuqala sihlukanise le nkambu ibe yizinkambu ezihlukene. Lokhu kungenziwa, isibonelo, ku-logstash. Uzoba umamukeli wamarekhodi asuka ku-filebeat. Nakhu okungase kubukeke kufayela lokucushwa kwe-filebeat:

output.logstash:
  hosts: ["logstash1.domain.com:5044"]

Uma kukhona amaseva amaningana, khona-ke ungakwazi ukunika amandla ukulinganisa: khona-ke i-filebeat ngeke ithumele izingodo kuseva yokuqala etholakalayo ohlwini, kodwa izosabalalisa izingodo ezithunyelwe phakathi kwamaseva amaningana:

hosts: ["logstash1.domain.com:5044", "logstash2.domain.com:5044"]
  loadbalance: true

I-Filebeat, lapho icubungula amalogi ku-json ethunyelwe, ngaphezu kokungena kwelogi equkethwe ensimini yomlayezo, ingeza inani elithile lemethadatha, elithinta usayizi wedokhumenti egcina ngokunwebeka. Le methadatha ingasuswa ngokukhetha kokuthunyelwe. Lokhu kwenziwa ku-block processor kusetshenziswa iprosesa drop_fields. Isibonelo, ungakhipha izinkambu ezilandelayo:

processors:
- drop_fields:
	fields: ["agent.ephemeral_id", "agent.hostname", "agent.id", "agent.type", "agent.version", "agent", "ecs.version", "ecs", "input.type", "input", "log.offset", "version"]

Kufanele usondele ekukhethweni kwezinkambu ezingabaliwe ngokucophelela, ngoba ezinye zazo zingasetshenziswa ohlangothini olunwebekayo ukwakha izinkomba.

Ngakho, ibhulokhi yokuthumela ilogu izobukeka kanje:

output.logstash:
  hosts: ["logstash1.domain.com:5044", "logstash2.domain.com:5044"]
  loadbalance: true
 
processors:
- drop_fields:
	fields: ["agent.ephemeral_id", "agent.hostname", "agent.id", "agent.type", "agent.version", "agent", "ecs.version", "ecs", "input.type", "input", "log.offset", "version"]

izilungiselelo zokungena kwefayela

Kunengqondo ukusetha izilungiselelo zokungena ezilandelayo:

Ulwazi lwezinga lokungena;
Sibhala izingodo kumafayela atholakala ngokuzenzakalelayo (inkomba yamalogi, kuhla lwemibhalo yokufaka ibhithi);
Igama lefayela lokungena - i-filebeat;
Gcina amafayela okungena angu-10 okugcina;
Qala ukuzungezisa lapho usayizi ufinyelela ku-1MB.

Ibhulokhi yokugcina yokumisa yokungena izobukeka kanje:

logging.level: info
logging.to_files: true
logging.files:
  name: filebeat
  keepfiles: 10
  rotateeverybytes: 1048576

Ukucushwa kokugcina

Sihlanganise ukucushwa futhi manje kubukeka kanje:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - C:inetpublogsLogFilesW3SVC1*.log
    - C:inetpublogsLogFilesW3SVC2*.log
  multiline:
    pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
    negate: true
    match: after
  tags: ['IIS', 'ex-srv1']
  exclude_lines: ['^#']
 
output.logstash:
  hosts: ["logstash1.domain.com:5044", "logstash2.domain.com:5044"]
  loadbalance: true
 
processors:
- drop_fields:
    fields: ["agent.ephemeral_id", "agent.hostname", "agent.id", "agent.type", "agent.version", "agent", "ecs.version", "ecs", "input.type", "input", "log.offset", "version"]
 
logging.level: info
logging.to_files: true
logging.files:
  name: filebeat
  keepfiles: 10
  rotateeverybytes: 1048576

Kubalulekile ukuqonda ukuthi ifomethi yefayela yokumisa ithi yml. Ngakho-ke, kubalulekile ukubeka izikhala kanye nezimpawu zokususa ngendlela efanele.

I-Filebeat ingahlola ifayela lokumisa futhi, uma i-syntax iqukethe amaphutha, izobonisa ukuthi yimuphi umugqa nokuthi kuphi kulayini i-syntax engalungile. Ukuhlolwa kwenziwa kanje: