1.Isitaki se-Elastic: ukuhlaziywa kwamalogi okuvikela. Isingeniso

Ngokuphathelene nokuphela kokuthengisa eRussia yohlelo lokugawula i-Splunk kanye ne-analytics, kwaphakama umbuzo: yini engashintsha lesi sixazululo? Ngemuva kokuchitha isikhathi ngizijwayelanisa nezixazululo ezahlukahlukene, ngaxazulula isixazululo sendoda yangempela - "Isitaki se-ELK". Lolu hlelo luthatha isikhathi ukusetha, kodwa ngenxa yalokho ungathola isistimu enamandla kakhulu yokuhlaziya isimo futhi uphendule ngokushesha izehlakalo zokuphepha kolwazi enhlanganweni. Kulolu chungechunge lwezihloko, sizobheka amakhono ayisisekelo (noma mhlawumbe cha) we-ELK stack, sicabangele ukuthi ungawahlukanisa kanjani izingodo, indlela yokwakha amagrafu namadeshibhodi, nokuthi yimiphi imisebenzi ethakazelisayo engenziwa usebenzisa isibonelo samalogi avela. i-Firewall ye-Check Point noma isithwebuli sokuvikela se-OpenVas. Okokuqala, ake sibheke ukuthi siyini - isitaki se-ELK, nokuthi siqukethe ziphi izingxenye.

"Isitaki se-ELK" isifinyezo samaphrojekthi amathathu omthombo ovulekile: Islastiki, Logstash и Kibana. Kuthuthukiswe i-Elastic kanye nawo wonke amaphrojekthi ahlobene. I-Elasticsearch iwumgogodla walo lonke uhlelo, oluhlanganisa imisebenzi yesizindalwazi, ukusesha kanye nohlelo lokuhlaziya. I-Logstash iyiphayiphi yokucubungula idatha eseceleni kweseva ethola idatha emithonjeni eminingi ngesikhathi esisodwa, idlulise ilogu, bese iyithumela kusizindalwazi se-Elasticsearch. I-Kibana ivumela abasebenzisi ukuthi babone idatha ngeso lengqondo besebenzisa amashadi namagrafu ku-Elasticsearch. Ungakwazi futhi ukuphatha i-database ngokusebenzisa i-Kibana. Okulandelayo, sizocubungula uhlelo ngalunye ngokwehlukana ngokuningiliziwe.

Logstash

I-Logstash iyinsiza yokucubungula imicimbi yelogi evela emithonjeni ehlukahlukene, ongakhetha ngayo izinkambu namanani azo kumlayezo, futhi ungalungisa ukuhlunga nokuhlela idatha. Ngemuva kwakho konke ukukhohlisa, i-Logstash iqondisa kabusha imicimbi esitolo sedatha yokugcina. Uhlelo lokusebenza lulungiswa kuphela ngamafayela wokucushwa.
Ukucushwa kwe-logstash okuvamile kuyifayela(ama) eliqukethe ukusakazwa kolwazi (okokufaka) okumbalwa, izihlungi ezimbalwa zalolu lwazi (isihlungi) kanye nokusakaza okuningana okuphumayo (okukhiphayo). Kubukeka njengefayela elilodwa noma amaningi okumisa, okuthi enguqulweni elula (engenzi lutho nhlobo) abukeke kanje:

input {
}

filter {
}

output {
}

Ku-INPUT silungiselela ukuthi iyiphi imbobo amalogi azothunyelwa kuyo nokuthi iyiphi iphrothokholi, noma kuyiphi ifolda ukuze kufundwe amafayela amasha noma abuyekezwa njalo. Ku-ISIHLUNGI silungiselela umhlahleli welogi: izinkambu zokuhlaziya, amanani okuhlela, ukwengeza amapharamitha amasha noma ukuwasusa. ISIHLUNGI yinkambu yokuphatha umlayezo oza ku-Logstash ngezinketho eziningi zokuhlela. Kokukhiphayo silungisa lapho sithumela khona ilogu esele ehlukanisiwe, uma i-elasticsearch isicelo se-JSON sithunyelwa lapho izinkambu ezinamavelu zithunyelwa khona, noma njengengxenye yokulungisa iphutha zingakhishelwa ku-stdout noma zibhalwe efayeleni.

I-ElasticSearch

Ekuqaleni, i-Elasticsearch iyisixazululo sokusesha umbhalo ogcwele, kodwa ngezinsiza ezengeziwe ezifana nokukala okulula, ukuphindaphinda nezinye izinto, okwenze umkhiqizo waba lula kakhulu futhi waba yisixazululo esihle samaphrojekthi anomthwalo omningi onemininingwane eminingi. I-Elasticsearch iyisitolo sedokhumenti esingahlobene (NoSQL) JSON kanye nenjini yokusesha esekelwe ekusesheni kombhalo ogcwele we-Lucene. Inkundla yehadiwe i-Java Virtual Machine, ngakho uhlelo ludinga inani elikhulu leprosesa nezinsiza ze-RAM ukuze zisebenze.
Umlayezo ngamunye ongenayo, kungaba nge-Logstash noma usebenzisa i-API yombuzo, ukhonjwa “njengombhalo” - ofana netafula ku-SQL ehlobene. Yonke imibhalo igcinwa kunkomba - i-analogue yesizindalwazi ku-SQL.

Isibonelo sedokhumenti kusizindalwazi:

{
  "_index": "checkpoint-2019.10.10",
  "_type": "_doc",
  "_id": "yvNZcWwBygXz5W1aycBy",
  "_version": 1,
  "_score": null,
  "_source": {
	"layer_uuid": [
      "dae7f01c-4c98-4c3a-a643-bfbb8fcf40f0",
      "dbee3718-cf2f-4de0-8681-529cb75be9a6"
	],
	"outzone": "External",
	"layer_name": [
  	"TSS-Standard Security",
  	"TSS-Standard Application"
	],
	"time": "1565269565",
	"dst": "103.5.198.210",
	"parent_rule": "0",
	"host": "10.10.10.250",
	"ifname": "eth6",
    ]
}

Wonke umsebenzi ngesizindalwazi usekelwe ezicelweni ze-JSON zisebenzisa i-REST API, ekhiqiza amadokhumenti ngenkomba noma ngezibalo ezithile ngefomethi: umbuzo - impendulo. Ukuze ubone ngeso lengqondo zonke izimpendulo zezicelo, i-Kibana yabhalwa, okuyisevisi yewebhu.

Kibana

I-Kibana ikuvumela ukuthi useshe, ubuyise idatha kanye nezibalo zemibuzo kusuka kusizindalwazi se-elasticsearch, kodwa amagrafu amaningi amahle namadeshibhodi akhiwe ngokusekelwe ezimpendulo. Uhlelo luphinde lube nokusebenza kokulawulwa kwesizindalwazi se-elasticsearch; ezihlokweni ezilandelayo sizobheka le sevisi ngokuningiliziwe. Manje ake sibonise isibonelo samadeshibhodi we-Firewall ye-Check Point kanye nesikena sobungozi se-OpenVas esingakhiwa.

Isibonelo sedeshibhodi ye-Check Point, isithombe siyachofozeka:

Isibonelo sedeshibhodi ye-OpenVas, isithombe siyachofozeka:

isiphetho

Sibheke ukuthi ihlanganisani Idatha ye-ELK, sajwayelana kancane nemikhiqizo eyinhloko, kamuva esifundweni sizocabangela ngokuhlukile ukubhala ifayela lokumisa le-Logstash, ukusetha amadeshibhodi ku-Kibana, ukujwayelana nezicelo ze-API, ukuzenzekelayo nokunye okuningi!

Ngakho hlala ubukele (yocingo, Facebook, VK, I-TS Solution Blog), I-Yandex.Zen.

Source: www.habr.com