I-1.I-Elastic stack: uhlalutyo lweelogi zokhuseleko. Intshayelelo

Ngokumalunga nokuphela kweentengiso eRashiya yeSplunk yokugawulwa kwemithi kunye nenkqubo yokuhlalutya, kwavela umbuzo: yintoni enokutshintsha esi sisombululo? Emva kokuchitha ixesha ndiziqhelanisa nezisombululo ezahlukeneyo, ndazinza kwisisombululo sendoda yokwenyani- "Isitaki se-ELK". Le nkqubo ithatha ixesha lokumisa, kodwa ngenxa yoko unokufumana inkqubo enamandla kakhulu yokuhlalutya isimo kunye nokuphendula ngokukhawuleza kwiziganeko zokhuseleko lolwazi kwintlangano. Kolu luhlu lwamanqaku, siya kujonga kwisiseko (okanye mhlawumbi akunjalo) ubuchule be-ELK stack, qwalasela indlela onokuthi ucazulule ngayo izigodo, indlela yokwakha iigrafu kunye needeshibhodi, kwaye yeyiphi imisebenzi enomdla enokwenziwa usebenzisa umzekelo welogi ukusuka. i-Check Point firewall okanye iskena sokhuseleko se-OpenVas. Ukuqala, makhe sijonge ukuba yintoni na - isitakhi se-ELK, kwaye iqulathe ntoni na.

"Isitaki se-ELK" sisishunqulelo seeprojekthi ezintathu zomthombo ovulekileyo: Elasticsearch, I-Logstash и IKibana. Iphuhliswe yi-Elastic kunye nazo zonke iiprojekthi ezinxulumeneyo. I-Elasticsearch ngundoqo wenkqubo yonke, edibanisa imisebenzi yesiseko sedatha, ukukhangela kunye nenkqubo yohlalutyo. I-Logstash ngumbhobho wokusetyenzwa kwedatha kwicala leseva ofumana idatha kwimithombo emininzi ngaxeshanye, ucazulule ilogi, emva koko uyithumele kwisiseko sedatha se-Elasticsearch. I-Kibana ivumela abasebenzisi ukuba babonise idatha ngokusebenzisa iitshathi kunye neegrafu kwi-Elasticsearch. Unako kwakhona ukulawula isiseko sedatha ngokusebenzisa i-Kibana. Okulandelayo, siya kuqwalasela inkqubo nganye ngokwahlukileyo ngokubanzi.

I-Logstash

I-Logstash sisixhobo sokucwangcisa imicimbi yelog kwimithombo eyahlukeneyo, onokuthi ukhethe ngayo amasimi kunye namaxabiso awo kumyalezo, kwaye unokuqwalasela ukucocwa kwedatha kunye nokuhlelwa. Emva kwayo yonke i-manipulations, i-Logstash iphinda iqondise iziganeko kwindawo yokugcina idatha. Into eluncedo iqwalaselwe kuphela ngeefayile zoqwalaselo.
Uqwalaselo lwelogstash eqhelekileyo yifayile(ii) equlathe imijelo emininzi engenayo yolwazi (igalelo), izihluzo ezininzi zolu lwazi (icebo) kunye nemisinga ephumayo (imveliso). Ijongeka njengefayile enye okanye ezininzi zoqwalaselo, ezikwinguqulelo elula (engenzi nto konke konke) ijongeka ngolu hlobo:

input {
}

filter {
}

output {
}

Kwi-INPUT siqwalasela ukuba leliphi izibuko iilog eziya kuthunyelwa kuyo kwaye ngeyiphi iprothokholi, okanye kwesiphi isiqulathi seefayili zokufunda ezintsha okanye ezihlaziywa rhoqo iifayile. Kwi-FILTER siqwalasela isihluli selog: imihlaba yokwahlula, ukuhlela amaxabiso, ukongeza iiparameters ezintsha okanye ukuzicima. ISIHLUNGI yintsimi yokulawula umyalezo oza kwi-Logstash enokhetho oluninzi lokuhlela. Kwimveliso siqwalasela apho sithumela khona ilog esele yahluliwe, ukuba i-elasticsearch isicelo se-JSON sithunyelwa apho iindawo ezinamaxabiso athunyelwa khona, okanye njengenxalenye ye-debug inokukhutshwa kwi-stdout okanye ibhalwe kwifayile.

ElasticSearch

Ekuqaleni, i-Elasticsearch sisisombululo sokukhangela okubhaliweyo okugcweleyo, kodwa ngezixhobo ezongezelelweyo ezinjengokulinganisa okulula, ukuphindaphinda kunye nezinye izinto, ezenza imveliso ibe lula kakhulu kwaye isisombululo esihle kwiiprojekthi ezinomthwalo ophezulu kunye nomthamo omkhulu wedatha. I-Elasticsearch yi-non-relationship (NoSQL) ivenkile yoxwebhu lwe-JSON kunye ne-injini yokukhangela esekelwe kuphendlo olupheleleyo lwe-Lucene. Iqonga le-hardware nguMatshini weJava Virtual, ngoko inkqubo idinga inani elikhulu leprosesa kunye nezixhobo ze-RAM ukuze zisebenze.
Umyalezo ngamnye ongenayo, nokuba nge-Logstash okanye usebenzisa i-API yombuzo, ufakwe kwi-indexed as "uxwebhu" - olufana netheyibhile ku-SQL yobudlelwane. Onke amaxwebhu agcinwe kwisalathiso - i-analogue yesiseko sedatha kwiSQL.

Umzekelo woxwebhu olukwisiseko sedatha:

{
  "_index": "checkpoint-2019.10.10",
  "_type": "_doc",
  "_id": "yvNZcWwBygXz5W1aycBy",
  "_version": 1,
  "_score": null,
  "_source": {
	"layer_uuid": [
      "dae7f01c-4c98-4c3a-a643-bfbb8fcf40f0",
      "dbee3718-cf2f-4de0-8681-529cb75be9a6"
	],
	"outzone": "External",
	"layer_name": [
  	"TSS-Standard Security",
  	"TSS-Standard Application"
	],
	"time": "1565269565",
	"dst": "103.5.198.210",
	"parent_rule": "0",
	"host": "10.10.10.250",
	"ifname": "eth6",
    ]
}

Wonke umsebenzi kunye ne-database isekelwe kwizicelo ze-JSON usebenzisa i-REST API, enokuthi ivelise amaxwebhu ngesalathisi okanye ezinye izibalo kwifomathi: umbuzo - impendulo. Ukuze ubone ngeso lengqondo zonke iimpendulo kwizicelo, i-Kibana yabhalwa, eyinkonzo yewebhu.

IKibana

I-Kibana ikuvumela ukuba ukhangele, ufumane idatha kunye neenkcukacha zemibuzo kwi-database ye-elasticsearch, kodwa ezininzi iigrafu ezintle kunye needeshibhodi zakhiwe ngokusekelwe kwiimpendulo. Inkqubo ikwanolawulo lwedatha ye-elasticsearch; kumanqaku alandelayo siza kujonga le nkonzo ngokweenkcukacha ngakumbi. Ngoku makhe sibonise umzekelo wedeshibhodi ye-Check Point firewall kunye ne-OpenVas vulnerability scanner enokwakhiwa.

Umzekelo wedeshibhodi yeCheck Point, umfanekiso uyacofa:

Umzekelo wedeshibhodi ye-OpenVas, umfanekiso uyacofa:

isiphelo

Sijonge ukuba iqulathe ntoni na ELK isitakhi, siye saqhelana kancinci neemveliso eziphambili, kamva kwikhosi siza kuqwalasela ngokwahlukileyo ukubhala ifayile ye-Logstash yoqwalaselo, ukuseta iideshibhodi kwi-Kibana, ukuqhelana nezicelo ze-API, ukuzenzekelayo kunye nokunye okuninzi!

Ngoko hlala ubukele (yocingo, Facebook, VK, TS Solution Blog), Yandex.Zen.

umthombo: www.habr.com