1.Elastic stack: kusanthula zipika zachitetezo. Mawu Oyamba

Pokhudzana ndi kutha kwa malonda ku Russia a Splunk logging and analytics system, funso lidabuka: chomwe chingalowe m'malo mwa yankho ili? Nditatha nthawi ndikuzidziwa bwino ndi mayankho osiyanasiyana, ndidakhazikika pa yankho la mwamuna weniweni - "ELK". Dongosololi limatenga nthawi kuti likhazikitsidwe, koma chifukwa chake mutha kupeza dongosolo lamphamvu kwambiri losanthula momwe zinthu zilili ndikuyankha mwachangu zochitika zachitetezo chazidziwitso mu bungwe. M'nkhani zotsatizanazi, tiwona mphamvu zoyambira (kapena ayi) za stack ya ELK, taganizirani momwe mungasinthire zipika, momwe mungamangire ma graph ndi ma dashboards, ndi ntchito zotani zomwe zingagwiritsidwe ntchito pogwiritsa ntchito chitsanzo cha zipika kuchokera. ndi Check Point firewall kapena OpenVas chitetezo scanner. Poyamba, tiyeni tiwone chomwe chiri - stack ya ELK, ndi zigawo ziti zomwe zimapangidwira.

"ELK" ndi chidule cha mapulojekiti atatu otseguka: Elasticsearch, Logstash и Kibana. Yopangidwa ndi Elastic pamodzi ndi ma projekiti onse okhudzana. Elasticsearch ndiye maziko a dongosolo lonse, lomwe limaphatikiza ntchito za database, kufufuza ndi kusanthula dongosolo. Logstash ndi mapaipi opangira data pa seva omwe amalandira deta kuchokera kumagwero angapo nthawi imodzi, amachotsa chipikacho, kenako ndikuchitumiza ku database ya Elasticsearch. Kibana amalola ogwiritsa ntchito kuwona deta pogwiritsa ntchito ma chart ndi ma graph mu Elasticsearch. Mutha kuyang'aniranso database kudzera ku Kibana. Kenako, tikambirana dongosolo lililonse padera mwatsatanetsatane.

Logstash

Logstash ndi chida chothandizira kukonza zochitika za chipika kuchokera kumagwero osiyanasiyana, momwe mungasankhire minda ndi mfundo zake mu uthenga, komanso mutha kukonza kusefa ndikusintha kwa data. Pambuyo pakusintha konse, Logstash imawongolera zochitika kumalo osungira omaliza. Zomwe zimagwiritsidwa ntchito zimapangidwira kokha kupyolera mu mafayilo osinthika.
Kusintha kwanthawi zonse kwa logstash ndi fayilo (ma) yomwe ili ndi mitsinje ingapo yolowera (zolowetsa), zosefera zingapo za chidziwitsochi (sefa) ndi mitsinje ingapo yotuluka (zotulutsa). Zikuwoneka ngati fayilo imodzi kapena zingapo zosinthira, zomwe mu mtundu wosavuta (womwe sumachita kalikonse) umawoneka motere:

input {
}

filter {
}

output {
}

Mu INPUT timakonza malo omwe zipika zidzatumizidwa ndi kudzera mu protocol, kapena kuchokera ku foda yomwe mungawerenge mafayilo atsopano kapena osinthidwa nthawi zonse. Mu FILTER timakonza zolemba zolembera: magawo odulira, kusintha mikhalidwe, kuwonjezera magawo atsopano kapena kuwachotsa. FILTER ndi gawo lowongolera uthenga womwe umabwera ku Logstash ndi zosankha zambiri zosintha. Pazotulutsa timakonza pomwe timatumiza chipika chomwe chagawika kale, ngati ndi elasticsearch pempho la JSON limatumizidwa momwe magawo omwe ali ndi zikhalidwe amatumizidwa, kapena ngati gawo la debug amatha kutulutsa kuti stdout kapena kulembedwa ku fayilo.

Elasticsearch

Poyambirira, Elasticsearch ndi njira yothetsera kusaka zolemba zonse, koma ndi zina zowonjezera monga makulitsidwe osavuta, kubwerezabwereza ndi zinthu zina, zomwe zidapangitsa kuti malondawo akhale osavuta komanso yankho labwino pama projekiti olemetsa kwambiri okhala ndi ma data ambiri. Elasticsearch ndi malo osungira zolemba osakhudzana (NoSQL) JSON ndi injini yosakira kutengera kusaka kwa Lucene. Pulatifomu ya Hardware ndi Java Virtual Machine, kotero dongosololi limafunikira kuchuluka kwa purosesa ndi RAM zothandizira.
Uthenga uliwonse womwe ukubwera, kaya ndi Logstash kapena kugwiritsa ntchito API ya mafunso, amalembedwa ngati "document" - yofanana ndi tebulo mu SQL yogwirizana. Zolemba zonse zimasungidwa mu index - analogue ya database mu SQL.

Chitsanzo cha chikalata mu nkhokwe:

{
  "_index": "checkpoint-2019.10.10",
  "_type": "_doc",
  "_id": "yvNZcWwBygXz5W1aycBy",
  "_version": 1,
  "_score": null,
  "_source": {
	"layer_uuid": [
      "dae7f01c-4c98-4c3a-a643-bfbb8fcf40f0",
      "dbee3718-cf2f-4de0-8681-529cb75be9a6"
	],
	"outzone": "External",
	"layer_name": [
  	"TSS-Standard Security",
  	"TSS-Standard Application"
	],
	"time": "1565269565",
	"dst": "103.5.198.210",
	"parent_rule": "0",
	"host": "10.10.10.250",
	"ifname": "eth6",
    ]
}

Ntchito zonse ndi nkhokwe zimatengera zopempha za JSON pogwiritsa ntchito REST API, yomwe imatulutsa zolemba ndi index kapena ziwerengero zina mumpangidwe: funso - yankho. Kuti muwonetsetse mayankho onse pazopempha, Kibana adalembedwa, yomwe ndi ntchito yapaintaneti.

Kibana

Kibana amakulolani kuti mufufuze, mutengenso ziwerengero zamafunso kuchokera ku nkhokwe ya elasticsearch, koma ma graph ambiri okongola ndi ma dashboards amamangidwa potengera mayankho. Dongosololi lilinso ndi magwiridwe antchito a database ya elasticsearch; m'nkhani zotsatila tiwona ntchitoyi mwatsatanetsatane. Tsopano tiyeni tiwonetse chitsanzo cha ma dashboards a Check Point firewall ndi OpenVas vulnerability scanner yomwe ingamangidwe.

Chitsanzo cha dashboard ya Check Point, chithunzicho ndi chosavuta:

Chitsanzo cha dashboard ya OpenVas, chithunzicho ndi chosavuta:

Pomaliza

Tinayang'ana zomwe zikuphatikiza Mtengo wa ELK, tidadziwa pang'ono zinthu zazikuluzikulu, pambuyo pake m'maphunzirowa tidzakambirana padera kulemba fayilo yosinthira ya Logstash, kukhazikitsa ma dashboards pa Kibana, kudziwa zopempha za API, zodziwikiratu ndi zina zambiri!

Ndiye khalani maso (uthengawo, Facebook, VK, TS Solution Blog), Yandex.Zen.

Source: www.habr.com