🥇2. Xirmooyinka Elastic: falanqaynta diiwaannada amniga. Logstash

Waagii hore maqaal waanu kulanay kaydka ELK, waa maxay alaabta software ay ka kooban tahay. Hawsha ugu horraysa ee uu injineerku wajaho marka uu la shaqaynayo kaydka ELK waxa uu u soo dirayaa kaydinta elasticsearch ee falanqaynta xigta. Si kastaba ha ahaatee, tani waa adeeg dibinta kaliya, elasticsearch waxay ku kaydisaa diiwaannada qaab dukumeenti leh goobo iyo qiyam gaar ah, taas oo macnaheedu yahay injineerku waa inuu isticmaalo qalab kala duwan si uu u kala saaro fariinta laga soo diro nidaamyada dhammaadka. Tan waxaa loo samayn karaa dhowr siyaabood - adigu qor barnaamij kaas oo ku dari doona dukumentiyada kaydka xogta adoo isticmaalaya API, ama isticmaal xalal diyaarsan. Koorsadan waxaan ka fiirsan doonaa xalka Logstash, taas oo qayb ka ah xirmooyinka ELK. Waxaan eegi doonaa sida aan uga diri karno diiwaannada nidaamyada dhamaadka ilaa Logstash, ka dibna waxaan dejin doonaa faylka qaabeynta si aan u kala saarno oo aan u jiheyno xogta Elasticsearch. Si tan loo sameeyo, waxaan ka soo qaadannaa logs ka Check Point firewall sida nidaamka soo socda.

Koorasku ma daboolayo rakibidda xirmooyinka ELK, maadaama ay jiraan tiro badan oo maqaallo ah mowduucan; waxaan tixgelin doonaa qaybta qaabeynta.

Aynu dejinno qorshe hawleedka qaabeynta Logstash:

Hubinta in elasticsearch ay aqbali doonto diiwaannada ( hubinta shaqeynta iyo furnaanta dekedda).
Waxaan tixgelineynaa sida aan ugu diri karno dhacdooyinka Logstash, dooro habka, oo u hirgelinno.
Waxa aanu ku habaynaynaa gelinta faylka qaabaynta Logstash.
Waxaan ku habeyneynaa wax soo saarka faylka qaabeynta Logstash ee qaabka debug si aan u fahanno sida fariinta logu u eg tahay.
Dejinta shaandhaynta
Dejinta wax soo saarka saxda ah ee ElasticSearch.
Logstash ayaa bilaabay
Hubinta logyada Kibana.

Aynu si faahfaahsan u eegno qodob kasta:

Hubinta in elasticsearch ay aqbali doonto diiwaannada

Si tan loo sameeyo, waxaad isticmaali kartaa amarka curlka si aad u hubiso gelitaanka Elasticsearch ee nidaamka Logstash lagu rakibay. Haddii aad leedahay sugida habaysan, markaa waxaan sidoo kale ku wareejinaa isticmaalaha / erayga sirta ah iyada oo loo marayo curl, oo tilmaamaya dekedda 9200 haddii aadan bedelin. Haddii aad hesho jawaab la mid ah tan hoose, markaa wax walba waa hagaagsan yihiin.

[elastic@elasticsearch ~]$ curl -u <<user_name>> : <<password>> -sS -XGET "<<ip_address_elasticsearch>>:9200"
{
  "name" : "elastic-1",
  "cluster_name" : "project",
  "cluster_uuid" : "sQzjTTuCR8q4ZO6DrEis0A",
  "version" : {
    "number" : "7.4.1",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "fc0eeb6e2c25915d63d871d344e3d0b45ea0ea1e",
    "build_date" : "2019-10-22T17:16:35.176724Z",
    "build_snapshot" : false,
    "lucene_version" : "8.2.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
[elastic@elasticsearch ~]$

Haddii jawaabta aan la helin, markaa waxaa jiri kara dhowr nooc oo khaladaad ah: habka elasticsearch ma soconayo, dekedda khaldan ayaa la cayimay, ama dekedda waxaa xannibay dab-damis ku yaal server-ka halkaasoo elasticsearch lagu rakibay.

Aynu eegno sida aad Logstash uga soo diri karto logstash meesha jeegga dabka

Laga soo bilaabo server-ka maamulka Check Point waxaad u diri kartaa Logstash adoo isticmaalaya syslog adoo isticmaalaya log_exporter utility, waxaad wax badan oo ku saabsan ka akhriyi kartaa halkan maqaal, halkan waxaan ku dhaafi doonaa kaliya amarka abuura qulqulka:

cp_log_export ku dar magaca check_point_syslog target-server > target-dekedda 5555 borotokool tcp qaabka akhriska guud semi-midaysan

> - cinwaanka server-ka uu ku shaqeeyo Logstash, target-port 5555 - dekedda aan u diri doono logs, diritaanka logs via tcp waxay ku shuban kartaa server-ka, markaa xaaladaha qaarkood way ka saxsan tahay isticmaalka udp .

Dejinta INPUT ee faylka qaabeynta Logstash

Sida caadiga ah, faylka qaabeynta wuxuu ku yaalaa /etc/logstash/conf.d/ directory. Faylka qaabeynta wuxuu ka kooban yahay 3 qaybood oo macne leh: INPUT, FILTER, OUTPUT. IN aqbasho waxaanu tilmaamaynaa meesha uu nidaamku ka qaadan doono log, gudaha Kalasooc parse log - samee sida fariinta loogu qaybiyo goobo iyo qiyam, gudaha wax soo saarka waxa aanu habaynaynaa qulqulka wax soo saarka -halkaas oo la soo diri doono qoraalada la soo saaray.

Marka hore, aan habeyno INPUT, tixgeli qaar ka mid ah noocyada noqon kara - file, tcp iyo exe.

Tcp:

input {
tcp {
    port => 5555
    host => “10.10.1.205”
    type => "checkpoint"
    mode => "server"
}
}

qaabka => "server"
Waxay muujineysaa in Logstash ay aqbashay isku xirka.

dekedda => 5555
martigeliyaha => "10.10.1.205"
Waxaan ku aqbalnaa isku xirka cinwaanka IP 10.10.1.205 (Logstash), dekedda 5555 - dekeddu waa inay ogolaato siyaasadda dab-damiska.

nooca => "koontarool"
Waxaan calaamadeynaa dukumeentiga, aad u habboon haddii aad leedahay xiriiro dhowr ah oo soo galaya. Ka dib, xiriir kasta waxaad qori kartaa shaandhadaada adiga oo isticmaalaya macquul haddii la dhisayo.

Faylka:

input {
  file {
    path => "/var/log/openvas_report/*"
    type => "openvas"
    start_position => "beginning"
    }
}

Sharaxaada dejinta:
wadada => "/var/log/openvas_report/*"
Waxaan ku tusinaa hagaha loo baahan yahay in lagu akhriyo faylalka.

nooca => "furan"
Nooca dhacdada.

start_position => "bilaw"
Markaad beddesho faylka, waxay akhridaa faylka oo dhan; haddii aad dejiso "dhamaadka", nidaamku wuxuu sugayaa in diiwaanno cusub ay ka soo baxaan dhammaadka faylka.

Tusaale:

input {
  exec {
    command => "ls -alh"
    interval => 30
  }
}

Iyada oo la adeegsanayo gelintan, amar (kaliya!) ayaa bilaabmaya oo wax soo saarkiisa waxa loo rogaa fariin qoraal ah.

amar => "ls -alh"
Taliska aan danaynayno wax soo saarkiisa.

dhexda => 30
Muddada baaqa amarka ee ilbidhiqsiyo gudahood.

Si aan logu uga helno dab-damiska, waxaanu diiwaangelinaynaa shaandhada tcp ama pud, iyadoo ku xiran sida logstash loogu diro Logstash.

Waxaan ku habeyneynaa wax soo saarka faylka qaabeynta Logstash ee qaabka debug si aan u fahanno fariinta loggu waxay u egtahay

Ka dib markii aan habeynay INPUT, waxaan u baahanahay inaan fahanno fariinta logu waxay u ekaan doontaa iyo hababka loo baahan yahay in la isticmaalo si loo habeeyo filtarka log (parser).

Si tan loo sameeyo, waxaan isticmaali doonaa shaandheyn soo saarta natiijada si loo stdout si loo eego fariinta asalka ah; faylka qaabeynta oo dhameystiran wuxuu u ekaan doonaa sidan:

input 
{
         tcp 
         {
                port => 5555
  	  	type => "checkpoint"
    		mode => "server"
                host => “10.10.1.205”
   	 }
}

output 
{
	if [type] == "checkpoint" 
       {
		stdout { codec=> json }
	}
}

Orod amarka si aad u hubiso:
sudo /usr/share/logstash/bin//logstash -f /etc/logstash/conf.d/checkpoint.conf
Waxaan aragnaa natiijada, sawirka waa la gujin karaa:

Haddii aad nuqul ka sameysid waxay u ekaan doontaa sidan:

action="Accept" conn_direction="Internal" contextnum="1" ifdir="outbound" ifname="bond1.101" logid="0" loguid="{0x5dfb8c13,0x5,0xfe0a0a0a,0xc0000000}" origin="10.10.10.254" originsicname="CN=ts-spb-cpgw-01,O=cp-spb-mgmt-01.tssolution.local.kncafb" sequencenum="8" time="1576766483" version="5" context_num="1" dst="10.10.10.10" dst_machine_name="[email protected]" layer_name="TSS-Standard Security" layer_name="TSS-Standard Application" layer_uuid="dae7f01c-4c98-4c3a-a643-bfbb8fcf40f0" layer_uuid="dbee3718-cf2f-4de0-8681-529cb75be9a6" match_id="8" match_id="33554431" parent_rule="0" parent_rule="0" rule_action="Accept" rule_action="Accept" rule_name="Implicit Cleanup" rule_uid="6dc2396f-9644-4546-8f32-95d98a3344e6" product="VPN-1 & FireWall-1" proto="17" s_port="37317" service="53" service_id="domain-udp" src="10.10.1.180" ","type":"qqqqq","host":"10.10.10.250","@version":"1","port":50620}{"@timestamp":"2019-12-19T14:50:12.153Z","message":"time="1576766483" action="Accept" conn_direction="Internal" contextnum="1" ifdir="outbound" ifname="bond1.101" logid="0" loguid="{0x5dfb8c13,

Markaan eegno fariimahan, waxaan fahamsanahay in logu u eg yahay: field = qiimaha ama furaha = qiimaha, taas oo macnaheedu yahay filtar la yiraahdo kv ayaa ku habboon. Si aad u dooratid shaandhada saxda ah kiis kasta oo gaar ah, waxa ay ahaan lahayd fikrad wanaagsan in aad is barato iyaga oo ku jira dukumentiyada farsamada, ama weydii saaxiib.

Dejinta shaandhaynta

Marxaladda ugu dambeysa waxaan doorannay kv, qaabeynta filtarkan ayaa lagu soo bandhigay hoos:

filter {
if [type] == "checkpoint"{
	kv {
		value_split => "="
		allow_duplicate_values => false
	}
}
}

Waxaan dooraneynaa calaamadda aan u qaybin doono goobta iyo qiimaha - "=". Haddii aan galno isku mid ah ku hayno galka, waxaan ku keydineynaa hal tusaale oo kaliya database-ka, haddii kale waxaad ku dhamaan doontaa qiimo isku mid ah, taas oo ah, haddii aan haysano fariinta "foo = some foo= some" waxaan qoreynaa foo kaliya = qaar.

Dejinta wax soo saarka saxda ah ee ElasticSearch

Marka Filter la habeeyo, waxaad geli kartaa diiwaannada kaydka xogta laascaanood:

output 
{
if [type] == "checkpoint"
{
 	elasticsearch 
        {
		hosts => ["10.10.1.200:9200"]
		index => "checkpoint-%{+YYYY.MM.dd}"
    		user => "tssolution"
    		password => "cool"
  	}
}
}

Haddii dukumeentiga lagu saxiixo nooca isbaarada, waxaan ku badbaadineynaa dhacdada keydka xogta elasticsearch, kaas oo aqbala isku xirka 10.10.1.200 ee dekada 9200 si caadi ah. Dukumeenti kasta waxa lagu kaydiyaa tusmooyin gaar ah, kiiskan waxa aanu ku kaydinaynaa tusmada “koontarool-” + taariikhda wakhtiga hadda. Tusi kastaa wuxuu yeelan karaa goobo gaar ah, ama si toos ah ayaa loo abuuraa marka goob cusubi ka soo muuqato fariinta; goobaha goobta iyo noocooda waxaa lagu arki karaa khariidado.

Haddii aad haysato sugida habaysan (waynu eegi doonaa mardambe), aqoonsiga qoraalka tusmada gaarka ah waa in la cayimaa, tusaale ahaan waa “tssolution” oo wata erayga sirta ah “qabow”. Waxaad kala saari kartaa xuquuqaha isticmaalaha si aad u qorto diiwaanka tusmada gaarka ah oo keliya oo aan ka badnayn.

Fur Logstash

Logstash qaabeynta faylka:

input 
{
         tcp 
         {
                port => 5555
  	  	type => "checkpoint"
    		mode => "server"
                host => “10.10.1.205”
   	 }
}

filter {
        if [type] == "checkpoint"{
	kv {
		value_split => "="
		allow_duplicate_values => false
	}
        }
}

output 
{
if [type] == "checkpoint"
{
 	elasticsearch 
        {
		hosts => ["10.10.1.200:9200"]
		index => "checkpoint-%{+YYYY.MM.dd}"
    		user => "tssolution"
    		password => "cool"
  	}
}
}

Waxaan ka hubinaynaa faylka qaabeynta inuu sax yahay:
/usr/share/logstash/bin//logstash -f checkpoint.conf

Bilow habka Logstash:
sudo systemctl bilaw logstash

Waxaan hubineynaa in hawshu bilaabatay:
sudo systemctl status logstash

Aynu eegno haddii godku kor u kacay:
netstat -nat |grep 5555

Hubinta logyada Kibana.

Ka dib markii wax walba ay socdaan, u tag Kibana - Discover, hubi in wax walba si sax ah loo habeeyey, sawirku waa la riixi karaa!

Dhammaan qoraallada ayaa ku yaal meel waxaanan arki karnaa dhammaan beeraha iyo qiyamkooda!

gunaanad

Waxaan eegnay sida loo qoro faylka qaabeynta Logstash, natiijaduna waxay tahay waxaan helnay baarer dhammaan goobaha iyo qiyamka. Hadda waxaan ka shaqeyn karnaa raadinta iyo qorsheynta meelo gaar ah. Marka xigta koorsada waxaan eegi doonaa muuqaalaynta Kibana waxaanan samayn doonaa dashboard fudud. Waxaa habboon in la sheego in faylka qaabeynta Logstash uu u baahan yahay in si joogto ah loo cusbooneysiiyo xaaladaha qaarkood, tusaale ahaan, marka aan rabno inaan bedelno qiimaha beerta laga bilaabo lambar ilaa kelmad. Maqaalada soo socda waxaan samayn doonaa tan si joogto ah.

Hadaba la socotelegraam, Facebook, VK, TS Solution Blog), Yandex Zen.

Source: www.habr.com

2. Xirmooyinka Elastic: falanqaynta diiwaannada amniga. Logstash