Ukuqulunqa idatha engacwangciswanga nge-GROK
Ukuba usebenzisa isitaki se-Elastic (ELK) kwaye unomdla wokwenza imephu yelog yeLogstash yesiko kwi-Elasticsearch, ke le posi yeyakho.
Isitaki se-ELK sisishunqulelo seeprojekthi ezintathu zomthombo ovulekileyo: i-Elasticsearch, i-Logstash kunye ne-Kibana. Ngokudibeneyo benza iqonga lolawulo lwelog.
- Elasticsearch lukhangelo kunye nenkqubo yohlalutyo.
- I-Logstash ngumbhobho wokusetyenzwa kwedatha kwicala leseva ongenisa idatha kwimithombo emininzi ngaxeshanye, iyiguqule, kwaye emva koko iyithumele βkugcinoβ olufana ne-Elasticsearch.
- IKibana ivumela abasebenzisi ukuba bajonge idatha besebenzisa iitshathi kunye neegrafu kwi-Elasticsearch.
Beats yeza emva kwexesha kwaye ingumthumeli wedatha ekhaphukhaphu. Ukuqaliswa kweeBeats kuguqule i-Elk Stack kwi-Elastic Stack, kodwa ayisiyiyo loo nto.
Eli nqaku limalunga neGrok, eyona nto ibonakalayo kwiLogstash enokuguqula iilogi zakho ngaphambi kokuba zithunyelwe kwi-stash. Ngeenjongo zethu, ndiza kuthetha kuphela malunga nokucubungula idatha ukusuka kwi-Logstash ukuya kwi-Elasticsearch.
I-Grok sisihluzo esingaphakathi kweLogstash esisetyenziselwa ukwahlula idatha engacwangciswanga ibe yinto eyakhiweyo nenokubuzwa. Ihlala phezu kwentetho eqhelekileyo (regex) kwaye isebenzisa iipatheni zokubhaliweyo ukutshatisa imitya kwiifayile zelog.
Njengoko siza kubona kumacandelo alandelayo, ukusebenzisa i-Grok kwenza umahluko omkhulu xa kuziwa kulawulo lwelogi olusebenzayo.
Ngaphandle kweGrok idatha yakho yelog ayilungiswanga
Ngaphandle kweGrok, xa iilogi zithunyelwa ukusuka kwi-Logstash ukuya kwi-Elasticsearch kwaye zinikezelwe e-Kibana, zivela kuphela kwixabiso lomyalezo.
Ukubuza ulwazi olunentsingiselo kule meko kunzima kuba yonke idatha yelog igcinwa kwisitshixo esisodwa. Kuya kuba ngcono ukuba imiyalezo yelog ilungelelaniswe ngcono.
Idatha engacwangciswanga evela kwiilogi
localhost GET /v2/applink/5c2f4bb3e9fda1234edc64d 400 46ms 5bc6e716b5d6cb35fc9687c0Ukuba ujongisisa idatha ekrwada, uya kubona ukuba eneneni inamalungu ahlukeneyo, ngalinye lahlulwe sisithuba.
Kubaphuhlisi abanamava ngakumbi, unokuqikelela ukuba icandelo ngalinye lithetha ukuthini kwaye lo myalezo welogi usuka kwifowuni ye-API. Umboniso wento nganye udweliswe apha ngezantsi.
Imbonakalo ecwangcisiweyo yedatha yethu
- localhost == indawo engqongileyo
- GET == indlela
- β /v2/applink/5c2f4bb3e9fda1234edc64d == url
- 400 ==imeko_yempendulo
- 46ms == impendulo_ixesha
- β 5bc6e716b5d6cb35fc9687c0 == user_id
Njengoko sibona kwiinkcukacha ezicwangcisiweyo, kukho umyalelo weelogi ezingacwangciswanga. Inyathelo elilandelayo kukusetyenzwa kwesoftware yedatha ekrwada. Kulapho iGrok ikhanya khona.
Izakhelo zeGrok
Itemplates eyakhelwe-ngaphakathi yeGrok
I-Logstash iza neetemplates ezingaphezulu kwe-100 ezakhelwe ngaphakathi zokucwangcisa idatha engacwangciswanga. Ngokuqinisekileyo kuya kufuneka uthathe ithuba loku nanini na xa kunokwenzeka kwii-syslogs ngokubanzi ezifana ne-apache, i-linux, i-haproxy, i-aws njalo njalo.
Nangona kunjalo, kwenzeka ntoni xa uneelogi zesiko njengakumzekelo ongasentla? Kuya kufuneka uzenzele eyakho itemplate yeGrok.
Iitemplates zesiko leGrok
Kuya kufuneka uzame ukwakha itemplate yakho yeGrok. ndidla ngoku ΠΈ .
Qaphela ukuba i-syntax yetemplate ye-Grok imi ngolu hlobo lulandelayo: %{SYNTAX:SEMANTIC}
Into yokuqala endiye ndazama ukuyenza kukuya kwi-tab Discover kwi-Grok debugger. Ndicinge ukuba kuya kuba kuhle ukuba esi sixhobo sinokuvelisa ngokuzenzekelayo ipateni ye-Grok, kodwa ayizange ibe luncedo kakhulu kuba ifumene imidlalo emibini kuphela.
Ndisebenzisa oku kufunyanisiweyo, ndiqale ukwenza eyam ithemplate kwiGrok debugger ndisebenzisa i-syntax efunyenwe kwiphepha le-Elastic Github.
Emva kokudlala ngeenxa zonke ngeesintaksi ezahlukeneyo, ekugqibeleni ndakwazi ukucwangcisa idatha yelog ngendlela endifuna ngayo.
Grok Debugger Link
Isiqendu sokuqala:
localhost GET /v2/applink/5c2f4bb3e9fda1234edc64d 400 46ms 5bc6e716b5d6cb35fc9687c0Umzekelo:
%{WORD:environment} %{WORD:method} %{URIPATH:url} %{NUMBER:response_status} %{WORD:response_time} %{USERNAME:user_id}Kwenzeka ntoni ekugqibeleni
{
"environment": [
[
"localhost"
]
],
"method": [
[
"GET"
]
],
"url": [
[
"/v2/applink/5c2f4bb3e9fda1234edc64d"
]
],
"response_status": [
[
"400"
]
],
"BASE10NUM": [
[
"400"
]
],
"response_time": [
[
"46ms"
]
],
"user_id": [
[
"5bc6e716b5d6cb35fc9687c0"
]
]
}Ngetemplate yeGrok kunye nedatha emephu esandleni, inyathelo lokugqibela kukuyongeza kwiLogstash.
Ukuhlaziya ifayile yoqwalaselo yeLogstash.conf
Kumncedisi apho ufake khona isitaki se-ELK, yiya kuqwalaselo lweLogstash:
sudo vi /etc/logstash/conf.d/logstash.confNcamathelisa utshintsho.
input {
file {
path => "/your_logs/*.log"
}
}
filter{
grok {
match => { "message" => "%{WORD:environment} %{WORD:method} %{URIPATH:url} %{NUMBER:response_status} %{WORD:response_time} %{USERNAME:user_id}"}
}
}
output {
elasticsearch {
hosts => [ "localhost:9200" ]
}
}Emva kokugcina utshintsho lwakho, qalisa kwakhona i-Logstash kwaye ujonge isimo sayo ukuze uqiniseke ukuba isasebenza.
sudo service logstash restart
sudo service logstash statusEkugqibeleni, ukuqinisekisa ukuba utshintsho luyasebenza, Qiniseka ukuba uhlaziya isalathiso sakho se-Elasticsearch seLogstash eKibana!
NgeGrok, idatha yakho yelog yakhiwe!
Njengoko sibona kumfanekiso ongasentla, iGrok iyakwazi ukuthelekisa ngokuzenzekelayo idatha yelog kunye ne-Elasticsearch. Oku kwenza kube lula ukulawula iilog kwaye ngokukhawuleza ubuze ulwazi. Endaweni yokugrumba kwiifayile zelog ukulungisa ingxaki, ungahluza ngokulula ngale nto uyifunayo, njengendawo yendawo okanye i-url.
Zama iintetho zikaGrok! Ukuba unenye indlela yokwenza oku okanye unayo nayiphi na ingxaki ngemizekelo engentla, bhala nje uluvo olungezantsi ukuze undazise.
Enkosi ngokufunda-kwaye ndicela undilandele apha kwiMedium ukufumana amanqaku anomdla ngakumbi ngobunjineli besoftware!
Izibonelelo
PS
Ijelo leTelegram ngu
umthombo: www.habr.com