Ho theha data e sa hlophisoang ka GROK
Haeba u sebelisa stack ea Elastic (ELK) 'me u thahasella ho etsa limmapa tsa Logstash tse tloaelehileng ho Elasticsearch, joale poso ena ke ea hau.
ELK stack ke khutsufatso ea merero e meraro ea mohloli o bulehileng: Elasticsearch, Logstash le Kibana. Hammoho ba theha sethala sa taolo ea log.
- Elasticsearch ke mokhoa oa ho batla le oa ho hlahloba.
- Logstash ke phaephe ea ts'ebetso ea data e lehlakoreng la seva e kenyang data ho tsoa mehloling e mengata ka nako e le 'ngoe, e e fetole, ebe e e romella ho "stash" joalo ka Elasticsearch.
- Kibana e lumella basebelisi ho bona data ka mahlo a kelello ba sebelisa lichate le li-graph ho Elasticsearch.
Beats e tlile hamorao mme ke morekisi oa data o bobebe. Kenyelletso ea Beats e fetotse Elk Stack hore e be Elastic Stack, empa ha se eona ntlha.
Sengoliloeng sena se bua ka Grok, e leng karolo ea Logstash e ka fetolang lits'oants'o tsa hau pele li romelloa ho stash. Bakeng sa merero ea rona, ke tla bua feela ka ho sebetsana le data ho tloha Logstash ho Elasticsearch.
Grok ke filthara ka hare ho Logstash e sebelisetsoang ho hlalosa lintlha tse sa hlophisoang hore e be ntho e hlophisitsoeng le e ka botsoang. E lutse ka holim'a polelo e tloaelehileng (regex) 'me e sebelisa mekhoa ea mongolo ho tsamaisana le likhoele ho lifaele tsa log.
Joalokaha re tla bona likarolong tse latelang, ho sebelisa Grok ho etsa phapang e kholo ha ho tluoa tabeng ea tsamaiso e nepahetseng ea log.
Ntle le Grok data ea hau ea log ha e hlophisehe
Ntle le Grok, ha lits'oants'o li romelloa ho tloha Logstash ho ea Elasticsearch mme li fanoa ka Kibana, li hlaha feela ka boleng ba molaetsa.
Ho botsa tlhahisoleseding e nang le moelelo boemong bona ho thata hobane data eohle ea log e bolokiloe ka senotlolo se le seng. Ho ka ba molemo haeba melaetsa ea log e ne e hlophisitsoe hantle.
Lintlha tse sa hlophisoang tse tsoang ho li-log
localhost GET /v2/applink/5c2f4bb3e9fda1234edc64d 400 46ms 5bc6e716b5d6cb35fc9687c0Haeba u sheba ka hloko data e tala, u tla bona hore e hlile e na le likarolo tse fapaneng, e 'ngoe le e' ngoe e arotsoe ke sebaka.
Bakeng sa bahlahisi ba nang le boiphihlelo, o kanna oa hakanya hore na karolo ka 'ngoe e bolela'ng le hore na molaetsa oo oa log o tsoa mohaleng oa API. Tlhahiso ea ntlha ka 'ngoe e hlalositsoe ka tlase.
Pono e hlophisitsoeng ea data ea rona
- localhost == tikoloho
- FUMANA == mokhoa
- /v2/applink/5c2f4bb3e9fda1234edc64d == url
- 400 ==poelo_boemo
- 46ms == karabo_nako
- 5bc6e716b5d6cb35fc9687c0 == user_id
Joalokaha re bona ho data e hlophisitsoeng, ho na le taelo ea li-log tse sa hlophisoang. Mohato o latelang ke ts'ebetso ea software ea data e tala. Mona ke moo Grok e khanyang teng.
Li-template tsa Grok
Lithempleite tsa Grok tse hahiloeng kahare
Logstash e tla le litempele tse fetang 100 tse hahelletsoeng kahare bakeng sa ho hlophisa data e sa hlophisoang. Ka sebele u lokela ho nka monyetla oa sena neng kapa neng ha ho khoneha bakeng sa li-syslogs tse akaretsang tse kang apache, linux, haproxy, aws joalo-joalo.
Leha ho le joalo, ho etsahala'ng ha u e-na le li-logs tse tloaelehileng joaloka mohlala o ka holimo? U tlameha ho iketsetsa template ea hau ea Grok.
Custom Grok templates
U tlameha ho leka ho iketsetsa template ea hau ea Grok. Ke sebelisitse и .
Hlokomela hore syntax ea template ea Grok e tjena: %{SYNTAX:SEMANTIC}
Ntho ea pele eo ke ileng ka leka ho e etsa ke ho ea tab Discover ka Grok debugger. Ke ne ke nahana hore ho tla ba monate haeba sesebelisoa sena se ka iketsetsa mohlala oa Grok, empa se ne se se na thuso haholo kaha se fumane lipapali tse peli feela.
Ke sebelisa tšibollo ena, ke ile ka qala ho iketsetsa template ea ka ho Grok debugger ke sebelisa syntax e fumanoang leqepheng la Elastic Github.
Kamora ho bapala ka li-syntaxes tse fapaneng, qetellong ke ile ka khona ho hlophisa data ea log ka tsela eo ke neng ke e batla.
Sehokelo sa Grok Debugger
Sengoloa sa mantlha:
localhost GET /v2/applink/5c2f4bb3e9fda1234edc64d 400 46ms 5bc6e716b5d6cb35fc9687c0Mohlala:
%{WORD:environment} %{WORD:method} %{URIPATH:url} %{NUMBER:response_status} %{WORD:response_time} %{USERNAME:user_id}Se etsahetseng qetellong
{
"environment": [
[
"localhost"
]
],
"method": [
[
"GET"
]
],
"url": [
[
"/v2/applink/5c2f4bb3e9fda1234edc64d"
]
],
"response_status": [
[
"400"
]
],
"BASE10NUM": [
[
"400"
]
],
"response_time": [
[
"46ms"
]
],
"user_id": [
[
"5bc6e716b5d6cb35fc9687c0"
]
]
}Ka template ea Grok le data ea 'mapa e letsohong, mohato oa ho qetela ke ho e kenyelletsa ho Logstash.
Ho nchafatsa faele ea tlhophiso ea Logstash.conf
Ho seva moo u kentseng stack ea ELK, e-ea ho Logstash tlhophiso:
sudo vi /etc/logstash/conf.d/logstash.confBeha liphetoho.
input {
file {
path => "/your_logs/*.log"
}
}
filter{
grok {
match => { "message" => "%{WORD:environment} %{WORD:method} %{URIPATH:url} %{NUMBER:response_status} %{WORD:response_time} %{USERNAME:user_id}"}
}
}
output {
elasticsearch {
hosts => [ "localhost:9200" ]
}
}Kamora ho boloka liphetoho tsa hau, qala Logstash hape 'me u hlahlobe boemo ba eona ho netefatsa hore e ntse e sebetsa.
sudo service logstash restart
sudo service logstash statusQetellong, ho etsa bonnete ba hore liphetoho li sebetsa, Etsa bonnete ba hore o ntlafatsa index ea hau ea Elasticsearch bakeng sa Logstash e Kibana!
Ka Grok, data ea hau ea log e hlophisitsoe!
Joalo ka ha re bona setšoantšong se kaholimo, Grok e khona ho ipapisa le data ea log le Elasticsearch. Sena se nolofalletsa ho laola li-log le ho botsa lintlha kapele. Sebakeng sa ho cheka lifaele tsa log ho lokisa liphoso, o ka sefa feela ka seo o se batlang, joalo ka tikoloho kapa url.
Leka lipolelo tsa Grok! Haeba u na le tsela e 'ngoe ea ho etsa sena kapa u na le mathata leha e le afe ka mehlala e ka holimo, ngola feela maikutlo a ka tlase ho ntsebisa.
Ke leboha ho bala - 'me ka kopo ntatele mona ho Medium bakeng sa lingoliloeng tse khahlisang tsa boenjiniere ba software!
Lisebelisuoa
PS
Mocha oa thelekramo ka
Source: www.habr.com