Txhim kho cov ntaub ntawv tsis tsim nyog nrog GROK
Yog tias koj tab tom siv Elastic (ELK) pawg thiab txaus siab rau daim ntawv qhia kev cai Logstash cav rau Elasticsearch, tom qab ntawv tshaj tawm no yog rau koj.
ELK pawg yog cov lus luv rau peb qhov chaw qhib: Elasticsearch, Logstash thiab Kibana. Ua ke lawv tsim lub log tswj platform.
- Elasticsearch yog kev tshawb nrhiav thiab kev tshuaj ntsuam xyuas.
- Logstash yog lub server-sab cov ntaub ntawv ua cov raj xa dej uas noj cov ntaub ntawv los ntawm ntau qhov chaw ib txhij, hloov nws, thiab xa mus rau "stash" xws li Elasticsearch.
- Kibana tso cai rau cov neeg siv pom cov ntaub ntawv siv cov kab kos thiab cov duab hauv Elasticsearch.
neeg ntaus tuaj tom qab thiab yog ib lub teeb yuag cov ntaub ntawv shipper. Kev taw qhia ntawm Beats tau hloov pauv Elk Stack rau hauv Elastic Stack, tab sis qhov ntawd tsis yog lub ntsiab lus.
Kab lus no yog hais txog Grok, uas yog ib qho feature hauv Logstash uas tuaj yeem hloov koj cov cav ua ntej lawv raug xa mus rau lub stash. Rau peb lub hom phiaj, kuv tsuas yog tham txog kev ua cov ntaub ntawv los ntawm Logstash rau hauv Elasticsearch.
Grok yog lub lim hauv Logstash uas yog siv los txheeb xyuas cov ntaub ntawv tsis tsim nyog rau hauv ib yam dab tsi uas tau tsim thiab queryable. Nws zaum saum cov lus qhia tsis tu ncua (regex) thiab siv cov qauv ntawv kom phim cov hlua hauv cov ntaub ntawv teev cia.
Raws li peb yuav pom hauv ntu nram qab no, siv Grok ua qhov sib txawv loj thaum nws los txog rau kev tswj xyuas cov cav kom zoo.
Yog tsis muaj Grok koj cov ntaub ntawv cav yog unstructured
Yog tsis muaj Grok, thaum cov cav raug xa los ntawm Logstash mus rau Elasticsearch thiab muab tso rau hauv Kibana, lawv tsuas yog tshwm sim hauv cov lus tus nqi.
Kev nug cov ntaub ntawv tseem ceeb hauv qhov xwm txheej no yog qhov nyuaj vim tias tag nrho cov ntaub ntawv teev cia yog khaws cia rau hauv ib qho tseem ceeb. Nws yuav zoo dua yog tias cov ntawv teev lus tau zoo dua.
Cov ntaub ntawv tsis tsim nyog los ntawm cov cav
localhost GET /v2/applink/5c2f4bb3e9fda1234edc64d 400 46ms 5bc6e716b5d6cb35fc9687c0Yog tias koj saib ze ntawm cov ntaub ntawv nyoos, koj yuav pom tias nws muaj qhov sib txawv, txhua qhov sib cais los ntawm qhov chaw.
Rau cov neeg tsim khoom paub ntau dua, koj tuaj yeem twv seb txhua qhov txhais tau li cas thiab cov lus kaw lus yog dab tsi los ntawm API hu. Kev nthuav qhia ntawm txhua yam khoom tau piav qhia hauv qab no.
Structured saib ntawm peb cov ntaub ntawv
- localhost == ib puag ncig
- GET == method
- β /v2/applink/5c2f4bb3e9fda1234edc64d == url
- 400 == response_status
- 46ms == response_time
- β 5bc6e716b5d6cb35fc9687c0 == user_id
Raws li peb pom hauv cov ntaub ntawv tsim qauv, muaj ib qho kev txiav txim rau cov cav tsis sib xws. Cov kauj ruam tom ntej yog software ua cov ntaub ntawv nyoos. Qhov no yog qhov uas Grok ci.
Grok Templates
Ua-hauv Grok templates
Logstash los nrog ntau dua 100 tus qauv tsim rau kev tsim cov ntaub ntawv tsis tsim nyog. Koj yuav tsum tau siv qhov zoo ntawm qhov no thaum twg ua tau rau cov syslogs dav dav xws li apache, linux, haproxy, aws thiab lwm yam.
Txawm li cas los xij, ua li cas thaum koj muaj cov ntawv teev kev cai zoo li hauv qhov piv txwv saum toj no? Koj yuav tsum tsim koj tus kheej Grok template.
Kev cai Grok templates
Koj yuav tsum sim tsim koj tus kheej Grok template. kuv siv ΠΈ .
Nco ntsoov tias Grok template syntax yog raws li hauv qab no: %{SYNTAX:SEMANTIC}
Thawj qhov kuv sim ua yog mus rau lub tab Nrhiav hauv Grok debugger. Kuv xav tias nws yuav txias yog tias cov cuab yeej no tuaj yeem tsim tus qauv Grok, tab sis nws tsis muaj txiaj ntsig zoo vim nws tsuas pom ob qhov sib tw.
Siv qhov kev tshawb pom no, kuv pib tsim kuv tus kheej tus qauv hauv Grok debugger siv cov syntax pom ntawm nplooj Elastic Github.
Tom qab ua si ib puag ncig nrog cov syntaxes sib txawv, kuv thaum kawg tuaj yeem tsim cov ntaub ntawv teev tseg raws li qhov kuv xav tau.
Grok Debugger Link
Original text:
localhost GET /v2/applink/5c2f4bb3e9fda1234edc64d 400 46ms 5bc6e716b5d6cb35fc9687c0Txawv:
%{WORD:environment} %{WORD:method} %{URIPATH:url} %{NUMBER:response_status} %{WORD:response_time} %{USERNAME:user_id}Ua li cas thaum kawg
{
"environment": [
[
"localhost"
]
],
"method": [
[
"GET"
]
],
"url": [
[
"/v2/applink/5c2f4bb3e9fda1234edc64d"
]
],
"response_status": [
[
"400"
]
],
"BASE10NUM": [
[
"400"
]
],
"response_time": [
[
"46ms"
]
],
"user_id": [
[
"5bc6e716b5d6cb35fc9687c0"
]
]
}Nrog Grok template thiab mapped cov ntaub ntawv hauv tes, cov kauj ruam kawg yog ntxiv rau Logstash.
Hloov kho cov ntaub ntawv Logstash.conf configuration
Ntawm tus neeg rau zaub mov uas koj tau nruab ELK pawg, mus rau Logstash configuration:
sudo vi /etc/logstash/conf.d/logstash.confPaste cov kev hloov.
input {
file {
path => "/your_logs/*.log"
}
}
filter{
grok {
match => { "message" => "%{WORD:environment} %{WORD:method} %{URIPATH:url} %{NUMBER:response_status} %{WORD:response_time} %{USERNAME:user_id}"}
}
}
output {
elasticsearch {
hosts => [ "localhost:9200" ]
}
}Tom qab txuag koj cov kev hloov pauv, rov pib Logstash thiab xyuas nws cov xwm txheej kom paub tseeb tias nws tseem ua haujlwm.
sudo service logstash restart
sudo service logstash statusThaum kawg, kom paub tseeb tias cov kev hloov pauv tau siv, Nco ntsoov hloov kho koj qhov Elasticsearch Performance index rau Logstash hauv Kibana!
Nrog Grok, koj cov ntaub ntawv teev tseg tau teeb tsa!
Raws li peb tuaj yeem pom hauv daim duab saum toj no, Grok muaj peev xwm ua tau raws li cov ntaub ntawv teev cia nrog Elasticsearch. Qhov no ua rau nws yooj yim dua los tswj cov cav thiab nug cov ntaub ntawv sai. Es tsis txhob khawb los ntawm cov ntaub ntawv log rau kev debug, koj tuaj yeem lim qhov koj tab tom nrhiav, xws li ib puag ncig lossis url.
Muab cov lus qhia Grok sim! Yog tias koj muaj lwm txoj hauv kev los ua qhov no lossis muaj teeb meem nrog cov piv txwv saum toj no, tsuas yog sau ib nqe lus hauv qab no qhia rau kuv paub.
Ua tsaug rau kev nyeem ntawv-thiab thov ua raws kuv ntawm no ntawm Nruab Nrab rau cov lus nthuav dav software engineering ntxiv!
Cov kev pab
PS
Telegram channel los ntawm
Tau qhov twg los: www.hab.com