Ƙirƙirar bayanan da ba a tsara su ba tare da GROK
Idan kuna amfani da tarin Elastic (ELK) kuma kuna sha'awar yin taswirar logstash na al'ada zuwa Elasticsearch, to wannan post ɗin naku ne.
Tarin ELK gajarta ce don ayyukan buɗaɗɗen tushe guda uku: Elasticsearch, Logstash da Kibana. Tare suna samar da dandalin sarrafa log.
- Elasticsearch tsarin bincike ne da nazari.
- Logstash bututun sarrafa bayanai ne na gefen uwar garken wanda ke shigar da bayanai daga tushe da yawa lokaci guda, canza shi, sannan a tura shi zuwa “stash” kamar Elasticsearch.
- Kibana yana bawa masu amfani damar hange bayanai ta amfani da zane-zane da zane-zane a cikin Elasticsearch.
Barazana yazo daga baya kuma shine mai jigilar bayanai mara nauyi. Gabatarwar Beats ya canza Elk Stack zuwa Stack Elastic, amma wannan ba shine batun ba.
Wannan labarin yana game da Grok, wanda sifa ce a cikin Logstash wanda zai iya canza rajistan ayyukan ku kafin a aika su zuwa stash. Don dalilanmu, zan yi magana ne kawai game da sarrafa bayanai daga Logstash zuwa Elasticsearch.
Grok tacewa ne a cikin Logstash wanda ake amfani dashi don tantance bayanan da ba a tsara su ba cikin wani abu da aka tsara kuma mai iya tambaya. Yana zaune a saman magana ta yau da kullun (regex) kuma yana amfani da tsarin rubutu don daidaita kirtani a cikin fayilolin log.
Kamar yadda za mu gani a cikin sassan masu zuwa, amfani da Grok yana haifar da babban bambanci idan ya zo ga ingantaccen sarrafa log.
Ba tare da Grok ba an tsara bayanan log ɗin ku ba
Ba tare da Grok ba, lokacin da aka aika rajistan ayyukan daga Logstash zuwa Elasticsearch kuma ana yin su cikin Kibana, suna bayyana ne kawai a cikin ƙimar saƙon.
Neman bayanai masu ma'ana a wannan yanayin yana da wahala saboda duk bayanan log ɗin ana adana su a maɓalli ɗaya. Zai fi kyau idan an tsara saƙonnin log ɗin da kyau.
Bayanan da ba a tsara su ba daga rajistan ayyukan
localhost GET /v2/applink/5c2f4bb3e9fda1234edc64d 400 46ms 5bc6e716b5d6cb35fc9687c0Idan ka kalli danyen bayanan da kyau, za ka ga cewa a zahiri ya kunshi sassa daban-daban, kowanne ya rabu da sarari.
Don ƙwararrun ƙwararrun masu haɓakawa, ƙila za ku iya hasashen abin da kowane ɓangaren ke nufi da abin da saƙon log ɗin ya fito daga kiran API. An bayyana gabatarwar kowane abu a ƙasa.
Tsarin duba bayanan mu
- localhost == muhalli
- SAMU = Hanyar
- /v2/applink/5c2f4bb3e9fda1234edc64d == url
- 400 == Matsayin amsawa
- 46ms = lokacin amsawa
- 5bc6e716b5d6cb35fc9687c0 == user_id
Kamar yadda muke gani a cikin bayanan da aka tsara, akwai oda don rajistan ayyukan da ba a tsara su ba. Mataki na gaba shine sarrafa software na danyen bayanai. Wannan shine inda Grok ke haskakawa.
Samfuran Grok
Samfuran Grok da aka gina a ciki
Logstash ya zo da samfuran ginanni sama da 100 don tsara bayanan da ba a tsara su ba. Ya kamata ku yi amfani da wannan a duk lokacin da zai yiwu don janar syslogs kamar apache, Linux, haproxy, aws da sauransu.
Koyaya, menene zai faru idan kuna da rajistan ayyukan al'ada kamar a cikin misalin da ke sama? Dole ne ku gina samfurin Grok naku.
Samfuran Grok na Musamman
Dole ne ku yi ƙoƙarin gina samfurin Grok na ku. na yi amfani и .
Lura cewa tsarin tsarin tsarin Grok shine kamar haka: %{SYNTAX:SEMANTIC}
Abu na farko da na yi ƙoƙarin yi shi ne zuwa shafin Discover a cikin Grok debugger. Ina tsammanin zai yi kyau idan wannan kayan aikin zai iya samar da tsarin Grok ta atomatik, amma ba shi da amfani sosai tunda ya sami matches biyu kawai.
Yin amfani da wannan binciken, na fara ƙirƙirar samfura na a cikin Grok debugger ta amfani da sintax da aka samo akan shafin Elastic Github.
Bayan yin wasa tare da kalmomi daban-daban, a ƙarshe na sami damar tsara bayanan log ɗin yadda nake so.
Grok Debugger Link
Rubutun asali:
localhost GET /v2/applink/5c2f4bb3e9fda1234edc64d 400 46ms 5bc6e716b5d6cb35fc9687c0juna:
%{WORD:environment} %{WORD:method} %{URIPATH:url} %{NUMBER:response_status} %{WORD:response_time} %{USERNAME:user_id}Me ya faru a karshe
{
"environment": [
[
"localhost"
]
],
"method": [
[
"GET"
]
],
"url": [
[
"/v2/applink/5c2f4bb3e9fda1234edc64d"
]
],
"response_status": [
[
"400"
]
],
"BASE10NUM": [
[
"400"
]
],
"response_time": [
[
"46ms"
]
],
"user_id": [
[
"5bc6e716b5d6cb35fc9687c0"
]
]
}Tare da samfurin Grok da bayanan taswira a hannu, mataki na ƙarshe shine ƙara shi zuwa Logstash.
Ana ɗaukaka fayil ɗin daidaitawar Logstash.conf
A kan uwar garken inda kuka shigar da tarin ELK, je zuwa tsarin Logstash:
sudo vi /etc/logstash/conf.d/logstash.confManna canje-canje.
input {
file {
path => "/your_logs/*.log"
}
}
filter{
grok {
match => { "message" => "%{WORD:environment} %{WORD:method} %{URIPATH:url} %{NUMBER:response_status} %{WORD:response_time} %{USERNAME:user_id}"}
}
}
output {
elasticsearch {
hosts => [ "localhost:9200" ]
}
}Bayan adana canje-canjen ku, sake kunna Logstash kuma duba matsayinsa don tabbatar da cewa har yanzu yana aiki.
sudo service logstash restart
sudo service logstash statusA ƙarshe, don tabbatar da cewa canje-canjen sun yi tasiri. Tabbatar sabunta fihirisar Elasticsearch don Logstash a Kibana!
Tare da Grok, an tsara bayanan log ɗin ku!
Kamar yadda muke iya gani a hoton da ke sama, Grok yana da ikon daidaita bayanan log ta atomatik tare da Elasticsearch. Wannan yana ba da sauƙi don sarrafa rajistan ayyukan da sauri da neman bayanin tambaya. Maimakon tona ta cikin fayilolin log don gyara kuskure, kawai kuna iya tace ta abin da kuke nema, kamar muhalli ko url.
Gwada maganganun Grok! Idan kuna da wata hanyar yin wannan ko kuna da wata matsala tare da misalan da ke sama, kawai ku rubuta sharhi a ƙasa don sanar da ni.
Na gode da karantawa-kuma da fatan za a biyo ni nan akan Matsakaici don ƙarin labaran injiniyan software masu ban sha'awa!
Resources
PS
Telegram channel by
source: www.habr.com