ืกืืจืึทืงืืฉืขืจืื ื ืึทื ืกืืจืึทืงืืฉืขืจื ืืึทืื ืืื GROK
ืืืื ืืืจ ื ืืฆื ืื Elastic (ELK) ืึธื ืืืืื ืืื ืืืจ ืืขื ื ืืื ืืขืจืขืกืืจื ืืื ืืึทืคึผืื ื ืื ืื Logstash ืืึธืืก ืฆื Elasticsearch, ืืขืจ ืคึผืึธืกืื ืืื ืคึฟืึทืจ ืืืจ.
ืื ELK ืึธื ืืืืื ืืื ืึทื ืึทืงืจืึทื ืื ืคึฟืึทืจ ืืจืื ืึธืคึฟื ืืงืืจ ืคึผืจืึทืืืฉืขืงืก: Elasticsearch, Logstash ืืื Kibana. ืฆืืืึทืืขื ืืื ืคืึธืจืขื ืึท ืงืืึธืฅ ืคืึทืจืืืึทืืืื ื ืคึผืืึทืืคืึธืจืืข.
- ืขืืึทืกืืืงืกืึทืจืืฉ ืืื ืึท ืืืื ืืื ืึทื ืึทืืืกืืก ืกืืกืืขื.
- ืืึธืืกืืึทืฉ ืืื ืึท ืกืขืจืืืขืจ-ืืืึทื ืืึทืื ืคึผืจืึทืกืขืกืื ื ืจืขืจื - ืืื ืืข ืืืึธืก ืื ืืืฉืขืกืืื ืืึทืื ืคืื ืงืืืคื ืงืืืืื ืกืืืืึทืืืืื ืืึทืกืื, ืืจืึทื ืกืคืึธืจืื ืขืก ืืื ืืขืจื ืึธื ืกืขื ืื ืขืก ืฆื ืึท "ืกืืฉ" ืึทืืึท ืืื Elasticsearch.
- ืงืืืึทื ืึท ืึทืืึทืื ื ืืฆืขืจืก ืฆื ืืืืืฉืืืึทืืืื ืืึทืื ืืื ืืฉืึทืจืฅ ืืื ืืจืึทืคืก ืืื Elasticsearch.
ืฉืืึธืื ืืขืงืืืขื ืฉืคึผืขืืขืจ ืืื ืืื ืึท ืืืืืืืืื ืืึทืื ืฉืืคึผืขืจ. ืื ืืงืืื ืคืื ืืืฅ ืคืืจืืืื ืืืขื ืื ืขืืง ืกืืึทืง ืืื ืื ืขืืึทืกืืืง ืกืืึทืง, ืึธืืขืจ ืืึธืก ืืื ื ืืฉื ืื ืคืื ื.
ืืขืจ ืึทืจืืืงื ืืื ืืืขืื Grok, ืืืึธืก ืืื ืึท ืฉืืจืื ืืื Logstash ืืืึธืก ืงืขื ืขื ืืืขืจืืึทืื ืืืื ืืึธืืก ืืืืืขืจ ืืื ืืขื ืขื ืืขืฉืืงื ืฆื ืื ืกืืึทืฉ. ืคึฟืึทืจ ืืื ืืืขืจ ืฆืืืขืงื, ืืื ืืืขื ื ืึธืจ ืจืขืื ืืืขืื ืคึผืจืึทืกืขืกืื ื ืืึทืื ืคึฟืื Logstash ืืื Elasticsearch.
Grok ืืื ืึท ืคืืืืขืจ ืืื Logstash ืืืึธืก ืืื ืืขื ืืฆื ืฆื ืคึผืึทืจืก ืึทื ืกืืจืึทืงืืฉืขืจื ืืึทืื ืืื ืขืคึผืขืก ืกืืจืึทืงืืฉืขืจื ืืื ืงืืืขืจืืึทืืืข. ืขืก ืืืฆื ืืืืฃ ืฉืคึผืืฅ ืคืื ืึท ืจืขืืืืขืจ ืืืืกืืจืืง (ืจืขืืขืงืก) ืืื ื ืืฆื ืืขืงืกื ืคึผืึทืืขืจื ื ืฆื ืืืืึทืื ืกืืจืื ืืก ืืื ืงืืึธืฅ ืืขืงืขืก.
ืืื ืืืจ ืืืขืื ืืขื ืืื ืื ืคืืืืขื ืืข ืกืขืงืฉืึทื ื, ืื ื ืืฆื ืคืื Grok ืืืื ืึท ืืจืืืก ืืืืืง ืืืขื ืขืก ืงืืื ืฆื ืขืคืขืงืืืื ืงืืึธืฅ ืคืึทืจืืืึทืืืื ื.
ืึธื ืืจืึธืง, ืืืื ืงืืึธืฅ ืืึทืื ืืขื ืขื ืึทื ืกืืจืึทืงืืฉืขืจื
ืึธื ืืจืึธืง, ืืืขื ืืึธืืก ืืขื ืขื ืืขืฉืืงื ืคึฟืื Logstash ืฆื Elasticsearch ืืื ืจืขื ืืขืจื ืืื Kibana, ืืื ืืขืจืฉืืึทื ืขื ืืืืื ืืื ืื ืึธื ืืึธื ืืืขืจื.
ืขืก ืืื ืฉืืืขืจ ืฆื ืืืื ืืื ืคึฟืึธืจืืึทืฆืืข ืืื ืืขื ืกืืืืึทืฆืืข, ืืืืึทื ืึทืืข ืงืืึธืฅ ืืึทืื ืืขื ืขื ืกืืึธืจื ืืื ืึท ืืืื ืฉืืืกื. ืขืก ืืืึธืื ืืืื ืืขืกืขืจ ืืืื ืื ืงืืึธืฅ ืึทืจืืืงืืขื ืืขื ืขื ืืขืกืขืจ ืึธืจืืึทื ืืืืจื.
ืึทื ืกืืจืึทืงืืฉืขืจื ืืึทืื ืคืื ืืึธืืก
localhost GET /v2/applink/5c2f4bb3e9fda1234edc64d 400 46ms 5bc6e716b5d6cb35fc9687c0ืืืื ืืืจ ืงืืง ืขื ื ืืื ืื ืจืื ืืึทืื, ืืืจ ืืืขื ืืขื ืึทื ืขืก ืึทืงืฉืื ืืืฉืืืื ืคืื ืคืึทืจืฉืืืขื ืข ืืืืื, ืืขืืขืจ ืืคืืขืฉืืืื ืืืจื ืึท ืคึผืืึทืฅ.
ืคึฟืึทืจ ืืขืจ ืืงืกืคึผืืจืืึทื ืกื ืืขืืืขืืึธืคึผืขืจืก, ืืืจ ืงืขื ืขื ืืืกืืึธืืข ืืจืขืคื ืืืึธืก ืืขืืขืจ ืคืื ืื ืคึผืึทืจืฅ ืืืื ืืื ืืืึธืก ืื ืงืืึธืฅ ืึธื ืืึธื ืืื ืคึฟืื ืื ืึทืคึผื ืจืืคื. ืื ืคึผืจืขืืขื ืืืจืื ื ืคืื ืืขืืขืจ ื ืืืขืจ ืืื ืึทืืืืืื ื ืืื ืื.
ืกืืจืึทืงืืฉืขืจื ืืืื ืื ื ืคืื ืืื ืืืขืจ ืืึทืื
- ืืึธืงืึทืืืึธืกื == ืกืืืืืืข
- ืืึทืงืืืขื == ืืืคึฟื
- โ /v2/applink/5c2f4bb3e9fda1234edc64d == url
- 400 == ืขื ืืคืขืจ_ืกืืึทืืืก
- 46ms == ืขื ืืคืขืจ_ืฆืืื
- โ 5bc6e716b5d6cb35fc9687c0 == user_id
ืืื ืืืจ ืืขื ืืื ืกืืจืึทืงืืฉืขืจื ืืึทืื, ืขืก ืืื ืึท ืกืืจ ืคึฟืึทืจ ืึทื ืกืืจืึทืงืืฉืขืจื ืืึธืืก. ืืขืจ ืืืืึทืืขืจ ืฉืจืื ืืื ืืืืืืืืืจื ืคึผืจืึทืกืขืกืื ื ืคืื ืจืื ืืึทืื. ืื ืฉืฒื ื ืืจืืง.
ืืจืึธืง ืืขืืคึผืืึทืืขืก
ืืขืืืื-ืืื ืืจืึธืง ืืขืืคึผืืึทืืขืก
Logstash ืงืืื ืืื ืืืืขืจ 100 ืืขืืืื-ืืื ืืขืืคึผืืึทืืขืก ืคึฟืึทืจ ืกืืจืึทืงืืฉืขืจืื ื ืึทื ืกืืจืึทืงืืฉืขืจื ืืึทืื. ืืืจ ืืึธื ืืืฉืืืื ื ืืฆื ืืขื ืืืขื ืืขืืืขื ืคึฟืึทืจ ืืขื ืขืจืึทื ืกืืกืืึธืืก ืืื ืึทืคึผืึทืืฉื, ืืื ืืงืก, ืืึทืคึผืจืึธืงืกื, ืึทืืก ืืื ืึทืืื ืืืืฃ.
ืึธืืขืจ, ืืืึธืก ืืึทืคึผืึทื ื ืืืขื ืืืจ ืืึธืื ืื ืื ืืึธืืก ืืื ืืื ืืขื ืืืึทืฉืคึผืื ืืืืื? ืืืจ ืืืื ืืืืขื ืืืื ืืืืืขื ืข ืืจืึธืง ืืืกืืขืจ.
ืื ืื ืืจืึธืง ืืขืืคึผืืึทืืขืก
ืืืจ ืืึธืื ืฆื ืคึผืจืืืืจื ืฆื ืืืืขื ืืืื ืืืืืขื ืข ืืจืึธืง ืืืกืืขืจ. ืืื ืืื ืืขื ืืฆื ะธ .
ืืึทืืขืจืงืื ื ืึทื ืื Grok ืืืกืืขืจ ืกืื ืืึทืงืก ืืื ืืื ืืืื: %{SYNTAX:SEMANTIC}
ืืขืจ ืขืจืฉืืขืจ ืืึทื ืืื ืืขืคืจืืืื ืฆื ืืึธื ืืื ืฆื ืืืื ืฆื ืื ืงืืืืื ืึทื ืืืขืงื ืืื ืื ืืจืึธืง ืืืืืืืขืจ. ืืื ืืขืืึทื ืง ืขืก ืืืึธืื ืืืื ืงืื ืืืื ืื ืืขืฆืืึทื ืงืขื ืืืืืึธืืึทืืืฉ ืืืฉืขื ืขืจืืื ืึท ืืจืึธืง ืืืกืืขืจ, ืึธืืขืจ ืขืก ืืื ื ืืฉื ืฆื ื ืืฆืืง ืืืืึทื ืขืก ืืืืื ืืขืคึฟืื ืขื ืฆืืืื ืฉืืืขืืขืืขื.
ืืื ืืขื ืืคืืขืงืื ื, ืืื ืกืืึทืจืืขื ืงืจืืืืืื ื ืืืื ืืืืืขื ืข ืืืกืืขืจ ืืื ืื ืืจืึธืง ืืขืืืืืขืจ ื ืืฆื ืื ืกืื ืืึทืงืก ืืขืคึฟืื ืขื ืืืืฃ ืื Elastic Github ืืืึทื.
ื ืึธื ืคึผืืืืื ื ืืื ืคืึทืจืฉืืืขื ืข ืกืื ืืึทืงืกืื, ืืื ืืขืกืึธืฃ ืงืขื ืขื ืฆื ืกืืจืืงืืืจ ืื ืงืืึธืฅ ืืึทืื ืืื ืืื ืืขืืืืื.
ืืจืึธืง ืืืืืืืขืจ ืืื ืง
ืึธืจืืืื ืขื ืืขืงืกื:
localhost GET /v2/applink/5c2f4bb3e9fda1234edc64d 400 46ms 5bc6e716b5d6cb35fc9687c0ืืืกืืขืจ:
%{WORD:environment} %{WORD:method} %{URIPATH:url} %{NUMBER:response_status} %{WORD:response_time} %{USERNAME:user_id}ืืืึธืก ืืขืืจืืคื ืืื ืื ืกืืฃ
{
"environment": [
[
"localhost"
]
],
"method": [
[
"GET"
]
],
"url": [
[
"/v2/applink/5c2f4bb3e9fda1234edc64d"
]
],
"response_status": [
[
"400"
]
],
"BASE10NUM": [
[
"400"
]
],
"response_time": [
[
"46ms"
]
],
"user_id": [
[
"5bc6e716b5d6cb35fc9687c0"
]
]
}ืืื ืื ืืจืึธืง ืืืกืืขืจ ืืื ืืึทืคึผื ืืึทืื ืืื ืืึทื ื, ืื ืืขืฆืืข ืฉืจืื ืืื ืฆื ืืืืื ืขืก ืฆื Logstash.
ืึทืคึผืืืืืื ื ืื Logstash.conf ืงืึทื ืคืืืืขืจืืืฉืึทื ืืขืงืข
ืืืืฃ ืื ืกืขืจืืืขืจ ืืื ืืืจ ืืื ืกืืึทืืืจื ืื ELK ืึธื ืืืืื, ืืืื ืฆื ืื Logstash ืงืึทื ืคืืืืขืจืืืฉืึทื:
sudo vi /etc/logstash/conf.d/logstash.confืคึผืึทืคึผ ืื ืขื ืืขืจืื ืืขื.
input {
file {
path => "/your_logs/*.log"
}
}
filter{
grok {
match => { "message" => "%{WORD:environment} %{WORD:method} %{URIPATH:url} %{NUMBER:response_status} %{WORD:response_time} %{USERNAME:user_id}"}
}
}
output {
elasticsearch {
hosts => [ "localhost:9200" ]
}
}ื ืึธื ืฉืคึผืึธืจื ืืืื ืขื ืืขืจืื ืืขื, ืจืืกืืึทืจื Logstash ืืื ืงืึธื ืืจืึธืืืจื ืืืื ืกืืึทืืืก ืฆื ืืึทืื ืืืืขืจ ืึทื ืขืก ืืื ื ืึธื ืืจืืขืื.
sudo service logstash restart
sudo service logstash statusืฆืื ืกืืฃ, ืฆื ืืึทืื ืืืืขืจ ืึทื ืื ืขื ืืขืจืื ืืขื ืืึธืื ืืขื ืืืขื ืืืืจืงืื ื, ืืืื ืืืืขืจ ืฆื ืืขืจืืืึทื ืืืงื ืืืื Elasticsearch ืืื ืืขืงืก ืคึฟืึทืจ Logstash ืืื Kibana!
ืืื Grok, ืืืื ืงืืึธืฅ ืืึทืื ืืื ืกืืจืึทืงืืฉืขืจื!
ืืื ืืืจ ืงืขื ืขื ืืขื ืืื ืื ืืืื ืืืืื, Grok ืืื ืืืืืขืืืืืง ืคืื ืืืืืึธืืึทืืืฉ ืืืืึทืื ืืึธื ืืึทืื ืืื Elasticsearch. ืืึธืก ืืืื ืขืก ืืจืื ืืขืจ ืฆื ืคืืจื ืืึธืืก ืืื ืืขืฉืืืื ื ืึธื ืคึฟืจืขื ืืื ืคึฟืึธืจืืึทืฆืืข. ืึทื ืฉืืึธื ืืืืื ื ืืืจื ืงืืึธืฅ ืืขืงืขืก ืฆื ืืืืึทื, ืืืจ ืงืขื ืขื ืคืฉืื ืคืืืืขืจ ืืืจื ืืืึธืก ืืืจ ืืืื ืคึฟืึทืจ, ืึทืืึท ืืื ืึท ืกืืืืืืข ืึธืืขืจ ืึท URL.
ืคึผืจืึผืืื ืืจืึธืง ืืืืกืืจืืงื! ืืืื ืืืจ ืืึธื ืื ืื ืืขืจ ืืืขื ืฆื ืืึธื ืืึธืก ืึธืืขืจ ืืึธืื ืงืืื ืคืจืืืืขืืขื ืืื ืื ืืืืฉืคืืื ืืืืื, ื ืึธืจ ืฉืจืืึทืื ืึท ืืึทืืขืจืงืื ื ืืื ืื ืฆื ืืึธืื ืืืจ ืืืืกื.
ืืึทื ืงืขื ืคึฟืึทืจ ืืืืขื ืขื - ืืื ืืืืข ื ืึธืืคืึธืืื ืืืจ ืืึธ ืืืืฃ ืืขืืืื ืคึฟืึทืจ ืืขืจ ืืฉืืงืึทืืืข ืืืืืืืืืจื ืื ืืฉืขื ืืขืจืืข ืึทืจืืืงืืขื!
ืจืขืกืืจืกื
ืคึผืก
ืืขืืขืืจืึทื ืงืึทื ืึทื ืืืจื
ืืงืืจ: www.habr.com