ืืงืืื
ืืฉืขืช ืืืคึผืืืืื ื ืื ืื ืืขืจ ืกืืกืืขื, ืืืจ ืืขื ืขื ืคืืืกื ืืื ืื ื ืืื ืฆื ืคึผืจืึธืฆืขืก ืึท ืืจืืืก ื ืืืขืจ ืคืื ืคืึทืจืฉืืืขื ืข ืืึธืืก. ELK ืืื ืืืืกืืขืจืืืืืื ืืื ืื ืืขืฆืืึทื. ืืขืจ ืึทืจืืืงื ืืืขื ืืืกืงืืืืจื ืืื ืืืขืจ ืืขืจืคืึทืจืื ื ืืื ืืึทืฉืืขืืืงื ืืขื ืึธื ืืืืื.
ืืืจ ืฉืืขืื ื ืืฉื ืึท ืฆืื ืฆื ืืึทืฉืจืืึทืื ืึทืืข ืืืึทื ืงืืืคึผืึทืืืืึทืืื, ืึธืืขืจ ืืืจ ืืืืื ืฆื ืงืึทื ืกืึทื ืืจืืื ืกืคึผืึทืกืืคืืงืื ืืืืฃ ืกืึทืืืืื ื ืคึผืจืึทืงืืืฉ ืคึผืจืึธืืืขืืก. ืืึธืก ืืื ืจืขืื ืฆื ืืขื ืคืึทืงื ืึทื ืืึธืืฉ ืขืก ืืื ืึท ืืึทื ืฅ ืืจืืืก ืกืืืข ืคืื โโืืึทืงืืืืขื ืืืืฉืึทื ืืื ืคืึทืจืืืง ืืืืืขืจ, ืขืก ืืขื ืขื ืืึทื ืฅ ืึท ืคึผืืึทืฅ ืคืื ืคึผืืืคืึธืื, ืืื ืืื ืืกืืขืจ ืืืจ ืืขืคึฟืื ืขื ืืื.
ืืืจ ืืืคึผืืืื ืืขื ืึธื ืืืืื ืืืจื docker-compose. ืืขืจืฆื, ืืืจ ืืึธืื ืึท ืืขืืื ื-ืืขืฉืจืืื docker-compose.yml, ืืืึธืก ืขืจืืืืื ืืื ืื ืฆื ืืึทืคึผื ืืขื ืึธื ืืืืื ืึผืืขื ืึธื ืคืจืืืืขืืขื. ืืื ืขืก ืืขืืืขื ืฆื ืืื ืื ืึทื ื ืฆืืื ืืื ืฉืืื ื ืึธืขื ื, ืืืฆื ืืืจ ืืืขื ืืืืืง ืขืก ืึท ืืืกื ืฆื ืคึผืึทืกื ืืื ืืืขืจ ืืืืขืจืคืขื ืืฉื ืืื ืึทื ืก ืขืก.
ืฆืื ืืึทืืืืขืจื, ืืขืจ ืคึผืจืืืื ืฆื ืงืึทื ืคืืืืขืจ ืื ืกืืกืืขื ืฆื ืืึทืงืืืขื ืืื ืคึผืจืึธืฆืขืก ืืึธืืก ืคึฟืื ืืื ืืืขืจ ืึทืคึผืืึทืงืืืฉืึทื ืืื ื ืืฉื ืืืืื ืืขืจืึธืื. ืืขืจืืืขืจ, ืืืจ ืืึทืฉืืึธืกื ืึทื ืขืก ืืื ืืืขืจื ืฆื ืืขืจื ืขื ืืขืืขืจ ืงืึธืืคึผืึธื ืขื ื ืกืขืคึผืขืจืึทืืื, ืืื ืืขืืึธืื ืฆืืจืืงืงืืืขื ืฆื ืืืืขืจ ืงืึทื ืขืงืฉืึทื ื.
ืึทืืื, ืืืจ ืื ืืขืืืืื ืืื ืืึธืืกืืึทืฉ.
ืกืืืืืืข, ืืืคึผืืืืืึทื ื, ืคืืืกื ืืืง Logstash ืืื ืึท ืงืึทื ืืืื ืขืจ
ืคึฟืึทืจ ืืืคึผืืืืืึทื ื ืืืจ ื ืืฆื docker-compose; ืื ืืงืกืคึผืขืจืึทืืึทื ืฅ ืืืกืงืจืืืื ืืึธ ืืขื ืขื ืืืจืืืขืงืึธืื ืืืืฃ MacOS ืืื Ubuntu 18.0.4.
ืื ืืึธืืกืืึทืฉ ืืืื ืืืึธืก ืืื ืืขืืืขื ืจืขืืืกืืจืืจื ืืื ืืื ืืืขืจ ืึธืจืืืื ืขื docker-compose.yml ืืื docker.elastic.co/logstash/logstash:6.3.2
ืืืจ ืืืขืื ื ืืฆื ืขืก ืคึฟืึทืจ ืืงืกืคึผืขืจืึทืืึทื ืฅ.
ืืืจ ืืขืฉืจืืื ืึท ืืึทืืื ืืขืจ docker-compose.yml ืฆื ืืืืคื logstash. ืืึธื, ืขืก ืืื ืืขืืืขื ืฆื ืงืึทืืขืจ ืื ืืืื ืคึฟืื ืื ืืึทืคึฟืขืื ืฉืืจื, ืึธืืขืจ ืืืจ ืืึธืื ืกืึทืืืื ืึท ืกืคึผืขืฆืืคืืฉ ืคึผืจืึธืืืขื, ืืื ืืืจ ืืืืคื ืึทืืฅ ืคึฟืื ืืึธืงืขืจ-ืงืึธืืคึผืึธืกืข.
ืืขืงืืฆืขืจ ืืืขืื ืงืึทื ืคืืืืขืจืืืฉืึทื ืืขืงืขืก
ืืื ืืืื ืคืื ืื ืืึทืฉืจืืึทืืื ื, ืืึธืืกืืึทืฉ ืงืขื ืขื ืืืื ืืืืคื ืึธืืขืจ ืคึฟืึทืจ ืืืื ืงืึทื ืึทื, ืืื ืืืึธืก ืคืึทื ืขืก ืืึทืจืฃ ืฆื ืคืึธืจื ืื *.conf ืืขืงืข, ืึธืืขืจ ืคึฟืึทืจ ืขืืืขืืข ืืฉืึทื ืึทืื, ืืื ืืืึธืก ืคืึทื ืขืก ืืึทืจืฃ ืฆื ืคืึธืจื ืื pipelines.yml ืืขืงืข, ืืืึธืก, ืืื ืงืขืจ , ืืืขื ืืื ืง ืฆื ืื ืืขืงืขืก .conf ืคึฟืึทืจ ืืขืืขืจ ืงืึทื ืึทื.
ืื ืจ ืืื ื ืืขื ืืืข ื ืืข ื ืฆืฐืฒื ื ืฐืขื . ืขืก ืืขืืืขื ืฆื ืืื ืื ืืขืจ ืื ืืืืขืจืกืึทื ืืื ืกืงืึทืืึทืืืข. ืืขืจืืืขืจ, ืืืจ ืืืฉืืคื pipelines.yml, ืืื ืืขืืืื ืึท ืคึผืืืคึผืืืื ื ืืืขืืืืืึทืืขืจ ืืื ืืืึธืก ืืืจ ืืืขืื ืฉืืขืื .conf ืืขืงืขืก ืคึฟืึทืจ ืืขืืขืจ ืงืึทื ืึทื.
ืื ืืขืจ ืงืึทื ืืืื ืขืจ ืขืก ืืื ืื ืื ืืขืจ ืงืึทื ืคืืืืขืจืืืฉืึทื ืืขืงืข - logstash.yml. ืืืจ ืึธื ืจืืจื ืขืก ื ืืฉื, ืืืจ ื ืืฆื ืขืก ืืื ืขืก ืืื.
ืึทืืื, ืืื ืืืขืจ ืืืขืืืืืึทืืขืจ ืกืืจืืงืืืจ:
ืฆื ืืึทืงืืืขื ืึทืจืืึทื ืฉืจืืึทื ืืึทืื, ืืืฆื ืืืจ ืืืขืจื ืขืืขื ืึทื ืืึธืก ืืื tcp ืืืืฃ ืคึผืึธืจื 5046, ืืื ืคึฟืึทืจ ืคึผืจืึธืืืงืฆืืข ืืืจ ืืืขืื ื ืืฆื ืกืืืึธืื.
ืืึธ ืืื ืึท ืคึผืฉืื ืงืึทื ืคืืืืขืจืืืฉืึทื ืคึฟืึทืจ ืืขืจ ืขืจืฉืืขืจ ืงืึทืืขืจ. ืืืืึทื ืื ืขืจืฉื ืึทืจืืขื ืืื ืฆื ืงืึทืืขืจ.
ืึทืืื, ืืืจ ืืึธืื ืืขื docker-compose.yml
version: '3'
networks:
elk:
volumes:
elasticsearch:
driver: local
services:
logstash:
container_name: logstash_one_channel
image: docker.elastic.co/logstash/logstash:6.3.2
networks:
- elk
ports:
- 5046:5046
volumes:
- ./config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
- ./config/pipelines:/usr/share/logstash/config/pipelines:ro
ืืืึธืก ืืึธื ืืืจ ืืขื ืืึธ?
- ื ืขืืืืึธืจืงืก ืืื ืืืึทืืืืื ืืขื ืขื ืืขื ืืืขื ืคืื ืืขืจ ืึธืจืืืื ืขื docker-compose.yml (ืืขืจ ืืืื ืขืจ ืืื ืื ืืื ืฆืข ืึธื ืืืืื ืืื ืืึธื ืืฉื) ืืื ืืื ืืจืึทืืื ืึทื ืืื ืืึธื ื ืื ืืืืขืจ ืืืืจืงื ืื ืงืืืืขืืืืง ืืืื ืืึธ.
- ืืืจ ืฉืึทืคึฟื ืืืื ืืึธืืกืืืฉ ืืื ืกื (s) ืคึฟืื ืื docker.elastic.co/logstash/logstash:6.3.2 ืืืื ืืื ื ืึธืืขื ืขืก logstash_one_channel.
- ืืืจ ืคึฟืึธืจืืืขืจืืก ืคึผืึธืจื 5046 ืืื ืืขื ืงืึทื ืืืื ืขืจ, ืฆื ืืขืจ ืืขืืืืงืขืจ ืื ืขืจืืขื ืคึผืึธืจื.
- ืืืจ ืืึทืคึผืข ืืื ืืืขืจ ืจืขืจ ืงืึทื ืคืืืืขืจืืืฉืึทื ืืขืงืข ./config/pipelines.yml ืฆื ืืขืจ ืืขืงืข /usr/share/logstash/config/pipelines.yml ืืื ืืขื ืงืึทื ืืืื ืขืจ, ืืื ืืึธืืกืืึทืฉ ืืืขื ืงืืืึทืื ืขืก ืืื ืืึทืื ืขืก ืืืืื ืืืืขื ืขื, ืืื ืคืึทื.
- ืืืจ ืืึทืคึผืก ืื ./config/pipelines ืืืขืืืืืึทืืขืจ, ืืื ืืืจ ืืึธืื ืืขืงืขืก ืืื ืงืึทื ืึทื ืกืขืืืื ืืก, ืืื ืื /usr/share/logstash/config/pipelines ืืืขืืืืืึทืืขืจ ืืื ืืืื ืืึทืื ืขืก ืืืืขื ืขื-ืืืืื.
Pipelines.yml ืืขืงืข
- pipeline.id: HABR
pipeline.workers: 1
pipeline.batch.size: 1
path.config: "./config/pipelines/habr_pipeline.conf"
ืืืื ืงืึทื ืึทื ืืื ืื HABR ืืืขื ืืืคืืขืจ ืืื ืืขืจ ืืจื ืฆื ืืืื ืงืึทื ืคืืืืขืจืืืฉืึทื ืืขืงืข ืืขื ืขื ืืืกืงืจืืืื ืืึธ.
ืืื ืืขืกืึธืฃ ืื ืืขืงืข "./config/pipelines/habr_pipeline.conf"
input {
tcp {
port => "5046"
}
}
filter {
mutate {
add_field => [ "habra_field", "Hello Habr" ]
}
}
output {
stdout {
}
}
ืืึธืืืจ ื ืืฉื ืืืื ืืื ืืืื ืืึทืฉืจืืึทืืื ื ืคึฟืึทืจ ืืืฆื, ืืึธืืืจ ืคึผืจืืืืจื ืฆื ืืืืคื ืขืก:
docker-compose up
ืืืึธืก ืืึธื ืืืจ ืืขื?
ืืขืจ ืงืึทื ืืืื ืขืจ ืืื ืื ืืขืืืืื. ืืืจ ืงืขื ืขื ืงืึธื ืืจืึธืืืจื ืืืึทื ืึธืคึผืขืจืึทืฆืืข:
echo '13123123123123123123123213123213' | nc localhost 5046
ืืื ืืืจ ืืขื ืื ืขื ืืคืขืจ ืืื ืื ืงืึทื ืืืื ืขืจ ืงืึทื ืกืึธืื:
ืึธืืขืจ ืืื ืืขืจ ืืขืืืืงืขืจ ืฆืืื, ืืืจ ืืืื ืืขื:
logstash_one_channel | [2019-04-29T11:28:59,790][ERROR][logstash.licensechecker.licensereader] ื ืื ืืขืงืขื ื ืฆืืจืืงืงืจืืื ืืขืจืืืืืขื ืืฉ ืืื ืคึฟืึธืจืืึทืฆืืข ืคืื โโืืขืจ ืืขืจืืืืืขื ืืฉ ืกืขืจืืืขืจ {:message=>โElasticsearch Unreachable: [http://elasticsearch:9200/][Manticore :: ResolutionFailure] elasticsearch", ...
logstash_one_channel | [2019-04-29T11:28:59,894][INFO ][logstash.pipeline ] ืคึผืืืคึผืืืื ืกืืึทืจืืขื ืืฆืืื {:pipeline_id=>.monitoring-logstash", :thread=>"# "}
logstash_one_channel | [2019-04-29T11:28:59,988][INFO ][logstash.agent ] ืคึผืืืคึผืืืื ื ืคืืืกื ืืืง {:count=>2, :running_pipelines=>[:HABR,:.monitoring-logstash"], :non_running_pipelines=>[ ]}
logstash_one_channel | [2019-04-29T11:29:00,015][ERROR][logstash.inputs.metrics] X-Pack ืืื ืืื ืกืืึทืืืจื ืืืืฃ Logstash ืึธืืขืจ ื ืืฉื ืืืืฃ Elasticsearch. ืืืืข ืื ืกืืึทืืืจื X-Pack ืืืืฃ Elasticsearch ืฆื ื ืืฆื ืื ืืึธื ืืืึธืจืื ื ืฉืืจืื. ืื ืืขืจืข ืคึฟืขืึดืงืืืื ืงืขื ืืืื ืืืจืขืืืืื.
logstash_one_channel | [2019-04-29T11:29:00,526][INFO ][logstash.agent] ืืฆืืื ืกืืึทืจืืขื Logstash API ืขื ืืคึผืืื ื {:port=>9600}
logstash_one_channel | [2019-04-29T11:29:04,478][INFO ][logstash.outputs.elasticsearch] ืคืืืกื ืืืง ืืขืืื ื ืืฉืขืง ืฆื ืืขื ืืืื ืึทื Elasticsearch ืงืฉืจ ืืื ืืจืืขืื {:healthcheck_url=>http://elasticsearch:9200/, :path=> "/"}
logstash_one_channel | [2019-04-29T11:29:04,487][WARN ][logstash.outputs.elasticsearch] ืคึผืจืืืื ืฆื ืจืขืืขืจืขืงื ืื ืงืฉืจ ืฆื ืืืื ES ืืืึทืฉืคึผืื, ืึธืืขืจ ืึท ืืขืืช. {: URL=>":9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch"}
logstash_one_channel | [2019-04-29T11:29:04,704][INFO ][logstash.licensechecker.licensereader] ืคืืืกื ืืืง ืืขืืื ื ืืฉืขืง ืฆื ืืขื ืืืื ืึทื Elasticsearch ืงืฉืจ ืืื ืืจืืขืื {:healthcheck_url=>http://elasticsearch:9200/, :path=> "/"}
logstash_one_channel | [2019-04-29T11:29:04,710][WARN ][logstash.licensechecker.licensereader] ืคึผืจืืืื ืฆื ืจืขืืขืจืขืงื ืื ืงืฉืจ ืฆื ืืืื ES ืืืึทืฉืคึผืื, ืึธืืขืจ ืึท ืืขืืช. {: URL=>":9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch"}
ืืื ืืื ืืืขืจ ืงืืึธืฅ ืืื ืงืจืืคึผืื ื ืึทืจืืืฃ ืึทืืข ืื ืฆืืึทื.
ืืึธ ืืื ืืืืืืืืื ืืื ืืจืื ืื ืึธื ืืึธื ืึทื ืื ืจืขืจื - ืืื ืืข ืืื ืืึธื ืืฉื ืืฆืืื, ืืื ืจืืื ืื ืืขืืช ืึธื ืืึธื ืืื ืืื ืืขื ืื ืึธื ืืึธื ืืืขืื ืึทื ืคึผืจืืืื ืฆื ืงืึธื ืืึทืงื : ืงืกื ืืืงืก.
ืืึธืก ืืึทืคึผืึทื ื ืืืืึทื logstash.conf, ืึทืจืืึทื ืืขืจืขืื ื ืืื ืื ืืืื, ืึผืืื ืึท ืืฉืขืง ืคึฟืึทืจ ืึทืืืืืืึทืืืืึทืื ืคืื Elasticsearch. ื ืึธื ืึทืืข, ืืึธืืกืืึทืฉ ืึทืกืืื ืึทื ืขืก ืึทืจืืขื ืืื ืึท ืืืื ืคืื ืื ืขืืง ืึธื ืืืืื, ืึธืืขืจ ืืืจ ืืคืืขืฉืืืื ืขืก.
ืขืก ืืื ืืขืืืขื ืฆื ืึทืจืืขืื, ืึธืืขืจ ืขืก ืืื ื ืืฉื ืืึทืงืืืขื.
ืื ืืืืืื ื ืืื ืฆื ืืืกืืืืึทื ืืขื ืืฉืขืง ืืืจื ืื XPACK_MONITORING_ENABLED ืกืืืืืืข ืืืึทืืขืืืืืง.
ืืึธืืืจ ืืึทืื ืึท ืขื ืืขืจืื ื ืฆื docker-compose.yml ืืื ืืืืคื ืขืก ืืืืืขืจ:
version: '3'
networks:
elk:
volumes:
elasticsearch:
driver: local
services:
logstash:
container_name: logstash_one_channel
image: docker.elastic.co/logstash/logstash:6.3.2
networks:
- elk
environment:
XPACK_MONITORING_ENABLED: "false"
ports:
- 5046:5046
volumes:
- ./config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
- ./config/pipelines:/usr/share/logstash/config/pipelines:ro
ืืืฆื, ืึทืืฅ ืืื ืืื. ืืขืจ ืงืึทื ืืืื ืขืจ ืืื ืืจืืื ืคึฟืึทืจ ืืงืกืคึผืขืจืึทืืึทื ืฅ.
ืืืจ ืงืขื ืขื ืืจืืงื ืืืืืขืจ ืืื ืืขืจ ืืืืึทืืขืจ ืงืึทื ืกืึธืื:
echo '13123123123123123123123213123213' | nc localhost 5046
ืืื ืืขื:
logstash_one_channel | {
logstash_one_channel | "message" => "13123123123123123123123213123213",
logstash_one_channel | "@timestamp" => 2019-04-29T11:43:44.582Z,
logstash_one_channel | "@version" => "1",
logstash_one_channel | "habra_field" => "Hello Habr",
logstash_one_channel | "host" => "gateway",
logstash_one_channel | "port" => 49418
logstash_one_channel | }
ืึทืจืืขื ืืื ืืืื ืงืึทื ืึทื
ืึทืืื ืืืจ ืืึธื ืืฉื. ืืืฆื ืืืจ ืงืขื ืขื ืืึทืงืข ื ืขืืขื ืื ืฆืืื ืฆื ืงืึทื ืคืืืืขืจ ืืึธืืกืืึทืฉ ืืื. ืืึธืืืจ ื ืืฉื ืึธื ืจืืจื ืื pipelines.yml ืืขืงืข ืคึฟืึทืจ ืืืฆื, ืืึธืืืจ ืืขื ืืืึธืก ืืืจ ืงืขื ืขื ืืึทืงืืืขื ืืืจื ืืจืืขืื ืืื ืืืื ืงืึทื ืึทื.
ืืื ืืืื ืืึธืื ืึทื ืืขืจ ืืขื ืขืจืึทื ืคึผืจืื ืฆืืคึผ ืคืื ืืจืืขืื ืืื ืื ืงืึทื ืึทื ืงืึทื ืคืืืืขืจืืืฉืึทื ืืขืงืข ืืื ืืขืืื ื ืืืกืงืจืืืื ืืื ืืขืจ ืืึทืึทืืืขืจ ืืึทื ืืึทื ืืึธ
ืืืื ืืืจ ืืืืื ืืืืขื ืขื ืืืืฃ ืจืืกืืฉ, ืืืจ ืืขืืืืื ื ืืขื (ืึธืืขืจ ืื ืึธื ืคึฟืจืขื ืกืื ืืึทืงืก ืืึธืจื ืืื ืึทืื, ืืืจ ืืึทืจืคึฟื ืฆื ื ืขืืขื ืืขื ืืื ืืฉืืื).
ืืื ืก ืืืื ืกืึทืงืืืขื ืืฉืึทืื ืคืื ืื ืึทืจืืึทื ืฉืจืืึทื ืึธืคึผืืืืืื ื. ืืืจ ืืึธืื ืฉืืื ืืขืืขื ืึทืจืืขื ืืืืฃ TCP. ืืืึธืก ืึทื ืืขืจืฉ ืงืขื ืืืื ืืฉืืงืึทืืืข ืืึธ?
ืคึผืจืืืืจื ืึทืจืืืงืืขื ื ืืฆื ืืึทืจืืืื
ืขืก ืืื ืึทืืึท ืึท ืืฉืืงืึทืืืข ืืขืืขืื ืืืื ืฆื ืืืฉืขื ืขืจืืื ืึธืืึทืืึทืืืง ืคึผืจืืืืจื ืึทืจืืืงืืขื.
ืฆื ืืึธื ืืึธืก, ืืืจ ืืึทืจืคึฟื ืฆื ืืขืื ืื ืืขืึทืจืืืขืึทื ืคึผืืืืื ืืื ืื ืึทืจืืึทื ืฉืจืืึทื ืึธืคึผืืืืืื ื.
input {
heartbeat {
message => "HeartBeat!"
}
}
ืงืขืจ ืขืก ืืืืฃ, ืึธื ืืืื ืจืืกืืืืื ื ืึทืืึธื ืึท ืืื ืื
logstash_one_channel | {
logstash_one_channel | "@timestamp" => 2019-04-29T13:52:04.567Z,
logstash_one_channel | "habra_field" => "Hello Habr",
logstash_one_channel | "message" => "HeartBeat!",
logstash_one_channel | "@version" => "1",
logstash_one_channel | "host" => "a0667e5c57ec"
logstash_one_channel | }
ืืืื ืืืจ ืืืืื ืฆื ืืึทืงืืืขื ืืขืจ ืึธืคื, ืืืจ ืืึทืจืคึฟื ืฆื ืืืืื ืื ืื ืืขืจืืืึทื ืคึผืึทืจืึทืืขืืขืจ.
ืึทืืื ืืืจ ืืืขืื ืืึทืงืืืขื ืึท ืึธื ืืึธื ืืขืืขืจ 10 ืกืขืงืื ืืขืก.
input {
heartbeat {
message => "HeartBeat!"
interval => 10
}
}
ืจืืืจืืืืื ื ืืึทืื ืคืื ืึท ืืขืงืข
ืืืจ ืืืื ืืึทืฉืืึธืกื ืฆื ืงืืงื ืืื ืื ืืขืงืข ืืึธืืข. ืืืื ืขืก ืึทืจืืขื ืคืืึทื ืืื ืืขืจ ืืขืงืข, ืืึธืืขืจ ืงืืื ืึทืืขื ื ืืื ืืืจืฃ, ืืคึผืืืช ืคึฟืึทืจ ืืืืข ื ืืฆื.
ืืืื ืื ืืึทืฉืจืืึทืืื ื, ืื ืึทืคึผืขืจืืืืื ื ืืึธืืข ืืึธื ืืืื ืขื ืืขื ืฆื ืขืง -f, ื.ืข. ืืืืขื ื ื ืืึทืข ืฉืืจืืช ืึธืืขืจ, ืืื ืึทื ืึธืคึผืฆืืข, ืืืืขื ื ืื ืืื ืฆืข ืืขืงืข.
ืึทืืื ืืืึธืก ืืืจ ืืืืื ืฆื ืืึทืงืืืขื:
- ืืืจ ืืืืื ืฆื ืืึทืงืืืขื ืฉืืจืืช ืืืึธืก ืืขื ืขื ืึทืคึผืคึผืขื ืืขื ืฆื ืืืื ืงืืึธืฅ ืืขืงืข.
- ืืืจ ืืืืื ืฆื ืืึทืงืืืขื ืืึทืื ืืืึธืก ืืขื ืขื ืืขืฉืจืืื ืฆื ืขืืืขืืข ืืึธื ืืขืงืขืก, ืืฉืขืช ืืืจ ืงืขื ืขื ืึธืคึผืืืืื ืืืึธืก ืืื ืืืงืืืขื ืคืื ืืื.
- ืืืจ ืืืืื ืฆื ืืึทืื ืืืืขืจ ืึทื ืืืขื ืืึธืืกืืึทืฉ ืืื ืจืืกืืึทืจืืื, ืขืก ืืืขื ื ืืฉื ืืึทืงืืืขื ืื ืืึทืื ืืืืืขืจ.
- ืืืจ ืืืืื ืฆื ืงืึธื ืืจืึธืืืจื ืึทื ืืืื ืืึธืืกืืึทืฉ ืืื ืืืืกืืขืืจืืื ืึทืืืขืง ืืื ืืึทืื ืคืึธืจืืขืฆื ืฆื ืืืื ืืขืฉืจืืื ืฆื ืืขืงืขืก, ืืืขื ืืืจ ืืืืคื ืขืก, ืืืจ ืืืขืื ืืึทืงืืืขื ืื ืืึทืื.
ืฆื ืึธื ืคืืจื ืืขื ืขืงืกืคึผืขืจืืืขื ื, ืืึธืื ืืื ืื ืืืืื ืื ืื ืืขืจ ืฉืืจื ืฆื docker-compose.yml, ืขืคื ืื ืืืขืืืืืึทืืขืจ ืืื ืืืึธืก ืืืจ ืฉืืขืื ืื ืืขืงืขืก.
version: '3'
networks:
elk:
volumes:
elasticsearch:
driver: local
services:
logstash:
container_name: logstash_one_channel
image: docker.elastic.co/logstash/logstash:6.3.2
networks:
- elk
environment:
XPACK_MONITORING_ENABLED: "false"
ports:
- 5046:5046
volumes:
- ./config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
- ./config/pipelines:/usr/share/logstash/config/pipelines:ro
- ./logs:/usr/share/logstash/input
ืืื ืืืืฉื ืื ืึทืจืืึทื ืฉืจืืึทื ืึธืคึผืืืืืื ื ืืื habr_pipeline.conf
input {
file {
path => "/usr/share/logstash/input/*.log"
}
}
ืืืืืจ ืื ืืืืื:
docker-compose up
ืฆื ืฉืึทืคึฟื ืืื ืฉืจืืึทืื ืงืืึธืฅ ืืขืงืขืก ืืืจ ืืืขืื ื ืืฆื ืืขื ืืึทืคึฟืขื:
โจecho '1' >> logs/number1.log
{
logstash_one_channel | "host" => "ac2d4e3ef70f",
logstash_one_channel | "habra_field" => "Hello Habr",
logstash_one_channel | "@timestamp" => 2019-04-29T14:28:53.876Z,
logstash_one_channel | "@version" => "1",
logstash_one_channel | "message" => "1",
logstash_one_channel | "path" => "/usr/share/logstash/input/number1.log"
logstash_one_channel | }
ืืึธ, ืขืก ืึทืจืืขื!
ืืื ืืขืจ ืืขืืืืงืขืจ ืฆืืื, ืืืจ ืืขื ืึทื ืืืจ ืืึธืื ืืืืืึธืืึทืืืฉ ืฆืืืขืืขืื ืืขื ืืจื ืคืขืื. ืืึธืก ืืืื ื ืึทื ืืื ืืขืจ ืฆืืงืื ืคึฟื, ืืืจ ืืืขืื ืงืขื ืขื ืฆื ืคืืืืขืจ ืจืขืงืึธืจืืก ืืืจื ืขืก.
ืืืืืจ ืคืจืืืืจื ื ืืืืืื:
echo '2' >> logs/number1.log
{
logstash_one_channel | "host" => "ac2d4e3ef70f",
logstash_one_channel | "habra_field" => "Hello Habr",
logstash_one_channel | "@timestamp" => 2019-04-29T14:28:59.906Z,
logstash_one_channel | "@version" => "1",
logstash_one_channel | "message" => "2",
logstash_one_channel | "path" => "/usr/share/logstash/input/number1.log"
logstash_one_channel | }
ืืื ืืืฆื ืฆื ืื ืื ืืขืจ ืืขืงืข:
echo '1' >> logs/number2.log
{
logstash_one_channel | "host" => "ac2d4e3ef70f",
logstash_one_channel | "habra_field" => "Hello Habr",
logstash_one_channel | "@timestamp" => 2019-04-29T14:29:26.061Z,
logstash_one_channel | "@version" => "1",
logstash_one_channel | "message" => "1",
logstash_one_channel | "path" => "/usr/share/logstash/input/number2.log"
logstash_one_channel | }
ืืจืืืก! ืืขืจ ืืขืงืข ืืื ืคึผืืงื ืืื, ืืขืจ ืืจื ืืื ืืขืืืขื ืจืืืืืง ืกืคึผืขืกืืคืืขื, ืึทืืฅ ืืื ืืื.
ืืึทืืื ืืึธืืกืืึทืฉ ืืื ืึธื ืืืื ืืืืืขืจ. ืืึธืืืจ ืืืึทืจืื. ืฉืืืืงืืึทื. ืืขื ืข. ืืืจ ืืึทืงืืืขื ื ืืฉื ืื ืจืขืงืึธืจืืก ืืืืืขืจ.
ืืื ืืืฆื ืื ืืขืจืกื ืืขืจืื ื ืขืงืกืคึผืขืจืืืขื ื.
ืื ืกืืึทืืืจื ืืึธืืกืืืฉ ืืื ืืืกืคืืจื:
echo '3' >> logs/number2.log
echo '4' >> logs/number1.log
ืืืืคื ืืึธืืกืืึทืก ืืืืืขืจ ืืื ืืขื:
logstash_one_channel | {
logstash_one_channel | "host" => "ac2d4e3ef70f",
logstash_one_channel | "habra_field" => "Hello Habr",
logstash_one_channel | "message" => "3",
logstash_one_channel | "@version" => "1",
logstash_one_channel | "path" => "/usr/share/logstash/input/number2.log",
logstash_one_channel | "@timestamp" => 2019-04-29T14:48:50.589Z
logstash_one_channel | }
logstash_one_channel | {
logstash_one_channel | "host" => "ac2d4e3ef70f",
logstash_one_channel | "habra_field" => "Hello Habr",
logstash_one_channel | "message" => "4",
logstash_one_channel | "@version" => "1",
logstash_one_channel | "path" => "/usr/share/logstash/input/number1.log",
logstash_one_channel | "@timestamp" => 2019-04-29T14:48:50.856Z
logstash_one_channel | }
ืืืจืื! ืืืขืก ืืื ืืืืคืืขื ืืืขื ืืขืืืืจื.
ืืืขืจ ืืืจ ืืืื ืืืึธืจืขื ืขื ืืืจ ืืืขืื ืื ืคืืืืขื ืืข. ืืืื ืื ืืึธืืกืืึทืฉ ืงืึทื ืืืื ืขืจ ืืื ืืืืกืืขืืขืงื (ืืึธืงืงืขืจ ืืึทืืื logstash_one_channel && docker rm logstash_one_channel), ืืึธืจื ืืฉื ืืืขื ืืืื ืคึผืืงื ืึทืจืืืฃ. ืื ืฉืืขืืข ืคืื โโืืขืจ ืืขืงืข ืฆื ืืืึธืก ืขืก ืืื ืืขืืืขื ืืืืขื ืขื ืืื ืกืืึธืจื ืืื ืืขื ืงืึทื ืืืื ืขืจ. ืืืื ืืืจ ืืืืคื ืขืก ืคึฟืื ืงืจืึทืฆื, ืขืก ืืืขื ื ืึธืจ ืึธื ื ืขืืขื ื ืืึทืข ืฉืืจืืช.
ืืืืขื ืขื ืืืืืกืืื ื ืืขืงืขืก
ืืื ืก ืืึธืื ืืืจ ืืขื ืขื ืืึธื ืืฉืื ื ืืึธืืกืืึทืฉ ืคึฟืึทืจ ืื ืขืจืฉืืขืจ ืืึธื, ืึธืืขืจ ืืืจ ืืึธืื ืฉืืื ืืึธืืก ืืื ืืืจ ืืืึธืื ืืื ืฆื ืคึผืจืึธืฆืขืก ืืื.
ืืืื ืืืจ ืืืืคื ืืึธืืกืืืฉ ืืื ืื ืึทืจืืึทื ืฉืจืืึทื ืึธืคึผืืืืืื ื ืืืึธืก ืืืจ ืืขืืืืื ื ืืืืื, ืืืจ ืืืขืื ืืึทืงืืืขื ืืึธืจื ืืฉื. ืืืืื ื ืืึทืข ืฉืืจืืช ืืืขื ืืืื ืคึผืจืึทืกืขืกื ืืืจื ืืึธืืกืืึทืฉ.
ืึผืื ืื ืฉืืจืืช ืคืื ืืืืืกืืื ื ืืขืงืขืก ืืึธื ืืืื ืคึผืืื ืึทืจืืืฃ, ืืืจ ืืึธื ืืืืื ืึทื ื ืึธื ืฉืืจื ืฆื ืื ืึทืจืืึทื ืฉืจืืึทื ืึธืคึผืืืืืื ื:
input {
file {
start_position => "beginning"
path => "/usr/share/logstash/input/*.log"
}
}
ืืขืจืฆื, ืขืก ืืื ืึท ื ืืึทื ืก: ืืึธืก ืืืืื ืึทืคืขืงืฅ ื ืืึทืข ืืขืงืขืก ืืืึธืก ืืึธืืกืืึทืฉ ืืื ื ืืฉื ื ืึธื ืืขืืขื. ืคึฟืึทืจ ืื ืืขืืืข ืืขืงืขืก ืืืึธืก ืืขื ืขื ืฉืืื ืืื ืื ืคืขืื ืคืื ืืืื ืื ื ืคืื ืืึธืืกืืึทืฉ, ืขืก ืืื ืฉืืื ืืขืจืืื ื ืืืืขืจ ืืจืืืก ืืื ืืืขื ืืืฆื ื ืึธืจ ื ืขืืขื ื ืืึท ืืืื ืกื ืืื ืืื.
ืืื ืก ืืึทืืื ืืึธ ืืื ืืขืจื ืขื ืื ืึทืจืืึทื ืฉืจืืึทื ืึธืคึผืืืืืื ื. ืขืก ืืขื ืขื ื ืึธื ืคืืืข ืึธืคึผืฆืืขืก, ืึธืืขืจ ืืึธืก ืืื ืืขื ืื ืคึฟืึทืจ ืืื ืื ืคึฟืึทืจ ืืืืึทืืขืจ ืืงืกืคึผืขืจืึทืืึทื ืฅ ืืืฆื.
ืจืืืื ื ืืื ืืึทืืึท ืืจืึทื ืกืคืึธืจืืึทืฆืืข
ืืื ืก ืคึผืจืืืืจื ืฆื ืกืึธืืืืข ืื ืคืืืืขื ืืข ืคึผืจืึธืืืขื, ืืึธืื ืก ืืึธืื ืืืจ ืืึธืื ืึทืจืืืงืืขื ืคืื ืืืื ืงืึทื ืึทื, ืขืืืขืืข ืคืื โโืืื ืืขื ืขื ืื ืคืึธืจืืึทืืืื, ืืื ืขืืืขืืข ืืขื ืขื ืืขืืช ืึทืจืืืงืืขื. ืืื ืึทื ืืขืจืฉ ืืืจื ืงืืืืื. ืขืืืขืืข ืืขื ืขื ืืื ืคึฟืึธืจืืึทืฆืืข, ืื ืืขืจืข ืืขื ืขื ืืขืืช.
ืืืจ ืืึทืจืคึฟื ืฆื ืืึทืืื ืืขืจ ืืื ืืืึท ืื ืึทืจืืืกืืึทื ื. ืืขื ืข. ืืืจ ืฉืจืืึทืื ืืื ืคึฟืึธืจืืึทืฆืืข ืึทืจืืืงืืขื ืืื ืืืื ืงืึทื ืึทื, ืืื ืืขืืช ืึทืจืืืงืืขื ืืื ืื ืืขืจื.
ืฆื ืืึธื ืืึธืก, ืืึทื ืคืื ืื ืึทืจืืึทื ืฉืจืืึทื ืึธืคึผืืืืืื ื ืฆื ืคืืืืขืจ ืืื ืจืขืืืืืึทื.
ื ืืฆื ืื ืคืืืืขืจ ืึธืคึผืืืืืื ื, ืืืจ ืืืขืื ืคึผืึทืจืก ืื ืื ืงืึทืืื ื ืึธื ืืึธื, ืงืจืืื ืึท ืืึทืฉ (ืฉืืืกื-ืืืขืจื ืคึผืขืจื), ืืืึธืก ืืืจ ืงืขื ืขื ืฉืืื ืึทืจืืขืื ืืื, ื.ื. ืืืกืึทืกืขืืืึทื ืืืื ืฆื ืืืืื ืืื ืืขื. ืืื ืืื ืืขืจ ืจืขืืืืืึทื ืึธืคึผืืืืืื ื, ืืืจ ืืืขืื ืืืืกืงืืืึทืื ืึทืจืืืงืืขื ืืื ืฉืืงื ืืขืืขืจ ืฆื ืืืื ืืืืืขื ืข ืงืึทื ืึทื.
ืคึผืึทืจืกืื ื ืึท ืึธื ืืึธื ืืื ืืจืึธืง
ืืื ืกืืจ ืฆื ืคึผืึทืจืก ืืขืงืกื ืกืืจืื ืืก ืืื ืืึทืงืืืขื ืึท ืกืืื ืคืื ืคืขืืืขืจ ืคึฟืื ืืื, ืขืก ืืื ืึท ืกืคึผืขืฆืืขื ืคึผืืืืื ืืื ืื ืคืืืืขืจ ืึธืคึผืืืืืื ื - grok.
ืึธื ืืึทืฉืืขืืืงื ืืื ืืขื ืฆืื ืฆื ืืขืื ืึท ืืืืืืื ืืึทืฉืจืืึทืืื ื ืคืื ืขืก ืืึธ (ืคึฟืึทืจ ืืขื ืืื ืึธืคึผืฉืืงื ืฆื ), ืืื ืืืขื ืืขืื ืืืื ืคึผืฉืื ืืืึทืฉืคึผืื.
ืฆื ืืึธื ืืึธืก, ืืืจ ืืึทืจืคึฟื ืฆื ืืึทืฉืืืกื ืื ืคึฟืึธืจืืึทื ืคืื ืื ืึทืจืืึทื ืฉืจืืึทื ืกืืจืื ืืก. ืืื ืืึธืื ืืื ืืื ืืึธืก:
1 ืืื ืคึฟืึธืจืืึทืฆืืข ืึธื ืืึธื 1
2 ืืขืืช ืึธื ืืึธื 2
ืืขื ืข. ืืขืจ ืืืืขื ืืืคืืฆืขืจ ืงืืื ืขืจืฉืืขืจ, ืืขืืึธืื INFO / ืืขืืช, ืืื ืขืืืขืืข ืืืึธืจื ืึธื ืกืคึผืืืกืึทื.
ืขืก ืืื ื ืืฉื ืฉืืืขืจ, ืึธืืขืจ ืขืก ืืื ืืขื ืื ืฆื ืคึฟืึทืจืฉืืืื ืืขื ืคึผืจืื ืฆืืคึผ ืคืื ืึธืคึผืขืจืึทืฆืืข.
ืึทืืื, ืืื ืื ืคืืืืขืจ ืึธืคึผืืืืืื ื ืคืื ืื ืืจืึธืง ืคึผืืืืื, ืืืจ ืืืื ืืขืคืื ืืจื ืึท ืืืกืืขืจ ืคึฟืึทืจ ืคึผืึทืจืกืื ื ืืื ืืืขืจ ืกืืจืื ืืก.
ืขืก ืืืขื ืงืืงื ืืื ืืึธืก:
filter {
grok {
match => { "message" => ["%{INT:message_id} %{LOGLEVEL:message_type} %{WORD:message_text}"] }
}
}
ืืกืขื ืฉืึทืื ืขืก ืก ืึท ืจืขืืืืขืจ ืืืืกืืจืืง. ืืจืืื-ืืขืืืื ืคึผืึทืืขืจื ื ืืขื ืขื ืืขื ืืฆื, ืึทืืึท ืืื INT, LOGLEVEL, WORD. ืืืืขืจ ืืึทืฉืจืืึทืืื ื, ืืื ืืขืืื ื ืืื ืื ืืขืจืข ืคึผืึทืืขืจื ื, ืงืขื ืขื ืืืื ืืขืคึฟืื ืขื ืืึธ
ืืืฆื, ืืืจืืืขืืื ืืขื ืืืจื ืืขื ืคืืืืขืจ, ืืื ืืืขืจ ืฉืืจืืงื ืืืขื ืืืื ืึท ืืึทืฉ ืคืื ืืจืื ืคืขืืืขืจ: message_id, message_type, message_text.
ืืื ืืืขืื ืืืื ืืขืืืืื ืืื ืื ืจืขืืืืืึทื ืึธืคึผืืืืืื ื.
ืจืืืื ื ืึทืจืืืงืืขื ืฆื ืืขืจ ืจืขืืืืืึทื ืึธืคึผืืืืืื ื ื ืืฆื ืื ืืืื ืืึทืคึฟืขื
ืืื ืืขืจ ืจืขืืืืืึทื ืึธืคึผืืืืืื ื, ืืื ืืืจ ืืขืืขื ืงืขื, ืืืจ ืืขื ืขื ืืขืืื ืืขื ืฆื ืฉืคึผืึทืืื ืื ืึทืจืืืงืืขื ืืื ืฆืืืื ืกืืจืืื. ืขืืืขืืข - ืืืึธืก ืืขื ืขื ืื ืคืึธ, ืืืขื ืืืื ืจืขืืืืืึทื ืฆื ืื ืงืึทื ืกืึธืื, ืืื ืืื ืขืจืจืึธืจืก, ืืืจ ืืืขืื ืจืขืืืืืึทื ืฆื ืึท ืืขืงืข.
ืืื ืืึธื ืืืจ ืืึทืืื ืืขืจ ืื ืึทืจืืืงืืขื? ืืขืจ ืฆืืฉืืึทื ื ืคืื ืืขื ืคึผืจืึธืืืขื ืฉืืื ืกืึทืืืืฉืขืกืฅ ืึท ืืืืืื ื - ื ืึธื ืึทืืข, ืืืจ ืืึธืื ืฉืืื ืึท ืืขืืึทืงืืืืึทื ืึธื ืืึธื_ืืืคึผ ืคืขืื, ืืืึธืก ืงืขื ืขื ื ืึธืจ ื ืขืืขื ืฆืืืื ืืืึทืืืขืก: INFO ืืื ERROR. ืขืก ืืื ืืืืฃ ืืขื ืืงืขืจ ืึทื ืืืจ ืืืขืื ืืึทืื ืึท ืืจืืจื ื ืืฆื ืื ืืืื ืืืกืืึธืืื ื.
if [message_type] == "ERROR" {
# ะะดะตัั ะฒัะฒะพะดะธะผ ะฒ ัะฐะนะป
} else
{
# ะะดะตัั ะฒัะฒะพะดะธะผ ะฒ stdout
}
ื ืืึทืฉืจืืึทืืื ื ืคืื ืืจืืขืื ืืื ืคืขืืืขืจ ืืื ืึธืคึผืขืจืืืืขืจื ืงืขื ืขื ืืืื ืืขืคึฟืื ืขื ืืื ืืขื ืึธืคึผืืืืืื ื .
ืืืฆื, ืืืขืื ืื ืคืึทืงืืืฉ ืืกืงื ื ืืื.
ืงืึทื ืกืึธืื ืจืขืืืืืึทื, ืึทืืฅ ืืื ืงืืึธืจ ืืึธ - ืกืืืึธืื {}
ืึธืืขืจ ืืขืจ ืจืขืืืืืึทื ืฆื ืึท ืืขืงืข - ืืขืืขื ืงื ืึทื ืืืจ ืืืืคื ืึทืืข ืืขื ืคึฟืื ืึท ืงืึทื ืืืื ืขืจ ืืื ืึทืืื ืึทื ืื ืืขืงืข ืืื ืืืึธืก ืืืจ ืฉืจืืึทืื ืื ืจืขืืืืืึทื ืืื ืฆืืืจืืืืขื ืคึฟืื ืึทืจืืืก, ืืืจ ืืึทืจืคึฟื ืฆื ืขืคึฟืขื ืขื ืืขื ืืืขืืืืืึทืืขืจ ืืื docker-compose.yml.
ืืึทื ืฅ:
ืืขืจ ืจืขืืืืืึทื ืึธืคึผืืืืืื ื ืคืื ืืื ืืืขืจ ืืขืงืข ืงืืงื ืืื ืืึธืก:
โจoutput {
if [message_type] == "ERROR" {
file {
path => "/usr/share/logstash/output/test.log"
codec => line { format => "custom format: %{message}"}
}
} else
{stdout {
}
}
}
ืืื docker-compose.yml ืืืจ ืืืืื ืื ืื ืืขืจ ืืึทื ื ืคึฟืึทืจ ืจืขืืืืืึทื:
version: '3'
networks:
elk:
volumes:
elasticsearch:
driver: local
services:
logstash:
container_name: logstash_one_channel
image: docker.elastic.co/logstash/logstash:6.3.2
networks:
- elk
environment:
XPACK_MONITORING_ENABLED: "false"
ports:
- 5046:5046
volumes:
- ./config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
- ./config/pipelines:/usr/share/logstash/config/pipelines:ro
- ./logs:/usr/share/logstash/input
- ./output:/usr/share/logstash/output
ืืืจ ืงืึทืืขืจ ืขืก, ืคึผืจืืืืจื ืขืก ืืื ืืขื ืึท ืืืื ืืื ืฆืืืื ืกืืจืืื.
ืืงืืจ: www.habr.com