Whakataki
I a matou e tuku ana i tetahi atu punaha, i raru matou ki te tukatuka i te maha o nga momo raarangi rereke. I whiriwhiria a ELK hei taputapu. Ka matapakihia e tenei tuhinga o maatau wheako ki te whakarite i tenei puranga.
Kare matou e whakatakoto whainga ki te whakaahua i ona kaha katoa, engari e hiahia ana matou ki te aro ki te whakaoti rapanga mahi. Ko tenei na te mea ahakoa he nui te nui o nga tuhinga me nga whakaahua kua rite, he maha nga raru, i te mea kua kitea e matou.
I tukuna e matou te puranga ma te docker-compose. I tua atu, he pai te tuhi docker-compose.yml, i taea ai e matou te whakaara i te puranga tata kaore he raru. A ko te ahua ki a matou kua tata kee te wikitoria, inaianei ka whakatikahia e matou kia rite ki o maatau hiahia, katahi ano.
Heoi, ko te ngana ki te whirihora i te punaha ki te whiwhi me te tukatuka i nga raarangi mai i ta maatau tono kaore i angitu. Na reira, i whakatau maatau he pai ki te ako takitahi i ia waahanga, ka hoki ki o raatau hononga.
Na, i timata matou ki te logstash.
Taiao, horahanga, rere Logstash i roto i te ipu
Mo te tohatoha ka whakamahia e matou te docker-compose; ko nga whakamatautau i whakaahuahia i konei i mahia i runga i te MacOS me te Ubuntu 18.0.4.
Ko te ahua logstash i rehitatia ki ta maatau docker-compose.yml he docker.elastic.co/logstash/logstash:6.3.2
Ka whakamahia e matou mo nga whakamatautau.
I tuhia e matou he docker-compose.yml hei whakahaere i te logstash. Ae ra, ka taea te whakarewa i te ahua mai i te raina whakahau, engari kei te whakaoti rapanga motuhake, kei reira ka whakahaerehia e matou nga mea katoa mai i te docker-compose.
He poto mo nga konae whirihoranga
E whai ake nei mai i te whakaahuatanga, ka taea te whakahaere logstash mo te hongere kotahi, me tuku te *.conf file, mo etahi takere ranei, me tuku te konae pipelines.yml, ka mutu , ka hono ki nga konae .conf mo ia hongere.
I haere matou i te ara tuarua. Ko te ahua ki a maatau he nui ake te ao me te tauineine. Na reira, i hanga e matou te pipelines.yml, ka hanga he raarangi pipelines hei whakatakoto i nga konae .conf mo ia hongere.
Kei roto i te ipu tetahi atu konae whirihoranga - logstash.yml. Kaore matou e pa ki a ia, ka whakamahia tonu e matou.
Na, to tatou hanganga whaiaronga:
Ki te whiwhi raraunga whakauru, mo tenei wa ka whakaarohia ko tcp tenei i runga i te tauranga 5046, mo te putanga ka whakamahia e matou te stdout.
Anei he whirihoranga ngawari mo te whakarewatanga tuatahi. No te mea ko te mahi tuatahi ko te whakarewa.
Na, kei a matou tenei docker-compose.yml
version: '3'
networks:
elk:
volumes:
elasticsearch:
driver: local
services:
logstash:
container_name: logstash_one_channel
image: docker.elastic.co/logstash/logstash:6.3.2
networks:
- elk
ports:
- 5046:5046
volumes:
- ./config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
- ./config/pipelines:/usr/share/logstash/config/pipelines:ro
He aha ta tatou e kite nei?
- Ko nga whatunga me nga pukapuka i tangohia mai i te docker-compose.yml taketake (ko te mea i whakarewahia te puranga katoa) a ki taku whakaaro kaore e tino pa ki te pikitia katoa i konei.
- Ka waihangahia e matou tetahi (ng) ratonga logstash mai i te docker.elastic.co/logstash/logstash:6.3.2 whakaahua ka whakaingoatia he logstash_one_channel.
- Tukuna atu te tauranga 5046 ki roto i te ipu, ki te tauranga o roto ano.
- Ka mahere matou i ta matou konae whirihoranga putorino ./config/pipelines.yml ki te konae /usr/share/logstash/config/pipelines.yml i roto i te ipu, ka tikina e te logstash ka panui-anake, mena he take.
- Ka mahere matou i te raarangi ./config/pipelines, kei reira nga konae me nga tautuhinga hongere, ki roto i te raarangi /usr/share/logstash/config/pipelines me te panui-anake.
Pipelines.yml kōnae
- pipeline.id: HABR
pipeline.workers: 1
pipeline.batch.size: 1
path.config: "./config/pipelines/habr_pipeline.conf"
Ko tetahi hongere me te tohu tohu HABR me te ara ki tana konae whirihoranga e whakaahuatia ana i konei.
Ka mutu ko te konae "./config/pipelines/habr_pipeline.conf"
input {
tcp {
port => "5046"
}
}
filter {
mutate {
add_field => [ "habra_field", "Hello Habr" ]
}
}
output {
stdout {
}
}
Kaua e haere ki roto i tana whakaahuatanga mo tenei wa, me ngana ki te whakahaere:
docker-compose up
He aha ta tatou e kite ana?
Kua timata te ipu. Ka taea e taatau te tirotiro i tana mahi:
echo '13123123123123123123123213123213' | nc localhost 5046
A ka kite tatou i te whakautu i te papatohu ipu:
Engari i te wa ano, ka kite ano tatou:
logstash_one_channel | [2019-04-29T11:28:59,790][ERROR][logstash.licensechecker.licensereader] Kaore e taea te tiki korero raihana mai i te tūmau raihana {:message=>“Elasticsearch Kaore e taea te toro: [http://elasticsearch:9200/][Manticore ::ResolutionFailure] elasticsearch", ...
logstash_one_channel | [2019-04-29T11:28:59,894][INFO ][logstash.pipeline ] I timata pai te Pipeline {:pipeline_id="".monitoring-logstash", :thread=>"# "}
logstash_one_channel | [2019-04-29T11:28:59,988][INFO ][logstash.agent ] Paipa rere {:count=>2, :running_pipelines=>[:HABR, :".monitoring-logstash"], :non_running_pipelines=>[ ]}
logstash_one_channel | [2019-04-29T11:29:00,015][HAPA][logstash.inputs.metrics] Kua whakauruhia te X-Pack ki runga Logstash engari kaua ki Elasticsearch. Tena koa whakauruhia te X-Pack ki te Elasticsearch hei whakamahi i te waahanga aroturuki. Kei te waatea etahi atu waahanga.
logstash_one_channel | [2019-04-29T11:29:00,526][INFO ][logstash.agent ] I angitu te tiimata i te waahi mutunga API Logstash {:port=>9600}
logstash_one_channel | [2019-04-29T11:29:04,478][INFO ][logstash.outputs.elasticsearch] Te whakahaere tirotiro hauora kia kite mena kei te mahi tetahi hononga Elasticsearch {:healthcheck_url=>http://elasticsearch:9200/, :path=> "/"}
logstash_one_channel | [2019-04-29T11:29:04,487][WARN][logstash.outputs.elasticsearch] I ngana ki te whakaara hononga ki te tauira ES mate, engari he hapa. {:url =>":9200/", :error_type=>LogStash:: Putanga::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"ElasticsearchKare e taea te toro: [http://elasticsearch:9200/][Manticore::ResolutionFailure] rapunga elastic"}
logstash_one_channel | [2019-04-29T11:29:04,704][INFO ][logstash.licensechecker.licensereader] Te whakahaere tirotiro hauora kia kite mena kei te mahi tetahi hononga Elasticsearch {:healthcheck_url=>http://elasticsearch:9200/, :path=> "/"}
logstash_one_channel | [2019-04-29T11:29:04,710][WARN][logstash.licensechecker.licensereader] I ngana ki te whakaara hononga ki te tauira ES kua mate, engari he hapa. {:url =>":9200/", :error_type=>LogStash:: Putanga::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"ElasticsearchKare e taea te toro: [http://elasticsearch:9200/][Manticore::ResolutionFailure] rapunga elastic"}
A kei te ngokingoki to tatou pororakau i nga wa katoa.
I konei kua tohuhia e au ki te kakariki te panui kua pai te whakarewanga o te paipa, ki te whero te karere hapa me te kowhai te panui mo te ngana ki te whakapā atu. : 9200.
Ka tupu tenei na te mea kei roto i te logstash.conf, kei roto i te ahua, he haki mo te elasticsearch e waatea ana. I muri i nga mea katoa, e kii ana te logstash kei te mahi hei waahanga o te puranga Elk, engari i wehea e matou.
Ka taea te mahi, engari kaore i te watea.
Ko te otinga ko te whakakore i tenei haki ma te XPACK_MONITORING_ENABLED taurangi taiao.
Me huri tatou ki docker-compose.yml ka whakahaere ano:
version: '3'
networks:
elk:
volumes:
elasticsearch:
driver: local
services:
logstash:
container_name: logstash_one_channel
image: docker.elastic.co/logstash/logstash:6.3.2
networks:
- elk
environment:
XPACK_MONITORING_ENABLED: "false"
ports:
- 5046:5046
volumes:
- ./config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
- ./config/pipelines:/usr/share/logstash/config/pipelines:ro
Inaianei kei te pai nga mea katoa. Kua reri te ipu mo nga whakamatautau.
Ka taea e tatou te pato ano ki te papatohu e whai ake nei:
echo '13123123123123123123123213123213' | nc localhost 5046
A ka kite:
logstash_one_channel | {
logstash_one_channel | "message" => "13123123123123123123123213123213",
logstash_one_channel | "@timestamp" => 2019-04-29T11:43:44.582Z,
logstash_one_channel | "@version" => "1",
logstash_one_channel | "habra_field" => "Hello Habr",
logstash_one_channel | "host" => "gateway",
logstash_one_channel | "port" => 49418
logstash_one_channel | }
Mahi i roto i te hongere kotahi
Na ka whakarewahia e matou. Inaianei ka taea e koe te whai wa ki te whirihora i te logstash ano. Kaua tatou e pa ki te konae pipelines.yml mo tenei wa, kia kite tatou he aha te whiwhi ma te mahi tahi me te hongere kotahi.
Me kii ahau ko te kaupapa whanui mo te mahi me te konae whirihoranga hongere he pai te korero i roto i te pukapuka whai mana, i konei
Mena kei te pirangi koe ki te panui i te reo Ruhia, i whakamahia e matou tenei (engari he tawhito te wetereo patai, me whai whakaaro tatou ki tenei).
Me haere raupapa mai i te waahanga Whakauru. Kua kite matou i nga mahi mo TCP. He aha atu te mea whakamere ki konei?
Whakamātauhia nga karere ma te whakamahi i te ngakau
He waahi pai tenei ki te whakaputa i nga karere whakamatautau aunoa.
Ki te mahi i tenei, me whakahohe koe i te mono heartbean i te waahanga whakauru.
input {
heartbeat {
message => "HeartBeat!"
}
}
Whakakāhia, tīmata te whiwhi kotahi i te meneti
logstash_one_channel | {
logstash_one_channel | "@timestamp" => 2019-04-29T13:52:04.567Z,
logstash_one_channel | "habra_field" => "Hello Habr",
logstash_one_channel | "message" => "HeartBeat!",
logstash_one_channel | "@version" => "1",
logstash_one_channel | "host" => "a0667e5c57ec"
logstash_one_channel | }
Mena kei te pirangi tatou ki te whiwhi i nga wa maha, me taapiri te tawhā waahi.
Ma tenei ka whiwhi karere ia 10 hēkona.
input {
heartbeat {
message => "HeartBeat!"
interval => 10
}
}
Te tiki raraunga mai i te konae
I whakatau ano matou ki te titiro ki te aratau konae. Mena ka pai te mahi ki te konae, karekau pea e hiahiatia he kaihoko, ahakoa mo te whakamahi a rohe.
E ai ki te whakaahuatanga, me rite te ahua o te mahi ki te hiku -f, i.e. ka panui i nga rarangi hou, ka panui ranei i te konae katoa.
Na te mea e hiahia ana matou ki te tiki:
- Kei te pirangi matou ki te whiwhi rarangi ka apitihia ki tetahi konae rangitaki.
- E hiahia ana matou ki te whiwhi raraunga kua tuhia ki te maha o nga konae raarangi, me te wehe i nga mea ka riro mai i hea.
- E hiahia ana matou ki te whakarite kia timata ano te logstash, kaore e whiwhi ano i enei raraunga.
- E hiahia ana matou ki te tirotiro mena ka wetohia te logstash, ka tuhia tonu nga raraunga ki nga konae, katahi ka whakahaerea e matou, ka whiwhi matou i enei raraunga.
Hei whakahaere i te whakamatautau, me taapiri tetahi atu raina ki te docker-compose.yml, ka whakatuwhera i te raarangi whakauru i nga konae.
version: '3'
networks:
elk:
volumes:
elasticsearch:
driver: local
services:
logstash:
container_name: logstash_one_channel
image: docker.elastic.co/logstash/logstash:6.3.2
networks:
- elk
environment:
XPACK_MONITORING_ENABLED: "false"
ports:
- 5046:5046
volumes:
- ./config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
- ./config/pipelines:/usr/share/logstash/config/pipelines:ro
- ./logs:/usr/share/logstash/input
A huri i te waahanga whakauru i roto i te habr_pipeline.conf
input {
file {
path => "/usr/share/logstash/input/*.log"
}
}
Me timata tatou:
docker-compose up
Hei waihanga me te tuhi i nga konae rangitaki ka whakamahia e matou te whakahau:
echo '1' >> logs/number1.log
{
logstash_one_channel | "host" => "ac2d4e3ef70f",
logstash_one_channel | "habra_field" => "Hello Habr",
logstash_one_channel | "@timestamp" => 2019-04-29T14:28:53.876Z,
logstash_one_channel | "@version" => "1",
logstash_one_channel | "message" => "1",
logstash_one_channel | "path" => "/usr/share/logstash/input/number1.log"
logstash_one_channel | }
Ae, ka mahi!
I te wa ano, ka kite tatou kua taapiri aunoa i te mara ara. Ko te tikanga a nga ra kei te heke mai, ka taea e taatau te tarai i nga rekoata.
Kia ngana ano tatou:
echo '2' >> logs/number1.log
{
logstash_one_channel | "host" => "ac2d4e3ef70f",
logstash_one_channel | "habra_field" => "Hello Habr",
logstash_one_channel | "@timestamp" => 2019-04-29T14:28:59.906Z,
logstash_one_channel | "@version" => "1",
logstash_one_channel | "message" => "2",
logstash_one_channel | "path" => "/usr/share/logstash/input/number1.log"
logstash_one_channel | }
Na inaianei ki tetahi atu kōnae:
echo '1' >> logs/number2.log
{
logstash_one_channel | "host" => "ac2d4e3ef70f",
logstash_one_channel | "habra_field" => "Hello Habr",
logstash_one_channel | "@timestamp" => 2019-04-29T14:29:26.061Z,
logstash_one_channel | "@version" => "1",
logstash_one_channel | "message" => "1",
logstash_one_channel | "path" => "/usr/share/logstash/input/number2.log"
logstash_one_channel | }
Rawe! I tikina te konae, i tika te tohu i te ara, kei te pai nga mea katoa.
Kati te putea ka timata ano. Tatari tatou. Wahangu. Ko era. Kaore matou e whiwhi ano i enei rekoata.
Na inaianei ko te whakamatautau tino maia.
Tāutahia te logstash ka mahia:
echo '3' >> logs/number2.log
echo '4' >> logs/number1.log
Whakahaerehia ano te putea ka kite:
logstash_one_channel | {
logstash_one_channel | "host" => "ac2d4e3ef70f",
logstash_one_channel | "habra_field" => "Hello Habr",
logstash_one_channel | "message" => "3",
logstash_one_channel | "@version" => "1",
logstash_one_channel | "path" => "/usr/share/logstash/input/number2.log",
logstash_one_channel | "@timestamp" => 2019-04-29T14:48:50.589Z
logstash_one_channel | }
logstash_one_channel | {
logstash_one_channel | "host" => "ac2d4e3ef70f",
logstash_one_channel | "habra_field" => "Hello Habr",
logstash_one_channel | "message" => "4",
logstash_one_channel | "@version" => "1",
logstash_one_channel | "path" => "/usr/share/logstash/input/number1.log",
logstash_one_channel | "@timestamp" => 2019-04-29T14:48:50.856Z
logstash_one_channel | }
Hore! I kohia nga mea katoa.
Engari me whakatupato koe mo nga mea e whai ake nei. Mēnā ka mukua te ipu rākau (docker stop logstash_one_channel && docker rm logstash_one_channel), karekau he mea e kohia. Ko te tuunga o te konae i panuitia ai i rongoa i roto i te ipu. Mena ka rere koe mai i te wahanga, ka whakaaehia nga raina hou.
Te panui i nga konae kei roto
Me kii kei te whakarewahia e matou he pororakau mo te wa tuatahi, engari kei a matou nga raarangi ka hiahia matou ki te tukatuka.
Mena ka whakahaerehia e matou te logstash me te waahanga whakauru i whakamahia e matou i runga ake nei, kaore he mea e whiwhi. Ko nga raina hou anake ka tukatukahia e te logstash.
Kia toia ake nga rarangi mai i nga konae o mua, me taapiri he raina taapiri ki te waahanga whakauru:
input {
file {
start_position => "beginning"
path => "/usr/share/logstash/input/*.log"
}
}
Ano, he ahua ano: ka pa noa tenei ki nga konae hou kaore ano kia kitehia e te logstash. Mo nga konae ano kei roto i te waahi tirohanga o te logstash, kua maumahara ki te rahi o te waa, ka uru noa ki nga whakaurunga hou ki roto.
Ka mutu i konei ka ako i te waahanga whakauru. He maha tonu nga whiringa, engari he ranea maau mo etahi atu whakamatautau inaianei.
Te Ararere me te Huringa Raraunga
Me ngana ki te whakaoti i te raru e whai ake nei, me kii he karere mai i tetahi hongere, he korero korero etahi, he karere hapa etahi. He rereke nga tohu. Ko etahi he INFO, ko etahi he HAPA.
Me wehe ke i te putanga. Ko era. Ka tuhia e matou nga korero korero ki tetahi hongere, me nga karere hapa ki tetahi atu.
Ki te mahi i tenei, neke mai i te waahanga whakauru ki te tātari me te whakaputa.
Ma te whakamahi i te waahanga tātari, ka whakamaaramahia e matou te karere taumai, ka whiwhi i te hash (nga takirua uara matua) mai i a ia, ka taea e taatau te mahi, ara. wetewete kia rite ki nga tikanga. A, i roto i te waahanga whakaputa, ka tohua e matou nga karere ka tukuna ia tangata ki tana ake hongere.
Te poroporoaki i te karere me te grok
Hei tarai i nga aho kuputuhi me te tiki i tetahi huinga mara mai i a raatau, he mono motuhake kei roto i te waahanga tātari - grok.
Ma te kore e whakatau i a au ano te whainga ki te tuku korero taipitopito mo konei (mo tenei ka korero ahau ), ka hoatu e ahau taku tauira ngawari.
Ki te mahi i tenei, me whakatau koe i te whakatakotoranga o nga aho whakauru. Kei a au enei penei:
1 INFO message1
2 Karere HAPA2
Ko era. Ko te kaitautuhi ka tae tuatahi, katahi ko INFO/HAPA, katahi ko etahi kupu kore mokowhiti.
Ehara i te mea uaua, engari he nui ki te mohio ki te kaupapa o te mahi.
Na, i roto i te waahanga tātari o te mono grok, me tautuhi he tauira mo te tarai i a tatou aho.
Ka penei te ahua:
filter {
grok {
match => { "message" => ["%{INT:message_id} %{LOGLEVEL:message_type} %{WORD:message_text}"] }
}
}
Ko te tikanga he korero auau. Ka whakamahia nga tauira kua rite, penei i te INT, LOGLEVEL, WORD. Ko o raatau whakaahuatanga, me etahi atu tauira, ka kitea i konei
Inaianei, ka haere i roto i tenei tātari, ka huri to tatou aho ki te hash o nga mara e toru: message_id, message_type, message_text.
Ka whakaatuhia ki te waahanga whakaputa.
Te ararere i nga karere ki te waahanga whakaputa ma te whakamahi i te whakahau if
I roto i te waahanga whakaputa, e maumahara ana matou, ka wehea e matou nga karere kia rua nga awa. Ko etahi - he iNFO, ka puta ki te papatohu, me nga hapa, ka whakaputahia ki tetahi konae.
Me pehea e wehewehe ai i enei karere? Ko te ahuatanga o te raru kua whakaatu he otinga - i muri i nga mea katoa, kua whakatapua he mara karere_type, e rua noa nga uara: INFO me te HAPA. I runga i tenei ka whiriwhiri maatau ma te whakamahi i te korero mena.
if [message_type] == "ERROR" {
# Здесь выводим в файл
} else
{
# Здесь выводим в stdout
}
He whakaahuatanga mo te mahi me nga mara me nga kaiwhakahaere ka kitea i tenei waahanga .
Na, mo te mutunga ake.
Putanga papatohu, he maamaa nga mea katoa i konei - stdout {}
Engari ko te whakaputanga ki tetahi konae - mahara kei te whakahaerehia e matou enei mea katoa mai i te ipu, kia taea ai te uru o te konae i tuhia ai te hua mai i waho, me whakatuwhera tenei raarangi ki docker-compose.yml.
Te tapeke:
Ko te ahua o te waahanga whakaputa o ta maatau konae penei:
output {
if [message_type] == "ERROR" {
file {
path => "/usr/share/logstash/output/test.log"
codec => line { format => "custom format: %{message}"}
}
} else
{stdout {
}
}
}
I roto i te docker-compose.yml ka taapirihia e matou tetahi atu rōrahi mo te putanga:
version: '3'
networks:
elk:
volumes:
elasticsearch:
driver: local
services:
logstash:
container_name: logstash_one_channel
image: docker.elastic.co/logstash/logstash:6.3.2
networks:
- elk
environment:
XPACK_MONITORING_ENABLED: "false"
ports:
- 5046:5046
volumes:
- ./config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
- ./config/pipelines:/usr/share/logstash/config/pipelines:ro
- ./logs:/usr/share/logstash/input
- ./output:/usr/share/logstash/output
Ka whakarewahia, ka whakamatauhia, ka kite i te wehenga ki nga awa e rua.
Source: will.com