Selelekela
Ha re ntse re tsamaisa sistimi e 'ngoe, re ile ra tobana le tlhoko ea ho sebetsana le palo e kholo ea li-log tse fapaneng. ELK e ile ea khethoa e le sesebelisoa. Sengoliloeng sena se tla bua ka boiphihlelo ba rona ba ho theha stack ena.
Ha re ipehele pakane ea ho hlalosa bokhoni bohle ba eona, empa re batla ho tsepamisa maikutlo ho rarolleng mathata a sebetsang. Sena se bakoa ke taba ea hore ka palo e lekaneng ea litokomane le litšoantšo tse lokiselitsoeng, ho na le maraba a mangata, bonyane re a fumane.
Re tsamaisitse stack ka docker-compose. Ho feta moo, re ne re e-na le docker-compose.yml e ngotsoeng hantle e ileng ea re lumella ho phahamisa stack ntle le mathata. 'Me ho rona ho ne ho bonahala eka tlhōlo e se e le haufi, joale re tla e sotha hanyenyane hore e lumellane le litlhoko tsa rona' me ke eona.
Ka bomalimabe, teko ea ho lokisa sistimi ho amohela le ho sebetsana le lintlha tse tsoang ho kopo ea rona ha ea atleha hang hang. Ka hona, re nkile qeto ea hore ho bohlokoa ho ithuta karolo ka 'ngoe ka thoko, ebe re khutlela ho likhokahano tsa bona.
Kahoo a re qaleng ka logstash.
Tikoloho, phepelo, ho tsamaisa Logstash ka setshelo
Bakeng sa ho tsamaisoa, re sebelisa docker-compose, liteko tse hlalositsoeng mona li entsoe ho MacOS le Ubuntu 18.0.4.
Setšoantšo sa logstash seo re neng re e-na le sona ho docker-compose.yml ea rona ea mantlha ke docker.elastic.co/logstash/logstash:6.3.2
Re tla e sebelisa bakeng sa liteko.
Ho tsamaisa logstash, re ngotse docker-compose.yml e arohaneng. Ha e le hantle, ho ne ho ka khoneha ho qala setšoantšo ho tloha molaong oa taelo, empa ka mor'a moo, re ile ra rarolla mosebetsi o itseng, moo ntho e 'ngoe le e' ngoe e tsoang ho docker-compose e qalisoang bakeng sa rona.
Ka bokhutšoanyane ka lifaele tsa tlhophiso
Ka tsela e latelang ho tsoa ho tlhaloso, logstash e ka tsamaisoa e le mocha o le mong, tabeng ena, e hloka ho fetisetsa faele ea * .conf kapa bakeng sa litsela tse 'maloa, moo ho hlokahalang ho fetisetsa pipelines.yml file, eo, ka lehlakoreng le leng. , e tla bua ka lifaele .conf bakeng sa kanale ka 'ngoe.
Re ile ra nka tsela ea bobeli. Ho rona e ne e bonahala e le e feto-fetohang le e qhekellang. Ka hona, re thehile pipelines.yml, 'me ra etsa directory ea pipeline moo re tla kenya lifaele tsa .conf bakeng sa mocha ka mong.
Ka har'a setshelo ho na le faele e 'ngoe ea tlhophiso - logstash.yml. Ha re e ame, re e sebelisa kamoo e leng kateng.
Kahoo sebopeho sa rona sa directory ke:
Hajoale, re nka hore ena ke tcp ho port 5046 ho amohela data ea ho kenya, 'me re tla sebelisa stdout bakeng sa tlhahiso.
Mona ke tlhophiso e bonolo joalo bakeng sa ho matha pele. Hobane mosebetsi oa pele ke ho qala.
Kahoo re na le docker-compose.yml ena
version: '3'
networks:
elk:
volumes:
elasticsearch:
driver: local
services:
logstash:
container_name: logstash_one_channel
image: docker.elastic.co/logstash/logstash:6.3.2
networks:
- elk
ports:
- 5046:5046
volumes:
- ./config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
- ./config/pipelines:/usr/share/logstash/config/pipelines:ro
Re bona eng moo?
- Marang-rang le li-volumes li nkiloe ho docker-compose.yml ea pele (eo ho eona ho qalisoang stack kaofela) 'me ke nahana hore ha li ame setšoantšo ka kakaretso mona.
- Re theha tšebeletso e le 'ngoe (litšebeletso) logstash, ho tloha ho docker.elastic.co/logstash/logstash:6.3.2 setšoantšo ebe re se fa lebitso la logstash_one_channel.
- Re fetisetsa port 5046 ka har'a sets'oants'o, ho boema-kepe bo le bong bo kahare.
- Re etsa 'mapa oa faele ea rona ea ./config/pipelines.yml ho faele ea /usr/share/logstash/config/pipelines.yml ka har'a setshelo, moo logstash e tla e nka ebe e etsa hore e balehe feela, haeba ho ka etsahala.
- Re etsa 'mapa oa ./config/pipelines directory, moo re nang le lifaele tsa tlhophiso ea liphaephe, ho /usr/share/logstash/config/pipelines directory hape re etsa hore e bale feela.
piping.yml faele
- pipeline.id: HABR
pipeline.workers: 1
pipeline.batch.size: 1
path.config: "./config/pipelines/habr_pipeline.conf"
E hlalosa mocha o le mong o nang le sekhetho sa HABR le tsela e lebisang faeleng ea eona ea tlhophiso.
'Me qetellong faele "./config/pipelines/habr_pipeline.conf"
input {
tcp {
port => "5046"
}
}
filter {
mutate {
add_field => [ "habra_field", "Hello Habr" ]
}
}
output {
stdout {
}
}
Ha re na ho kena tlhalosong ea eona hajoale, re leka ho matha:
docker-compose up
Re bona eng?
Setshelo se qalile. Re ka hlahloba mosebetsi oa eona:
echo '13123123123123123123123213123213' | nc localhost 5046
'Me re bona karabo ho console ea setshelo:
Empa ka nako e ts'oanang, re boetse re bona:
logstash_one_channel | [2019-04-29T11:28:59,790][ERROR][logstash.licensechecker.licensereader] Ha re khone ho fumana lintlha tsa laesense ho tsoa ho seva sa laesense {:message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore ::ResolutionFailure]elasticsearch", ...
logstash_one_channel | [2019-04-29T11:28:59,894][INFO ][logstash.pipeline ] Pipeline e qadile ka katleho {:pipeline_id=>".monitoring-logstash", :thread=>"# »}
logstash_one_channel | [2019-04-29T11:28:59,988][INFO ][logstash.agent ] Liphaephe tse sebetsang {:count=>2, :running_pipelines=>[:HABR, :".monitoring-logstash"], :non_running_pipelines=>[ ]}
logstash_one_channel | [2019-04-29T11:29:00,015][ERROR][logstash.inputs.metrics ] X-Pack e kentsoe ho Logstash empa eseng ho Elasticsearch. Ka kopo kenya X-Pack ho Elasticsearch ho sebelisa ts'ebetso ea ho beha leihlo. Likarolo tse ling li ka ba teng.
logstash_one_channel | [2019-04-29T11:29:00,526][INFO ][logstash.agent ] E qadile ka katleho endpoint ya Logstash API {:port=>9600}
logstash_one_channel | [2019-04-29T11:29:04,478][INFO ][logstash.outputs.elasticsearch] Ho hlahloba bophelo bo botle ho bona hore na khokahanyo ea Elasticsearch ea sebetsa {:healthcheck_url=>http://elasticsearch:9200/, :path=> "/"}
logstash_one_channel | [2019-04-29T11:29:04,487][TEMOSO ][logstash.outputs.elasticsearch] O lekile ho tsosa khokahano le mohlala o shoeleng oa ES, empa o fumane phoso. {:url="":9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][ManticoreFailure::Reso] elasticsearch"}
logstash_one_channel | [2019-04-29T11:29:04,704][INFO ][logstash.licensechecker.licensereader] E etsa tlhahlobo ea bophelo bo botle ho bona hore na khokahanyo ea Elasticsearch ea sebetsa {:healthcheck_url=>http://elasticsearch:9200/, :path=> "/"}
logstash_one_channel | [2019-04-29T11:29:04,710][TEMOSO ][logstash.licensechecker.licensereader] O lekile ho tsosa khokahanyo ho mohlala o shoeleng oa ES, empa o fumane phoso. {:url="":9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][ManticoreFailure::Reso] elasticsearch"}
'Me logi ea rona e lula e phahama ka linako tsohle.
Mona ke totobalitse ka botala molaetsa oo pipeline e qalileng ka katleho, ka bofubelu molaetsa oa phoso le ka mosehla molaetsa o mabapi le ho leka ho ikopanya. : 9200.
Sena se etsahala ka lebaka la hore ho logstash.conf e kenyelelitsoeng setšoantšong, ho na le cheke bakeng sa ho fumaneha ha elasticsearch. Ntle le moo, logstash e nka hore e sebetsa e le karolo ea Elk stack, mme re e arotse.
O ka sebetsa, empa ha ho bonolo.
Tharollo ke ho tima cheke ena ka XPACK_MONITORING_ENABLED ea tikoloho.
Ha re etseng phetoho ho docker-compose.yml 'me re e tsamaise hape:
version: '3'
networks:
elk:
volumes:
elasticsearch:
driver: local
services:
logstash:
container_name: logstash_one_channel
image: docker.elastic.co/logstash/logstash:6.3.2
networks:
- elk
environment:
XPACK_MONITORING_ENABLED: "false"
ports:
- 5046:5046
volumes:
- ./config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
- ./config/pipelines:/usr/share/logstash/config/pipelines:ro
Jwale, tsohle di lokile. Setshelo se se se loketse liteko.
Re ka thaepa hape ho console e haufi:
echo '13123123123123123123123213123213' | nc localhost 5046
Mme bona:
logstash_one_channel | {
logstash_one_channel | "message" => "13123123123123123123123213123213",
logstash_one_channel | "@timestamp" => 2019-04-29T11:43:44.582Z,
logstash_one_channel | "@version" => "1",
logstash_one_channel | "habra_field" => "Hello Habr",
logstash_one_channel | "host" => "gateway",
logstash_one_channel | "port" => 49418
logstash_one_channel | }
Sebetsa ka har'a mocha o le mong
Kahoo re ile ra qala. Joale u ka nka nako ea ho lokisa logstash ka kotloloho. Ha re se keng ra ama faele ea pipelines.yml hajoale, a re boneng hore na re ka fumana eng ka ho sebetsa ka mocha o le mong.
Ke tlameha ho bolela hore molao-motheo o akaretsang oa ho sebetsa le faele ea tlhophiso ea mocha o hlalositsoe hantle bukeng ea molao, mona
Haeba u batla ho bala ka Serussia, joale re sebelisitse ena (empa syntax ea lipotso ke ea khale moo, o hloka ho ela sena hloko).
Ha re eeng ka tatellano ho tsoa karolong ea Input. Re se re bone mosebetsi ho tcp. Ke eng hape e ka khahlang mona?
Lekola melaetsa ka ho otla ha pelo
Ho na le monyetla o joalo o khahlisang oa ho hlahisa melaetsa ea tlhahlobo ea othomathiki.
Ho etsa sena, o hloka ho kenyelletsa plugin ea heartbean karolong ea ho kenya.
input {
heartbeat {
message => "HeartBeat!"
}
}
Re e bulela, re qala ho amohela hang ka motsotso
logstash_one_channel | {
logstash_one_channel | "@timestamp" => 2019-04-29T13:52:04.567Z,
logstash_one_channel | "habra_field" => "Hello Habr",
logstash_one_channel | "message" => "HeartBeat!",
logstash_one_channel | "@version" => "1",
logstash_one_channel | "host" => "a0667e5c57ec"
logstash_one_channel | }
Re batla ho amohela khafetsa, re hloka ho eketsa paramente ea nako.
Ena ke tsela eo re tla amohela molaetsa ka eona metsotsoana e meng le e meng e 10.
input {
heartbeat {
message => "HeartBeat!"
interval => 10
}
}
Ho fumana data ho tsoa faeleng
Hape re ile ra etsa qeto ea ho sheba mokhoa oa faele. Haeba e sebetsa hantle ka faele, joale ho ka khoneha hore ha ho moemeli ea hlokahalang, hantle, bonyane bakeng sa tšebeliso ea sebaka seo.
Ho ea ka tlhaloso, mokhoa oa ts'ebetso o lokela ho tšoana le mohatla -f, i.e. e bala mela e mecha kapa, ka boikhethelo, e bala faele eohle.
Kahoo seo re batlang ho se fumana:
- Re batla ho fumana mela e kenyellelitsoeng faeleng e le 'ngoe ea log.
- Re batla ho fumana lintlha tse ngotsoeng ho lifaele tse 'maloa tsa log, ha re ntse re khona ho arola se amohetsoeng ho tloha kae.
- Re batla ho etsa bonnete ba hore ha logstash e qala hape, e ke ke ea fumana data ena hape.
- Re batla ho hlahloba hore haeba logstash e holofetse, 'me data e ntse e tsoela pele ho ngoloa lifaeleng, joale ha re e tsamaisa, re tla fumana data ena.
Ho etsa liteko, a re kenye mola o mong hape ho docker-compose.yml, re bula bukana moo re behang lifaele teng.
version: '3'
networks:
elk:
volumes:
elasticsearch:
driver: local
services:
logstash:
container_name: logstash_one_channel
image: docker.elastic.co/logstash/logstash:6.3.2
networks:
- elk
environment:
XPACK_MONITORING_ENABLED: "false"
ports:
- 5046:5046
volumes:
- ./config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
- ./config/pipelines:/usr/share/logstash/config/pipelines:ro
- ./logs:/usr/share/logstash/input
'Me u fetole karolo ea ho kenya habr_pipeline.conf
input {
file {
path => "/usr/share/logstash/input/*.log"
}
}
Re qala:
docker-compose up
Ho theha le ho ngola lifaele tsa log, re tla sebelisa taelo:
echo '1' >> logs/number1.log
{
logstash_one_channel | "host" => "ac2d4e3ef70f",
logstash_one_channel | "habra_field" => "Hello Habr",
logstash_one_channel | "@timestamp" => 2019-04-29T14:28:53.876Z,
logstash_one_channel | "@version" => "1",
logstash_one_channel | "message" => "1",
logstash_one_channel | "path" => "/usr/share/logstash/input/number1.log"
logstash_one_channel | }
E, ea sebetsa!
Ka nako e ts'oanang, rea bona hore re kentse lebala la tsela ka bohona. Kahoo nakong e tlang, re tla khona ho sefa lirekoto ka eona.
Ha re leke hape:
echo '2' >> logs/number1.log
{
logstash_one_channel | "host" => "ac2d4e3ef70f",
logstash_one_channel | "habra_field" => "Hello Habr",
logstash_one_channel | "@timestamp" => 2019-04-29T14:28:59.906Z,
logstash_one_channel | "@version" => "1",
logstash_one_channel | "message" => "2",
logstash_one_channel | "path" => "/usr/share/logstash/input/number1.log"
logstash_one_channel | }
'Me joale ho faele e' ngoe:
echo '1' >> logs/number2.log
{
logstash_one_channel | "host" => "ac2d4e3ef70f",
logstash_one_channel | "habra_field" => "Hello Habr",
logstash_one_channel | "@timestamp" => 2019-04-29T14:29:26.061Z,
logstash_one_channel | "@version" => "1",
logstash_one_channel | "message" => "1",
logstash_one_channel | "path" => "/usr/share/logstash/input/number2.log"
logstash_one_channel | }
E kholo! Faele e ile ea nkuoa, tsela e boletsoeng ka nepo, tsohle li hantle.
Emisa logstash ebe u qala hape. Ha re emeng. Ho thola. Tseo. Ha re sa fumana lirekoto tsena hape.
'Me hona joale teko e matla ka ho fetisisa.
Re kenya logstash mme re phethe:
echo '3' >> logs/number2.log
echo '4' >> logs/number1.log
Matha logstash hape 'me u bone:
logstash_one_channel | {
logstash_one_channel | "host" => "ac2d4e3ef70f",
logstash_one_channel | "habra_field" => "Hello Habr",
logstash_one_channel | "message" => "3",
logstash_one_channel | "@version" => "1",
logstash_one_channel | "path" => "/usr/share/logstash/input/number2.log",
logstash_one_channel | "@timestamp" => 2019-04-29T14:48:50.589Z
logstash_one_channel | }
logstash_one_channel | {
logstash_one_channel | "host" => "ac2d4e3ef70f",
logstash_one_channel | "habra_field" => "Hello Habr",
logstash_one_channel | "message" => "4",
logstash_one_channel | "@version" => "1",
logstash_one_channel | "path" => "/usr/share/logstash/input/number1.log",
logstash_one_channel | "@timestamp" => 2019-04-29T14:48:50.856Z
logstash_one_channel | }
Hooray! Tsohle di ile tsa kokota.
Empa, hoa hlokahala ho lemosa ka tse latelang. Haeba setshelo sa logstash se tlositsoe (docker stop logstash_one_channel && docker rm logstash_one_channel), ha ho letho le tla nkuoa. Boemo ba faele eo e neng e baloa ho eona e ne e bolokiloe ka har'a setshelo. Haeba u qala ho tloha qalong, e tla amohela mela e mecha feela.
Ho bala lifaele tse teng
Ha re re re ntse re matha logstash lekhetlo la pele, empa re se re ntse re e-na le li-logs 'me re ka rata ho li sebetsa.
Haeba re matha logstash ka karolo ea ho kenya eo re e sebelisitseng ka holimo, re ke ke ra fumana letho. Ke mela e mecha feela e tla sebetsoa ke logstash.
Bakeng sa ho hula mela ho tsoa lifaeleng tse seng li ntse li le teng, eketsa mola o eketsehileng karolong ea ho kenya:
input {
file {
start_position => "beginning"
path => "/usr/share/logstash/input/*.log"
}
}
Ho feta moo, ho na le nuance, sena se ama feela lifaele tse ncha tseo logstash e e-s'o e bone. Bakeng sa lifaele tse tšoanang tse neng li se li ntse li le tšimong ea pono ea logstash, e se e ntse e hopola boholo ba eona 'me joale e tla nka litlaleho tse ncha feela ho tsona.
Ha re emise ho sena ka ho ithuta karolo ea ho kenya. Ho na le likhetho tse ling tse ngata, empa hajoale, re na le tse lekaneng bakeng sa liteko tse ling.
Litsela le phetoho ea data
Ha re leke ho rarolla bothata bo latelang, ha re re re na le melaetsa e tsoang ho mocha o le mong, e meng ke ea litaba, 'me e meng ke melaetsa ea liphoso. Li fapane ka li-tag. Tse ling ke INFO, tse ling ke LIPHOSO.
Re hloka ho li arola ha ho tsoa. Tseo. Re ngola melaetsa ea tlhahisoleseding mocha o mong, le melaetsa ea liphoso ho e 'ngoe.
Ho etsa sena, tloha karolong ea ho kenya ho ea ho sefa le tlhahiso.
Re sebelisa karolo ea filthara, re tla arola molaetsa o kenang, ho fumana hash (li-key-value pairs) ho eona, eo re seng re ntse re ka sebetsa ka eona, ke hore. arola ho ya ka maemo. 'Me karolong ea lihlahisoa, re tla khetha melaetsa ebe re romela e' ngoe le e 'ngoe ho mocha oa eona.
Ho fetisa molaetsa ka grok
E le hore u arole likhoele tsa mongolo le ho fumana lihlopha tsa masimo ho tsona, ho na le plugin e khethehileng karolong ea filthara - grok.
Ntle le ho ipehela sepheo sa ho fana ka tlhaloso e qaqileng ea eona mona (bakeng sa sena ke bua ka sona ), ke tla fana ka mohlala oa ka o bonolo.
Ho etsa sena, o hloka ho etsa qeto ka sebopeho sa mela ea ho kenya. Ke na le tsona tse kang tsena:
1 INFO molaetsa1
2 MOLAOTSA WA MOLAOTSA2
Tseo. Sekhetho pele, ebe INFO/ERROR, ebe lentsoe le se nang libaka.
Ha ho thata, empa ho lekaneng ho utloisisa molao-motheo oa mosebetsi.
Kahoo, karolong ea filthara, ho plugin ea grok, re hloka ho hlalosa mokhoa oa ho arola likhoele tsa rona.
E tla shebahala tjena:
filter {
grok {
match => { "message" => ["%{INT:message_id} %{LOGLEVEL:message_type} %{WORD:message_text}"] }
}
}
Ha e le hantle, ke polelo e tloaelehileng. Ho sebelisoa lipaterone tse seng li entsoe, joalo ka INT, LOGLEVEL, WORD. Tlhaloso ea bona, hammoho le mekhoa e meng, e ka bonoa mona.
Joale, ha re feta sefeneng sena, khoele ea rona e tla fetoha hashe ea likarolo tse tharo: message_id, message_type, message_text.
Li tla bontšoa karolong ea tlhahiso.
Ho tsamaisa melaetsa karolong ea tlhahiso ka taelo ea if
Karolong ea tlhahiso, joalo ka ha re hopola, re ne re tla arola melaetsa ka melapo e 'meli. Tse ling - tseo e leng iNFO, re tla li hlahisa ho console, 'me ka liphoso, re tla hlahisa faele.
Re ka arolelana melaetsa ee joang? Boemo ba bothata bo se bo ntse bo fana ka maikutlo a tharollo - ka mor'a tsohle, re se re ntse re e-na le tšimo e inehetseng ea mofuta oa molaetsa, e ka nkang lintlha tse peli feela INFO le ERROR. Ke ho eona re tla etsa khetho re sebelisa polelo ea if.
if [message_type] == "ERROR" {
# Здесь выводим в файл
} else
{
# Здесь выводим в stdout
}
Tlhaloso ea mosebetsi le masimo le basebetsi e ka fumanoa karolong ena .
Jwale, mabapi le sephetho ka bosona.
Tlhahiso ea Console, tsohle li hlakile mona - stdout {}
Empa sephetho ho faele - hopola hore re ntse re tsamaisa tsena tsohle ho tsoa ka setshelo mme e le hore faele eo re ngolang sephetho ho eona e ka fumaneha ho tsoa kantle, re hloka ho bula bukana ena ho docker-compose.yml.
Kakaretso:
Karolo ea tlhahiso ea faele ea rona e shebahala tjena:
output {
if [message_type] == "ERROR" {
file {
path => "/usr/share/logstash/output/test.log"
codec => line { format => "custom format: %{message}"}
}
} else
{stdout {
}
}
}
Kenya molumo o mong hape ho docker-compose.yml bakeng sa tlhahiso:
version: '3'
networks:
elk:
volumes:
elasticsearch:
driver: local
services:
logstash:
container_name: logstash_one_channel
image: docker.elastic.co/logstash/logstash:6.3.2
networks:
- elk
environment:
XPACK_MONITORING_ENABLED: "false"
ports:
- 5046:5046
volumes:
- ./config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
- ./config/pipelines:/usr/share/logstash/config/pipelines:ro
- ./logs:/usr/share/logstash/input
- ./output:/usr/share/logstash/output
Re qala, re leka, re bona karohano ka melapo e 'meli.
Source: www.habr.com