ProHoster > Blog > Utongi > Kushanda kushandisa ELK. Kugadzira logstash

Kushanda kushandisa ELK. Kugadzira logstash

Nhanganyaya

Tichiri kuendesa imwe hurongwa, takatarisana nekudiwa kwekugadzirisa nhamba huru yematanda akasiyana. ELK yakasarudzwa sechombo. Ichi chinyorwa chichakurukura chiitiko chedu mukugadzira iyi stack.

Isu hatiisi chinangwa chekutsanangura kugona kwayo kwese, asi tinoda kuisa pfungwa pakugadzirisa matambudziko anoshanda. Izvi zvinokonzerwa nekuti kunyangwe paine huwandu hwakati wandei hwezvinyorwa uye mifananidzo yakagadzirira-yakagadzirwa, kune akawanda makomba, zvirinani isu takaawana.

Isu takatumira stack kuburikidza ne docker-compose. Uyezve, takanga tiine yakanyatsonyorwa docker-compose.yml, iyo yakatibvumira kusimudza stack pasina matambudziko. Uye zvakaratidzika kwatiri kuti kukunda kwaive kwatove pedyo, ikozvino tichazvigadzirisa zvishoma kuti zvienderane nezvido zvedu uye ndizvozvo.

Nehurombo, kuyedza kugadzirisa iyo sisitimu kuti igamuchire uye kugadzirisa matanda kubva kune yedu application haina kubudirira nekukurumidza. Nokudaro, takasarudza kuti zvakakodzera kudzidza chikamu chimwe nechimwe zvakasiyana, uye kudzokera kune avo vakabatanidzwa.

Saka, takatanga ne logstash.

Nzvimbo, kutumirwa, kumhanya Logstash mumudziyo

Kuendesa tinoshandisa docker-compose; zviedzo zvinotsanangurwa pano zvakaitwa paMacOS uye Ubuntu 18.0.4.

Mufananidzo welogstash wakanyoreswa mune yedu yepakutanga docker-compose.yml is docker.elastic.co/logstash/logstash:6.3.2

Tichaishandisa pakuedza.

Isu takanyora yakaparadzana docker-compose.yml kumhanya logstash. Ehe, zvaigoneka kuvhura chifananidzo kubva kumutsara wekuraira, asi isu taigadzirisa dambudziko chairo, kwatinomhanyisa zvese kubva docker-compose.

Muchidimbu nezve configuration mafaira

Sezvinotevera kubva mutsanangudzo, logstash inogona kumhanyisa kana kune imwe chiteshi, panguva iyo inoda kupfuudza * .conf faira, kana kune akati wandei chiteshi, pakadai inoda kupfuudza pipelines.yml faira, iyo, zvakare. , ichabatanidza kune mafaira .conf yechiteshi chega chega.
Takatora nzira yechipiri. Kwatiri zvaiita sepasirese uye scalable. Nokudaro, takagadzira pipelines.yml, uye takagadzira dhairekitori remapaipi umo tichaisa .conf mafaira echiteshi chega chega.

Mukati memudziyo mune imwe faira yekugadzirisa - logstash.yml. Hatiibate, tinoishandisa sezvairi.

Saka, yedu dhairekitori chimiro:

Kushanda kushandisa ELK. Kugadzira logstash

Kuti tigamuchire data rekuisa, ikozvino tinofungidzira kuti iyi tcp pachiteshi 5046, uye yekubuda tichashandisa stdout.

Heino gadziriso yakapusa yekutanga kutanga. Nekuti basa rekutanga nderekutanga.

Saka, tine iyi docker-compose.yml

version: '3'

networks:
  elk:

volumes:
  elasticsearch:
    driver: local

services:

  logstash:
    container_name: logstash_one_channel
    image: docker.elastic.co/logstash/logstash:6.3.2
    networks:
      	- elk
    ports:
      	- 5046:5046
    volumes:
      	- ./config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
	- ./config/pipelines:/usr/share/logstash/config/pipelines:ro

Chii chatinoona pano?

  1. Manetiweki uye mavhoriyamu akatorwa kubva kune yekutanga docker-compose.yml (iyo inotangwa iyo yese stack) uye ndinofunga kuti haina zvakanyanya kukanganisa mufananidzo wakazara pano.
  2. Isu tinogadzira imwe logstash sevhisi kubva kudocker.elastic.co/logstash/logstash:6.3.2 mufananidzo uye toitumidza kuti logstash_one_channel.
  3. Isu tinoendesa mberi port 5046 mukati memudziyo, kune imwechete yemukati chiteshi.
  4. Isu tinoronga faira yedu yekugadzirisa pombi ./config/pipelines.yml kune faira /usr/share/logstash/config/pipelines.yml mukati memudziyo, apo logstash ichaitora uye iite kuti iverenge-chete, kana zvikaitika.
  5. Isu tinomepu iyo ./config/pipelines dhairekitori, kwatine mafaera ane chiteshi marongero, mu /usr/share/logstash/config/pipelines dhairekitori uye zvakare ita kuti iverenge-chete.

Kushanda kushandisa ELK. Kugadzira logstash

Pipelines.yml file

- pipeline.id: HABR
  pipeline.workers: 1
  pipeline.batch.size: 1
  path.config: "./config/pipelines/habr_pipeline.conf"

Imwe chiteshi ine HABR identifier uye nzira yekumisikidza faira inotsanangurwa pano.

Uye pakupedzisira faira "./config/pipelines/habr_pipeline.conf"

input {
  tcp {
    port => "5046"
   }
  }
filter {
  mutate {
    add_field => [ "habra_field", "Hello Habr" ]
    }
  }
output {
  stdout {
      
    }
  }

Ngatiregei kupinda murondedzero yayo ikozvino, ngatiedzei kuimhanyisa:

docker-compose up

Chii chatinoona?

Chigaba chatanga. Tinogona kutarisa kushanda kwayo:

echo '13123123123123123123123213123213' | nc localhost 5046

Uye isu tinoona mhinduro mumudziyo console:

Kushanda kushandisa ELK. Kugadzira logstash

Asi panguva imwecheteyo, tinoona zvakare:

logstash_one_channel | [2019-04-29T11:28:59,790][ERROR][logstash.licensechecker.licensereader] Hatikwanisi kutora ruzivo rwerezinesi kubva kune rezinesi server {:message=>β€œElasticsearch Unreachable: [http://elasticsearch:9200/][Manticore ::ResolutionFailure] elasticsearch", ...

logstash_one_channel | [2019-04-29T11:28:59,894][INFO ][logstash.pipeline ] Pipeline yakatanga zvinobudirira {:pipeline_id=>".monitoring-logstash", :thread=>"# "}

logstash_one_channel | [2019-04-29T11:28:59,988][INFO ][logstash.agent ] Mapaipi ari kushanda {:count=>2, :running_pipelines=>[:HABR, :".monitoring-logstash"], :non_running_pipelines=>[ ]}
logstash_one_channel | [2019-04-29T11:29:00,015][ERROR][logstash.inputs.metrics] X-Pack yakaiswa paLogstash asi kwete paElasticsearch. Ndokumbira uise X-Pack paElasticsearch kuti ushandise iyo yekutarisa chimiro. Zvimwe zvingave zviripo.
logstash_one_channel | [2019-04-29T11:29:00,526][INFO ][logstash.agent ] Takatanga Logstash API endpoint {:port=>9600}
logstash_one_channel | [2019-04-29T11:29:04,478][INFO ][logstash.outputs.elasticsearch] Kutarisa hutano kuona kana Elasticsearch yekubatanidza iri kushanda {:healthcheck_url=>http://elasticsearch:9200/, :path=> "/"}
logstash_one_channel | [2019-04-29T11:29:04,487][YAMBIRA ][logstash.outputs.elasticsearch] Akaedza kumutsa kubatana kune akafa ES muenzaniso, asi akawana chikanganiso. {:url=>β€œelasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][MantitionFair::Reso] elasticsearch"}
logstash_one_channel | [2019-04-29T11:29:04,704][INFO ][logstash.licensechecker.licensereader] Kutarisa hutano kuona kana Elasticsearch yekubatanidza iri kushanda {:healthcheck_url=>http://elasticsearch:9200/, :path=> "/"}
logstash_one_channel | [2019-04-29T11:29:04,710][YAMBIRA ][logstash.licensechecker.licensereader] Akaedza kumutsa kubatana kune akafa ES muenzaniso, asi akawana chikanganiso. {:url=>β€œelasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][MantitionFair::Reso] elasticsearch"}

Uye danda redu riri kufamba nguva dzose.

Pano ndaratidza mugirini meseji iyo pombi yakatangwa zvinobudirira, mutsvuku meseji yekukanganisa uye neyero meseji yekuyedza kubata. elasticsearch: 9200.
Izvi zvinoitika nekuti logstash.conf, inosanganisirwa mumufananidzo, ine cheki yekuwanikwa kweelasticsearch. Mushure mezvose, logstash inofunga kuti inoshanda sechikamu cheElk stack, asi isu takaiparadzanisa.

Zvinokwanisika kushanda, asi hazvisi nyore.

Mhinduro ndeyekudzima cheki iyi kuburikidza ne XPACK_MONITORING_ENABLED nharaunda inosiyana.

Ngatiite shanduko ku docker-compose.yml uye timhanye zvakare:

version: '3'

networks:
  elk:

volumes:
  elasticsearch:
    driver: local

services:

  logstash:
    container_name: logstash_one_channel
    image: docker.elastic.co/logstash/logstash:6.3.2
    networks:
      - elk
    environment:
      XPACK_MONITORING_ENABLED: "false"
    ports:
      - 5046:5046
   volumes:
      - ./config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
      - ./config/pipelines:/usr/share/logstash/config/pipelines:ro

Zvino, zvinhu zvose zvakanaka. Mugaba wakagadzirira kuyedza.

Tinogona kunyora zvakare mune inotevera console:

echo '13123123123123123123123213123213' | nc localhost 5046

Uye ona:

logstash_one_channel | {
logstash_one_channel |         "message" => "13123123123123123123123213123213",
logstash_one_channel |      "@timestamp" => 2019-04-29T11:43:44.582Z,
logstash_one_channel |        "@version" => "1",
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |            "host" => "gateway",
logstash_one_channel |            "port" => 49418
logstash_one_channel | }

Kushanda mukati meimwe chiteshi

Saka takatangisa. Iye zvino unogona kutora nguva yekugadzirisa logstash pachayo. Ngatisabate mapaipi.yml faira parizvino, ngationei zvatingawane nekushanda neimwe chiteshi.

Ndinofanira kutaura kuti iyo general musimboti wekushanda nechiteshi chegadziriso faira inotsanangurwa mugwaro repamutemo, pano pano
Kana iwe uchida kuverenga muchiRussia, isu takashandisa iyi article(asi mhinduro yemubvunzo iripo yekare, isu tinofanirwa kufunga nezve izvi).

Ngatiendei zvakatevedzana kubva kuInput chikamu. Tatoona basa paTCP. Chii chimwe chinganakidza pano?

Edza mameseji uchishandisa kurova kwemoyo

Pane mukana unonakidza wekugadzira otomatiki bvunzo mameseji.
Kuti uite izvi, unofanirwa kugonesa iyo heartbean plugin muchikamu chekuisa.

input {
  heartbeat {
    message => "HeartBeat!"
   }
  }

Batidza, tanga kugamuchira kamwe chete paminiti 

logstash_one_channel | {
logstash_one_channel |      "@timestamp" => 2019-04-29T13:52:04.567Z,
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |         "message" => "HeartBeat!",
logstash_one_channel |        "@version" => "1",
logstash_one_channel |            "host" => "a0667e5c57ec"
logstash_one_channel | }

Kana isu tichida kugamuchira kakawanda, tinoda kuwedzera iyo interval parameter.
Aya ndiwo matorero atichaita meseji mumasekonzi gumi ega ega.

input {
  heartbeat {
    message => "HeartBeat!"
    interval => 10
   }
  }

Kutora data kubva mufaira

Isu takasarudzawo kutarisa iyo faira mode. Kana ikashanda zvakanaka nefaira, saka pamwe hapana mumiririri anodiwa, zvirinani kushandiswa kwenzvimbo.

Maererano netsanangudzo, maitiro ekushanda anofanira kunge akafanana nemuswe -f, i.e. inoverenga mitsetse mitsva kana, sechisarudzo, inoverenga faira rese.

Saka izvo zvatinoda kuwana:

  1. Tinoda kugashira mitsara inowedzerwa kune imwe faira regi.
  2. Isu tinoda kugashira data inonyorerwa kune akati wandei mafaira egi, tichikwanisa kupatsanura izvo zvinogamuchirwa kubva kupi.
  3. Tinoda kuve nechokwadi chekuti kana logstash yatangwazve, haigamuchire iyi data zvakare.
  4. Tinoda kutarisa kuti kana logstash yakadzimwa, uye data inoramba ichinyorwa kune mafaira, zvino patinomhanya, tichagamuchira iyi data.

Kuti tiite kuyedza, ngatiwedzere imwe mutsara kune docker-compose.yml, tichivhura dhairekitori yatinoisa mafaera.

version: '3'

networks:
  elk:

volumes:
  elasticsearch:
    driver: local

services:

  logstash:
    container_name: logstash_one_channel
    image: docker.elastic.co/logstash/logstash:6.3.2
    networks:
      - elk
    environment:
      XPACK_MONITORING_ENABLED: "false"
    ports:
      - 5046:5046
   volumes:
      - ./config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
      - ./config/pipelines:/usr/share/logstash/config/pipelines:ro
      - ./logs:/usr/share/logstash/input

Uye shandura chikamu chekuisa mu habr_pipeline.conf

input {
  file {
    path => "/usr/share/logstash/input/*.log"
   }
  }

Ngatitange:

docker-compose up

Kugadzira nekunyora faira regi isu tichashandisa iwo murairo:


echo '1' >> logs/number1.log


{
logstash_one_channel |            "host" => "ac2d4e3ef70f",
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |      "@timestamp" => 2019-04-29T14:28:53.876Z,
logstash_one_channel |        "@version" => "1",
logstash_one_channel |         "message" => "1",
logstash_one_channel |            "path" => "/usr/share/logstash/input/number1.log"
logstash_one_channel | }

Hongu, inoshanda!

Panguva imwecheteyo, tinoona kuti isu tatowedzera nzira yenzira. Izvi zvinoreva kuti mune ramangwana, tichakwanisa kusefa zvinyorwa nazvo.

Ngatiedze zvakare:

echo '2' >> logs/number1.log


{
logstash_one_channel |            "host" => "ac2d4e3ef70f",
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |      "@timestamp" => 2019-04-29T14:28:59.906Z,
logstash_one_channel |        "@version" => "1",
logstash_one_channel |         "message" => "2",
logstash_one_channel |            "path" => "/usr/share/logstash/input/number1.log"
logstash_one_channel | }

Uye zvino kune imwe faira:

 echo '1' >> logs/number2.log


{
logstash_one_channel |            "host" => "ac2d4e3ef70f",
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |      "@timestamp" => 2019-04-29T14:29:26.061Z,
logstash_one_channel |        "@version" => "1",
logstash_one_channel |         "message" => "1",
logstash_one_channel |            "path" => "/usr/share/logstash/input/number2.log"
logstash_one_channel | }

Hukuru! Iyo faira yakatorwa, iyo nzira yakatsanangurwa nenzira kwayo, zvese zvakanaka.

Misa logstash uye tanga zvakare. Ngatimirirei. Kunyarara. Avo. Hatigamuchire zvinyorwa izvi zvakare.

Uye zvino kuyedza kwakanyanya.

Isa logstash uye ita:

echo '3' >> logs/number2.log
echo '4' >> logs/number1.log

Mhanya logstash zvakare uye ona:

logstash_one_channel | {
logstash_one_channel |            "host" => "ac2d4e3ef70f",
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |         "message" => "3",
logstash_one_channel |        "@version" => "1",
logstash_one_channel |            "path" => "/usr/share/logstash/input/number2.log",
logstash_one_channel |      "@timestamp" => 2019-04-29T14:48:50.589Z
logstash_one_channel | }
logstash_one_channel | {
logstash_one_channel |            "host" => "ac2d4e3ef70f",
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |         "message" => "4",
logstash_one_channel |        "@version" => "1",
logstash_one_channel |            "path" => "/usr/share/logstash/input/number1.log",
logstash_one_channel |      "@timestamp" => 2019-04-29T14:48:50.856Z
logstash_one_channel | }

Hooray! Zvese zvakatorwa.

Asi tinofanira kukuyambira pamusoro pezvinotevera. Kana iyo logstash mudziyo yadzimwa (docker stop logstash_one_channel && docker rm logstash_one_channel), hapana chichatorwa. Nzvimbo yefaira kusvika payakaverengwa yakachengetwa mukati memudziyo. Kana iwe ukaimhanyisa kubva pakutanga, inongobvuma mitsetse mitsva.

Kuverenga mafaira aripo

Ngatitii tiri kuvhura logstash kekutanga, asi isu tatova nematanda uye tinoda kuagadzirisa.
Kana tikamhanya logstash nechikamu chekuisa chatakashandisa pamusoro, hapana chatinowana. Mitsetse mitsva chete ndiyo inogadziriswa nelogstash.

Kuti mitsara kubva kumafaira aripo kuti idhonzwe kumusoro, iwe unofanirwa kuwedzera imwe mutsara kune chikamu chekuisa:

input {
  file {
    start_position => "beginning"
    path => "/usr/share/logstash/input/*.log"
   }
  }

Uyezve, pane nuance: izvi zvinongobata mafaera matsva ayo logstash isati yaonekwa. Kune mafaira akafanana akanga atova mumunda wekuona logstash, yakatoyeuka hukuru hwavo uye zvino ichangotora zvinyorwa zvitsva mavari.

Ngatimirei pano tidzidze chikamu chekuisa. Pachine zvakawanda zvingasarudzwa, asi izvo zvakatikwanira kune zvimwe zviedzo izvozvi.

Routing uye Data Shanduko

Ngatiedzei kugadzirisa dambudziko rinotevera, ngatitii tine mameseji kubva kune imwe chiteshi, mamwe acho ane ruzivo, uye mamwe mameseji ekukanganisa. Vanosiyana nema tag. Mamwe ari INFO, mamwe ari ERROR.

Tinofanira kuvaparadzanisa pakubuda. Avo. Isu tinonyora mameseji eruzivo mune imwe chiteshi, uye meseji yekukanganisa mune imwe.

Kuti uite izvi, famba kubva pachikamu chekuisa kusefa uye kubuda.

Tichishandisa chikamu chesefa, isu tichaparadzanisa meseji inouya, tichiwana hashi (key-value pairs) kubva mairi, iyo yatinogona kutoshanda nayo, i.e. disassemble maererano nemamiriro. Uye muchikamu chekubuda, isu tichasarudza mameseji uye totumira imwe neimwe kune yayo chiteshi.

Kutumira meseji na grok

Kuti upatsanure tambo dzemavara uye uwane seti yeminda kubva kwavari, kune yakakosha plugin muchikamu chesefa - grok.

Pasina kuzviisa pachangu chinangwa chekupa tsananguro yakadzama yayo pano (yeizvi ini ndinoreva zvinyorwa zvepamutemo), ndichapa muenzaniso wangu wakapfava.

Kuti uite izvi, unofanirwa kusarudza pane chimiro chetambo dzekupinza. Ndine kwavari seizvi:

1 INFO meseji1
2 ERROR meseji2

Avo. Chiziviso chinouya chekutanga, kozoti INFO/ERROR, kozoita rimwe izwi risina nzvimbo.
Hazvisi zvakaoma, asi zvakakwana kuti unzwisise nheyo yekushanda.

Saka, muchikamu chesefa chegrok plugin, isu tinofanirwa kutsanangura patani yekuisa tambo dzedu.

Ichaita seizvi:

filter {
  grok {
    match => { "message" => ["%{INT:message_id} %{LOGLEVEL:message_type} %{WORD:message_text}"] }
   }
  }

Chaizvoizvo kutaura kwenguva dzose. Yakagadzirirwa-yakagadzirwa mapatani anoshandiswa, senge INT, LOGLEVEL, WORD. Tsanangudzo yavo, pamwe nemamwe mapatani, inogona kuwanikwa pano pano

Zvino, tichipfuura nepasefa iyi, tambo yedu inoshanduka kuita hashi yezvikamu zvitatu: meseji_id, meseji_type, meseji_zvinyorwa.

Vacharatidzwa muchikamu chekubuda.

Kuendesa mameseji kuchikamu chinobuda uchishandisa iyo if command

Muchikamu chekubuda, sezvatinorangarira, taizopatsanura mameseji kuita hova mbiri. Mamwe - ari iNFO, anozobuditswa kune koni, uye nezvikanganiso, isu tinoburitsa kune faira.

Tinoparadzanisa sei mharidzo idzi? Mamiriro edambudziko anotoratidza mhinduro - mushure mezvose, isu tatova neakazvitsaurira meseji_type ndima, iyo inogona chete kutora maviri maitiro: INFO uye ERROR. Pahwaro huno ndipo patichaita sarudzo tichishandisa iyo if chirevo.

if [message_type] == "ERROR" {
        # Π—Π΄Π΅ΡΡŒ Π²Ρ‹Π²ΠΎΠ΄ΠΈΠΌ Π² Ρ„Π°ΠΉΠ»
       } else
     {
      # Π—Π΄Π΅ΡΡŒ Π²Ρ‹Π²ΠΎΠ΄ΠΈΠΌ Π² stdout
    }

Tsanangudzo yekushanda neminda nevashandisi inogona kuwanikwa muchikamu chino official manual.

Zvino, pamusoro pemhedziso chaiyo pachayo.

Console yakabuda, zvese zvakajeka pano - stdout {}

Asi zvakabuda kufaira - yeuka kuti tiri kuita zvese izvi kubva mumudziyo uye kuitira kuti faira yatinonyora mhedzisiro iwanikwe kubva kunze, tinoda kuvhura dhairekitori iri mu docker-compose.yml.

Zvose:

Chikamu chinobuda chefaira redu chinotaridzika seizvi:


output {
  if [message_type] == "ERROR" {
    file {
          path => "/usr/share/logstash/output/test.log"
          codec => line { format => "custom format: %{message}"}
         }
    } else
     {stdout {
             }
     }
  }

Mu docker-compose.yml isu tinowedzera imwe vhoriyamu yekubuda:

version: '3'

networks:
  elk:

volumes:
  elasticsearch:
    driver: local

services:

  logstash:
    container_name: logstash_one_channel
    image: docker.elastic.co/logstash/logstash:6.3.2
    networks:
      - elk
    environment:
      XPACK_MONITORING_ENABLED: "false"
    ports:
      - 5046:5046
   volumes:
      - ./config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
      - ./config/pipelines:/usr/share/logstash/config/pipelines:ro
      - ./logs:/usr/share/logstash/input
      - ./output:/usr/share/logstash/output

Isu tinoivhura, tiedze, uye toona kupatsanurwa kuita hova mbiri.

Source: www.habr.com

Voeg