ProHoster > Блог > Gudanarwa > Abubuwan da aka bayar na ELK. Saita logstash

Abubuwan da aka bayar na ELK. Saita logstash

Gabatarwar

Yayin da ake tura wani tsarin, mun fuskanci buƙatar aiwatar da babban adadin rajistan ayyukan. An zaɓi ELK azaman kayan aiki. Wannan labarin zai tattauna kwarewarmu wajen kafa wannan tari.

Ba mu kafa maƙasudi don bayyana dukkan ƙarfinsa ba, amma muna so mu mai da hankali musamman kan magance matsaloli masu amfani. Wannan shi ne saboda gaskiyar cewa ko da yake akwai adadi mai yawa na takardu da hotuna da aka shirya, akwai matsala masu yawa, aƙalla mun same su.

Mun tura tari ta hanyar docker-compose. Bugu da ƙari, muna da docker-compose.yml da aka rubuta da kyau, wanda ya ba mu damar ɗaga tarin kusan ba tare da matsala ba. Kuma ya zama kamar a gare mu cewa nasara ta rigaya ta kusa, yanzu za mu ɗanɗana shi don dacewa da bukatunmu kuma shi ke nan.

Abin takaici, ƙoƙarinmu na daidaita tsarin don karɓa da sarrafa rajistan ayyukan daga aikace-aikacenmu bai yi nasara nan da nan ba. Saboda haka, mun yanke shawarar cewa yana da daraja nazarin kowane bangare daban, sa'an nan kuma komawa ga haɗin gwiwar su.

Don haka, mun fara da logstash.

Muhalli, turawa, Gudun Logstash a cikin akwati

Don turawa muna amfani da docker-compose; gwaje-gwajen da aka kwatanta a nan an yi su akan MacOS da Ubuntu 18.0.4.

Hoton logstash da aka yi rajista a cikin ainihin docker-compose.yml shine docker.elastic.co/logstash/logstash:6.3.2

Za mu yi amfani da shi don gwaji.

Mun rubuta wani docker-compose.yml daban don gudanar da logstash. Tabbas, yana yiwuwa a ƙaddamar da hoton daga layin umarni, amma muna magance takamaiman matsala, inda muke gudanar da komai daga docker-compose.

A taƙaice game da fayilolin sanyi

Kamar yadda ya zo daga bayanin, logstash za a iya gudu ko dai don tashar daya, a cikin wannan yanayin yana buƙatar wucewa fayil ɗin * .conf, ko kuma tashoshi da yawa, a cikin wannan yanayin yana buƙatar wuce fayil ɗin pipelines.yml, wanda, bi da bi. , zai haɗa zuwa fayilolin .conf ga kowane tashoshi.
Mun dauki hanya ta biyu. Ya zama kamar a gare mu fiye da kowa da kowa kuma mai daidaitawa. Saboda haka, mun ƙirƙiri pipelines.yml, kuma mun yi kundin adireshi na bututun da za mu sanya fayilolin .conf ga kowane tashoshi.

A cikin akwati akwai wani fayil ɗin daidaitawa - logstash.yml. Ba mu taba shi ba, muna amfani da shi yadda yake.

Don haka, tsarin littafin mu:

Abubuwan da aka bayar na ELK. Saita logstash

Don karɓar bayanan shigarwa, a yanzu muna ɗauka cewa wannan shine tcp akan tashar jiragen ruwa 5046, kuma don fitarwa za mu yi amfani da stdout.

Anan akwai tsari mai sauƙi don ƙaddamar da farko. Domin aikin farko shine farawa.

Don haka, muna da wannan docker-compose.yml

version: '3'

networks:
  elk:

volumes:
  elasticsearch:
    driver: local

services:

  logstash:
    container_name: logstash_one_channel
    image: docker.elastic.co/logstash/logstash:6.3.2
    networks:
      	- elk
    ports:
      	- 5046:5046
    volumes:
      	- ./config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
	- ./config/pipelines:/usr/share/logstash/config/pipelines:ro

Me muke gani a nan?

  1. An ɗauko hanyoyin sadarwa da kundin bayanai daga ainihin docker-compose.yml (wanda aka ƙaddamar da duka tari) kuma ina tsammanin ba su da tasiri sosai kan hoto gaba ɗaya a nan.
  2. Mun ƙirƙiri sabis (s) logstash guda ɗaya daga docker.elastic.co/logstash/logstash:6.3.2 hoton kuma sanya masa suna logstash_one_channel.
  3. Muna tura tashar jiragen ruwa 5046 a cikin akwati, zuwa tashar ciki guda ɗaya.
  4. Muna taswirar fayil ɗin daidaitawar bututunmu ./config/pipelines.yml zuwa fayil /usr/share/logstash/config/pipelines.yml a cikin akwati, inda logstash zai ɗauka ya sanya shi karantawa kawai, kawai idan.
  5. Muna taswirar ./config/pipelines directory, inda muke da fayiloli tare da saitunan tashoshi, cikin /usr/share/logstash/config/pipelines directory kuma mu sanya shi karantawa kawai.

Abubuwan da aka bayar na ELK. Saita logstash

Pipelines.yml fayil

- pipeline.id: HABR
  pipeline.workers: 1
  pipeline.batch.size: 1
  path.config: "./config/pipelines/habr_pipeline.conf"

Tashar ɗaya tare da mai gano HABR da hanyar zuwa fayil ɗin sanyi an kwatanta su anan.

Kuma a ƙarshe fayil ɗin "./config/pipelines/habr_pipeline.conf"

input {
  tcp {
    port => "5046"
   }
  }
filter {
  mutate {
    add_field => [ "habra_field", "Hello Habr" ]
    }
  }
output {
  stdout {
      
    }
  }

Kada mu shiga cikin bayaninsa a yanzu, bari mu yi ƙoƙarin gudanar da shi:

docker-compose up

Me muke gani?

Kwantena ya fara. Za mu iya duba aikinsa:

echo '13123123123123123123123213123213' | nc localhost 5046

Kuma muna ganin martani a cikin na'ura mai kwakwalwa:

Abubuwan da aka bayar na ELK. Saita logstash

Amma a lokaci guda, muna kuma ganin:

logstash_one_channel | [2019-04-29T11:28:59,790][ERROR][logstash.licensechecker.licensereader] Rashin iya dawo da bayanan lasisi daga uwar garken lasisi {: saƙo=>“Ba a iya samun Elasticsearch: [http://elasticsearch:9200/] [Manticore] ::ResolutionFailure] elasticsearch", ...

logstash_one_channel | [2019-04-29T11:28:59,894] [INFO ][logstash.pipeline ] An fara bututun bututun cikin nasara {: pipeline_id=>".monitoring-logstash", :thread=>"# "}

logstash_one_channel | [2019-04-29T11:28:59,988] [INFO] [logstash.agent] Bututun da ke gudana {:count=>2, : running_pipelines=>[:HABR, :". monitoring-logstash"], :non_running_pipelines=>[ ]}
logstash_one_channel | [2019-04-29T11:29:00,015][ERROR][logstash.inputs.metrics] X-Pack an shigar dashi akan Logstash amma ba akan Elasticsearch ba. Da fatan za a shigar da fakitin X akan Elasticsearch don amfani da fasalin sa ido. Wasu fasaloli na iya samuwa.
logstash_one_channel | [2019-04-29T11:29:00,526] [INFO] [logstash.agent] Nasarar fara Logstash API ƙarshen ƙarshen {: tashar jiragen ruwa=>9600}
logstash_one_channel | [2019-04-29T11:29:04,478] [INFO] [logstash.outputs.elasticsearch] Gudanar da binciken lafiya don ganin ko haɗin binciken Elasticsearch yana aiki {:healthcheck_url=>http://elasticsearch:9200/, : hanya=> "/"}
logstash_one_channel | [2019-04-29T11:29:04,487] [WARN] [logstash.outputs.elasticsearch] An yi ƙoƙarin tayar da haɗi zuwa misalin ES da ya mutu, amma ya sami kuskure. {:url=>"maganin roba: 9200/", : error_type => LogStash :: Abubuwan da aka samu :: ElasticSearch :: HttpClient :: Pool :: HostUnreachableError, : error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/] [http://elasticsearch:XNUMX/] [Manticore :: ResolutionFailure] elasticsearch"}
logstash_one_channel | [2019-04-29T11:29:04,704] [INFO] [logstash.licensechecker.licensereader] Gudanar da binciken lafiya don ganin ko haɗin Elasticsearch yana aiki {:healthcheck_url=>http://elasticsearch:9200/, : hanya=> "/"}
logstash_one_channel | [2019-04-29T11:29:04,710] [WARN] [logstash.licensechecker.licensereader] Yayi ƙoƙarin tayar da haɗi zuwa misalin ES da ya mutu, amma ya sami kuskure. {:url=>"maganin roba: 9200/", : error_type => LogStash :: Abubuwan da aka samu :: ElasticSearch :: HttpClient :: Pool :: HostUnreachableError, : error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/] [http://elasticsearch:XNUMX/] [Manticore :: ResolutionFailure] elasticsearch"}

Kuma log ɗinmu yana rarrafe koyaushe.

Anan na yi tsokaci a koren sakon da aka kaddamar da bututun mai cikin nasara, a ja sakon kuskuren da kuma cikin launin rawaya sakon game da yunkurin tuntuɓar. maganin roba: 9200.
Wannan yana faruwa ne saboda logstash.conf, wanda aka haɗa a cikin hoton, ya ƙunshi cak don samun elasticsearch. Bayan haka, logstash yana ɗauka cewa yana aiki azaman ɓangare na tarin Elk, amma mun raba shi.

Yana yiwuwa a yi aiki, amma bai dace ba.

Maganin shine a kashe wannan cak ta hanyar canjin yanayi na XPACK_MONITORING_ENABLED.

Bari mu canza zuwa docker-compose.yml kuma mu sake gudanar da shi:

version: '3'

networks:
  elk:

volumes:
  elasticsearch:
    driver: local

services:

  logstash:
    container_name: logstash_one_channel
    image: docker.elastic.co/logstash/logstash:6.3.2
    networks:
      - elk
    environment:
      XPACK_MONITORING_ENABLED: "false"
    ports:
      - 5046:5046
   volumes:
      - ./config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
      - ./config/pipelines:/usr/share/logstash/config/pipelines:ro

Yanzu, komai yana da kyau. An shirya kwandon don gwaji.

Za mu iya sake rubutawa a cikin na'ura mai kwakwalwa ta gaba:

echo '13123123123123123123123213123213' | nc localhost 5046

Kuma duba:

logstash_one_channel | {
logstash_one_channel |         "message" => "13123123123123123123123213123213",
logstash_one_channel |      "@timestamp" => 2019-04-29T11:43:44.582Z,
logstash_one_channel |        "@version" => "1",
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |            "host" => "gateway",
logstash_one_channel |            "port" => 49418
logstash_one_channel | }

Yin aiki a cikin tashoshi ɗaya

Don haka muka kaddamar. Yanzu zaku iya ɗaukar lokaci don saita logstash kanta. Kada mu taɓa fayil ɗin pipelines.yml a yanzu, bari mu ga abin da za mu iya samu ta yin aiki tare da tashar ɗaya.

Dole ne in faɗi cewa babban ƙa'idar aiki tare da fayil ɗin daidaitawar tashar an bayyana shi da kyau a cikin jagorar hukuma, anan a nan
Idan kuna son karantawa cikin Rashanci, mun yi amfani da wannan labarin(amma kalmar tambaya a can akwai tsohuwar, muna buƙatar yin la'akari da wannan).

Bari mu tafi a jere daga sashin shigarwa. Mun riga mun ga aiki akan TCP. Menene kuma zai iya zama mai ban sha'awa a nan?

Gwada saƙon ta amfani da bugun zuciya

Akwai irin wannan dama mai ban sha'awa don samar da saƙonnin gwaji ta atomatik.
Don yin wannan, kuna buƙatar kunna plugin ɗin bugun zuciya a cikin sashin shigarwa.

input {
  heartbeat {
    message => "HeartBeat!"
   }
  }

Kunna shi, fara karba sau ɗaya a minti daya 

logstash_one_channel | {
logstash_one_channel |      "@timestamp" => 2019-04-29T13:52:04.567Z,
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |         "message" => "HeartBeat!",
logstash_one_channel |        "@version" => "1",
logstash_one_channel |            "host" => "a0667e5c57ec"
logstash_one_channel | }

Idan muna son karɓar sau da yawa, muna buƙatar ƙara ma'aunin tazara.
Ta haka ne za mu karɓi saƙo kowane daƙiƙa 10.

input {
  heartbeat {
    message => "HeartBeat!"
    interval => 10
   }
  }

Maido da bayanai daga fayil

Mun kuma yanke shawarar duba yanayin fayil. Idan yana aiki lafiya tare da fayil ɗin, to watakila ba a buƙatar wakili, aƙalla don amfanin gida.

Dangane da bayanin, yanayin aiki yakamata yayi kama da wutsiya -f, i.e. yana karanta sababbin layi ko, azaman zaɓi, karanta fayil ɗin gabaɗaya.

Don haka abin da muke so mu samu:

  1. Muna son karɓar layukan da aka liƙa zuwa fayil ɗin log ɗaya.
  2. Muna son karɓar bayanan da aka rubuta zuwa fayilolin log da yawa, yayin da muke iya raba abin da aka karɓa daga ina.
  3. Muna son tabbatar da cewa lokacin da aka sake kunna logstash, baya sake karɓar wannan bayanan.
  4. Muna so mu bincika cewa idan logstash yana kashe, kuma ana ci gaba da rubuta bayanai zuwa fayiloli, to lokacin da muka kunna shi, za mu karɓi wannan bayanan.

Don gudanar da gwajin, bari mu ƙara ƙarin layi ɗaya zuwa docker-compose.yml, buɗe kundin adireshi wanda muka sanya fayilolin.

version: '3'

networks:
  elk:

volumes:
  elasticsearch:
    driver: local

services:

  logstash:
    container_name: logstash_one_channel
    image: docker.elastic.co/logstash/logstash:6.3.2
    networks:
      - elk
    environment:
      XPACK_MONITORING_ENABLED: "false"
    ports:
      - 5046:5046
   volumes:
      - ./config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
      - ./config/pipelines:/usr/share/logstash/config/pipelines:ro
      - ./logs:/usr/share/logstash/input

Kuma canza sashin shigarwa a habr_pipeline.conf

input {
  file {
    path => "/usr/share/logstash/input/*.log"
   }
  }

Bari mu fara:

docker-compose up

Don ƙirƙira da rubuta fayilolin log za mu yi amfani da umarnin:


echo '1' >> logs/number1.log


{
logstash_one_channel |            "host" => "ac2d4e3ef70f",
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |      "@timestamp" => 2019-04-29T14:28:53.876Z,
logstash_one_channel |        "@version" => "1",
logstash_one_channel |         "message" => "1",
logstash_one_channel |            "path" => "/usr/share/logstash/input/number1.log"
logstash_one_channel | }

Ee, yana aiki!

A lokaci guda, muna ganin cewa mun ƙara filin hanya ta atomatik. Wannan yana nufin cewa a nan gaba, za mu iya tace bayanai da shi.

Mu sake gwadawa:

echo '2' >> logs/number1.log


{
logstash_one_channel |            "host" => "ac2d4e3ef70f",
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |      "@timestamp" => 2019-04-29T14:28:59.906Z,
logstash_one_channel |        "@version" => "1",
logstash_one_channel |         "message" => "2",
logstash_one_channel |            "path" => "/usr/share/logstash/input/number1.log"
logstash_one_channel | }

Kuma yanzu zuwa wani fayil:

 echo '1' >> logs/number2.log


{
logstash_one_channel |            "host" => "ac2d4e3ef70f",
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |      "@timestamp" => 2019-04-29T14:29:26.061Z,
logstash_one_channel |        "@version" => "1",
logstash_one_channel |         "message" => "1",
logstash_one_channel |            "path" => "/usr/share/logstash/input/number2.log"
logstash_one_channel | }

Mai girma! An ɗauki fayil ɗin, an ƙayyade hanyar daidai, komai yana da kyau.

Dakatar da logstash kuma sake farawa. Mu jira. Shiru. Wadancan. Ba mu sake samun waɗannan bayanan ba.

Kuma yanzu mafi m gwaji.

Shigar logstash kuma aiwatar:

echo '3' >> logs/number2.log
echo '4' >> logs/number1.log

Shigar logstash kuma duba:

logstash_one_channel | {
logstash_one_channel |            "host" => "ac2d4e3ef70f",
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |         "message" => "3",
logstash_one_channel |        "@version" => "1",
logstash_one_channel |            "path" => "/usr/share/logstash/input/number2.log",
logstash_one_channel |      "@timestamp" => 2019-04-29T14:48:50.589Z
logstash_one_channel | }
logstash_one_channel | {
logstash_one_channel |            "host" => "ac2d4e3ef70f",
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |         "message" => "4",
logstash_one_channel |        "@version" => "1",
logstash_one_channel |            "path" => "/usr/share/logstash/input/number1.log",
logstash_one_channel |      "@timestamp" => 2019-04-29T14:48:50.856Z
logstash_one_channel | }

Hooray! An dauko komai.

Amma dole ne mu gargaɗe ku game da waɗannan abubuwan. Idan an goge akwati mai logstash (docker stop logstash_one_channel && docker rm logstash_one_channel), to babu abin da zai ɗauka. Matsayin fayil ɗin har zuwa lokacin da aka karanta shi an adana shi a cikin akwati. Idan kun gudanar da shi daga karce, zai karɓi sababbin layi kawai.

Karanta fayilolin da ke akwai

Bari mu ce muna ƙaddamar da logstash a karon farko, amma muna da rajistan ayyukan kuma muna son sarrafa su.
Idan muka gudu logstash tare da sashin shigarwa da muka yi amfani da shi a sama, ba za mu sami komai ba. Sabbin layika kawai za a sarrafa ta logstash.

Domin a ciro layukan daga fayilolin da ke akwai, yakamata ku ƙara ƙarin layi zuwa sashin shigarwa:

input {
  file {
    start_position => "beginning"
    path => "/usr/share/logstash/input/*.log"
   }
  }

Haka kuma, akwai nuance: wannan kawai yana shafar sabbin fayilolin da logstash bai taɓa gani ba. Don fayilolin iri ɗaya waɗanda suka riga sun kasance a cikin filin ra'ayi na logstash, ya riga ya tuna girman su kuma yanzu kawai zai ɗauki sabbin shigarwar a cikinsu.

Bari mu tsaya anan mu yi nazarin sashin shigarwa. Har yanzu akwai zaɓuɓɓuka da yawa, amma wannan ya ishe mu don ƙarin gwaje-gwaje a yanzu.

Ragewa da Canjin Bayanai

Mu yi kokarin magance matsalar nan, a ce muna da sakonni daga wata tasha, wasu na bayanai ne, wasu kuma sakonni ne na kuskure. Sun bambanta ta tag. Wasu INFO ne, wasu kuma KUSKURE ne.

Muna buƙatar raba su a wurin fita. Wadancan. Muna rubuta saƙonnin bayanai a cikin tashoshi ɗaya, saƙon kuskure a wata.

Don yin wannan, matsa daga sashin shigarwa don tacewa da fitarwa.

Yin amfani da sashin tacewa, za mu rarraba saƙon mai shigowa, samun zanta (maɓalli-darajar nau'i-nau'i) daga gare ta, wanda za mu iya aiki da shi, watau. tarwatsa bisa ga sharadi. Kuma a bangaren fitarwa, za mu zabar sakonni mu aika kowanne zuwa tasharsa.

Fassarar sako tare da grok

Don rarraba kirtani na rubutu da samun saitin filayen daga gare su, akwai plugin na musamman a cikin sashin tace - grok.

Ba tare da kafa kaina ba na ba da cikakken bayaninsa a nan (don wannan ina nufin takardun shaida), Zan ba da misali mai sauƙi na.

Don yin wannan, kuna buƙatar yanke shawara akan tsarin kirtani shigarwa. Ina da su kamar haka:

1 saƙon INFO1
2 SAKON KUSKURE2

Wadancan. Mai ganowa ya fara zuwa, sannan INFO/KUSKURE, sannan wata kalma ba tare da sarari ba.
Ba shi da wahala, amma ya isa ya fahimci ka'idar aiki.

Don haka, a cikin ɓangaren tacewa na grok plugin, dole ne mu ayyana tsari don tantance igiyoyin mu.

Zai yi kama da haka:

filter {
  grok {
    match => { "message" => ["%{INT:message_id} %{LOGLEVEL:message_type} %{WORD:message_text}"] }
   }
  }

Ainihin magana ce ta yau da kullun. Ana amfani da tsarin da aka shirya, kamar INT, LOGLEVEL, WORD. Ana iya samun bayanin su, da sauran alamu a nan a nan

Yanzu, wucewa ta wannan tacewa, igiyoyin mu za su juya zuwa zanta na filayen guda uku: message_id, message_type, message_text.

Za a nuna su a cikin sashin fitarwa.

Gudanar da saƙon zuwa sashin fitarwa ta amfani da umarni idan

A cikin sashin fitarwa, kamar yadda muke tunawa, za mu raba saƙon zuwa rafuka biyu. Wasu - waɗanda suke iNFO, za a fitar da su zuwa na'ura wasan bidiyo, kuma tare da kurakurai, za mu fitar zuwa fayil.

Ta yaya zamu raba wadannan sakonni? Yanayin matsalar ya riga ya ba da shawarar mafita - bayan haka, mun riga mun sami takamaiman filin saƙon_type, wanda kawai zai iya ɗaukar dabi'u biyu: INFO da ERROR. A kan haka ne za mu yi zaɓi ta amfani da idan sanarwa.

if [message_type] == "ERROR" {
        # Здесь выводим в файл
       } else
     {
      # Здесь выводим в stdout
    }

Ana iya samun bayanin aiki tare da filaye da masu aiki a wannan sashe hukuma manual.

Yanzu, game da ainihin ƙarshe kanta.

Fitowar Console, komai a bayyane yake a nan - stdout {}

Amma fitarwa zuwa fayil - ku tuna cewa muna gudanar da wannan duka daga akwati kuma domin fayil ɗin da muka rubuta sakamakon ya zama mai isa daga waje, muna buƙatar buɗe wannan jagorar a docker-compose.yml.

Jimlar:

Sashen fitarwa na fayil ɗin mu yayi kama da haka:


output {
  if [message_type] == "ERROR" {
    file {
          path => "/usr/share/logstash/output/test.log"
          codec => line { format => "custom format: %{message}"}
         }
    } else
     {stdout {
             }
     }
  }

A cikin docker-compose.yml muna ƙara wani ƙara don fitarwa:

version: '3'

networks:
  elk:

volumes:
  elasticsearch:
    driver: local

services:

  logstash:
    container_name: logstash_one_channel
    image: docker.elastic.co/logstash/logstash:6.3.2
    networks:
      - elk
    environment:
      XPACK_MONITORING_ENABLED: "false"
    ports:
      - 5046:5046
   volumes:
      - ./config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
      - ./config/pipelines:/usr/share/logstash/config/pipelines:ro
      - ./logs:/usr/share/logstash/input
      - ./output:/usr/share/logstash/output

Mun kaddamar da shi, gwada shi, kuma mu ga rarraba zuwa koguna biyu.

source: www.habr.com

Add a comment