ProHoster > Блог > Administrasi > Aplikasi praktis ELK. Nyetel logstash

Aplikasi praktis ELK. Nyetel logstash

Pambuka

Nalika deploying sistem liyane, kita padha ngadhepi karo perlu kanggo proses nomer akeh log beda. ELK dipilih minangka alat. Artikel iki bakal ngrembug pengalaman kita nyetel tumpukan iki.

Kita ora nemtokake tujuan kanggo njlèntrèhaké kabeh kapabilitas, nanging kita pengin musataken khusus kanggo ngrampungake masalah praktis. Iki amarga kasunyatan sing sanajan ana jumlah cukup akeh dokumentasi lan gambar siap-digawe, ana cukup akèh pitfalls, paling kita ketemu.

Kita nyebarake tumpukan liwat docker-compose. Kajaba iku, kita duwe docker-compose.yml sing ditulis kanthi apik, sing ngidini kita ngunggahake tumpukan meh tanpa masalah. Lan misale jek kamenangan wis cedhak, saiki kita bakal ngapiki sethithik supaya cocog karo kabutuhan kita lan kabeh iku.

Sayange, upaya kanggo ngatur sistem kanggo nampa lan ngolah log saka aplikasi kita ora langsung kasil. Mulane, kita mutusake manawa kudu sinau saben komponen kanthi kapisah, banjur bali menyang sambungane.

Dadi, kita miwiti karo logstash.

Lingkungan, penyebaran, mlaku Logstash ing wadhah

Kanggo penyebaran kita nggunakake docker-compose; eksperimen sing diterangake ing kene ditindakake ing MacOS lan Ubuntu 18.0.4.

Gambar logstash sing didaftar ing docker-compose.yml asli yaiku docker.elastic.co/logstash/logstash:6.3.2

Kita bakal nggunakake kanggo eksperimen.

We wrote docker-compose.yml kapisah kanggo mbukak logstash. Mesthi, iku bisa kanggo miwiti gambar saka baris printah, nanging kita ngrampungake masalah tartamtu, ngendi kita mbukak kabeh saka docker-nyipta.

Sedhela babagan file konfigurasi

Kaya ing ngisor iki saka katrangan, logstash bisa ditindakake kanggo siji saluran, mula kudu ngliwati file *.conf, utawa kanggo sawetara saluran, ing kasus iki kudu ngliwati file pipelines.yml, sing banjur , bakal nyambung menyang file .conf kanggo saben saluran.
Kita njupuk dalan kapindho. Iku ketoke kanggo kita luwih universal lan keukur. Mulane, kita nggawe pipelines.yml, lan nggawe direktori pipelines kang bakal sijine file .conf kanggo saben saluran.

Ing wadhah kasebut ana file konfigurasi liyane - logstash.yml. Kita ora ndemek, kita nggunakake minangka apa.

Dadi, struktur direktori kita:

Aplikasi praktis ELK. Nyetel logstash

Kanggo nampa data input, saiki kita nganggep yen iki tcp ing port 5046, lan kanggo output kita bakal nggunakake stdout.

Punika konfigurasi prasaja kanggo peluncuran pisanan. Amarga tugas wiwitan yaiku kanggo miwiti.

Dadi, kita duwe docker-compose.yml iki

version: '3'

networks:
  elk:

volumes:
  elasticsearch:
    driver: local

services:

  logstash:
    container_name: logstash_one_channel
    image: docker.elastic.co/logstash/logstash:6.3.2
    networks:
      	- elk
    ports:
      	- 5046:5046
    volumes:
      	- ./config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
	- ./config/pipelines:/usr/share/logstash/config/pipelines:ro

Apa sing kita deleng ing kene?

  1. Jaringan lan volume dijupuk saka docker-compose.yml asli (sing kabeh tumpukan diluncurake) lan aku mikir yen ora mengaruhi gambar sakabèhé ing kene.
  2. Kita nggawe siji layanan logstash (s) saka docker.elastic.co/logstash/logstash:6.3.2 gambar lan jeneng logstash_one_channel.
  3. We nerusake port 5046 nang wadhah, menyang port internal padha.
  4. We map file konfigurasi pipe ./config/pipelines.yml menyang file /usr/share/logstash/config/pipelines.yml nang wadhah, ngendi logstash bakal Pick munggah lan nggawe mung diwaca, mung ing kasus.
  5. Kita peta direktori ./config/pipelines, ing ngendi kita duwe file kanthi setelan saluran, menyang direktori /usr/share/logstash/config/pipelines lan uga nggawe mung diwaca.

Aplikasi praktis ELK. Nyetel logstash

file Pipelines.yml

- pipeline.id: HABR
  pipeline.workers: 1
  pipeline.batch.size: 1
  path.config: "./config/pipelines/habr_pipeline.conf"

Siji saluran kanthi pengenal HABR lan path menyang file konfigurasi diterangake ing kene.

Lan pungkasane file "./config/pipelines/habr_pipeline.conf"

input {
  tcp {
    port => "5046"
   }
  }
filter {
  mutate {
    add_field => [ "habra_field", "Hello Habr" ]
    }
  }
output {
  stdout {
      
    }
  }

Ayo ora menyang katrangan saiki, ayo nyoba mbukak:

docker-compose up

Apa sing kita deleng?

Wadhah wis diwiwiti. Kita bisa mriksa operasi:

echo '13123123123123123123123213123213' | nc localhost 5046

Lan kita ndeleng respon ing konsol wadhah:

Aplikasi praktis ELK. Nyetel logstash

Nanging ing wektu sing padha, kita uga weruh:

logstash_one_channel | [2019-04-29T11:28:59,790][ERROR][logstash.licensechecker.licensereader] Ora bisa njupuk informasi lisensi saka server lisensi {:message=>“Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore ::ResolutionFailure] elasticsearch", ...

logstash_one_channel | [2019-04-29T11:28:59,894][INFO ][logstash.pipeline ] Pipa diwiwiti kanthi sukses {:pipeline_id=>".monitoring-logstash", :thread=>"# "}

logstash_one_channel | [2019-04-29T11:28:59,988][INFO ][logstash.agent ] Pipelines mlaku {:count=>2, :running_pipelines=>[:HABR, :".monitoring-logstash"], :non_running_pipelines=>[ ]}
logstash_one_channel | [2019-04-29T11: 29: 00,015, XNUMX] [ERROR] [logstash.inputs.metrics] X-Pack diinstal ing Logstash nanging ora ing Elasticsearch. Mangga instal X-Pack ing Elasticsearch kanggo nggunakake fitur ngawasi. Fitur liyane bisa uga kasedhiya.
logstash_one_channel | [2019-04-29T11:29:00,526][INFO ][logstash.agent ] Kasil miwiti Logstash API endpoint {:port=>9600}
logstash_one_channel | [2019-04-29T11:29:04,478][INFO ][logstash.outputs.elasticsearch] Mlaku mriksa kesehatan kanggo ndeleng apa sambungan Elasticsearch bisa digunakake {:healthcheck_url=>http://elasticsearch:9200/,:path=> "/"}
logstash_one_channel | [2019-04-29T11: 29: 04,487] [WARN] [logstash.outputs.elasticsearch] Nyoba kanggo urip maneh sambungan kanggo conto ES mati, nanging entuk kesalahan. {:url =>"elastisark:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch"}
logstash_one_channel | [2019-04-29T11:29:04,704][INFO ][logstash.licensechecker.licensereader] Priksa kesehatan kanggo ndeleng apa sambungan Elasticsearch bisa digunakake {:healthcheck_url=>http://elasticsearch:9200/,:path=> "/"}
logstash_one_channel | [2019-04-29T11: 29: 04,710] [WARN] [logstash.licensechecker.licensereader] Nyoba kanggo urip maneh sambungan kanggo conto ES mati, nanging entuk kesalahan. {:url =>"elastisark:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch"}

Lan log kita creeping munggah kabeh wektu.

Ing kene aku wis nyorot pesen ijo yen pipa wis diluncurake kanthi sukses, kanthi warna abang pesen kesalahan lan ing warna kuning pesen babagan upaya kanggo hubungi elastisark: 9200.
Iki kedadeyan amarga logstash.conf, kalebu ing gambar, ngemot priksa kasedhiyan elasticsearch. Sawise kabeh, logstash nganggep kerjane minangka bagéan saka tumpukan Elk, nanging kita misahake.

Bisa uga, nanging ora trep.

Solusi kanggo mateni mriksa iki liwat variabel lingkungan XPACK_MONITORING_ENABLED.

Ayo gawe owah-owahan menyang docker-compose.yml lan mbukak maneh:

version: '3'

networks:
  elk:

volumes:
  elasticsearch:
    driver: local

services:

  logstash:
    container_name: logstash_one_channel
    image: docker.elastic.co/logstash/logstash:6.3.2
    networks:
      - elk
    environment:
      XPACK_MONITORING_ENABLED: "false"
    ports:
      - 5046:5046
   volumes:
      - ./config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
      - ./config/pipelines:/usr/share/logstash/config/pipelines:ro

Saiki, kabeh apik. Wadhah wis siyap kanggo eksperimen.

Kita bisa ngetik maneh ing konsol sabanjure:

echo '13123123123123123123123213123213' | nc localhost 5046

Lan ndeleng:

logstash_one_channel | {
logstash_one_channel |         "message" => "13123123123123123123123213123213",
logstash_one_channel |      "@timestamp" => 2019-04-29T11:43:44.582Z,
logstash_one_channel |        "@version" => "1",
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |            "host" => "gateway",
logstash_one_channel |            "port" => 49418
logstash_one_channel | }

Makarya ing siji saluran

Dadi kita diluncurake. Saiki sampeyan bisa njupuk wektu kanggo ngatur logstash dhewe. Ayo saiki ora ndemek file pipelines.yml, ayo ndeleng apa sing bisa ditindakake kanthi nggarap saluran siji.

Aku kudu ngomong yen prinsip umum nggarap file konfigurasi saluran uga diterangake ing manual resmi, kene kene
Yen sampeyan pengin maca ing basa Rusia, kita nggunakake iki artikel(nanging sintaks query ana lawas, kita kudu njupuk menyang akun).

Ayo dadi sequentially saka bagean Input. Kita wis ndeleng karya ing TCP. Apa maneh sing bisa menarik ing kene?

Test pesen nggunakake deg-degan

Ana kesempatan sing menarik kanggo ngasilake pesen tes otomatis.
Kanggo nindakake iki, sampeyan kudu ngaktifake plugin heartbean ing bagean input.

input {
  heartbeat {
    message => "HeartBeat!"
   }
  }

Nguripake, miwiti nampa sapisan menit 

logstash_one_channel | {
logstash_one_channel |      "@timestamp" => 2019-04-29T13:52:04.567Z,
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |         "message" => "HeartBeat!",
logstash_one_channel |        "@version" => "1",
logstash_one_channel |            "host" => "a0667e5c57ec"
logstash_one_channel | }

Yen kita pengin nampa luwih kerep, kita kudu nambah parameter interval.
Iki carane kita bakal nampa pesen saben 10 detik.

input {
  heartbeat {
    message => "HeartBeat!"
    interval => 10
   }
  }

Njupuk data saka file

Kita uga mutusake kanggo ndeleng mode file. Yen bisa digunakake kanthi apik karo file kasebut, mula ora ana agen sing dibutuhake, paling ora kanggo panggunaan lokal.

Miturut katrangan, mode operasi kudu padha karo buntut -f, i.e. maca baris anyar utawa, minangka pilihan, maca kabeh file.

Dadi apa sing arep kita entuk:

  1. Kita pengin nampa baris sing ditambahake menyang siji file log.
  2. Kita pengin nampa data sing ditulis ing sawetara file log, nalika bisa misahake apa sing ditampa saka ngendi.
  3. Kita pengin mesthekake yen nalika logstash diwiwiti maneh, data kasebut ora bakal ditampa maneh.
  4. Kita pengin mriksa yen logstash dipateni, lan data terus ditulis menyang file, banjur nalika mbukak, kita bakal nampa data iki.

Kanggo nindakake eksperimen, ayo nambah baris liyane menyang docker-compose.yml, mbukak direktori sing dilebokake file kasebut.

version: '3'

networks:
  elk:

volumes:
  elasticsearch:
    driver: local

services:

  logstash:
    container_name: logstash_one_channel
    image: docker.elastic.co/logstash/logstash:6.3.2
    networks:
      - elk
    environment:
      XPACK_MONITORING_ENABLED: "false"
    ports:
      - 5046:5046
   volumes:
      - ./config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
      - ./config/pipelines:/usr/share/logstash/config/pipelines:ro
      - ./logs:/usr/share/logstash/input

Lan ngganti bagean input ing habr_pipeline.conf

input {
  file {
    path => "/usr/share/logstash/input/*.log"
   }
  }

Ayo miwiti:

docker-compose up

Kanggo nggawe lan nulis file log kita bakal nggunakake printah:


echo '1' >> logs/number1.log


{
logstash_one_channel |            "host" => "ac2d4e3ef70f",
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |      "@timestamp" => 2019-04-29T14:28:53.876Z,
logstash_one_channel |        "@version" => "1",
logstash_one_channel |         "message" => "1",
logstash_one_channel |            "path" => "/usr/share/logstash/input/number1.log"
logstash_one_channel | }

Ya, iku bisa!

Ing wektu sing padha, kita weruh manawa kita wis nambah lapangan path kanthi otomatis. Iki tegese ing mangsa ngarep, kita bakal bisa nyaring cathetan kasebut.

Ayo coba maneh:

echo '2' >> logs/number1.log


{
logstash_one_channel |            "host" => "ac2d4e3ef70f",
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |      "@timestamp" => 2019-04-29T14:28:59.906Z,
logstash_one_channel |        "@version" => "1",
logstash_one_channel |         "message" => "2",
logstash_one_channel |            "path" => "/usr/share/logstash/input/number1.log"
logstash_one_channel | }

Lan saiki menyang file liyane:

 echo '1' >> logs/number2.log


{
logstash_one_channel |            "host" => "ac2d4e3ef70f",
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |      "@timestamp" => 2019-04-29T14:29:26.061Z,
logstash_one_channel |        "@version" => "1",
logstash_one_channel |         "message" => "1",
logstash_one_channel |            "path" => "/usr/share/logstash/input/number2.log"
logstash_one_channel | }

apik tenan! File kasebut dijupuk, path kasebut ditemtokake kanthi bener, kabeh apik.

Mungkasi logstash lan miwiti maneh. Ayo ngenteni. meneng. Sing. Kita ora nampa cathetan iki maneh.

Lan saiki eksperimen paling wani.

Instal logstash lan eksekusi:

echo '3' >> logs/number2.log
echo '4' >> logs/number1.log

Mbukak logstash maneh lan ndeleng:

logstash_one_channel | {
logstash_one_channel |            "host" => "ac2d4e3ef70f",
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |         "message" => "3",
logstash_one_channel |        "@version" => "1",
logstash_one_channel |            "path" => "/usr/share/logstash/input/number2.log",
logstash_one_channel |      "@timestamp" => 2019-04-29T14:48:50.589Z
logstash_one_channel | }
logstash_one_channel | {
logstash_one_channel |            "host" => "ac2d4e3ef70f",
logstash_one_channel |     "habra_field" => "Hello Habr",
logstash_one_channel |         "message" => "4",
logstash_one_channel |        "@version" => "1",
logstash_one_channel |            "path" => "/usr/share/logstash/input/number1.log",
logstash_one_channel |      "@timestamp" => 2019-04-29T14:48:50.856Z
logstash_one_channel | }

Hore! Kabeh dijupuk.

Nanging kita kudu ngelingake sampeyan babagan ing ngisor iki. Yen wadhah logstash wis dibusak (docker mandeg logstash_one_channel && docker rm logstash_one_channel), banjur ora ana sing bakal dijupuk. Posisi file nganti sing diwaca disimpen ing wadhah kasebut. Yen sampeyan mbukak saka ngeruk, iku mung bakal nampa baris anyar.

Maca file sing ana

Contone, kita ngluncurake logstash kanggo pisanan, nanging kita wis duwe log lan pengin diproses.
Yen kita mbukak logstash karo bagean input sing digunakake ing ndhuwur, kita ora bakal entuk apa-apa. Mung baris anyar bakal diproses dening logstash.

Supaya baris saka file sing ana bisa ditarik munggah, sampeyan kudu nambah baris tambahan menyang bagean input:

input {
  file {
    start_position => "beginning"
    path => "/usr/share/logstash/input/*.log"
   }
  }

Kajaba iku, ana nuansa: iki mung mengaruhi file anyar sing durung katon logstash. Kanggo file sing padha sing wis ana ing lapangan tampilan logstash, wis ngelingi ukurane lan saiki mung bakal njupuk entri anyar.

Ayo mandheg ing kene lan sinau bagean input. Isih ana akeh opsi, nanging cukup kanggo kita kanggo eksperimen luwih saiki.

Routing lan Transformasi Data

Ayo nyoba kanggo ngatasi masalah ing ngisor iki, ayo ngomong kita duwe pesen saka siji saluran, sawetara iku informasi, lan sawetara pesen kesalahan. Padha beda-beda miturut tag. Sawetara INFO, liyane ERROR.

Kita kudu misahake wong-wong mau ing metu. Sing. Kita nulis pesen informasi ing siji saluran, lan pesen kesalahan ing saluran liyane.

Kanggo nindakake iki, pindhah saka bagean input kanggo nyaring lan output.

Nggunakake bagean panyaring, kita bakal ngurai pesen sing mlebu, entuk hash (pasangan kunci-nilai) saka iku, sing wis bisa digunakake, yaiku. disassemble miturut kahanan. Lan ing bagean output, kita bakal milih pesen lan ngirim saben siji menyang saluran dhewe.

Parsing pesen karo grok

Kanggo ngurai senar teks lan entuk sakumpulan kolom, ana plugin khusus ing bagean filter - grok.

Tanpa nyetel tujuan kanggo menehi katrangan rinci babagan iki ing kene (kanggo iki aku deleng dokumentasi resmi), Aku bakal menehi conto prasaja.

Kanggo nindakake iki, sampeyan kudu nemtokake format strings input. Aku duwe wong kaya iki:

1 pesen INFO1
2 Pesen ERROR2

Sing. Identifier teka dhisik, banjur INFO/ERROR, banjur sawetara tembung tanpa spasi.
Ora angel, nanging cukup kanggo ngerti prinsip operasi.

Dadi, ing bagean filter saka plugin grok, kita kudu nemtokake pola kanggo parsing strings kita.

Bakal katon kaya iki:

filter {
  grok {
    match => { "message" => ["%{INT:message_id} %{LOGLEVEL:message_type} %{WORD:message_text}"] }
   }
  }

Ateges iku ekspresi biasa. Pola sing wis siap digunakake, kayata INT, LOGLEVEL, WORD. Katrangane, uga pola liyane, bisa ditemokake ing kene kene

Saiki, liwat filter iki, string kita bakal dadi hash saka telung kolom: message_id, message_type, message_text.

Padha bakal ditampilake ing bagean output.

Nuntun pesen menyang bagean output nggunakake printah yen

Ing bagean output, kaya sing kita eling, kita bakal mbagi pesen dadi rong aliran. Sawetara - kang iNFO, bakal output kanggo console, lan karo kasalahan, kita bakal output file.

Kepiye carane misahake pesen kasebut? Kondisi masalah kasebut wis menehi solusi - sawise kabeh, kita wis duwe kolom message_type khusus, sing mung bisa njupuk rong nilai: INFO lan ERROR. Ing basis iki kita bakal nggawe pilihan nggunakake statement yen.

if [message_type] == "ERROR" {
        # Здесь выводим в файл
       } else
     {
      # Здесь выводим в stdout
    }

Katrangan babagan nggarap lapangan lan operator bisa ditemokake ing bagean iki manual resmi.

Saiki, babagan kesimpulan nyata dhewe.

Output konsol, kabeh wis jelas ing kene - stdout {}

Nanging output kanggo file - elinga yen kita mlaku kabeh iki saka wadhah lan supaya file kang kita nulis asil bisa diakses saka njaba, kita kudu mbukak direktori iki ing docker-compose.yml.

Total:

Bagean output file kita katon kaya iki:


output {
  if [message_type] == "ERROR" {
    file {
          path => "/usr/share/logstash/output/test.log"
          codec => line { format => "custom format: %{message}"}
         }
    } else
     {stdout {
             }
     }
  }

Ing docker-compose.yml kita nambah volume liyane kanggo output:

version: '3'

networks:
  elk:

volumes:
  elasticsearch:
    driver: local

services:

  logstash:
    container_name: logstash_one_channel
    image: docker.elastic.co/logstash/logstash:6.3.2
    networks:
      - elk
    environment:
      XPACK_MONITORING_ENABLED: "false"
    ports:
      - 5046:5046
   volumes:
      - ./config/pipelines.yml:/usr/share/logstash/config/pipelines.yml:ro
      - ./config/pipelines:/usr/share/logstash/config/pipelines:ro
      - ./logs:/usr/share/logstash/input
      - ./output:/usr/share/logstash/output

Kita miwiti, nyoba, lan ndeleng divisi dadi rong aliran.

Source: www.habr.com

Add a comment