🥇Splunk Universal Forwarder ing Docker minangka kolektor log sistem

Splunk minangka salah sawijining produk koleksi lan analisis log komersial sing paling dikenal. Malah saiki, nalika dodolan ora digawe maneh ing Rusia, iki dudu alesan kanggo ora nulis instruksi / cara kanggo produk iki.

Tujuan: ngumpulake log sistem saka kelenjar docker ing Splunk tanpa ngganti konfigurasi mesin inang

Aku kaya kanggo miwiti karo pendekatan resmi, kang katon rada aneh nalika nggunakake Docker.
Link menyang hub Docker
Apa sing kita duwe:

1. Gambar Pullim

$ docker pull splunk/universalforwarder:latest

2. Miwiti wadhah kanthi paramèter sing dibutuhake

$ docker run -d  -p 9997:9997 -e 'SPLUNK_START_ARGS=--accept-license' -e 'SPLUNK_PASSWORD=<password>' splunk/universalforwarder:latest

3. We menyang wadhah

docker exec -it <container-id> /bin/bash

Sabanjure, kita dijaluk menyang alamat sing dikenal ing dokumentasi.

Lan konfigurasi wadhah kasebut sawise diwiwiti:


./splunk add forward-server <host name or ip address>:<listening port>
./splunk add monitor /var/log
./splunk restart

Ngenteni. opo?

Nanging kejutan ora mungkasi ana. Yen sampeyan mbukak wadhah saka gambar resmi ing mode interaktif, sampeyan bakal weruh ing ngisor iki:

A bit saka kuciwo


$ docker run -it -p 9997:9997 -e 'SPLUNK_START_ARGS=--accept-license' -e 'SPLUNK_PASSWORD=password' splunk/universalforwarder:latest

PLAY [Run default Splunk provisioning] *******************************************************************************************************************************************************************************************************
Tuesday 09 April 2019  13:40:38 +0000 (0:00:00.096)       0:00:00.096 *********

TASK [Gathering Facts] ***********************************************************************************************************************************************************************************************************************
ok: [localhost]
Tuesday 09 April 2019  13:40:39 +0000 (0:00:01.520)       0:00:01.616 *********

TASK [Get actual hostname] *******************************************************************************************************************************************************************************************************************
changed: [localhost]
Tuesday 09 April 2019  13:40:40 +0000 (0:00:00.599)       0:00:02.215 *********
Tuesday 09 April 2019  13:40:40 +0000 (0:00:00.054)       0:00:02.270 *********

TASK [set_fact] ******************************************************************************************************************************************************************************************************************************
ok: [localhost]
Tuesday 09 April 2019  13:40:40 +0000 (0:00:00.075)       0:00:02.346 *********
Tuesday 09 April 2019  13:40:40 +0000 (0:00:00.067)       0:00:02.413 *********
Tuesday 09 April 2019  13:40:40 +0000 (0:00:00.060)       0:00:02.473 *********
Tuesday 09 April 2019  13:40:40 +0000 (0:00:00.051)       0:00:02.525 *********
Tuesday 09 April 2019  13:40:40 +0000 (0:00:00.056)       0:00:02.582 *********
Tuesday 09 April 2019  13:40:41 +0000 (0:00:00.216)       0:00:02.798 *********
included: /opt/ansible/roles/splunk_common/tasks/change_splunk_directory_owner.yml for localhost
Tuesday 09 April 2019  13:40:41 +0000 (0:00:00.087)       0:00:02.886 *********

TASK [splunk_common : Update Splunk directory owner] *****************************************************************************************************************************************************************************************
ok: [localhost]
Tuesday 09 April 2019  13:40:41 +0000 (0:00:00.324)       0:00:03.210 *********
included: /opt/ansible/roles/splunk_common/tasks/get_facts.yml for localhost
Tuesday 09 April 2019  13:40:41 +0000 (0:00:00.094)       0:00:03.305 *********

ну и так далее...

Agung. Gambar kasebut ora ngemot artefak. Sing, saben-saben sampeyan miwiti bakal njupuk wektu kanggo ngundhuh arsip karo binari, unpack lan ngatur.
Kepiye babagan docker-way lan liya-liyane?

Ora, matur suwun. Kita bakal njupuk rute sing beda. Apa yen kita nindakake kabeh operasi kasebut ing tahap perakitan? Banjur ayo padha lunga!

Supaya ora telat banget, aku bakal langsung nuduhake gambar pungkasan:

file docker

# Тут у кого какие предпочтения
FROM centos:7

# Задаём переменные, чтобы каждый раз при старте не указывать их
ENV SPLUNK_HOME /splunkforwarder
ENV SPLUNK_ROLE splunk_heavy_forwarder
ENV SPLUNK_PASSWORD changeme
ENV SPLUNK_START_ARGS --accept-license

# Ставим пакеты
# wget - чтобы скачать артефакты
# expect - понадобится для первоначального запуска Splunk на этапе сборки
# jq - используется в скриптах, которые собирают статистику докера
RUN yum install -y epel-release 
    && yum install -y wget expect jq

# Качаем, распаковываем, удаляем
RUN wget -O splunkforwarder-7.2.4-8a94541dcfac-Linux-x86_64.tgz 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=7.2.4&product=universalforwarder&filename=splunkforwarder-7.2.4-8a94541dcfac-Linux-x86_64.tgz&wget=true' 
    && wget -O docker-18.09.3.tgz 'https://download.docker.com/linux/static/stable/x86_64/docker-18.09.3.tgz' 
    && tar -xvf splunkforwarder-7.2.4-8a94541dcfac-Linux-x86_64.tgz 
    && tar -xvf docker-18.09.3.tgz  
    && rm -f splunkforwarder-7.2.4-8a94541dcfac-Linux-x86_64.tgz 
    && rm -f docker-18.09.3.tgz

# С shell скриптами всё понятно, а вот inputs.conf, splunkclouduf.spl и first_start.sh нуждаются в пояснении. Об этом расскажу после source тэга.
COPY [ "inputs.conf", "docker-stats/props.conf", "/splunkforwarder/etc/system/local/" ]
COPY [ "docker-stats/docker_events.sh", "docker-stats/docker_inspect.sh", "docker-stats/docker_stats.sh", "docker-stats/docker_top.sh", "/splunkforwarder/bin/scripts/" ]
COPY splunkclouduf.spl /splunkclouduf.spl
COPY first_start.sh /splunkforwarder/bin/

#  Даём права на исполнение, добавляем пользователя и выполняем первоначальную настройку
RUN chmod +x /splunkforwarder/bin/scripts/*.sh 
    && groupadd -r splunk 
    && useradd -r -m -g splunk splunk 
    && echo "%sudo ALL=NOPASSWD:ALL" >> /etc/sudoers 
    && chown -R splunk:splunk $SPLUNK_HOME 
    && /splunkforwarder/bin/first_start.sh 
    && /splunkforwarder/bin/splunk install app /splunkclouduf.spl -auth admin:changeme 
    && /splunkforwarder/bin/splunk restart

# Копируем инит скрипты
COPY [ "init/entrypoint.sh", "init/checkstate.sh", "/sbin/" ]

# По желанию. Кому нужно локально иметь конфиги/логи, кому нет.
VOLUME [ "/splunkforwarder/etc", "/splunkforwarder/var" ]

HEALTHCHECK --interval=30s --timeout=30s --start-period=3m --retries=5 CMD /sbin/checkstate.sh || exit 1

ENTRYPOINT [ "/sbin/entrypoint.sh" ]
CMD [ "start-service" ]

Dadi apa sing ana ing

first_start.sh

#!/usr/bin/expect -f
set timeout -1
spawn /splunkforwarder/bin/splunk start --accept-license
expect "Please enter an administrator username: "
send -- "adminr"
expect "Please enter a new password: "
send -- "changemer"
expect "Please confirm new password: "
send -- "changemer"
expect eof

Ing wiwitan pisanan, Splunk njaluk sampeyan menehi login / sandhi, nanging data iki digunakake mung kanggo nglakokake perintah administratif kanggo instalasi tartamtu, yaiku, ing jero wadhah. Ing kasus kita, kita mung pengin mbukak wadhah supaya kabeh bisa lan log mili kaya kali. Mesthi, iki hardcode, nanging aku ora nemokake cara liyane.

Salajengipun miturut naskah kaleksanan

/splunkforwarder/bin/splunk install app /splunkclouduf.spl -auth admin:changeme

splunkclouduf.spl - Iki minangka file kapercayan kanggo Splunk Universal Forwarder, sing bisa didownload saka antarmuka web.

Ngendi klik kanggo ngundhuh (ing gambar)

Iki minangka arsip biasa sing bisa dibongkar. Ing njero ana sertifikat lan sandhi kanggo nyambungake menyang SplunkCloud lan outputs.conf karo dhaptar kedadean input kita. Berkas iki bakal relevan nganti sampeyan nginstal maneh instalasi Splunk utawa nambah simpul input yen instalasi ana ing lokasi. Mula, ora ana sing salah yen ditambahake ing njero wadhah.

Lan sing pungkasan yaiku miwiti maneh. Ya, kanggo ngetrapake owah-owahan, sampeyan kudu miwiti maneh.

Ing kita inputs.conf kita nambah log sing arep kita kirim menyang Splunk. Ora perlu nambah file iki menyang gambar yen, contone, sampeyan nyebarake konfigurasi liwat wayang. Siji-sijine yaiku Forwarder ndeleng konfigurasi nalika daemon diwiwiti, yen ora, butuh ./splunk miwiti maneh.

Apa jenis skrip stats docker? Ana solusi lawas ing Github saka wong outcold, Tulisan dijupuk saka ing kono lan diowahi kanggo karya karo versi saiki Docker (ce-17.*) lan Splunk (7.*).

Kanthi data sing dipikolehi, sampeyan bisa mbangun ing ngisor iki

dashboard: (saperangan gambar)

Kode sumber kanggo dashes ana ing link sing kasedhiya ing pungkasan artikel. Wigati dimangerteni yen ana 2 kolom sing dipilih: 1 - pilihan indeks (digoleki nganggo topeng), pilihan host/wadhah. Sampeyan bisa uga kudu nganyari topeng indeks, gumantung saka jeneng sing sampeyan gunakake.

Ing kesimpulan, aku pengin narik kawigaten sampeyan menyang fungsi kasebut wiwit() в

entrypoint.sh

start() {
    trap teardown EXIT
	if [ -z $SPLUNK_INDEX ]; then
	echo "'SPLUNK_INDEX' env variable is empty or not defined. Should be 'dev' or 'prd'." >&2
	exit 1
	else
	sed -e "s/@index@/$SPLUNK_INDEX/" -i ${SPLUNK_HOME}/etc/system/local/inputs.conf
	fi
	sed -e "s/@hostname@/$(cat /etc/hostname)/" -i ${SPLUNK_HOME}/etc/system/local/inputs.conf
    sh -c "echo 'starting' > /tmp/splunk-container.state"
	${SPLUNK_HOME}/bin/splunk start
    watch_for_failure
}

Ing kasusku, kanggo saben lingkungan lan saben entitas individu, dadi aplikasi ing wadhah utawa mesin host, kita nggunakake indeks sing kapisah. Kanthi cara iki, kacepetan panelusuran ora bakal nandhang sangsara nalika ana akumulasi data sing signifikan. Aturan prasaja digunakake kanggo menehi jeneng indeks: _. Mulane, supaya wadhah bisa universal, sadurunge ngluncurake daemon dhewe, kita ngganti sed-th wildcard kanggo jeneng lingkungan. Variabel jeneng lingkungan dilewati variabel lingkungan. Swara lucu.

Iku uga worth kang lagi nyimak sing sakperangan alesan Splunk ora kena pengaruh dening ngarsane parameter docker hostname. Dheweke isih bakal stubbornly ngirim log karo id saka wadhah ing lapangan inang. Minangka solusi, sampeyan bisa masang / etc / hostname saka mesin inang lan ing wiwitan nggawe panggantos padha jeneng indeks.

Conto docker-compose.yml

version: '2'
services:
  splunk-forwarder:
    image: "${IMAGE_REPO}/docker-stats-splunk-forwarder:${IMAGE_VERSION}"
    environment:
      SPLUNK_INDEX: ${ENVIRONMENT}
    volumes:
    - /etc/hostname:/etc/hostname:ro
    - /var/log:/var/log
    - /var/run/docker.sock:/var/run/docker.sock:ro

Asile

Ya, mbok menawa solusi kasebut ora becik lan mesthi ora universal kanggo kabeh wong, amarga ana akeh "hardcode". Nanging adhedhasar iku, saben wong bisa mbangun gambar dhewe lan sijine ing artefactory pribadi, yen, kaya mengkono, sampeyan kudu Splunk Forwarder ing Docker.

Cathetan:

Solusi saka artikel
Solusi saka outcoldman sing menehi inspirasi kanggo nggunakake maneh sawetara fungsi kasebut
Saka. dokumentasi kanggo nyetel Universal Forwarder

Source: www.habr.com

Splunk Universal Forwarder ing docker minangka kolektor log sistem

Add a comment Отменить ответ