Splunk is one of the most recognizable commercial products for collecting and analyzing logs. Even now, when sales in Russia are no longer being made, it doesn’t prevent us from writing instructions/how-tos for this product.
Task: to collect system logs from Docker nodes in Splunk without changing the host machine’s configuration
We would like to start with the official approach, which seems a bit odd when using Docker.
So what do we have:
1. Pulling the image
$ docker pull splunk/universalforwarder:latest2. Starting the container with the required parameters
$ docker run -d -p 9997:9997 -e 'SPLUNK_START_ARGS=--accept-license' -e 'SPLUNK_PASSWORD=' splunk/universalforwarder:latest3. Entering the container
docker exec -it /bin/bashNext, we are prompted to follow a known address in the documentation.
And configure the container after it starts:
./splunk add forward-server :
./splunk add monitor /var/log
./splunk restart
Wait. What?
But the surprises don’t end here. If you run the container from the official image in interactive mode, you will see the following:
A bit of disappointment
$ docker run -it -p 9997:9997 -e 'SPLUNK_START_ARGS=--accept-license' -e 'SPLUNK_PASSWORD=password' splunk/universalforwarder:latest
PLAY [Run default Splunk provisioning] *******************************************************************************************************************************************************************************************************
Tuesday 09 April 2019 13:40:38 +0000 (0:00:00.096) 0:00:00.096 *********
TASK [Gathering Facts] ***********************************************************************************************************************************************************************************************************************
ok: [localhost]
Tuesday 09 April 2019 13:40:39 +0000 (0:00:01.520) 0:00:01.616 *********
TASK [Get actual hostname] *******************************************************************************************************************************************************************************************************************
changed: [localhost]
Tuesday 09 April 2019 13:40:40 +0000 (0:00:00.599) 0:00:02.215 *********
Tuesday 09 April 2019 13:40:40 +0000 (0:00:00.054) 0:00:02.270 *********
TASK [set_fact] ******************************************************************************************************************************************************************************************************************************
ok: [localhost]
Tuesday 09 April 2019 13:40:40 +0000 (0:00:00.075) 0:00:02.346 *********
Tuesday 09 April 2019 13:40:40 +0000 (0:00:00.067) 0:00:02.413 *********
Tuesday 09 April 2019 13:40:40 +0000 (0:00:00.060) 0:00:02.473 *********
Tuesday 09 April 2019 13:40:40 +0000 (0:00:00.051) 0:00:02.525 *********
Tuesday 09 April 2019 13:40:40 +0000 (0:00:00.056) 0:00:02.582 *********
Tuesday 09 April 2019 13:40:41 +0000 (0:00:00.216) 0:00:02.798 *********
included: /opt/ansible/roles/splunk_common/tasks/change_splunk_directory_owner.yml for localhost
Tuesday 09 April 2019 13:40:41 +0000 (0:00:00.087) 0:00:02.886 *********
TASK [splunk_common : Update Splunk directory owner] *****************************************************************************************************************************************************************************************
ok: [localhost]
Tuesday 09 April 2019 13:40:41 +0000 (0:00:00.324) 0:00:03.210 *********
included: /opt/ansible/roles/splunk_common/tasks/get_facts.yml for localhost
Tuesday 09 April 2019 13:40:41 +0000 (0:00:00.094) 0:00:03.305 *********
and so on...
Great. The image doesn’t even have the artifact. That means every time it starts, time will be wasted downloading the archive with the binaries, unpacking it, and configuring.
What about the docker way and all that?
No, thank you. We'll take a different path. What if we perform all these operations at build time? Let’s go!
To not drag it out, I'll show the finished image right away:
Dockerfile
# Тут у кого какие предпочтения
FROM centos:7
# Задаём переменные, чтобы каждый раз при старте не указывать их
ENV SPLUNK_HOME /splunkforwarder
ENV SPLUNK_ROLE splunk_heavy_forwarder
ENV SPLUNK_PASSWORD changeme
ENV SPLUNK_START_ARGS --accept-license
# Ставим пакеты
# wget - чтобы скачать артефакты
# expect - понадобится для первоначального запуска Splunk на этапе сборки
# jq - используется в скриптах, которые собирают статистику докера
RUN yum install -y epel-release
&& yum install -y wget expect jq
# Качаем, распаковываем, удаляем
RUN wget -O splunkforwarder-7.2.4-8a94541dcfac-Linux-x86_64.tgz 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=7.2.4&product=universalforwarder&filename=splunkforwarder-7.2.4-8a94541dcfac-Linux-x86_64.tgz&wget=true'
&& wget -O docker-18.09.3.tgz 'https://download.docker.com/linux/static/stable/x86_64/docker-18.09.3.tgz'
&& tar -xvf splunkforwarder-7.2.4-8a94541dcfac-Linux-x86_64.tgz
&& tar -xvf docker-18.09.3.tgz
&& rm -f splunkforwarder-7.2.4-8a94541dcfac-Linux-x86_64.tgz
&& rm -f docker-18.09.3.tgz
# С shell скриптами всё понятно, а вот inputs.conf, splunkclouduf.spl и first_start.sh нуждаются в пояснении. Об этом расскажу после source тэга.
COPY [ "inputs.conf", "docker-stats/props.conf", "/splunkforwarder/etc/system/local/" ]
COPY [ "docker-stats/docker_events.sh", "docker-stats/docker_inspect.sh", "docker-stats/docker_stats.sh", "docker-stats/docker_top.sh", "/splunkforwarder/bin/scripts/" ]
COPY splunkclouduf.spl /splunkclouduf.spl
COPY first_start.sh /splunkforwarder/bin/
# Даём права на исполнение, добавляем пользователя и выполняем первоначальную настройку
RUN chmod +x /splunkforwarder/bin/scripts/*.sh
&& groupadd -r splunk
&& useradd -r -m -g splunk splunk
&& echo "%sudo ALL=NOPASSWD:ALL" >> /etc/sudoers
&& chown -R splunk:splunk $SPLUNK_HOME
&& /splunkforwarder/bin/first_start.sh
&& /splunkforwarder/bin/splunk install app /splunkclouduf.spl -auth admin:changeme
&& /splunkforwarder/bin/splunk restart
# Копируем инит скрипты
COPY [ "init/entrypoint.sh", "init/checkstate.sh", "/sbin/" ]
# По желанию. Кому нужно локально иметь конфиги/логи, кому нет.
VOLUME [ "/splunkforwarder/etc", "/splunkforwarder/var" ]
HEALTHCHECK --interval=30s --timeout=30s --start-period=3m --retries=5 CMD /sbin/checkstate.sh || exit 1
ENTRYPOINT [ "/sbin/entrypoint.sh" ]
CMD [ "start-service" ]So, what does it contain
first_start.sh
#!/usr/bin/expect -f
set timeout -1
spawn /splunkforwarder/bin/splunk start --accept-license
expect "Please enter an administrator username: "
send -- "adminr"
expect "Please enter a new password: "
send -- "changemer"
expect "Please confirm new password: "
send -- "changemer"
expect eofAt first start, Splunk asks for a username/password, BUT this data is used only for executing administrative commands for this specific installation, that is, inside the container. In our case, we just want to run the container so that everything works and logs flow freely. Of course, this is hardcoded, but I haven't found other ways.
Next in the script, the following actions are performed
/splunkforwarder/bin/splunk install app /splunkclouduf.spl -auth admin:changemesplunkclouduf.spl — This is a credentials file for the Splunk Universal Forwarder, which can be downloaded from the web interface.
Where to click to download (in pictures)
This is a regular archive that can be unpacked. Inside are the certificates and password to connect to our SplunkCloud and outputs.conf with a list of our input instances. This file will remain relevant until you reinstall your Splunk installation or add input nodes if the installation is on-premise. So, it's perfectly fine to add it to the container.
And finally — restart. Yes, to apply the changes, you need to restart it.
In our inputs.conf we add the logs that we want to send to Splunk. It's not mandatory to include this file in the image if, for example, you are distributing configs via Puppet. The main thing is that the Forwarder sees the configs when the daemon starts; otherwise, you'll need ./splunk restart.
What are the docker stats scripts about? There is an old solution on GitHub from , scripts taken from there and modified to work with current versions of Docker (ce-17.*) and Splunk (7.*).
Using the gathered data, you can build such
dashboards: (a couple of pictures)
The source code for the dashboards is in the repository mentioned at the end of the article. Note that there are 2 select fields: 1 — index selection (searches by mask), selection of host/container. You will likely need to update the index mask depending on the names you use.
In conclusion, I want to highlight the function start() downward API support (simultaneously with this in
entrypoint.sh
start() {
trap teardown EXIT
if [ -z $SPLUNK_INDEX ]; then
echo "'SPLUNK_INDEX' env variable is empty or not defined. Should be 'dev' or 'prd'." >2
exit 1
else
sed -e "s/@index@/$SPLUNK_INDEX/" -i ${SPLUNK_HOME}/etc/system/local/inputs.conf
fi
sed -e "s/@hostname@/$(cat /etc/hostname)/" -i ${SPLUNK_HOME}/etc/system/local/inputs.conf
sh -c "echo 'starting' > /tmp/splunk-container.state"
${SPLUNK_HOME}/bin/splunk start
watch_for_failure
}In my case, for each environment and each separate entity, whether it's an application in a container or a host machine, we use a separate index. This ensures that search speed is not impacted with significant data accumulation. The naming of indexes follows a simple rule: _. Therefore, to make the container universal, before starting the daemon, we replace sedwith the environment name. The variable with the environment name is passed through environment variables. Sounds funny.
It is also worth noting that for some reason, the presence of the Docker parameter does not affect Splunk. hostnameIt will still relentlessly send logs with the ID of its container in the host field. As a solution, you can mount /etc/hostname from the host machine and upon startup, make a replacement similar to index names.
Example docker-compose.yml
version: '2'
services:
splunk-forwarder:
image: "${IMAGE_REPO}/docker-stats-splunk-forwarder:${IMAGE_VERSION}"
environment:
SPLUNK_INDEX: ${ENVIRONMENT}
volumes:
- /etc/hostname:/etc/hostname:ro
- /var/log:/var/log
- /var/run/docker.sock:/var/run/docker.sock:roSummary
Yes, the solution may not be ideal and certainly not universal for everyone, as there is a lot of ‘hardcoding’.However, based on it, anyone can build their own image and place it in their private artifact repository, if, by chance, you need Splunk Forwarder specifically in Docker.
Links:
Source: habr.com
