Hey Habr!
In today's reality, due to the increasing role of containerization in development processes, the issue of ensuring the security of various stages and entities associated with containers is not in the last place. Performing checks manually is a laborious task, so it would be nice to take at least the initial steps towards automating this process.
In this article, I will share ready-made scripts for implementing several Docker security utilities and instructions on how to set up a small demo stand to test this process. You can use the materials to experiment with how to organize the process of testing the security of Dockerfile images and instructions. It is clear that the development and implementation infrastructure is different for everyone, so below I will give several possible options.
Security Check Utilities
There are a large number of different helper applications and scripts that perform checks on various aspects of the Docker infrastructure. Some of them have already been described in a previous article (), and in this article I would like to focus on three of them, which cover the bulk of the security requirements for Docker images that are built during the development process. In addition, I will also show an example of how these three utilities can be combined into one pipeline to perform security checks.
Hadolint
A fairly simple console utility that helps to first assess the correctness and safety of Dockerfile instructions (for example, using only allowed image registries or using sudo).
Dockle
A console utility that works on an image (or on a saved image tarball) that checks the correctness and security of a particular image as such by analyzing its layers and configuration - what users are created, what instructions are in use, what volumes are mounted, the presence of a blank password, etc. e. While the number of checks is not very large and is based on several own checks and recommendations for docker.
Trives
This utility is aimed at finding two types of vulnerabilities - OS build problems (Alpine, RedHat (EL), CentOS, Debian GNU, Ubuntu are supported) and dependency problems (Gemfile.lock, Pipfile.lock, composer.lock, package-lock.json , yarn.lock, Cargo.lock). Trivy can scan both the image in the repository and the local image, and also scan based on the transferred .tar file with the Docker image.
Utilities Implementation Options
In order to try out the described applications in isolated conditions, I will provide instructions for installing all the utilities as part of a simplified process.
The main idea is to demonstrate how you can implement automatic content checking for Dockerfiles and Docker images that are created during development.
The verification itself consists of the following steps:
- Checking the correctness and safety of Dockerfile instructions with a linter utility Hadolint
- Checking the correctness and security of the final and intermediate images - a utility Dockle
- Checking for Commonly Known Vulnerabilities (CVE) in the base image and a number of dependencies - by the utility Trives
Later in the article I will give three options for implementing these steps:
The first one is by configuring the CI / CD pipeline using the example of GitLab (with a description of the process of raising a test instance).
The second is using a shell script.
The third is with building a Docker image to scan Docker images.
You can choose the option that suits you best, transfer it to your infrastructure and adapt it to your needs.
All necessary files and additional instructions are also in the repository:
GitLab CI/CD integration
In the first option, we will look at how security checks can be implemented using the GitLab repository system as an example. Here we will go through the steps and see how to set up a test environment with GitLab from scratch, create a scan process and run utilities to test a test Dockerfile and a random image - the JuiceShop application.
Installing GitLab
1. Install Docker:
sudo apt-get update && sudo apt-get install docker.io2. Add the current user to the docker group so that you can work with docker without using sudo:
sudo addgroup <username> docker3. Find your IP:
ip addr4. Install and run GitLab in the container, replacing the IP address in hostname with your own:
docker run --detach
--hostname 192.168.1.112
--publish 443:443 --publish 80:80
--name gitlab
--restart always
--volume /srv/gitlab/config:/etc/gitlab
--volume /srv/gitlab/logs:/var/log/gitlab
--volume /srv/gitlab/data:/var/opt/gitlab
gitlab/gitlab-ce:latestWe are waiting for GitLab to complete all the necessary installation procedures (you can follow the process through the output of the log file: docker logs -f gitlab).
5. Open your local IP in the browser and see a page offering to change the password for the root user:
Set a new password and go to GitLab.
6. Create a new project, for example cicd-test and initialize it with a start file README.md:
7. Now we need to install the GitLab Runner: an agent that will run all the necessary operations on request.
Download the latest version (in this case, under Linux 64-bit):
sudo curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd648. Make it executable:
sudo chmod +x /usr/local/bin/gitlab-runner9. Add an OS user for the Runner and start the service:
sudo useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash
sudo gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
sudo gitlab-runner startIt should look something like this:
local@osboxes:~$ sudo gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
Runtime platform arch=amd64 os=linux pid=8438 revision=0e5417a3 version=12.0.1
local@osboxes:~$ sudo gitlab-runner start
Runtime platform arch=amd64 os=linux pid=8518 revision=0e5417a3 version=12.0.1
10. Now we register the Runner so that it can interact with our GitLab instance.
To do this, open the Settings-CI/CD page (http://OUR_ IP_ADDRESS/root/cicd-test/-/settings/ci_cd) and on the Runners tab find the URL and Registration token:
11. Register the Runner by substituting the URL and the Registration token:
sudo gitlab-runner register
--non-interactive
--url "http://<URL>/"
--registration-token "<Registration Token>"
--executor "docker"
--docker-privileged
--docker-image alpine:latest
--description "docker-runner"
--tag-list "docker,privileged"
--run-untagged="true"
--locked="false"
--access-level="not_protected"As a result, we get a ready-made working GitLab, in which we need to add instructions to start our utilities. In this demo, we don't have application build and containerization steps, but in a real environment they will precede the scan steps and generate images and a Dockerfile for analysis.
pipeline configuration
1. Add files to the repository mydockerfile.df (this is a test Dockerfile that we will test) and the GitLab CI/CD process configuration file .gitlab-cicd.yml, which lists instructions for scanners (note the dot in the file name).
The .yaml configuration file contains instructions for running three utilities (Hadolint, Dockle, and Trivy) that will parse the selected Dockerfile and the image specified in the DOCKERFILE variable. All the necessary files can be taken from the repository:
Excerpt from mydockerfile.df (this is an abstract file with a set of arbitrary instructions just to demonstrate how the utility works). Direct link to the file:
Contents of mydockerfile.df
FROM amd64/node:10.16.0-alpine@sha256:f59303fb3248e5d992586c76cc83e1d3700f641cbcd7c0067bc7ad5bb2e5b489 AS tsbuild
COPY package.json .
COPY yarn.lock .
RUN yarn install
COPY lib lib
COPY tsconfig.json tsconfig.json
COPY tsconfig.app.json tsconfig.app.json
RUN yarn build
FROM amd64/ubuntu:18.04@sha256:eb70667a801686f914408558660da753cde27192cd036148e58258819b927395
LABEL maintainer="Rhys Arkins <[email protected]>"
LABEL name="renovate"
...
COPY php.ini /usr/local/etc/php/php.ini
RUN cp -a /tmp/piik/* /var/www/html/
RUN rm -rf /tmp/piwik
RUN chown -R www-data /var/www/html
ADD piwik-cli-setup /piwik-cli-setup
ADD reset.php /var/www/html/
## ENTRYPOINT ##
ADD entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
USER rootThe configuration YAML looks like this (the file itself can be taken from the direct link here: ):
Contents of .gitlab-ci.yml
variables:
DOCKER_HOST: "tcp://docker:2375/"
DOCKERFILE: "mydockerfile.df" # name of the Dockerfile to analyse
DOCKERIMAGE: "bkimminich/juice-shop" # name of the Docker image to analyse
# DOCKERIMAGE: "knqyf263/cve-2018-11235" # test Docker image with several CRITICAL CVE
SHOWSTOPPER_PRIORITY: "CRITICAL" # what level of criticality will fail Trivy job
TRIVYCACHE: "$CI_PROJECT_DIR/.cache" # where to cache Trivy database of vulnerabilities for faster reuse
ARTIFACT_FOLDER: "$CI_PROJECT_DIR"
services:
- docker:dind # to be able to build docker images inside the Runner
stages:
- scan
- report
- publish
HadoLint:
# Basic lint analysis of Dockerfile instructions
stage: scan
image: docker:git
after_script:
- cat $ARTIFACT_FOLDER/hadolint_results.json
script:
- export VERSION=$(wget -q -O - https://api.github.com/repos/hadolint/hadolint/releases/latest | grep '"tag_name":' | sed -E 's/.*"v([^"]+)".*/1/')
- wget https://github.com/hadolint/hadolint/releases/download/v${VERSION}/hadolint-Linux-x86_64 && chmod +x hadolint-Linux-x86_64
# NB: hadolint will always exit with 0 exit code
- ./hadolint-Linux-x86_64 -f json $DOCKERFILE > $ARTIFACT_FOLDER/hadolint_results.json || exit 0
artifacts:
when: always # return artifacts even after job failure
paths:
- $ARTIFACT_FOLDER/hadolint_results.json
Dockle:
# Analysing best practices about docker image (users permissions, instructions followed when image was built, etc.)
stage: scan
image: docker:git
after_script:
- cat $ARTIFACT_FOLDER/dockle_results.json
script:
- export VERSION=$(wget -q -O - https://api.github.com/repos/goodwithtech/dockle/releases/latest | grep '"tag_name":' | sed -E 's/.*"v([^"]+)".*/1/')
- wget https://github.com/goodwithtech/dockle/releases/download/v${VERSION}/dockle_${VERSION}_Linux-64bit.tar.gz && tar zxf dockle_${VERSION}_Linux-64bit.tar.gz
- ./dockle --exit-code 1 -f json --output $ARTIFACT_FOLDER/dockle_results.json $DOCKERIMAGE
artifacts:
when: always # return artifacts even after job failure
paths:
- $ARTIFACT_FOLDER/dockle_results.json
Trivy:
# Analysing docker image and package dependencies against several CVE bases
stage: scan
image: docker:git
script:
# getting the latest Trivy
- apk add rpm
- export VERSION=$(wget -q -O - https://api.github.com/repos/knqyf263/trivy/releases/latest | grep '"tag_name":' | sed -E 's/.*"v([^"]+)".*/1/')
- wget https://github.com/knqyf263/trivy/releases/download/v${VERSION}/trivy_${VERSION}_Linux-64bit.tar.gz && tar zxf trivy_${VERSION}_Linux-64bit.tar.gz
# displaying all vulnerabilities w/o failing the build
- ./trivy -d --cache-dir $TRIVYCACHE -f json -o $ARTIFACT_FOLDER/trivy_results.json --exit-code 0 $DOCKERIMAGE
# write vulnerabilities info to stdout in human readable format (reading pure json is not fun, eh?). You can remove this if you don't need this.
- ./trivy -d --cache-dir $TRIVYCACHE --exit-code 0 $DOCKERIMAGE
# failing the build if the SHOWSTOPPER priority is found
- ./trivy -d --cache-dir $TRIVYCACHE --exit-code 1 --severity $SHOWSTOPPER_PRIORITY --quiet $DOCKERIMAGE
artifacts:
when: always # return artifacts even after job failure
paths:
- $ARTIFACT_FOLDER/trivy_results.json
cache:
paths:
- .cache
Report:
# combining tools outputs into one HTML
stage: report
when: always
image: python:3.5
script:
- mkdir json
- cp $ARTIFACT_FOLDER/*.json ./json/
- pip install json2html
- wget https://raw.githubusercontent.com/shad0wrunner/docker_cicd/master/convert_json_results.py
- python ./convert_json_results.py
artifacts:
paths:
- results.htmlIf necessary, you can also scan saved images as a .tar archive (however, you will need to change the input parameters for the utilities in the YAML file)
NB: Trivy requires installed rpm ΠΈ git. Otherwise, it will generate errors when scanning RedHat-based images and getting updates to the vulnerability database.
2. After adding the files to the repository, according to the instructions in our configuration file, GitLab will automatically start the build and scan process. On the CI / CD β Pipelines tab, you can see the progress of the instructions.
As a result, we have four tasks. Three of them are directly involved in scanning, and the last one (Report) collects a simple report from scattered files with scan results.
By default, Trivy stops its execution if CRITICAL vulnerabilities are found in the image or dependencies. At the same time, Hadolint always returns a Success code, since its execution always has remarks, which leads to a build stop.
Depending on your specific requirements, you can configure an exit code so that these utilities also stop the build process when problems of a certain criticality are detected. In our case, the build will stop only if Trivy detects a vulnerability with a severity that we have specified in the SHOWSTOPPER variable in .gitlab-ci.yml.
The result of the operation of each utility can be viewed in the log of each scanning task, directly in json files in the artifacts section, or in a simple HTML report (more on that below):
3. To present utility reports in a slightly more human-readable form, a small Python script is used to convert three json files into one HTML file with a table of defects.
This script is launched by a separate Report task, and its final artifact is an HTML file with a report. The script source is also in the repository and can be adapted to your needs, colors, etc.
Shell script
The second option is suitable for cases where you need to check Docker images not within the CI / CD system, or you need to have all instructions in a form that can be executed directly on the host. This option is covered by a ready-made shell script that can be run on a clean virtual (or even real) machine. The script follows the same instructions as the gitlab-runner above.
For the script to work successfully, Docker must be installed on the system and the current user must be in the docker group.
The script itself can be found here:
At the beginning of the file, variables specify which image should be scanned and what severity of defects will cause the Trivy utility to exit with the specified error code.
During the script execution, all utilities will be downloaded to the directory docker_tools, the results of their work - in the directory docker_tools/json, and the HTML with the report will be in the file results.html.
Example script output
~/docker_cicd$ ./docker_sec_check.sh
[+] Setting environment variables
[+] Installing required packages
[+] Preparing necessary directories
[+] Fetching sample Dockerfile
2020-10-20 10:40:00 (45.3 MB/s) - βDockerfileβ saved [8071/8071]
[+] Pulling image to scan
latest: Pulling from bkimminich/juice-shop
[+] Running Hadolint
...
Dockerfile:205 DL3015 Avoid additional packages by specifying `--no-install-recommends`
Dockerfile:248 DL3002 Last USER should not be root
...
[+] Running Dockle
...
WARN - DKL-DI-0006: Avoid latest tag
* Avoid 'latest' tag
INFO - CIS-DI-0005: Enable Content trust for Docker
* export DOCKER_CONTENT_TRUST=1 before docker pull/build
...
[+] Running Trivy
juice-shop/frontend/package-lock.json
=====================================
Total: 3 (UNKNOWN: 0, LOW: 1, MEDIUM: 0, HIGH: 2, CRITICAL: 0)
+---------------------+------------------+----------+---------+-------------------------+
| LIBRARY | VULNERABILITY ID | SEVERITY | VERSION | TITLE |
+---------------------+------------------+----------+---------+-------------------------+
| object-path | CVE-2020-15256 | HIGH | 0.11.4 | Prototype pollution in |
| | | | | object-path |
+---------------------+------------------+ +---------+-------------------------+
| tree-kill | CVE-2019-15599 | | 1.2.2 | Code Injection |
+---------------------+------------------+----------+---------+-------------------------+
| webpack-subresource | CVE-2020-15262 | LOW | 1.4.1 | Unprotected dynamically |
| | | | | loaded chunks |
+---------------------+------------------+----------+---------+-------------------------+
juice-shop/package-lock.json
============================
Total: 20 (UNKNOWN: 0, LOW: 1, MEDIUM: 6, HIGH: 8, CRITICAL: 5)
...
juice-shop/package-lock.json
============================
Total: 5 (CRITICAL: 5)
...
[+] Removing left-overs
[+] Making the output look pretty
[+] Converting JSON results
[+] Writing results HTML
[+] Clean exit ============================================================
[+] Everything is done. Find the resulting HTML report in results.htmlDocker image with all utilities
As a third alternative, I compiled two simple Dockerfiles to create an image with security utilities. One Dockerfile will help build a set to scan the image from the repository, the second (Dockerfile_tar) will build a set to scan the tar file with the image.
1. We take the appropriate Docker file and scripts from the repository .
2. Run it for assembly:
docker build -t dscan:image -f docker_security.df .
3. After the build is complete, create a container from the image. At the same time, we pass the DOCKERIMAGE environment variable with the name of the image we are interested in and mount the Dockerfile that we want to analyze from our machine to the file /dockerfile (note that an absolute path to this file is required):
docker run --rm -v $(pwd)/results:/results -v $(pwd)/docker_security.df:/Dockerfile -e DOCKERIMAGE="bkimminich/juice-shop" dscan:image
[+] Setting environment variables
[+] Running Hadolint
/Dockerfile:3 DL3006 Always tag the version of an image explicitly
[+] Running Dockle
WARN - DKL-DI-0006: Avoid latest tag
* Avoid 'latest' tag
INFO - CIS-DI-0005: Enable Content trust for Docker
* export DOCKER_CONTENT_TRUST=1 before docker pull/build
INFO - CIS-DI-0006: Add HEALTHCHECK instruction to the container image
* not found HEALTHCHECK statement
INFO - DKL-LI-0003: Only put necessary files
* unnecessary file : juice-shop/node_modules/sqlite3/Dockerfile
* unnecessary file : juice-shop/node_modules/sqlite3/tools/docker/architecture/linux-arm64/Dockerfile
* unnecessary file : juice-shop/node_modules/sqlite3/tools/docker/architecture/linux-arm/Dockerfile
[+] Running Trivy
...
juice-shop/package-lock.json
============================
Total: 20 (UNKNOWN: 0, LOW: 1, MEDIUM: 6, HIGH: 8, CRITICAL: 5)
...
[+] Making the output look pretty
[+] Starting the main module ============================================================
[+] Converting JSON results
[+] Writing results HTML
[+] Clean exit ============================================================
[+] Everything is done. Find the resulting HTML report in results.htmlThe results
We've covered just one basic set of Docker artifact scanning tools, which I think covers a fair amount of image security requirements quite effectively. There are many other paid and free tools that can perform the same checks, draw beautiful reports or work purely in console mode, cover container management systems, etc. An overview of these tools and how to integrate them may appear a bit later.
The positive side of the set of tools described in the article is that they are all built on open source and you can experiment with them and other similar tools to find what exactly suits your requirements and infrastructure features. Of course, all vulnerabilities that are found should be studied for applicability in specific conditions, but this is a topic for a future large article.
I hope these instructions, scripts and utilities will help you and become a starting point for creating a more secure infrastructure in the field of containerization.
Source: habr.com