Hello, Habr!
In today's reality, due to the increasing role of containerization in development processes, the question of ensuring security at various stages and entities related to containers is paramount. Conducting checks manually is a labor-intensive task, so it would be beneficial to at least take initial steps toward automating this process.
In this article, I will share ready-made scripts for implementing several Docker security utilities and instructions on how to deploy a small demo setup to test this process. The materials can be used to experiment with how to organize the security testing of Docker images and Dockerfile instructions. It's clear that everyone's development and deployment infrastructure is different, so below I will provide several possible options.
Security Checking Utilities
There are many various auxiliary applications and scripts that perform checks on diverse aspects of Docker infrastructure. Some of them were described in the previous article (), and in this material, I would like to focus on three of them that cover the main security requirements for Docker images built during the development process. Additionally, I will show an example of how these three utilities can be combined into a single pipeline for performing security checks.
Hadolint
A fairly simple console utility that helps to initially assess the correctness and security of Dockerfile instructions (e.g., using only allowed image registries or using sudo).
Dockle
A console utility that works with an image (or a saved tar archive of the image), which checks the correctness and security of that specific image by analyzing its layers and configuration – which users are created, which instructions are used, which volumes are mounted, presence of empty passwords, and so on. Currently, the number of checks is not very large and is based on a few custom checks and recommendations. for Docker.
Trivy
This utility is aimed at finding vulnerabilities of two types – operating system build issues (supporting Alpine, RedHat (EL), CentOS, Debian GNU, Ubuntu) and dependency issues (Gemfile.lock, Pipfile.lock, composer.lock, package-lock.json, yarn.lock, Cargo.lock). Trivy can scan both images in the repository and local images, as well as conduct scans based on the provided .tar file with the Docker image.
Options for implementing utilities
To try the described applications in an isolated environment, I will provide instructions for installing all utilities within a simplified process.
The main idea is to demonstrate how to implement automated checks for the contents of Dockerfiles and Docker images created during development.
The check consists of the following steps:
- Checking the correctness and security of Dockerfile instructions using a linter utility Hadolint
- Checking the correctness and security of final and intermediate images using a utility Dockle
- Checking for known vulnerabilities (CVE) in the base image and a series of dependencies using a utility Trivy
Next in the article, I will provide three options for implementing these steps:
The first – by configuring a CI/CD pipeline using a GitLab example (with a description of the process of setting up a test instance).
The second – using a shell script.
The third – by building a Docker image for scanning Docker images.
You can choose the option that suits you best, transfer it to your infrastructure, and adapt it to your needs.
All necessary files and additional instructions can also be found in the repository:
Integration into GitLab CI/CD
In the first option, we will explore how to implement security checks using the GitLab repository system as an example. Here we will walk through the steps and explain how to set up a test environment with GitLab from scratch, create a scanning process, and run utilities to check a test Dockerfile and a random image – the JuiceShop application.
Installing GitLab
1. Install Docker:
sudo apt-get update && sudo apt-get install docker.io2. Add the current user to the docker group so you can work with Docker without sudo:
sudo addgroup docker3. Find your IP:
ip addr4. Install and run GitLab in a container, replacing the IP address in the hostname with your own:
docker run --detach
--hostname 192.168.1.112
--publish 443:443 --publish 80:80
--name gitlab
--restart always
--volume /srv/gitlab/config:/etc/gitlab
--volume /srv/gitlab/logs:/var/log/gitlab
--volume /srv/gitlab/data:/var/opt/gitlab
gitlab/gitlab-ce:latestWaiting for GitLab to complete all necessary installation procedures (you can monitor the process through the log file output: docker logs -f gitlab).
5. Open your local IP in a browser and you will see a page prompting you to change the password for the root user:
Set a new password and log into GitLab.
6. Create a new project, for example cicd-test, and initialize it with a starter file. README.md:
7. Now we need to install GitLab Runner: an agent that will execute all necessary operations upon request.
Download the latest version (in this case — for Linux 64-bit):
sudo curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd648. Make it executable:
sudo chmod +x /usr/local/bin/gitlab-runner9. Add an OS user for the Runner and start the service:
sudo useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash
sudo gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
sudo gitlab-runner startIt should look something like this:
local@osboxes:~$ sudo gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
Runtime platform arch=amd64 os=linux pid=8438 revision=0e5417a3 version=12.0.1
local@osboxes:~$ sudo gitlab-runner start
Runtime platform arch=amd64 os=linux pid=8518 revision=0e5417a3 version=12.0.1 10. Now register the Runner so that it can interact with our GitLab instance.
To do this, open the Settings-CI/CD page (http://OUR_IP_ADDRESS/root/cicd-test/-/settings/ci_cd) and on the Runners tab, find the URL and Registration token:
11. Register the Runner by inserting the URL and Registration token:
sudo gitlab-runner register
--non-interactive
--url "http:///"
--registration-token ""
--executor "docker"
--docker-privileged
--docker-image alpine:latest
--description "docker-runner"
--tag-list "docker,privileged"
--run-untagged="true"
--locked="false"
--access-level="not_protected"As a result, we get a fully working GitLab, to which we need to add instructions for starting our utilities. In this demo case, we have no application build steps and containerization, but in a real environment, they will precede the scanning steps and create images and Dockerfiles for analysis.
Pipeline configuration
1. Let's add files to the repository mydockerfile.df (this is a test Dockerfile that we will verify) and the configuration file for the GitLab CI/CD process .gitlab-cicd.yml, which lists the instructions for the scanners (note the dot in the file name).
The YAML configuration file contains instructions for launching three utilities (Hadolint, Dockle, and Trivy) that will analyze the specified Dockerfile and the image defined in the DOCKERFILE variable. All necessary files can be obtained from the repository:
Excerpt from mydockerfile.df (this is an abstract file with a set of arbitrary instructions solely for demonstrating how the utility works). Direct link to the file:
Contents of mydockerfile.df
FROM amd64/node:10.16.0-alpine@sha256:f59303fb3248e5d992586c76cc83e1d3700f641cbcd7c0067bc7ad5bb2e5b489 AS tsbuild
COPY package.json .
COPY yarn.lock .
RUN yarn install
COPY lib lib
COPY tsconfig.json tsconfig.json
COPY tsconfig.app.json tsconfig.app.json
RUN yarn build
FROM amd64/ubuntu:18.04@sha256:eb70667a801686f914408558660da753cde27192cd036148e58258819b927395
LABEL maintainer="Rhys Arkins "
LABEL name="renovate"
...
COPY php.ini /usr/local/etc/php/php.ini
RUN cp -a /tmp/piik/* /var/www/html/
RUN rm -rf /tmp/piwik
RUN chown -R www-data /var/www/html
ADD piwik-cli-setup /piwik-cli-setup
ADD reset.php /var/www/html/
## ENTRYPOINT ##
ADD entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]
USER rootThe YAML configuration looks like this (the file can be obtained via the direct link here: ):
Contents of .gitlab-ci.yml
variables:
DOCKER_HOST: "tcp://docker:2375/"
DOCKERFILE: "mydockerfile.df" # name of the Dockerfile to analyze
DOCKERIMAGE: "bkimminich/juice-shop" # name of the Docker image to analyze
# DOCKERIMAGE: "knqyf263/cve-2018-11235" # test Docker image with several CRITICAL CVE
SHOWSTOPPER_PRIORITY: "CRITICAL" # criticality level that will fail Trivy job
TRIVYCACHE: "$CI_PROJECT_DIR/.cache" # where to cache Trivy database of vulnerabilities for faster reuse
ARTIFACT_FOLDER: "$CI_PROJECT_DIR"
services:
- docker:dind # needed to build Docker images inside the Runner
stages:
- scan
- report
- publish
HadoLint:
# Basic lint analysis of Dockerfile instructions
stage: scan
image: docker:git
after_script:
- cat $ARTIFACT_FOLDER/hadolint_results.json
script:
- export VERSION=$(wget -q -O - https://api.github.com/repos/hadolint/hadolint/releases/latest | grep '"tag_name":' | sed -E 's/.*"v([^"]+)".*//1/')
- wget https://github.com/hadolint/hadolint/releases/download/v${VERSION}/hadolint-Linux-x86_64 && chmod +x hadolint-Linux-x86_64
# NB: hadolint will always exit with 0 exit code
- ./hadolint-Linux-x86_64 -f json $DOCKERFILE > $ARTIFACT_FOLDER/hadolint_results.json || exit 0
artifacts:
when: always # return artifacts even after job failure
paths:
- $ARTIFACT_FOLDER/hadolint_results.json
Dockle:
# Analyzing best practices regarding Docker image (user permissions, instructions followed when image was built, etc.)
stage: scan
image: docker:git
after_script:
- cat $ARTIFACT_FOLDER/dockle_results.json
script:
- export VERSION=$(wget -q -O - https://api.github.com/repos/goodwithtech/dockle/releases/latest | grep '"tag_name":' | sed -E 's/.*"v([^"]+)".*//1/')
- wget https://github.com/goodwithtech/dockle/releases/download/v${VERSION}/dockle_${VERSION}_Linux-64bit.tar.gz && tar zxf dockle_${VERSION}_Linux-64bit.tar.gz
- ./dockle --exit-code 1 -f json --output $ARTIFACT_FOLDER/dockle_results.json $DOCKERIMAGE
artifacts:
when: always # return artifacts even after job failure
paths:
- $ARTIFACT_FOLDER/dockle_results.json
Trivy:
# Analyzing Docker image and package dependencies against several CVE databases
stage: scan
image: docker:git
script:
# getting the latest Trivy
- apk add rpm
- export VERSION=$(wget -q -O - https://api.github.com/repos/knqyf263/trivy/releases/latest | grep '"tag_name":' | sed -E 's/.*"v([^"]+)".*//1')
- wget https://github.com/knqyf263/trivy/releases/download/v${VERSION}/trivy_${VERSION}_Linux-64bit.tar.gz && tar zxf trivy_${VERSION}_Linux-64bit.tar.gz
# displaying all vulnerabilities without failing the build
- ./trivy -d --cache-dir $TRIVYCACHE -f json -o $ARTIFACT_FOLDER/trivy_results.json --exit-code 0 $DOCKERIMAGE
# write vulnerabilities info to stdout in human-readable format (reading pure json is not fun, eh?). You can remove this if you don't need this.
- ./trivy -d --cache-dir $TRIVYCACHE --exit-code 0 $DOCKERIMAGE
# failing the build if the SHOWSTOPPER priority is found
- ./trivy -d --cache-dir $TRIVYCACHE --exit-code 1 --severity $SHOWSTOPPER_PRIORITY --quiet $DOCKERIMAGE
artifacts:
when: always # return artifacts even after job failure
paths:
- $ARTIFACT_FOLDER/trivy_results.json
cache:
paths:
- .cache
Report:
# combining tool outputs into one HTML document
stage: report
when: always
image: python:3.5
script:
- mkdir json
- cp $ARTIFACT_FOLDER/*.json ./json/
- pip install json2html
- wget https://raw.githubusercontent.com/shad0wrunner/docker_cicd/master/convert_json_results.py
- python ./convert_json_results.py
artifacts:
paths:
- results.htmlIf necessary, you can also scan and save images in the form of a .tar archive (however, you will need to change the input parameters for the utilities in the YAML file).
Note: Trivy requires installed rpm and git. Otherwise, it will throw errors when scanning RedHat-based images and fetching vulnerability database updates.
2. After adding files to the repository, following the instructions in our configuration file, GitLab will automatically start the build and scanning process. You can monitor the progress of the instructions on the CI/CD → Pipelines tab.
As a result, we have four tasks. Three of them are directly involved in scanning, and the last one (Report) compiles a simple report from the disparate files containing the scanning results.
By default, Trivy halts execution if critical vulnerabilities are found in the image or dependencies. Meanwhile, Hadolint always returns a Success exit code, as its output always includes notes, which leads to the build being stopped.
Depending on specific requirements, you can configure the exit code so that these utilities also halt the build process when problems of a certain criticality are detected. In our case, the build will stop only if Trivy detects a vulnerability with the criticality specified in the SHOWSTOPPER variable in .gitlab-ci.yml.
The results of each utility's execution can be viewed in the log of each scanning task, directly in the JSON files in the artifacts section or in a simple HTML report (more on this below):
3. A small Python script is used to represent the utilities' reports in a slightly more human-readable format, converting three JSON files into a single HTML file with a defect table.
This script is run as a separate Report task, and its output artifact is an HTML report file. The source code of the script is also in the repository and can be adapted for your own needs, colors, etc.
Shell script
The second option is suitable for scenarios where Docker images need to be checked outside of a CI/CD system or where it's necessary to have all instructions in a format that can be executed directly on the host. This option is covered by a ready-made shell script that can be run on a clean virtual (or even real) machine. The script executes the same instructions as the aforementioned gitlab-runner.
For the script to work successfully, Docker must be installed on the system, and the current user must be part of the docker group.
You can get the script here:
At the beginning of the file, variables are set for which image needs to be scanned and which severity defects will cause the Trivy utility to exit with the specified error code.
During the execution of the script, all utilities will be downloaded to the directory docker_tools, and the results of their work will be in the directory docker_tools/json, while the HTML report will be found in the file results.html.
Example output of the script
~\/docker_cicd$ .\/docker_sec_check.sh
[+] Setting environment variables
[+] Installing required packages
[+] Preparing necessary directories
[+] Fetching sample Dockerfile
2020-10-20 10:40:00 (45.3 MB\/s) - 'Dockerfile' saved [8071\/8071]
[+] Pulling image to scan
latest: Pulling from bkimminich\/juice-shop
[+] Running Hadolint
...
Dockerfile:205 DL3015 Avoid additional packages by specifying `--no-install-recommends`
Dockerfile:248 DL3002 Last USER should not be root
...
[+] Running Dockle
...
WARN - DKL-DI-0006: Avoid latest tag
* Avoid 'latest' tag
INFO - CIS-DI-0005: Enable Content trust for Docker
* export DOCKER_CONTENT_TRUST=1 before docker pull\/build
...
[+] Running Trivy
juice-shop\/frontend\/package-lock.json
=====================================
Total: 3 (UNKNOWN: 0, LOW: 1, MEDIUM: 0, HIGH: 2, CRITICAL: 0)
+---------------------+------------------+----------+---------+-------------------------+
| LIBRARY | VULNERABILITY ID | SEVERITY | VERSION | TITLE |
+---------------------+------------------+----------+---------+-------------------------+
| object-path | CVE-2020-15256 | HIGH | 0.11.4 | Prototype pollution in |
| | | | | object-path |
+---------------------+------------------+ +---------+-------------------------+
| tree-kill | CVE-2019-15599 | | 1.2.2 | Code Injection |
+---------------------+------------------+----------+---------+-------------------------+
| webpack-subresource | CVE-2020-15262 | LOW | 1.4.1 | Unprotected dynamically |
| | | | | loaded chunks |
+---------------------+------------------+----------+---------+-------------------------+
juice-shop\/package-lock.json
============================
Total: 20 (UNKNOWN: 0, LOW: 1, MEDIUM: 6, HIGH: 8, CRITICAL: 5)
...
juice-shop\/package-lock.json
============================
Total: 5 (CRITICAL: 5)
...
[+] Removing left-overs
[+] Making the output look pretty
[+] Converting JSON results
[+] Writing results HTML
[+] Clean exit ============================================================
[+] Everything is done. Find the resulting HTML report in results.htmlDocker image with all utilities
As a third alternative, I created two simple Dockerfiles for building an image with security utilities. One Dockerfile will help build a set for scanning an image from the repository, while the other (Dockerfile_tar) will build a set for scanning a tar file with the image.
1. Take the appropriate Docker file and scripts from the repository .
2. Run it to build:
docker build -t dscan:image -f docker_security.df .
3. After the build is complete, create a container from the image. Pass the environment variable DOCKERIMAGE with the name of the image we are interested in, and mount the Dockerfile we want to analyze from our machine to the file /Dockerfile (note that an absolute path to this file is required):
docker run --rm -v $(pwd)/results:/results -v $(pwd)/docker_security.df:/Dockerfile -e DOCKERIMAGE="bkimminich/juice-shop" dscan:image
[+] Setting environment variables
[+] Running Hadolint
/Dockerfile:3 DL3006 Always tag the version of an image explicitly
[+] Running Dockle
WARN - DKL-DI-0006: Avoid latest tag
* Avoid 'latest' tag
INFO - CIS-DI-0005: Enable Content trust for Docker
* export DOCKER_CONTENT_TRUST=1 before docker pull/build
INFO - CIS-DI-0006: Add HEALTHCHECK instruction to the container image
* not found HEALTHCHECK statement
INFO - DKL-LI-0003: Only put necessary files
* unnecessary file : juice-shop/node_modules/sqlite3/Dockerfile
* unnecessary file : juice-shop/node_modules/sqlite3/tools/docker/architecture/linux-arm64/Dockerfile
* unnecessary file : juice-shop/node_modules/sqlite3/tools/docker/architecture/linux-arm/Dockerfile
[+] Running Trivy
...
juice-shop/package-lock.json
============================
Total: 20 (UNKNOWN: 0, LOW: 1, MEDIUM: 6, HIGH: 8, CRITICAL: 5)
...
[+] Making the output look pretty
[+] Starting the main module ============================================================
[+] Converting JSON results
[+] Writing results HTML
[+] Clean exit ============================================================
[+] Everything is done. Find the resulting HTML report in results.htmlResults
We have only covered a basic set of utilities for scanning Docker artifacts, which I believe effectively addresses a considerable portion of the security requirements for images. There are many more paid and free tools available that can perform the same checks, generate attractive reports, or operate purely in console mode, covering container management systems, etc. A review of these tools and integration methods may appear later.
A positive aspect of the toolset described in this article is that all of them are built on open source, allowing you to experiment with them and other similar tools to find what best meets your requirements and infrastructure features. Undoubtedly, all vulnerabilities found should be studied for applicability in specific conditions, but this is a topic for a future comprehensive article.
I hope this guide, scripts, and utilities will help you and serve as a starting point for creating a more secure infrastructure in the realm of containerization.
Source: habr.com
