What is the beauty of separating the container runtime environment into distinct tool components? Specifically, the fact that these tools can be combined to protect each other.
Many are drawn to the idea of building container OCI images within or a similar system. Suppose we have CI/CD that constantly builds images, then something like /Kubernetes было бы весьма полезно с точки зрения распределения нагрузки при сборке. До недавних пор большинство людей просто давали контейнерам доступ к Docker-сокету и разрешали выполнять команду docker build. , that it is very insecure, in fact, it's even worse than giving passwordless root or sudo.
That’s why people constantly try to run Buildah in a container. In short, we created what we believe is the best way to run Buildah inside a container, and we’ve uploaded the corresponding images to . Let's get started…
Settings
These images are built from Dockerfiles that can be found in the Buildah repository in the .
Here we will discuss .
# stable/Dockerfile
#
# Build a Buildah container image from the latest
# stable version of Buildah on the Fedoras Updates System.
# https://bodhi.fedoraproject.org/updates/?search=buildah
# This image can be used to create a secured container
# that runs safely with privileges within the container.
#
FROM fedora:latest
# Don't include container-selinux and remove
# directories used by dnf that are just taking
# up space.
RUN yum -y install buildah fuse-overlayfs --exclude container-selinux; rm -rf /var/cache /var/log/dnf* /var/log/yum.*
# Adjust storage.conf to enable Fuse storage.
RUN sed -i -e 's|^#mount_program|mount_program|g' -e '/additionalimage.*/a "/var/lib/shared",' /etc/containers/storage.conf
Instead of OverlayFS implemented at the host Linux kernel level, we use a program inside the container called , since currently, OverlayFS can only mount if given SYS_ADMIN privileges using Linux capabilities. And we want to run our Buildah containers without any root-level privileges. Fuse-overlay works quite quickly and performs better than the VFS storage driver. Note that when running a Buildah container that uses Fuse, you need to provide the device /dev/fuse.
podman run --device /dev/fuse quay.io/buildahctr ...
RUN mkdir -p /var/lib/shared/overlay-images /var/lib/shared/overlay-layers; touch /var/lib/shared/overlay-images/images.lock; touch /var/lib/shared/overlay-layers/layers.lock
Next, we create a directory for additional stores. supports the concept of mounting additional read-only image stores. For example, one can set up an overlay storage area on one machine and then mount this storage on another machine via NFS and use images from it without downloading through pull. We need this storage to be able to mount some kind of image store from the host as a volume and use it inside the container.
# Set up environment variables to note that this is
# not starting with user namespace and default to
# isolate the filesystem with chroot.
ENV _BUILDAH_STARTED_IN_USERNS="" BUILDAH_ISOLATION=chroot
Finally, by using the environment variable BUILDAH_ISOLATION, we indicate that by default, the Buildah container should run with chroot isolation. Additional isolation is not required here since we are already operating within a container. To allow Buildah to create its own containers with namespace separation, SYS_ADMIN privilege is needed, which will require relaxing SELinux and SECCOMP rules for the container, conflicting with our goal of performing builds from a secure container.
Running Buildah Inside a Container
The image scheme for the Buildah container discussed above allows for flexible variations in how such containers are launched.
Speed versus Security
Computer security is always a trade-off between process execution speed and the level of protection surrounding it. This statement holds true during container builds as well; below, we will examine options for this compromise.
The container image discussed above will keep its storage in /var/lib/containers. Therefore, we need to mount content into this folder, and the manner in which we do this will significantly impact the speed of building container images.
Let’s consider three options.
Option 1. If maximum security is required, a separate folder for containers/image can be created for each container and mounted to the container via volume-mount. Additionally, the context directory can be placed inside the container in the /build folder:
# mkdir /var/lib/containers1
# podman run -v ./build:/build:z -v /var/lib/containers1:/var/lib/containers:Z quay.io/buildah/stable
buildah -t image1 bud /build
# podman run -v /var/lib/containers1:/var/lib/containers:Z quay.io/buildah/stable buildah push image1 registry.company.com/myuser
# rm -rf /var/lib/containers1
Security. When running in such a container, Buildah has maximum security: it is not granted any root privileges through capabilities, and all SECOMP and SELinux restrictions are applied to it. This container can even be run with User Namespace isolation by adding an option like --uidmap 0:100000:10000.
Performance. However, the performance here is minimal, since any images from container registries are copied to the host each time, and caching does not work at all. Upon completion, the Buildah container must send the image to the registry and destroy the content on the host. When the container image is built next time, it will need to be downloaded again from the registry, as there will be nothing left on the host by that time.
Option 2. If you need Docker-level performance, you can mount the host's container/storage directly inside the container.
# podman run -v ./build:/build:z -v /var/lib/containers:/var/lib/containers --security-opt label:disabled quay.io/buildah/stable buildah -t image2 bud /build
# podman run -v /var/lib/containers:/var/lib/containers --security-opt label:disabled quay.io/buildah/stable buildah push image2 registry.company.com/myuser
Security. This is the least secure way to build containers, as it allows the container to modify the storage on the host, potentially leading to malicious images being injected into Podman or CRI-O. Additionally, SELinux separation will need to be disabled for processes within the Buildah container to interact with the host storage. Note that this option is still better than the Docker socket because the container is restricted by the remaining security features and cannot simply run any container on the host.
Performance. Here it is maximized as caching is fully utilized. If Podman or CRI-O has already downloaded the required image to the host, the Buildah process inside the container will not need to download it again, and subsequent builds based on that image can also retrieve what they need from the cache.
Option 3. The essence of this method is to combine several images into one project with a shared folder for container images.
# mkdir /var/lib/project3
# podman run --security-opt label_level=s0:C100, C200 -v ./build:/build:z
-v /var/lib/project3:/var/lib/containers:Z quay.io/buildah/stable buildah -t image3 bud /build
# podman run --security-opt label_level=s0:C100, C200
-v /var/lib/project3:/var/lib/containers quay.io/buildah/stable buildah push image3 registry.company.com/myuser
In this example, we do not delete the project folder (/var/lib/project3) between runs, so all subsequent builds within the project benefit from caching.
Security. A middle ground between options 1 and 2. On one hand, containers do not have access to content on the host and therefore cannot inject anything malicious into the Podman/CRI-O image storage. On the other hand, within its project, the container can interfere with the building of other containers.
Performance. Here it is worse than using a shared cache at the host level since previously downloaded images via Podman/CRI-O cannot be used. However, once Buildah downloads an image, it can be used in any subsequent builds within the project.
Additional storage
The There is such a cool thing as additional stores, which allows container engines to use external image storage in read-only overlay mode during the launch and assembly of containers. Essentially, you can add one or more 'read-only' storage options to the storage.conf file, so that when the container is started, the container engine will look for the required image in these stores. It will only download the image from the registry if it does not find it in any of these storages. The container engine will only be able to write to writable storage...
If you scroll up and look at the Dockerfile we are using to build the image quay.io/buildah/stable, you will see such lines:
# Adjust storage.conf to enable Fuse storage.
RUN sed -i -e 's|^#mount_program|mount_program|g' -e '/additionalimage.*/a "/var/lib/shared",' /etc/containers/storage.conf
RUN mkdir -p /var/lib/shared/overlay-images /var/lib/shared/overlay-layers; touch /var/lib/shared/overlay-images/images.lock; touch /var/lib/shared/overlay-layers/layers.lock
In the first line, we modify /etc/containers/storage.conf within the container image, instructing the storage driver to use 'additionalimagestores' in the /var/lib/shared folder. In the next line, we create a shared folder and add a couple of lock files to avoid conflicts from containers/storage. Essentially, we are just creating an empty container image storage.
If we mount containers/storage one level higher than this folder, Buildah will be able to use images.
Now let's return to the previously discussed Option 2, where the Buildah container can read and write to containers/store on hosts, and consequently achieves maximum performance by caching images at the Podman/CRI-O level but offers minimal security since it can write directly to the storages. Now, if we add additional stores here, we will get the best of both worlds.
# mkdir /var/lib/containers4
# podman run -v ./build:/build:z -v /var/lib/containers/storage:/var/lib/shared:ro -v /var/lib/containers4:/var/lib/containers:Z quay.io/buildah/stable
buildah -t image4 bud /build
# podman run -v /var/lib/containers/storage:/var/lib/shared:ro
-v >/var/lib/containers4:/var/lib/containers:Z quay.io/buildah/stable buildah push image4 registry.company.com/myuser
# rm -rf /var/lib/continers4
Note that /var/lib/containers/storage on the host is mounted to /var/lib/shared inside the container in read-only mode. Therefore, while working in the container, Buildah can use any images that have previously been downloaded via Podman/CRI-O (hello, speed), but can only write to its own storage (hello, security). Also note that this is done without disabling SELinux separation for the container.
Important nuance
Under no circumstances should any images be deleted from the underlying storage. Otherwise, the Buildah container may crash.
And this is by no means all the advantages
The capabilities of additional storage are not limited to the above scenario. For example, you can place all container images in a shared network storage and grant access to all Buildah containers. Suppose we have hundreds of images that our CI/CD system regularly uses to build container images. We centralize all these images on a single storage host and then, using preferred network storage methods (NFS, Gluster, Ceph, ISCSI, S3...), open this storage for all Buildah or Kubernetes nodes.
Now, it's enough to mount this network storage in the Buildah container at /var/lib/shared, and that's it—Buildah containers will never need to download images via pull again. This way, we eliminate the pre-population phase and are ready to roll out containers immediately.
Of course, this can be utilized within an existing Kubernetes or container infrastructure to run and execute containers anywhere without downloading images via pull. Moreover, when the container registry receives a push request to upload an updated image, it can automatically send that image to the shared network storage, where it instantly becomes available to all nodes.
Container image sizes can sometimes reach several gigabytes. The additional storage functionality allows us to avoid cloning such images across nodes and makes container launches practically instantaneous.
In addition, we are currently working on a new feature for overlay volume mounts, which will make container builds even faster.
Conclusion
Running Buildah inside a container in a Kubernetes/CRI-O, Podman, or even Docker environment is quite feasible, and it's simpler and much safer than using docker.socket. We have significantly increased flexibility in working with images, and now you can run them in various ways for optimal balance between security and performance.
The additional storage functionality allows for speeding up or even completely eliminating the downloading of images to nodes.
Source: habr.com
