We have been following the topic of using systemd in containers for quite some time. Back in 2014, our security engineer Daniel Walsh wrote an article , and a couple of years later, another one titled , in which he stated that the situation had not improved much. In particular, he wrote, "unfortunately, even two years later, if you google 'Docker system', his old article still pops up first. Thus, it's time for a change." Moreover, we have previously discussed the .
In this article, we will show what has changed over time and how Podman can help us with this.
There are many reasons to run systemd inside a container, such as:
- Multi-service containers – many want to extract their multi-service applications from virtual machines and run them in containers. It would be better to break such applications into microservices, but not everyone is skilled at it yet or simply doesn't have the time. Therefore, running such applications as services managed by systemd from unit files makes sense.
- Systemd unit files – most applications running inside containers are built from code that previously ran on virtual or physical machines. These applications have unit files specifically written for them that understand how to run them. Therefore, it’s better to launch services using supported methods rather than hacking your own init service.
- Systemd is a process manager. It manages services (stopping, restarting services, or killing zombie processes) better than any other tool.
However, there are also many reasons not to run systemd in containers. The main one is that systemd/journald controls the output of containers, while tools like or expect containers to log directly to stdout and stderr. Therefore, if you plan to manage containers through orchestration tools like the ones mentioned above, you should seriously consider the use of systemd-based containers. Furthermore, the developers of Docker and Moby have often been strongly opposed to the use of systemd in containers.
The Arrival of Podman
We are pleased to announce that the situation has finally started to progress. The team at Red Hat responsible for container launches has decided to develop . It has been named and offers the same command-line interface (CLI) as Docker. In fact, almost all Docker commands can be used in Podman in exactly the same way. We often hold workshops that are now called , and the very first slide encourages the writing of: alias docker=podman.
Many do just that.
Our take on Podman does not oppose the use of systemd-based containers. After all, systemd is often used as the init subsystem in Linux, and not allowing it to work properly in containers means ignoring how thousands of people are accustomed to launching containers.
Podman knows what needs to be done to ensure systemd operates correctly in a container. It requires things like mounting tmpfs on /run and /tmp. It prefers when a 'container' environment is enabled and looks for write permissions in its section of the cgroup directory and in /var/log/journald.
When starting a container where the first command is init or systemd, Podman automatically configures tmpfs and cgroups to ensure that systemd runs smoothly. To disable this automatic launch mode, the option --systemd=false is used. Note that Podman uses the systemd mode only when it detects that a systemd or init command needs to be executed.
Here is an excerpt from the manual:
man podman run
…--systemd=true|false
Run the container in systemd mode. Enabled by default.
If a systemd or init command is executed inside the container, Podman will configure tmpfs mount points in the following directories:
/run, /run/lock, /tmp, /sys/fs/cgroup/systemd, /var/lib/journal
Additionally, by default, SIGRTMIN+3 will be used as the stop signal.
All this allows systemd to work in an isolated container without any modifications.
NOTE: Systemd attempts to write to the cgroup filesystem. However, SELinux by default prevents containers from doing this. To allow writing, enable the boolean parameter container_manage_cgroup:
setsebool -P container_manage_cgroup true
Now take a look at what the Dockerfile looks like for running systemd in a container using Podman:
# cat Dockerfile
FROM fedora
RUN dnf -y install httpd; dnf clean all; systemctl enable httpd
EXPOSE 80
CMD [ "/sbin/init" ]
That's it.
Now we build the container:
# podman build -t systemd .
Tell SELinux to allow systemd to modify the cgroup configuration:
# setsebool -P container_manage_cgroup true
Many, by the way, forget about this step. Fortunately, it only needs to be done once, and the configuration is retained after the system reboots.
Now just start the container:
# podman run -ti -p 80:80 systemd
systemd 239 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid)
Detected virtualization container-other.
Detected architecture x86-64.
Welcome to Fedora 29 (Container Image)!
Set hostname to <1b51b684bc99>.
Failed to install release agent, ignoring: Read-only file system
File /usr/lib/systemd/system/systemd-journald.service:26 configures an IP firewall (IPAddressDeny=any), but the local system does not support BPF/cgroup based firewalling.
Proceeding WITHOUT firewalling in effect! (This warning is only shown for the first loaded unit using IP firewalling.)
[ OK ] Listening on initctl Compatibility Named Pipe.
[ OK ] Listening on Journal Socket (/dev/log).
[ OK ] Started Forward Password Requests to Wall Directory Watch.
[ OK ] Started Dispatch Password Requests to Console Directory Watch.
[ OK ] Reached target Slices.
…
[ OK ] Started The Apache HTTP Server.
That's it, the service has started and is running:
$ curl localhost
<html xml_lang="en" lang="en">
…
</html>
NOTE: Do not attempt to replicate this on Docker! It still requires a lot of workarounds to run such containers through the daemon. (Additional fields and packages will be needed for everything to work seamlessly in Docker, or you will have to run it in a privileged container. For details, see .)
A couple more cool things about Podman and systemd
Podman works better than Docker in systemd unit files
If containers need to be started at system boot, you can simply insert the corresponding Podman commands into the systemd unit file; it will start the service and monitor it. Podman uses the standard forking model for execution (fork-exec). In other words, container processes are child processes of the Podman process, so systemd can easily monitor them.
Docker uses a client-server model, and Docker CLI commands can also be placed directly in the unit file. However, once the Docker client connects to the Docker daemon, it (the client) becomes just another process handling stdin and stdout. Consequently, systemd has no concept of the relationship between the Docker client and the container running under the Docker daemon, and therefore, within this model, systemd fundamentally cannot monitor the service.
Activating systemd via socket
Podman correctly handles activation through the socket. Since Podman uses the fork-exec model, it can pass the socket to its child container processes. Docker cannot do this as it employs a client-server model.
The varlink service that Podman uses to interact with remote clients and containers is actually activated via a socket. The cockpit-podman package, written in Node.js and part of the cockpit project, allows users to interact with Podman containers through a web interface. The web daemon running cockpit-podman sends messages to the varlink socket that is monitored by systemd. Subsequently, systemd activates the Podman program to receive messages and start managing the containers. Activating systemd via the socket allows bypassing the need for a constantly running daemon for implementing remote APIs.
Additionally, we are developing another client for Podman called podman-remote, which implements the same Podman CLI but invokes varlink to start containers. Podman-remote can operate over SSH sessions, allowing secure interaction with containers on different machines. Over time, we plan to utilize podman-remote to support MacOS and Windows alongside Linux, so that developers on these platforms can run a Linux virtual machine with Podman varlink running and have the full feel that containers are executing on their local machine.
SD_NOTIFY
Systemd allows delaying the startup of auxiliary services until their required containerized service starts. Podman can pass the SD_NOTIFY socket to the containerized service so that the service notifies systemd of its readiness to operate. Again, Docker, using a client-server model, does not support this.
Plans
We plan to add the command podman generate systemd CONTAINERID, which will generate a systemd unit file for managing a specific given container. This should work in both root and rootless modes for unprivileged containers. We have even seen requests for the creation of an OCI-compliant systemd-nspawn runtime environment.
Conclusion
Running systemd in a container is a perfectly understandable need. And thanks to Podman, we finally have a container runtime environment that does not clash with systemd, but rather allows its easy use.
Source: habr.com
