Note: translation.: We present the translation of an article by a senior application security engineer from the British company ASOS.com. He kicks off a series of publications about enhancing security in Kubernetes through the use of seccomp. If readers enjoy the introduction, we will follow the author and continue with his future materials on this topic.
This article is the first in a series of publications on how to create seccomp profiles in the spirit of SecDevOps, without resorting to magic or witchcraft. In the first part, I will discuss the fundamentals and inner workings of seccomp implementation in Kubernetes.
The Kubernetes ecosystem offers a wide variety of ways to ensure the security and isolation of containers. This article focuses on Secure Computing Mode, also known as seccomp. Its essence lies in filtering the system calls available for execution by containers.
Why is this important? A container is merely a process running on a specific machine. It shares the kernel with other applications. If containers could execute any system calls, malware would quickly exploit this to bypass container isolation and impact other applications: intercepting information, altering system settings, and so on.
Seccomp profiles define which system calls should be allowed or denied. The container runtime activates them at startup so that the kernel can monitor their execution. Utilizing such profiles helps limit the attack surface and reduce damage if any program inside the container (that is, your dependencies or their dependencies) starts performing actions it isn’t permitted to.
Understanding the Basics
A basic seccomp profile consists of three elements: defaultAction, architectures (or archMap) and syscalls:
{
"defaultAction": "SCMP_ACT_ERRNO",
"architectures": [
"SCMP_ARCH_X86_64",
"SCMP_ARCH_X86",
"SCMP_ARCH_X32"
],
"syscalls": [
{
"names": [
"arch_prctl",
"sched_yield",
"futex",
"write",
"mmap",
"exit_group",
"madvise",
"rt_sigprocmask",
"getpid",
"gettid",
"tgkill",
"rt_sigaction",
"read",
"getpgrp"
],
"action": "SCMP_ACT_ALLOW"
}
]
}()
defaultAction defines the default fate of any system call not specified in the section syscalls. To simplify the task, let’s focus on two main values that will be used:
-
SCMP_ACT_ERRNO— blocks the execution of the system call, -
SCMP_ACT_ALLOW— allows.
In the section architectures the target architectures are listed. This is important because the filter applied at the kernel level depends on the system call identifiers rather than their names as specified in the profile. Before applying, the container runtime will match them to the identifiers. The point is that system calls can have completely different IDs depending on the architecture of the system. For example, the system call recvfrom (used to get information from a socket) has ID = 64 in x64 systems and ID = 517 in x86. you can find a list of all system calls for architectures x86-x64.
In the section syscalls all system calls are listed along with instructions on what to do with them. For example, you can create a whitelist by setting defaultAction to SCMP_ACT_ERRNO, and assign to the calls in the section syscalls . By doing so, you allow only the calls listed in the section SCMP_ACT_ALLOW, and prohibit all others. For a blacklist, you should swap the values syscallsand actions for the opposite. defaultAction Now a word about nuances that are not so obvious. Note that the recommendations below are based on the assumption that you are deploying a line of business applications in Kubernetes and that it's important for them to operate with the least privileges.
1. AllowPrivilegeEscalation=false
the container has the parameter
In securityContext AllowPrivilegeEscalation . If it is set to, containers will run with the set ( false) bitonno_new_priv A side effect of this parameter being set to
is that the container runtime applies the seccomp profile at the very beginning of the startup process. Thus, all system calls necessary for running internal processes of the runtime (for example, setting user/group IDs, dropping certain capabilities) must be allowed in the profile. true A container that runs a simple
echo hi , will require the following permissions:the following permissions will be required:
{
"defaultAction": "SCMP_ACT_ERRNO",
"architectures": [
"SCMP_ARCH_X86_64",
"SCMP_ARCH_X86",
"SCMP_ARCH_X32"
],
"syscalls": [
{
"names": [
"arch_prctl",
"brk",
"capget",
"capset",
"chdir",
"close",
"execve",
"exit_group",
"fstat",
"fstatfs",
"futex",
"getdents64",
"getppid",
"lstat",
"mprotect",
"nanosleep",
"newfstatat",
"openat",
"prctl",
"read",
"rt_sigaction",
"statfs",
"setgid",
"setgroups",
"setuid",
"stat",
"uname",
"write"
],
"action": "SCMP_ACT_ALLOW"
}
]
}()
… instead of these:
{
"defaultAction": "SCMP_ACT_ERRNO",
"architectures": [
"SCMP_ARCH_X86_64",
"SCMP_ARCH_X86",
"SCMP_ARCH_X32"
],
"syscalls": [
{
"names": [
"arch_prctl",
"brk",
"close",
"execve",
"exit_group",
"futex",
"mprotect",
"nanosleep",
"stat",
"write"
],
"action": "SCMP_ACT_ALLOW"
}
]
}()
But again, why is this a problem? Personally, I would avoid whitelisting the following system calls (unless absolutely necessary): capset, set_tid_address, setgid, setgroups and setuid. However, the real challenge is that by allowing processes you have no control over, you tie the profiles to the implementation of the container runtime. In other words, you might find that after an update to the container runtime (by you or, more likely, by the cloud provider), containers suddenly stop starting.
Tip #1: Run containers with AllowPrivilegeEscaltion=false. This will reduce the size of seccomp profiles and make them less sensitive to changes in the container runtime environment.
2. Setting seccomp profiles at the container level
Seccomp profiles can be set at the pod level:
annotations:
seccomp.security.alpha.kubernetes.io/pod: "localhost/profile.json"… or at the container level:
annotations:
container.security.alpha.kubernetes.io/: "localhost/profile.json"Note that the syntax above will change when Kubernetes seccomp (this event is expected in the next Kubernetes release — 1.18 — editor's note).
Few know that Kubernetes has always had , which caused seccomp profiles to be applied to The execution environment partially compensates for this shortcoming; however, this container does not disappear from the pods, as it is used for configuring their infrastructure.
The issue is that this container always starts with AllowPrivilegeEscalation=true, leading to the problems outlined in point 1, and this cannot be changed.
By applying seccomp profiles at the container level, you avoid this trap and can create a profile that is tailored for a specific container. This will need to be done until the developers fix the bug and a new version (perhaps 1.18?) becomes available for everyone.
Tip #2: Set seccomp profiles at the container level.
In practical terms, this rule usually serves as a universal answer to the question: "Why does my seccomp profile work with docker run, but fails after deployment in the Kubernetes cluster?"
3. Use runtime/default only as a last resort
Kubernetes offers two built-in profile options: runtime/default and docker/default. Both are implemented by the container runtime, not Kubernetes. Therefore, they may differ depending on the runtime environment and its version.
In other words, due to the change in the runtime, the container may gain access to a different set of system calls that it may or may not use. Most runtimes utilize . If you wish to use this profile, ensure it is suitable for you.
The profile docker/default has been deprecated since Kubernetes 1.11, so avoid using it.
In my opinion, the profile runtime/default is well-suited for the purposes for which it was created: to protect users from risks associated with executing commands docker run on their machines. However, when it comes to business applications running in Kubernetes clusters, I would dare to say that this profile is too open, and developers should focus on creating profiles tailored to their applications (or types of applications).
Tip #3: Create seccomp profiles for specific applications. If this is not possible, work on profiles for application types, for example, create an extended profile that incorporates all web APIs of an application in Golang. Only as a last resort use runtime/default.
In future publications, I will discuss how to create seccomp profiles in the spirit of SecDevOps, automate them, and test them in pipelines. In other words, you will have no excuses not to switch to profiles tailored for specific applications.
4. Unconfined is NOT an option
From it turned out that by default . This means that if you do not specify a PodSecurityPolicy, which enables it in the cluster, all pods for which no seccomp profile is defined will operate in seccomp=unconfined.
Operating in such a mode means losing a whole layer of isolation that protects the cluster. This approach is not recommended by security experts.
Tip #4: No container in the cluster should run in mode seccomp=unconfined, especially in production environments.
5. 'Audit Mode'
This point is not unique to Kubernetes, but it still falls into the category of 'what you should know before you begin.'
Creating seccomp profiles has traditionally been a challenging task and largely relied on trial and error. The fact is that users simply do not have the ability to test them in production environments without risking 'taking down' the application.
With the introduction of Linux kernel 4.14, it became possible to run parts of the profile in audit mode, logging information about all system calls to syslog without blocking them. This mode can be activated using the parameter SCMT_ACT_LOG:
SCMP_ACT_LOG: seccomp will not affect the execution of the thread making the system call if it does not match any rule in the filter, but information about the system call will be logged.
Here is a typical strategy for using this feature:
- Allow system calls that are necessary.
- Block system calls that are known to be unnecessary.
- Log information about all other calls.
A simplified example looks like this:
{
"defaultAction": "SCMP_ACT_LOG",
"architectures": [
"SCMP_ARCH_X86_64",
"SCMP_ARCH_X86",
"SCMP_ARCH_X32"
],
"syscalls": [
{
"names": [
"arch_prctl",
"sched_yield",
"futex",
"write",
"mmap",
"exit_group",
"madvise",
"rt_sigprocmask",
"getpid",
"gettid",
"tgkill",
"rt_sigaction",
"read",
"getpgrp"
],
"action": "SCMP_ACT_ALLOW"
},
{
"names": [
"add_key",
"keyctl",
"ptrace"
],
"action": "SCMP_ACT_ERRNO"
}
]
}()
But remember that it is necessary to block all calls that are known not to be used and which potentially could harm the cluster. A good foundation for compiling the list is the official . It explains in detail which system calls are blocked in the default profile and why.
However, there is one catch. Although SCMT_ACT_LOG it has been supported by the Linux kernel since the end of 2017, it has only recently entered the Kubernetes ecosystem. Therefore, you will need Linux kernel 4.14 and runC version no earlier than .
Tip No. 5: An audit mode profile for testing in production can be created by combining black and white lists, with all exceptions logged.
6. Use white lists
Creating white lists requires additional effort because you have to identify every call that the application may need, but this approach significantly increases security:
It is strongly recommended to use a white list-based approach, as it is simpler and more reliable. The black list will need to be updated every time a potentially dangerous system call (or a dangerous flag/option, if they are on the black list) is added. Additionally, you can often change the representation of a parameter without changing its essence, thus circumventing the limitations of the black list.
For applications written in Go, I developed a special tool that accompanies the application and collects all calls made during execution. For example, for the following application:
package main
import "fmt"
func main() {
fmt.Println("test")
} … let's run gosystract as follows:
go install https://github.com/pjbgf/gosystract
gosystract --template='{{- range . }}{{printf "%s", .Name}}{{- end}}' application-path… and we will get the following result:
"sched_yield",
"futex",
"write",
"mmap",
"exit_group",
"madvise",
"rt_sigprocmask",
"getpid",
"gettid",
"tgkill",
"rt_sigaction",
"read",
"getpgrp",
"arch_prctl",This is just an example — details about the toolkit will follow.
Tip #6: Only allow those calls that you truly need, and block all others.
7. Lay the right foundations (or prepare for unexpected behavior)
The kernel will enforce the profile regardless of what you have defined in it. Even if it's not exactly what you wanted. For instance, if you block access to calls like sigreturn or exit_group, the container will not be able to terminate correctly, and even a simple command like , will require the following permissions: indefinitely. As a result, you'll experience high CPU load in the cluster:
In such cases, a utility may come to the rescue strace — it will show what might be causing the problem:
sudo strace -c -p 9331
Make sure the profiles include all system calls that the application requires during its operation.
Tip #7: Pay close attention to details and ensure that all necessary system calls are whitelisted.
This concludes the first part of the series on using seccomp in Kubernetes in the spirit of SecDevOps. In the following parts, we will discuss why this is important and how to automate the process.
P.S. from the translator
Also read in our blog:
- «»;
- «»;
- «»;
- «».
Source: habr.com
