Hello, Habr users! The BPF virtual machine is one of the most important components of the Linux kernel. Its proper application will allow system engineers to identify failures and resolve even the most complex issues. You will learn how to create programs that track and modify kernel behavior, safely inject code for event monitoring in the kernel, and much more. David Kalavera and Lorenzo Fontana will help you unlock the potential of BPF. Expand your knowledge of performance optimization, networking, and security. β Use BPF to track and modify Linux kernel behavior. β Inject code for safe event monitoring in the kernel β without the need to recompile the kernel or reboot the system. β Benefit from convenient code examples in C, Go, or Python. β Manage the situation while mastering the BPF program lifecycle.
Linux kernel security, its capabilities, and Seccomp
BPF provides a powerful way to extend the kernel without compromising stability, security, and speed. For this reason, kernel developers thought it would be a good idea to leverage its versatility to enhance process isolation in Seccomp by implementing Seccomp filters supported by BPF programs, also known as Seccomp BPF. In this chapter, we will explain what Seccomp is and how it is applied. You will then learn how to write Seccomp filters using BPF programs. Finally, we will discuss the built-in BPF traps available in the kernel for Linux security modules.
Linux Security Modules (LSM) are a platform providing a set of functions that can be applied for standardized implementation of various security models. LSM can be used directly in the kernel source code tree, for example, AppArmor, SELinux, and Tomoyo.
Let's start by discussing the capabilities of Linux.
Capabilities
The essence of Linux capabilities is that you need to provide an unprivileged process the permission to perform a specific task, without using suid for that purpose, or otherwise make the process privileged, thus reducing attack possibilities while allowing the process to execute certain tasks. For instance, if your application needs to open a privileged port, say 80, instead of running the process as root, you can simply grant it the CAP_NET_BIND_SERVICE capability.
Let's consider a Go program named main.go:
package main
import (
"net/http"
"log"
)
func main() {
log.Fatalf("%v", http.ListenAndServe(":80", nil))
}This program serves an HTTP server on port 80 (this is a privileged port). Typically, we run it right after compilation:
$ go build -o capabilities main.go
$ ./capabilitiesHowever, since we are not providing root privileges, this code will throw an error when binding to the port:
2019/04/25 23:17:06 listen tcp :80: bind: permission denied
exit status 1capsh (capability shell) is a tool that runs a shell with a specific set of capabilities.
In this case, as previously mentioned, instead of granting full root rights, you can allow binding to privileged ports by providing the cap_net_bind_service capability along with all the others already present in the program. To accomplish this, we can wrap our program in capsh:
# capsh --caps='cap_net_bind_service+eip cap_setpcap,cap_setuid,cap_setgid+ep'
--keep=1 --user="nobody"
--addamb=cap_net_bind_service -- -c "./capabilities"Let's break down this command a bit.
- capsh - we use capsh as the shell.
- --caps='cap_net_bind_service+eip cap_setpcap,cap_setuid,cap_setgid+ep' - since we need to switch the user (we don't want to run with root rights), we specify cap_net_bind_service and the capability to actually change the user ID from root to nobody, namely cap_setuid and cap_setgid.
- --keep=1 - we want to retain the set capabilities when switching from the root account.
- --user='nobody' - the end user running the program will be nobody.
- --addamb=cap_net_bind_service - we set the clearing of related capabilities after switching from root mode.
- -- -c './capabilities' - we simply run the program.
Inherited capabilities are a special kind of capabilities that are passed on to child programs when the current program executes them via execve(). Only capabilities marked as inheritable, or in other words, as ambient capabilities, can be inherited.
You may be wondering what +eip means after specifying a capability in the option βcaps. These flags are used to define what the capability:
-must be activated (p);
-is available for use (e);
-can be inherited by child processes (i).
Since we want to use cap_net_bind_service, we need to do this with the e flag. Then we will launch a shell in the command. As a result, the capabilities binary will be executed, and we need to mark it with the i flag. Finally, we want the capability to be activated (we did this without changing the UID) using p. It looks like cap_net_bind_service+eip.
You can check the result using ss. We will slightly shorten the output to fit on the page, but it will show a related port and a user ID different from 0, in this case, 65,534:
# ss -tulpn -e -H | cut -d' ' -f17-
128 *:80 *:*
users:(("capabilities",pid=30040,fd=3)) uid:65534 ino:11311579 sk:2c v6only:0In this example, we used capsh, but you can write a shell using libcap. For more information, refer to man 3 libcap.
When writing programs, developers often do not know in advance all the capabilities required by the program at runtime; moreover, these capabilities can change in new versions.
To better understand the capabilities of our program, we can take the BCC capable tool, which sets a kprobe for the kernel function cap_capable:
/usr/share/bcc/tools/capable
TIME UID PID TID COMM CAP NAME AUDIT
10:12:53 0 424 424 systemd-udevd 12 CAP_NET_ADMIN 1
10:12:57 0 1103 1101 timesync 25 CAP_SYS_TIME 1
10:12:57 0 19545 19545 capabilities 10 CAP_NET_BIND_SERVICE 1We can achieve the same result using bpftrace with a one-liner kprobe in the kernel function cap_capable:
bpftrace -e
'kprobe:cap_capable {
time("%H:%M:%S ");
printf("%-6d %-6d %-16s %-4d %dn", uid, pid, comm, arg2, arg3);
}'
| grep -i capabilitiesThis will output something like the following if the capabilities of our program are activated after the kprobe:
12:01:56 1000 13524 capabilities 21 0
12:01:56 1000 13524 capabilities 21 0
12:01:56 1000 13524 capabilities 21 0
12:01:56 1000 13524 capabilities 12 0
12:01:56 1000 13524 capabilities 12 0
12:01:56 1000 13524 capabilities 12 0
12:01:56 1000 13524 capabilities 12 0
12:01:56 1000 13524 capabilities 10 1The fifth column represents the capabilities that the process needs, and since this output includes non-audit events, we see all non-audit checks, and finally, the required capability with the audit flag (the last in the output) set to 1. The capability we are interested in is CAP_NET_BIND_SERVICE, which is defined as a constant in the kernel source code in the file include/uapi/linux/ability.h with an identifier of 10:
/* Allows binding to TCP/UDP sockets below 1024 */
/* Allows binding to ATM VCIs below 32 */
#define CAP_NET_BIND_SERVICE 10<source lang="go">Capabilities are often used during the execution of containers, such as runC or Docker, to run them in an unprivileged mode while granting only the capabilities necessary to run most applications. When an application requires specific capabilities, they can be granted in Docker using βcap-add:
docker run -it --rm --cap-add=NET_ADMIN ubuntu ip link add dummy0 type dummyThis command will grant the container the CAP_NET_ADMIN capability, allowing it to configure the network link to add the dummy0 interface.
The next section shows the use of such capabilities, like filtering, but with another method that allows us to implement our own filters programmatically.
Seccomp
Seccomp stands for Secure Computing, a security feature implemented in the Linux kernel that allows developers to filter certain system calls. While Seccomp is comparable to Linux capabilities, its ability to manage specific system calls makes it much more flexible than them.
Seccomp and Linux capabilities are not mutually exclusive; they are often used together to benefit from both approaches. For example, you might want to grant a process the CAP_NET_ADMIN capability but not allow it to accept connections through a socket by blocking the accept and accept4 system calls.
The Seccomp filtering method is based on BPF filters operating in SECCOMP_MODE_FILTER, with system call filtering occurring in the same way as for packets.
Seccomp filters are loaded using prctl via the PR_SET_SECCOMP operation. These filters take the form of BPF programs, which are executed for each Seccomp packet presented via the seccomp_data structure. This structure contains the reference architecture, a CPU instruction pointer during the system call, and up to six system call arguments expressed as uint64.
Here is what the seccomp_data structure looks like from the kernel source in the file linux/seccomp.h:
struct seccomp_data {
int nr;
__u32 arch;
__u64 instruction_pointer;
__u64 args[6];
};As can be seen from this structure, we can filter by system call, its arguments, or a combination of both.
After receiving each packet, the Seccomp filter must process it to make a final decision and inform the kernel what to do next. The final decision is expressed as one of the returned values (status codes).
β SECCOMP_RET_KILL_PROCESS β terminates the entire process immediately after filtering the system call, which is therefore not executed.
β SECCOMP_RET_KILL_THREAD β terminates the current thread immediately after filtering the system call, which is therefore not executed.
β SECCOMP_RET_KILL β an alias for SECCOMP_RET_KILL_THREAD, retained for backward compatibility.
β SECCOMP_RET_TRAP β the system call is forbidden, and the SIGSYS (Bad System Call) signal is sent to the calling task.
β SECCOMP_RET_ERRNO β the system call is not executed, and part of the returned value of the filter SECCOMP_RET_DATA is passed to user space as the errno value. Different errno values are returned depending on the error cause. A list of error numbers is provided in the next section.
β SECCOMP_RET_TRACE β used to notify the ptrace tracer by using β PTRACE_O_TRACESECCOMP to intercept when the system call is made, to see and control this process. If no tracer is attached, an error is returned, errno is set to -ENOSYS, and the system call is not executed.
β SECCOMP_RET_LOG β the system call is allowed and logged.
β SECCOMP_RET_ALLOW β the system call is simply allowed.
ptrace is a system call for implementing tracing mechanisms in a process called tracee, with the capability of observing and controlling the execution of the process. The tracing program can effectively influence the execution and alter the memory registers of the tracee. In the context of Seccomp, ptrace is used when the status code SECCOMP_RET_TRACE is triggered, allowing the tracer to prevent the execution of the system call and implement its own logic.
Seccomp Errors
From time to time, when working with Seccomp, you will encounter various errors that are identified by the return value of type SECCOMP_RET_ERRNO. To report an error, the seccomp system call will return -1 instead of 0.
The following errors may occur:
β EACCESS β the calling party is not allowed to make the system call. This usually happens because it lacks CAP_SYS_ADMIN privileges or the no_new_privs is not set via prctl (we'll discuss this later);
β EFAULT β the provided arguments (args in the seccomp_data structure) do not have a valid address;
β EINVAL β there can be four reasons for this:
- the requested operation is unknown or not supported by the kernel in its current configuration;
- the specified flags are invalid for the requested operation;
- the operation includes BPF_ABS, but there are issues with the specified offset that may exceed the size of the seccomp_data structure;
- the number of instructions passed to the filter exceeds the maximum;
β ENOMEM β insufficient memory to execute the program;
β EOPNOTSUPP β the operation indicated that corresponding action was available with SECCOMP_GET_ACTION_AVAIL, but the kernel does not support returning in the arguments;
β ESRCH β there was a problem synchronizing another thread;
β ENOSYS β there is no tracer attached to the SECCOMP_RET_TRACE action.
prctl is a system call that allows a user-space program to manage (set and get) specific aspects of a process, such as byte order, thread names, secure computing mode (Seccomp), privileges, Perf events, etc.
Seccomp may seem like sandbox technology to you, but it is not. Seccomp is a utility that allows users to develop a sandboxing mechanism. Now letβs look at how user interaction programs are created using a filter directly invoked by the Seccomp system call.
Example of a Seccomp BPF filter
Here we will show how to combine the two previously discussed actions, namely:
β we will write a Seccomp BPF program that will be applied as a filter with various return codes based on the decisions made;
β we will load the filter using prctl.
First, we need the headers from the standard library and the Linux kernel:
#include <errno.h>
#include <linux/audit.h>
#include <linux/bpf.h>
#include <linux/filter.h>
#include <linux/seccomp.h>
#include <linux/unistd.h>
#include <stddef.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/prctl.h>
#include <unistd.h>Before attempting to run this example, we must ensure that the kernel is compiled with CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER set to y. On a working machine, this can be checked like this:
cat /proc/config.gz | zcat | grep -i CONFIG_SECCOMP
The rest of the code consists of the install_filter function, which is made up of two parts. The first part contains our list of BPF filtering instructions:
static int install_filter(int nr, int arch, int error) {
struct sock_filter filter[] = {
BPF_STMT(BPF_LD + BPF_W + BPF_ABS, (offsetof(struct seccomp_data, arch))),
BPF_JUMP(BPF_JMP + BPF_JEQ + BPF_K, arch, 0, 3),
BPF_STMT(BPF_LD + BPF_W + BPF_ABS, (offsetof(struct seccomp_data, nr))),
BPF_JUMP(BPF_JMP + BPF_JEQ + BPF_K, nr, 0, 1),
BPF_STMT(BPF_RET + BPF_K, SECCOMP_RET_ERRNO | (error & SECCOMP_RET_DATA)),
BPF_STMT(BPF_RET + BPF_K, SECCOMP_RET_ALLOW),
}; The instructions are set up using the BPF_STMT and BPF_JUMP macros defined in the linux/filter.h file.
Let's go through the instructions.
β BPF_STMT(BPF_LD + BPF_W + BPF_ABS (offsetof(struct seccomp_data, arch))) β the system loads and accumulates using BPF_LD in word form BPF_W, the packet data is located at a fixed offset BPF_ABS.
β BPF_JUMP(BPF_JMP + BPF_JEQ + BPF_K, arch, 0, 3) β checks using BPF_JEQ whether the architecture value in the constant accumulator BPF_K equals arch. If so, it jumps with offset 0 to the next instruction; otherwise, it skips with offset 3 (in this case) to issue an error because arch does not match.
β BPF_STMT(BPF_LD + BPF_W + BPF_ABS (offsetof(struct seccomp_data, nr))) β loads and accumulates using BPF_LD in word form BPF_W, which is the syscall number located at a fixed offset BPF_ABS.
β BPF_JUMP(BPF_JMP + BPF_JEQ + BPF_K, nr, 0, 1) β compares the syscall number with the value of the variable nr. If they are equal, it moves to the next instruction and denies the syscall; otherwise, it allows the syscall with SECCOMP_RET_ALLOW.
β BPF_STMT(BPF_RET + BPF_K, SECCOMP_RET_ERRNO | (error & SECCOMP_RET_DATA)) β terminates the program with BPF_RET and consequently issues an error SECCOMP_RET_ERRNO with the number from the error variable.
β BPF_STMT(BPF_RET + BPF_K, SECCOMP_RET_ALLOW) β terminates the program with BPF_RET and allows the execution of the syscall using SECCOMP_RET_ALLOW.
SECCOMP IS CBPF
You may be wondering why a list of instructions is used instead of a compiled ELF object or a C program compiled with JIT.There are two reasons for this.
β’ Firstly, Seccomp applies cBPF (classic BPF), not eBPF, which means: it has no registers and only an accumulator to store the last result of calculations, as can be seen in the example.
First, Seccomp accepts a pointer to an array of BPF instructions directly and nothing else. The macros we've used merely help to specify these instructions in a programmer-friendly format.
If you need additional help understanding this assembly, consider the pseudocode that does the same thing:
if (arch != AUDIT_ARCH_X86_64) {
return SECCOMP_RET_ALLOW;
}
if (nr == __NR_write) {
return SECCOMP_RET_ERRNO;
}
return SECCOMP_RET_ALLOW;After defining the filter code in the socket_filter structure, you need to define sock_fprog, which contains the code and the computed length of the filter. This data structure is necessary as an argument for further process declaration:
struct sock_fprog prog = {
.len = (unsigned short)(sizeof(filter) / sizeof(filter[0])),
.filter = filter,
};There's just one more thing to do in the install_filter function: load the program itself! For this, we use prctl, taking PR_SET_SECCOMP as an option to enter the protected computing mode. Then we specify the mode to load the filter using SECCOMP_MODE_FILTER, which is contained in the prog variable of type sock_fprog:
if (prctl(PR_SET_SECCOMP, SECCOMP_MODE_FILTER, &prog)) {
perror("prctl(PR_SET_SECCOMP)");
return 1;
}
return 0;
}Finally, we can use our install_filter function, but first we need to invoke prctl to set PR_SET_NO_NEW_PRIVS for the current execution, thus avoiding a situation where child processes gain broader privileges than their parents. This way, we can make the following prctl calls in the install_filter function without root privileges.
Now we can call the install_filter function. We will block all write system calls related to the X86-64 architecture and simply grant permission, blocking all attempts. After setting the filter, we continue execution using the first argument:
int main(int argc, char const *argv[]) {
if (prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0)) {
perror("prctl(NO_NEW_PRIVS)");
return 1;
}
install_filter(__NR_write, AUDIT_ARCH_X86_64, EPERM);
return system(argv[1]);
}Let's get started. To compile our program, we can use either clang or gcc, in any case, it's just compiling the main.c file without any special options:
clang main.c -o filter-writeAs noted, we have blocked all writes in the program. To check this, we need a program that outputs somethingβls seems like a good candidate. Here's how it usually behaves:
ls -la
total 36
drwxr-xr-x 2 fntlnz users 4096 Apr 28 21:09 .
drwxr-xr-x 4 fntlnz users 4096 Apr 26 13:01 ..
-rwxr-xr-x 1 fntlnz users 16800 Apr 28 21:09 filter-write
-rw-r--r-- 1 fntlnz users 19 Apr 28 21:09 .gitignore
-rw-r--r-- 1 fntlnz users 1282 Apr 28 21:08 main.c
Great! Hereβs what the use of our shell program looks like: we simply pass the program we want to test as the first argument:
./filter-write "ls -la"After execution, this program produces completely empty output. However, we can use strace to see what is happening:
strace -f ./filter-write "ls -la"The output is significantly shortened, but the relevant part shows that entries are blocked with the error EPERM β the same one we configured. This means the program outputs nothing because it cannot access the write system call:
[pid 25099] write(2, "ls: ", 4) = -1 EPERM (Operation not permitted)
[pid 25099] write(2, "write error", 11) = -1 EPERM (Operation not permitted)
[pid 25099] write(2, "n", 1) = -1 EPERM (Operation not permitted)Now you understand how Seccomp BPF works and have a good idea of what can be done with it. But wouldn't it be nice to achieve the same with eBPF instead of cBPF to harness all its power?
When thinking about eBPF programs, most people assume that they just write them and load them with administrator privileges. While this statement is generally true, the kernel implements a set of mechanisms to protect eBPF objects at various levels. These mechanisms are called BPF LSM hooks.
BPF LSM hooks
To provide architecture-independent control over system events, LSM implements the concept of hooks. Technically, a hook call is similar to a system call, however, it is independent from the system and integrated with the infrastructure. LSM provides a new concept in which the level of abstraction can help avoid problems that arise while working with system calls across different architectures.
At the time this book was written, the kernel had seven hooks related to BPF programs, and SELinux is the only built-in LSM that implements them.
The source code of the hooks is located in the kernel tree in the file include/linux/security.h:
extern int security_bpf(int cmd, union bpf_attr *attr, unsigned int size);
extern int security_bpf_map(struct bpf_map *map, fmode_t fmode);
extern int security_bpf_prog(struct bpf_prog *prog);
extern int security_bpf_map_alloc(struct bpf_map *map);
extern void security_bpf_map_free(struct bpf_map *map);
extern int security_bpf_prog_alloc(struct bpf_prog_aux *aux);
extern void security_bpf_prog_free(struct bpf_prog_aux *aux);Each of them will be called at different stages of execution:
β security_bpf β performs the initial check of system calls made by BPF;
β security_bpf_map β checks when the kernel returns a file descriptor for the map;
β security_bpf_prog β checks when the kernel returns a file descriptor for the eBPF program;
β security_bpf_map_alloc β checks whether the security field in BPF maps is initialized;
β security_bpf_map_free β checks whether the security field in BPF maps is cleared;
β security_bpf_prog_alloc β checks whether the security field in BPF programs is initialized;
β security_bpf_prog_free β checks whether the security field in BPF programs is cleared.
Now, seeing all this, we understand: the idea of LSM BPF interceptors is that they can provide protection for each eBPF object, ensuring that only those with the proper privileges can perform operations on maps and programs.
Summary
Security is not something you can universally enforce for everything you want to protect. Itβs important to be able to secure systems at different levels and in various ways. Believe it or not, the best way to secure a system is to establish different layers of protection from various positions, so that compromising one level of security does not allow access to the entire system. Kernel developers have done a lot of work providing us with a set of different layers and interaction points. We hope weβve given you a good understanding of what layers are and how to use BPF programs to work with them.
About the Authors
David Calavera is the CTO at Netlify. He has worked in Docker support and contributed to the development of Runc, Go, BCC tools, and other open source projects. He is known for his work on Docker projects and the development of the Docker plugin ecosystem. David is very passionate about flame graphs and always strives for performance optimization.
Lorenzo Fontana He works in a team of software developers at Sysdig, primarily focusing on Falco β a project by the Cloud Native Computing Foundation that secures container runtime environments and detects anomalies through the kernel module and eBPF. He is passionate about distributed systems, software-defined networking, the Linux kernel, and performance analysis.
Β» Learn more about the book at
Β»
Β»
For Habr users, a 25% discount with the coupon β Linux
Upon payment for the print version of the book, an electronic version will be sent to your email.
Source: habr.com
