In Unix-like operating systems, a program interacts with the outside world and the operating system through a small set of functions — system calls. Therefore, for debugging purposes, it can be useful to observe the processes being performed by system calls.
Monitoring the "intimate life" of programs on Linux is aided by a utility strace, which this article is dedicated to. Attached to the examples of using "spy" equipment is a brief history strace and a description of how such programs work.
Content
The Origin of Species
The main interface between programs and the OS kernel in Unix is system calls ( system calls, syscalls), and the interaction of programs with the outside world occurs solely through them.
However, in the first public version of Unix (, 1975) there were no convenient ways to track the behavior of user processes. To solve this problem, Bell Labs proposed a new system call for the next version (, 1979) — ptrace.
Ptrace was primarily developed for interactive debuggers, but by the late '80s (during the commercial era of ) specialized debuggers — system call tracers — emerged based on it and gained widespread popularity.
The same version of strace was published by Paul Kronenburg in the comp.sources.sun mailing list in 1992 as an alternative to the closed utility trace from Sun. Both the clone and the original were intended for SunOS, but by 1994 strace it was ported to System V, Solaris, and the rapidly growing Linux.
Today, strace only supports Linux and relies on the same ptrace, which has gained numerous extensions.
The current (and quite active) maintainer strace — . Thanks to him, the utility has acquired advanced features like error injection in system calls, support for a wide range of architectures, and most importantly, Unofficial sources claim that the choice fell on the ostrich due to the sound similarity between the Russian word «страус» and the English "strace".
It is also important to note that the ptrace system call and debuggers were never included in POSIX, despite their long history and implementations in Linux, FreeBSD, OpenBSD, and traditional Unix.
In short, strace is: Piglet Trace
"You are not expected to understand this" (Dennis Ritchie, comment in the source code of Version 6 Unix)
From early childhood, I have never liked black boxes: I didn’t play with toys, but tried to understand how they worked (adults used the word "broke", but don't believe those wicked tongues). Perhaps that’s why I feel so close to the informal culture of the early Unix and the modern open-source movement.
In this article, it is unreasonable to delve into the source code of strace, which has matured over decades. However, no secrets should remain for readers. Therefore, to demonstrate the principle of how such strace programs work, I will provide the code of a miniature tracer — (ptr). It doesn’t do anything special, but the main thing — it outputs the system calls of the program:
$ gcc examples/piglet-trace.c -o ptr
$ ptr echo test > /dev/null
BRK(12) -> 94744690540544
ACCESS(21) -> 18446744073709551614
ACCESS(21) -> 18446744073709551614
unknown(257) -> 3
FSTAT(5) -> 0
MMAP(9) -> 140694657216512
CLOSE(3) -> 0
ACCESS(21) -> 18446744073709551614
unknown(257) -> 3
READ(0) -> 832
FSTAT(5) -> 0
MMAP(9) -> 140694657208320
MMAP(9) -> 140694650953728
MPROTECT(10) -> 0
MMAP(9) -> 140694655045632
MMAP(9) -> 140694655070208
CLOSE(3) -> 0
unknown(158) -> 0
MPROTECT(10) -> 0
MPROTECT(10) -> 0
MPROTECT(10) -> 0
MUNMAP(11) -> 0
BRK(12) -> 94744690540544
BRK(12) -> 94744690675712
unknown(257) -> 3
FSTAT(5) -> 0
MMAP(9) -> 140694646390784
CLOSE(3) -> 0
FSTAT(5) -> 0
IOCTL(16) -> 18446744073709551591
WRITE(1) -> 5
CLOSE(3) -> 0
CLOSE(3) -> 0
unknown(231)
Tracee terminatedPiglet Trace recognizes about a hundred Linux system calls (see ) and works only on the x86-64 architecture. This is sufficient for educational purposes.
Let’s analyze the operation of our clone. In the case of Linux, debuggers and tracers use, as mentioned above, the ptrace system call. It works by passing command identifiers as the first argument, of which we only need PTRACE_TRACEME, PTRACE_SYSCALL and PTRACE_GETREGS.
The work of the tracer begins in the usual Unix style: fork(2) starts a child process, and that in turn uses exec(3) to launch the program being examined. The only nuance here is the call ptrace(PTRACE_TRACEME) before exec: the child process waits for the parent process to trace it:
pid_t child_pid = fork();
switch (child_pid) {
case -1:
err(EXIT_FAILURE, "fork");
case 0:
/* Child here */
/* Tracing mode must be enabled. A parent will have to wait(2) for it
* to happen. */
ptrace(PTRACE_TRACEME, 0, NULL, NULL);
/* Replace itself with a program to be run. */
execvp(argv[1], argv + 1);
err(EXIT_FAILURE, "exec");
}The parent process must now call wait(2) in the child process, meaning ensuring that the switch to tracing mode has occurred:
/* Parent */
/* First we wait for the child to set the traced mode (see
* ptrace(PTRACE_TRACEME) above) */
if (waitpid(child_pid, NULL, 0) == -1)
err(EXIT_FAILURE, "traceme -> waitpid");This concludes the preparations, and we can proceed directly to monitoring system calls in an infinite loop.
Call ptrace(PTRACE_SYSCALL) ensures that the subsequent wait of the parent will either finish before the execution of the system call or immediately after it is completed. Between the two calls, some actions may be performed: replacing the call with an alternative, modifying arguments, or changing the return value.
We only need to call the command twice ptrace(PTRACE_GETREGS), to capture the state of the register rax before the call (system call number) and immediately after (return value).
Actually, the loop:
/* A system call tracing loop, one interation per call. */
for (;;) {
/* A non-portable structure defined for ptrace/GDB/strace usage mostly.
* It allows to conveniently dump and access register state using
* ptrace. */
struct user_regs_struct registers;
/* Enter syscall: continue execution until the next system call
* beginning. Stop right before syscall.
*
* It's possible to change the system call number, system call
* arguments, return value or even avoid executing the system call
* completely. */
if (ptrace(PTRACE_SYSCALL, child_pid, NULL, NULL) == -1)
err(EXIT_FAILURE, "enter_syscall");
if (waitpid(child_pid, NULL, 0) == -1)
err(EXIT_FAILURE, "enter_syscall -> waitpid");
/* According to the x86-64 system call convention on Linux (see man 2
* syscall) the number identifying a syscall should be put into the rax
* general purpose register, with the rest of the arguments residing in
* other general purpose registers (rdi,rsi, rdx, r10, r8, r9). */
if (ptrace(PTRACE_GETREGS, child_pid, NULL, ®isters) == -1)
err(EXIT_FAILURE, "enter_syscall -> getregs");
/* Note how orig_rax is used here. That's because on x86-64 rax is used
* both for executing a syscall, and returning a value from it. To
* differentiate between the cases both rax and orig_rax are updated on
* syscall entry/exit, and only rax is updated on exit. */
print_syscall_enter(registers.orig_rax);
/* Exit syscall: execute of the syscall, and stop on system
* call exit.
*
* More system call tinkering possible: change the return value, record
* time it took to finish the system call, etc. */
if (ptrace(PTRACE_SYSCALL, child_pid, NULL, NULL) == -1)
err(EXIT_FAILURE, "exit_syscall");
if (waitpid(child_pid, NULL, 0) == -1)
err(EXIT_FAILURE, "exit_syscall -> waitpid");
/* Retrieve register state again as we want to inspect system call
* return value. */
if (ptrace(PTRACE_GETREGS, child_pid, NULL, ®isters) == -1) {
/* ESRCH is returned when a child terminates using a syscall and no
* return value is possible, e.g. as a result of exit(2). */
if (errno == ESRCH) {
fprintf(stderr, "nTracee terminatedn");
break;
}
err(EXIT_FAILURE, "exit_syscall -> getregs");
}
/* Done with this system call, let the next iteration handle the next
* one */
print_syscall_exit(registers.rax);
}That's the entire tracer. Now you know where to start with the next porting to Linux.
Basics: running a program under strace
As a first example of usage strace, it's worth mentioning the simplest way — running an application under strace.
To avoid digging through an endless list of calls of a typical program, let's write around exit:
int main(int argc, char *argv[])
{
char str[] = "write me to stdoutn";
/* write(2) is a simple wrapper around a syscall so it should be easy to
* find in the syscall trace. */
if (sizeof(str) != write(STDOUT_FILENO, str, sizeof(str))){
perror("write");
return EXIT_FAILURE;
}
return EXIT_SUCCESS;
}
Let's compile the program and ensure it works:
$ gcc examples/write-simple.c -o write-simple
$ ./write-simple
write me to stdoutAnd finally, run it under strace:
$ strace ./write-simple
execve("./write", ["./write"], 0x7ffebd6145b0 /* 71 vars */) = 0
brk(NULL) = 0x55ff5489e000
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=197410, ...}) = 0
mmap(NULL, 197410, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7a2a633000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "177ELF2113 3 >