An experimental release of the dynamic debugging tool DTrace for Linux 2.0.0-1.14 has been introduced, implemented as a user-space process using the eBPF subsystem and standard tracing mechanisms provided by the Linux kernel. In terms of functionality, the eBPF-based implementation of DTrace is close to the first DTrace implementation for Linux, which was done as a kernel module. The project code is distributed under the GPLv2 license.
The toolkit can be used with standard Linux kernels that support BPF. It requires the libctf library with support for the CTF (Compact Type Format) debugging format, which is part of the binutils package starting from release 2.40, or the libdtrace-ctf library, ported from Solaris. Two optional patches for kernel 6.7 are offered, enabling enhanced capabilities for obtaining additional data about modules and the kernel.
The DTrace technology was developed to address dynamic tracing tasks of the kernel and end-user applications in the Solaris operating system. DTrace allows users to closely monitor system behavior and perform real-time problem diagnostics. During debugging, DTrace does not affect the operation of the analyzed applications and does not impact their performance, enabling on-the-fly analysis of running systems. Among DTrace's strengths is the high-level D language, similar to AWK, which significantly simplifies the creation of tracing scripts compared to the handler writing tools available for eBPF in C, Python, and Lua with external libraries.
Main Features:
- Available providers:
- cpc (CPU Performance Counter) — retrieving performance information.
- dtrace — BEGIN, END, and ERROR handlers that are invoked before or after other checks, as well as in the event of errors.
- fbt (Function Boundary Tracing) — tracking calls to kernel functions.
- lockstat — tracking lock states.
- pid — tracking calls to functions in user-space processes.
- proc — tracking process-related activities such as starting and stopping (fork, exec, exit, fault).
- profile — saving performance statistics at specified intervals.
- sdt, rawtp — static kernel tracing (SDT — Statically Defined Tracing).
- usdt: static application tracing (USDT — Userspace Statically Defined Tracing)
- sched — tracking CPU resource allocation.
- sycall — tracking entry into and exit from system calls.
- Aggregation of collected data: ability to apply aggregate functions (avg, count, llquantize, lquantize, max, min, quantize, stddev, and sum) and aggregate actions (clear, normalize, normalize, printa). Support for storing the results of applying aggregate functions in regular and associative arrays.
- Support for speculative tracing, allowing ephemeral tracking of data, deciding which data to keep in the trace buffer and which to discard. Functions available: speculation, speculate, commit, and discard.
- Support for global and local variables, TLS (Thread-Local Storage), associative arrays, and strings.
- Provision of built-in variables: arg0 — arg9, args[], caller, curcpu, curthread, epid, errno, execname, gid, id, pid, ppid, probefunc, probemod, probename, probeprov, stackdepth, tid, timestamp, ucaller, uid, uregs[], ustackdepth, walltimestamp.
- Support for actions: exit, freopen, ftruncate, mod, printa, printf, raise, setopt, stack, sym, system, trace, tracemem, uaddr, umod, ustack, and usym.
- Availability of built-in functions: alloca, basename, bcopy, copyin, copyinstr, copyinto, copyout, copyoutstr, dirname, getmajor, getminor, htonl, htonll, htons, index, inet_ntoa, lltostr, mutex_owned, mutex_owner, mutex_type_adaptive, mutex_type_spin, ntohl, ntohll, ntohs, progenyof, rand, rindex, rw_iswriter, rw_read_held, rw_write_held, strchr, strjoin, strlen, strrchr, strstr, strtok, substr.
- Availability of data counters that do not enter trace buffers.
- Support for compiling tracing scripts in D language into BPF programs.
- Precompilation of BPF functions for library calls.
- bpflog option for obtaining logs of verified loaded BPF programs.
- Dynamic code generation and precompilation of code for reuse in BPF programs.
Changes in the new version:
- In the pid provider, used for tracking user space processes, support for tracking instruction execution by their offset in the code has been added.
- Support for using stack() and ustack() functions to aggregate collected data has been added.
- Ability to remove elements of any type from associative arrays by assigning a literal 0.
- Added the print action for outputting structured data with type annotations.
- New built-in functions cleanpath(), d_path(), and link_ntop() have been added.
- The parameter "-xcpu" has been added to bind checks to specific CPUs.
- The parameter "-xlockmem" has been introduced to limit memory size.
- Support for preserving process trace data (USDT) between dtprobed restarts has been ensured.
- Support for the BTF (BPF Type Format) mechanism, providing information for type checking in BPF pseudocode.
- Support for using the configure script for building.
Source: opennet.ru
