Introduction

Modern software systems are composed of complex chains of programs that invoke each other dynamically during execution. Shell scripts, build systems, and applications frequently spawn multiple subprocesses, each replacing itself through exec system calls. Under normal condition, the execution flow succeeds silently and few cares about it. However, when something breaks in the exec chains, many applications fail to precisely report what goes wrong and produce confusing or even misleading logs. Tracing and understanding the hidden execution flows thus are critical for debugging. Aside from that, tracing command execution is also a vital part of auditing program behavior and security monitoring.

tracexec is an exec tracer written in Rust that collects rich and accurate information and supports human-friendly output. It supports both ptrace-based backend and eBPF-based backend, where the ptrace-based backend is usually suitable for scoped tracing that does not involve setuid binaries while the eBPF-based backend could be use for system-wide tracing.

tracexec supports multiple user interfaces like logs, Terminal User Interface (TUI) and exporting structured traces.

Showcases

The following examples illustrate how tracexec can be used to inspect builds, trace command execution, and launch a debugger.

Perfetto Trace Export

tracexec supports exporting exec traces to perfetto trace format, which could be viewed in the Perfetto UI. The trace follows a tree format in the UI, where processes resulting from successful execs are represented as slices and exec failures are represented as instant events.

The following video shows analyzing the build process of tracexec with itself:

The shape of the traces in the Perfetto UI could give you a rough idea of how parallel the build is at process-level. The trace tree and details of slices enable identification of bottlenecks, troubleshooting, and a deep understanding of how the build works.

Start collecting a perfetto trace with the following command:

tracexec collect --format=perfetto -o out.pftrace -- cmd

See Perfetto Trace Export for instructions on collecting and interpreting a trace.

TUI mode with pseudo terminal

TUI mode allocates a pseudo terminal by default, allowing you to view the details of exec events and interact with the processes within the pseudo terminal. Use --no-tty when a pseudo terminal is not wanted; the tracee’s stdin, stdout, and stderr will be redirected to /dev/null.

Tracing setuid binaries

With root privileges, you can also trace setuid binaries and see how they work. But do note that this is not compatible with seccomp-bpf optimization so it is much less performant. You can use eBPF mode which is more performant in such scenarios.

sudo tracexec --user $(whoami) tui -- sudo ls

Nested setuid binary tracing is also possible: A real world use case is to trace extra-x86_64-build(Arch Linux’s build tool that requires sudo):

In this real world example, we can easily see that _FORTIFY_SOURCE is redefined from 2 to 3, which led to a compiler error.

Use tracexec as a debugger launcher

tracexec can also be used as a debugger launcher to make debugging programs easier. For example, it’s not trivial or convenient to debug a program executed by a shell/python script(which can use pipes as stdio for the program). The following video shows how to use tracexec to launch GDB to attach to two simple programs piped together by a shell script.

See the debugger-launcher tutorial for the complete example.

eBPF mode

Please check platform support status before using the eBPF backend.

The following examples show how to use eBPF in TUI mode. The ebpf command also supports regular log and collect subcommands.

System-wide Exec Tracing

System-wide tracing has no command to attach to a pseudo terminal, so it runs without one automatically:

sudo -E tracexec ebpf tui

Follow Fork mode with eBPF

sudo -E tracexec --user $(whoami) ebpf tui -- bash

Log mode

In log mode, by default, tracexec will print filename, argv and the diff of the environment variables and file descriptors.

example: tracexec log -- bash (In an interactive bash shell)

Reconstruct the command line with --show-cmdline

$ tracexec log --show-cmdline -- <command>
# example:
$ tracexec log --show-cmdline -- firefox

Try to reproduce stdio in the reconstructed command line

--stdio-in-cmdline and --fd-in-cmdline can be used to reproduce(hopefully) the stdio used by a process.

But do note that the result might be inaccurate when pipes, sockets, etc are involved.

tracexec log --show-cmdline --stdio-in-cmdline -- bash

Show the interpreter indicated by shebang with --show-interpreter

And show the cwd with --show-cwd.

$ tracexec log --show-interpreter --show-cwd -- <command>
# example: Running Arch Linux makepkg
$ tracexec log --show-interpreter --show-cwd -- makepkg -f

Installation

Different people have different opinions when it comes to how to install a program. Some may prefer using system package manager while others might like downloading a prebuilt binary. But don’t worry, tracexec supports a wide variety of installation methods.

Before installation, you should check the platform support status.

Install via Package Managers

tracexec is packaged in the following distributions.

Arch Linux (And Arch-based distributions)

tracexec is available in extra repository for Arch Linux. You can install it via

sudo pacman -S tracexec

Nix

To try tracexec without system-wide installation, running

nix-shell -p tracexec

will drop you into a shell where tracexec is available.

NixOS

If you are using NixOS, you should already have your preferred way to install packages.

e.g. by adding pkgs.tracexec to environment.systemPackages

Prebuilt Binaries

For stable versions, we release binaries in GitHub Releases.

Currently we offer two flavors of binaries

• Normal builds that dynamically links most dependencies except libbpf.
• Fully statically-linked builds which statically links all libraries including glibc.

Install from Source

Please refer to Building from Source for dependencies and feature flags.

To install the current stable version of tracexec. Run

cargo install tracexec --bin tracexec

To install the bleeding-edge development version of tracexec from git main branch. Run

cargo install --git https://github.com/kxxt/tracexec --bin tracexec

Platform Support

Currently tracexec only supports Linux operating system. Because the core of tracexec is implemented via ptrace, seccomp-bpf and eBPF, it is difficult to port to Windows, MacOS or other operating systems. (Well, technically speaking, ptrace itself is enough for initializing a port to other operating systems, but ptrace without seccomp-bpf is painfully slow.)

Architecture Support Status

Currently we support the following three architectures. You are welcome to submit PR for supporting more architectures.

*: for riscv64, some kernel versions has bugs in the ptrace implementation that would cause tracexec to display some information as errors. See this strace issue and the kernel mailing list discussion for more details if you got errors when using tracexec on riscv64.

Linux Kernel Support Status

LLVM Support Status

tracexec requires clang from LLVM for building the eBPF backend. We typically test the latest 3 versions of LLVM to ensure that the eBPF program compiled by them could be successfully loaded into the Linux kernels documented in Linux Kernel Support Status.

It is very likely that using other recent LLVM versions would work. If you encounter bugs with an LLVM version that is not covered in our CI, please open an issue and we are happy to help out.

Build from Source

To build tracexec from source, the following dependencies are needed:

• A working rust compiler and cargo.
  • Refer to package.rust-version in Cargo.toml for MSRV.
• libbpf: if not using vendored-libbpf
• zlib: if not using vendored
• libelf: if not using vendored
• libseccomp: For seccomp-bpf.
• If any library vendoring feature is enabled:
  • build-essential autopoint gettext for Debian based distros
  • base-devel for Arch Linux
• protoc for compiling ProtoBuf proto files if protobuf-binding-from-source feature is enabled.
  • By default, protoc from PATH is used. PROTOC environment variable could be used to specify the full path to the desired protoc compiler.
• clang for compiling eBPF program.
  • By default, clang from PATH is used. CLANG environment variable could be used to specify the full path to the desired clang compiler.

Library Linkage

By default, we dynamically link to libseccomp because most distros ship it out of the box. In order to statically link to libseccomp, please set LIBSECCOMP_LINK_TYPE to static and set LIBSECCOMP_LIB_PATH to the path of the directory containing libseccomp.a.

To control whether or not to dynamically link to libbpf, libelf and zlib, consult the next Feature Flags section.

Feature Flags

• recommended: This enables the recommended functionalities of tracexec
  • ebpf: eBPF backend that doesn’t use ptrace and can be used for system-wide tracing
• ebpf-debug: Not meant for end users. This flag enables debug logging to /sys/kernel/debug/tracing/trace_pipe and some debug checks.
• static: Statically link libelf, zlib and libbpf.
• vendored: Vendoring libelf, zlib and libbpf, implies static.
• vendored-libbpf: Vendoring libbpf and statically link to it.

By default, we enable the recommended and vendored-libbpf features. This means that we are dynamically linking zlib and libelf but statically linking libbpf. This choice is made because zlib and libelf are usually installed on most systems but libbpf is usually not.

To dynamically link to libbpf, turn off default features and enable recommended feature:

cargo build --release --no-default-features -F recommended

Features

tracexec supports many features, which will be explained in detail in this chapter.

At a high level, tracexec has two backends and several frontends. A backend collects exec events; a frontend decides how those events are presented or stored.

The default ptrace backend follows a command and its descendants. The eBPF backend can also trace execs across the whole system. Both feed the same event model.

Choose a frontend based on the job:

• Log prints events as they arrive and works well in a pipeline or CI log.
• TUI keeps an interactive event list next to the traced program’s terminal.
• Collect writes JSON, NDJSON, or a Perfetto trace for later analysis.

Filtering, privilege elevation, and most data-collection options are shared across frontends.

To save your preferred settings, see Configuration.

Configuration Profile

If you find yourself passing the same options every time you run tracexec, you can put them in a configuration file. For example, you might want the TUI to start with the Events pane focused, or always show timestamps in the log.

Configuring tracexec

tracexec looks for config.toml in $XDG_CONFIG_HOME/tracexec/, or $HOME/.config/tracexec/ if XDG_CONFIG_HOME is not set.

Create the directory if needed:

mkdir -p "${XDG_CONFIG_HOME:-$HOME/.config}/tracexec"

A configuration template is provided in the repository, which documents all available options with their default values and descriptions.

The file uses TOML. Setting names usually use underscores, such as active_pane, while CLI flags use hyphens, such as --active-pane. Enum values are case-sensitive: write "Events" in TOML, even though the CLI spelling is --active-pane events. The tables below use the TOML spellings.

You can start with the configuration template and adjust settings according to your preference.

Using a Different Profile

Use --profile (or -P) to load a different profile than the default profile; for example:

tracexec --profile ./build.toml tui -- bash

This loads build.toml instead of the default config.toml. The two files are not merged, so a setting omitted from the profile uses its built-in default.

Relative profile paths are resolved from tracexec’s working directory. --cwd changes that directory before loading the profile. For example, tracexec --cwd /path/to/project --profile build.toml tui -- bash reads /path/to/project/build.toml and starts Bash in that directory.

To run with the built-in defaults, use --no-profile:

tracexec --no-profile log -- ls

When using privilege elevation through --elevate, tracexec preserves the original user’s configuration directory.

Ptrace Backend

ptrace(2) is the default backend for tracexec. To use this backend, simply run tracexec with the desired frontend subcommand (log, tui and collect).

A Simple Introduction to Ptrace

ptrace(2) is the interface designed for implementing a debugger. It allows a tracer process to attach to a tracee process and do basically almost anything to it, such as reading/writing its registers and memories, intercepting its syscall and single-step debugging. A single tracer could trace multiple tracees concurrently but a single tracee could only be traced by one tracer at any given time.

strace is a generic syscall tracing tool built upon ptrace(2), while tracexec is a specialized tool for tracing exec syscall and related contexts.

But wait, isn’t ptrace slow since it is a syscall interface meant for debuggers? Would it slow down workloads significantly? It is indeed slow when used in default configuration because we need to stop/resume the program at every syscall it makes. But when combined with seccomp(2), the overhead could actually be reduced to minimal. seccomp(2) implements a fast syscall filtering interface with classic BPF, by combining ptrace(2) with a seccomp(2) filter that only notifies us when the exec syscalls happen, we avoid incurring overhead on other syscalls the tracee makes. In case you want to learn more about this optimization, read the well-written blog post from strace developer.

Strengths

• Works out of the box.
• Low overhead when combined with seccomp(2). (default in tracexec)
• The minimum required Linux kernel version is 5.3.
• Makes it possible to conveniently attach a debugger to a newly spawned process.

Weaknesses

• Cannot perform system-wide tracing.
• Does not work with setuid/setgid binaries out of the box.
• Significant overhead when seccomp(2) optimization is not used.
• ptrace(2) is a very complex interface abusing waitpid(2) and signals.

eBPF Backend

To use this backend, run tracexec with ebpf as subcommand and the desired frontend as sub-subcommand (tracexec ebpf log or tracexec ebpf tui for example).

A Brief Introduction to eBPF

eBPF is a revolutionary technology for running sandboxed and verified programs directly in the Linux kernel. The in-kernel BPF verifier verifies the program before loading it into the kernel to ensure its safety. For tracing exec, eBPF enables us to attach tracing eBPF programs to kernel functions that handle execve and execveat syscalls and other scheduler tracepoints like sched_process_fork that fires when a process creates a new thread or a new process.

Strengths

• System-wide tracing makes the eBPF backend well-suited for system observability.
• Scoped tracing is also implemented.
• Does not use ptrace(2) so
  • Tracing setuid/setgid binaries is supported.
  • You could combine it with other tools that use ptrace(2), e.g. gdb.

Weaknesses

• Requires root privilege. (or a bunch of capabilities like CAP_SYS_ADMIN and CAP_BPF)
• When sleepable eBPF is not available, sometimes reading userspace memory will fail due to page fault, causing the trace to miss some information.
  • See also https://mozillazg.com/2024/03/ebpf-tracepoint-syscalls-sys-enter-execve-can-not-get-filename-argv-values-case-en.html
• Requires loading eBPF code into Linux kernel, which might be forbidden in kernel lockdown mode.
• Sometimes there are kernel eBPF bugs that could reject the eBPF program.

Required Kernel Configs for eBPF Backend

Required Config Entries

The eBPF backend of course needs a kernel with eBPF and ftrace enabled:

CONFIG_DEBUG_INFO_BTF=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_EVENTS=y
CONFIG_FTRACE=y
CONFIG_FUNCTION_TRACER=y
CONFIG_KPROBES=y
CONFIG_KPROBE_EVENTS=y

We need the JIT of eBPF enabled and turned on because when the JIT is disabled, the verifier rejects our program.

CONFIG_BPF_JIT=y

An optional but highly recommended config entry is:

CONFIG_FUNCTION_ERROR_INJECTION=y

It enables tracexec to use sleepable eBPF programs for tracing the entry of exec syscalls. If this config is turned off, tracexec will use non-sleepable eBPF programs, which should work fine for most cases but might fail to read some data from user-space when the data is not yet loaded into the RAM. This problem is thoroughly explained in a blog post: https://mozillazg.com/2024/03/ebpf-tracepoint-syscalls-sys-enter-execve-can-not-get-filename-argv-values-case-en.html.

Example Config

The config used in our UKCI could serve as a reference for building a custom kernel that supports tracexec. It is written in Nix. To obtain a raw kernel config, build the .#ukci target and then dig /nix/store/*linux-config* out of the nix store.

Advanced Parameters for eBPF Backend

The parameters listed here are not considered a stable interface. They may be MODIFIED or completely REMOVED and it would not be considered as a breaking change.

You should only use parameters from this page if you understand it.

TRACEXEC_NO_SLEEP env var

By default, tracexec automatically detects whether the kernel supports sleepable fentry eBPF programs. If this environment variable is set to a non-empty value, tracexec will use non-sleepable eBPF programs for fentry of exec syscalls.

When fentry is disabled and kprobe is used, this setting has no effect.

TRACEXEC_USE_FENTRY/KPROBE env vars

By default, tracexec automatically detects whether the kernel supports CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS to decide whether to use fentry/fexit or kprobe/kretprobe. These two environment variables could override it.

• When TRACEXEC_USE_FENTRY is set to a non-empty value, tracexec will use fentry/fexit eBPF programs.
• When TRACEXEC_USE_KPROBE is set to a non-empty value, tracexec will use kprobe/kretprobe eBPF programs.
• Setting both variables simultaneously is not supported and may produce unpredictable results.

Log Frontend

Logging is the most simple frontend for tracexec. This frontend simply logs exec events to the terminal or a file.

To use the log frontend,

• run tracexec log to use ptrace backend,
• or run tracexec ebpf log to use eBPF backend.

By default, the log frontend shows the filename, argv and diff of environment variables, along with basic information like PID, comm before exec and syscall result.

Log Format

This section introduces the UI elements in the log lines. For example, let’s simply run ls with a new environment variable:

Basic Elements

tracexec log -- env A=B ls

As shown in the output,

1. The tracer forks and executes /usr/bin/env, with our supplied command line arguments.
   • The exec syscall is successful so the pid at the start of the line is in green color.
   • The comm of the original process is tracer before exec.
2. /usr/bin/env tried to execute ls program from paths listed in the PATH environment variable.
   • The first few attempts failed because ls program is not found in the attempted directory. The return value -2 is displayed at the end of the line with a user-friendly error message. The pid at the start of the line is displayed in yellow color, meaning a usually non-fatal error occurred.
   • Since we specified A=B in the commandline, env adds this environment variable when executing ls. tracexec shows this added environment variable in green color with a plus sign, indicating it is a new env var.
3. At last, /usr/bin/env successfully executed /usr/bin/ls and the output from ls is shown.

Timestamp

By default, tracexec does not show the timestamp for the events. Use --timestamp to enable it.

For example:

tracexec log --timestamp -- env ls /

It is possible to customize the format of the timestamp with --inline-timestamp-format <INLINE_TIMESTAMP_FORMAT>. See https://docs.rs/chrono/latest/chrono/format/strftime/index.html for available variables that can be used in the format string.

Verbosity of Environment Variables

By default, tracexec only outputs the diff of environment variables against the initial environment.

To increase the verbosity to show all environment variables, use --show-env. e.g.

tracexec log --show-env -- ls

To hide the environment variables entirely, use --no-show-env.

Reconstruct the Shell Commandline

A handy feature of tracexec is to show the equivalent shell commandlines of the exec events. This feature makes it easy to reproduce the exec events in a shell.

For example:

tracexec log --show-cmdline -- env -u LANG A=B ls

In this example, we use -u LANG to remove the LANG environment variable and A=B to set A env to B for the ls command. tracexec shows the equivalent shell commandline of the successful execution as env -a ls -u LANG A=B /usr/bin/ls, which could be directly copy-and-pasted into a bash shell.

File Descriptor Tracking

File descriptors are inherited during exec unless the file descriptor is marked with O_CLOEXEC. Bugs or even security vulnerabilities may occur if a program forgets to close file descriptors after fork and before exec. However, sometimes it is normal to keep some file descriptors open in order to pass them to the child process.

tracexec tracks the file descriptors used during exec and shows a diff of file descriptors by default.

In the following example, we run cat with stdin closed and stdout redirected to /dev/null and a new file descriptor of /dev/random.

tracexec log -- bash -c "cat <&- > /dev/null 4</dev/random"

tracexec shows the file descriptor diff with three entries.

• closed: stdin in red color for the closed stdin.
• stdout="/dev/null" in yellow color, meaning the stdout fd is modified and the current value is /dev/null.
• 4="/dev/random" in green color, indicating a new fd numbered 4 pointing to /dev/random.

By default, tracexec hides the file descriptors marked O_CLOEXEC that will be closed upon exec. To show such file descriptors, use --no-hide-cloexec-fds option. As demonstrated in the following example, a python script opened a file descriptor with O_CLOEXEC, which gets closed when executing the shell. tracexec shows the file descriptor in red color as cloexec: 3="/".

Log Output Destination

By default, the log frontend outputs to stderr.

To output to stdout, use --output - or -o- (e.g. target/debug/tracexec log -o- -- ls). To output to a file, use --output <PATH> where <PATH> is the path to the file for output. tracexec will truncate the file if it already exists.

(EXPERIMENTAL) Reconstruct Shell Commandline with File Descriptors

Previously, we showed how to reconstruct shell commandlines and track inherited file descriptors. We can combine them to reconstruct a full shell commandline with the file descriptors.

To reconstruct the commandline with stdio descriptors, use --stdio-in-cmdline:

tracexec log --show-cmdline --stdio-in-cmdline -- bash -c "cat <&- > /dev/null 4</dev/random"

To reconstruct the commandline with all file descriptors, use --fd-in-cmdline:

tracexec log --show-cmdline --fd-in-cmdline -- bash -c "cat <&- > /dev/null 4</dev/random"

This feature is currently experimental. It may produce inaccurate command lines. For example,

• in the above example, the reconstructed cmdline shows FD 4 as both readable and writable, but actually we only used it as an input file descriptor.
• Shells like zsh have limit on the file descriptor number that could be used in the cmdline. Thus the reconstructed cmdline may not work in some shells. For instance,
  • tracexec log -- bash -c "ls 114514</dev/null" succeeds,
  • but tracexec log -- zsh -c "ls 114514</dev/null" failed with ls: cannot access '114514': No such file or directory

Terminal User Interface

The Terminal User Interface (TUI) of tracexec supports most of features and provides an interactive way for tracing exec events.

By default, the TUI uses a built-in terminal pane for the tracee.

For example, here is how it usually looks like.

# Ptrace backend
tracexec tui -- env -u LANG A=B ls
# eBPF backend
tracexec --elevate ebpf tui -- env -u LANG A=B ls

The TUI is designed to be intitutive, that is, you should be able to use it without reading the rest of this docs. It shows available actions at the bottom. When focused on the event list, press F1 key to view the help within the TUI.

But if you like reading docs, feel free to continue.

Basics

In this section, I will introduce the basics of the TUI of tracexec.

Navigation

By default, the TUI comes with two panes: the Events pane and the Terminal pane.

The Terminal pane is focused upon launch but you can configure tracexec to focus the Events pane with a configuration file.

Terminal Pane

If you launch the TUI to trace a shell, you can type commands in the Terminal pane and tracexec will show the exec events in the Events pane.

If you launch the TUI to trace a program, you can interact with it in the Terminal pane.

Use Ctrl+U shortcut to enter scrollback mode, in which you can view the history of the terminal pane. Use ↑/↓/PgUp/PgDn/Home/End to navigate through the scroll buffer. Use Ctrl+U shortcut again to exit the scrollback mode.

Events Pane

Switch to the Events pane by shortcut Ctrl+S.

• Use ↑/↓ to scroll up/down the events list.
• Use PgUp/PgDn/Ctrl+↑/Ctrl+↓ to scroll faster.
• Use Home/End to jump to the start and the end of the list.
• Use ←/→ to scroll left/right in the events list.
• Use Ctrl+←/Ctrl+→ to scroll faster.
• Use Shift+Home/Shift+End to jump to the left end and the right end of the view.

To locate events by their text, use Ctrl+F. See Search for matching rules and result navigation.

How to Exit Vim Tracexec

When the Events pane is focused, press Q to exit.

You can switch to the Events pane by shortcut Ctrl+S if the Terminal pane is focused.

If there are still tracees running in the terminal pane, tracexec will wait for them after the TUI is closed. Press Ctrl+C to terminate them.

Layout

When the Events pane is focused,

• Press Alt+L to change between vertical and horizontal layout.
• Hold G/S to grow/shrink the Events pane.

Search

Search locates events by matching text in the Events pane. It supports literal text and regular expressions, with optional case sensitivity. Matching events are highlighted while the surrounding events remain visible, preserving the context of each invocation.

Search Procedure

With the Events pane focused, press Ctrl+F to open the search prompt. Enter a query and press Enter to execute the search.

The first matching event is selected. Press N to move to the next match or P to move to the previous match. The result counter reports the position within the matching events and their total number; 2/5, for example, denotes the second of five matching events. Each event contributes one result, even if the query occurs several times in its text. An unsuccessful search displays No match.

Press Ctrl+F again to edit the query. The existing text and matching options are retained. To close search and remove the highlights, press Esc while editing. Thus, after submitting a query, use Ctrl+F followed by Esc to close it. Submitting an empty query has the same effect.

The following table lists the default bindings. They can be changed through Key Bindings.

Matching Rules

A new search uses case-insensitive literal matching by default. For example, sample.c matches that text anywhere in an event line, including SAMPLE.C. The period is treated as an ordinary character. With case sensitivity enabled, only the specified letter case matches.

Regular-expression mode interprets the query as a pattern. For example, sample-[ab]\.c matches sample-a.c or sample-b.c, while gcc|clang matches either compiler name wherever it occurs in a line. Case sensitivity can be changed independently of regular-expression mode. The footer identifies the current modes while the query is being edited.

An invalid regex expression would cause a Regex Error popup to show.

The following recording compares literal and regular-expression matching for sample-[ab]\.c. Enabling case sensitivity excludes SAMPLE-A.C; an invalid pattern then illustrates error reporting and correction.

The displayed environment and working directory also form part of the search text. After submission, E and W toggle these fields in the Events pane and recompute the results. A hidden field does not contribute matches.

Example

Start a shell under tracexec.

tracexec tui -- bash --noprofile --norc

In the Terminal pane, execute the following commands:

/usr/bin/printf '%s\n' sample-a.c
/usr/bin/printf '%s\n' sample-b.c
/usr/bin/printf '%s\n' notes.txt

The explicit path invokes the external printf instead of shell builtin, so each command produces an exec event. Switch to Events with Ctrl+S, open search with Ctrl+F, and submit sample-. The first two printf events match because their argument lists contain that text. The notes.txt event remains visible but is not highlighted.

Use N and P to move between the results. To repeat the search with a pattern, press Ctrl+F, clear the text with Ctrl+U, enter sample-[ab]\.c, toggle regular-expression mode with Alt+R, and press Enter.

Real-time Search

An active search query is applied to newly arriving events. Collection continues while the query is edited and while its results are inspected.

In this recording, the query is submitted before any matching events exist. The result count increases as commands are executed in the Terminal pane. An unrelated command leaves the count unchanged. A counter such as 0/2 indicates that two matches have arrived but neither has been selected through search navigation yet.

Event Details

The events list gives you a quick overview of what was executed. When you want to know more about a particular event, select it in the Events pane and press V to open its details. If the Terminal pane is focused, use Ctrl+S to switch panes first.

For exec events, the details popup has three tabs: Info, Environment and FdInfo. Other events, such as warnings, only have the Info tab.

Navigation

• Use ←/→ to switch tabs, or Tab to cycle through them.
• Use ↑/↓ to scroll up/down.
• Use PgUp/PgDn to scroll a page at a time.
• Use Home/End to jump to the top or bottom of the current tab.
• Press Q to close the popup.

The following recording opens an exec event, selects a field, scrolls through its details and uses U to view the parent event.

Info Tab

The Info tab shows the command line and the information collected about the exec call. The fields include:

Syscall Result tells you whether the exec call succeeded. To see how the program ended, look at Process Status. Similarly, Duration is not the time spent inside the exec syscall. If the process is still running, close and reopen the popup later to see its updated status.

To collect the Cgroup field, start tracexec with --collect-cgroup:

tracexec tui --collect-cgroup -- bash

The two experimental command line fields try to include shell redirections for stdio or all file descriptors. They can help you understand the setup, but commands involving pipes or sockets may not be runnable as shown.

Copying a Field

In the Info tab, press W/S to select the previous/next field. The selected field is highlighted and marked with an arrow. Press C to copy its value to the system clipboard, when clipboard access is available.

Scrolling and field selection are separate: ↑/↓ move the view, while W/S change which value will be copied. To copy the environment or a reconstructed command line directly from the events list, see Copy.

Viewing the Parent Event

If the parent event is still available, its command line appears in the Info tab. Parent(Spawner) means a process spawned a child to execute this program. Parent(Becomer) means the same process replaced its previous program with this one.

Press U from any tab to open the parent event’s details. This is useful when you want to work backwards through a script or build process. Parent details are only available for events that tracexec has recorded and still keeps in the events list. See Backtrace for more about these relationships.

Environment Tab

The Environment tab shows the environment passed to the exec call, with changes relative to tracexec’s starting environment:

• + marks an added variable or the new value of a modified variable.
• - marks a removed variable or the old value of a modified variable.
• A leading space marks an unchanged variable.

For a modified variable, the old value appears first and the new value follows it. Unchanged variables appear after the changes. All events use the same baseline, so this is not a recursive diff.

In this example, env adds GREETING, removes LANG and changes DEMO_MODE from before to after before executing true.

File Descriptors Tab

The FdInfo tab shows the file descriptors collected at exec. Each entry includes its number, path, flags, mount information, file position and inode number. Some descriptors also have extra information, depending on their type.

Descriptors 0, 1 and 2 are standard input, output and error. Their targets can help explain why output went to a file or pipe instead of your terminal. Other descriptors can reveal files that a parent process left open for the new program.

Here, cat has its stdin and stdout redirected to /dev/null, and an extra descriptor 3 open for reading from /dev/zero. Switch to FdInfo and scroll down to see it.

By default, tracexec hides descriptors marked close-on-exec, since they are closed when exec succeeds. To include them, use --no-hide-cloexec-fds:

tracexec tui --no-hide-cloexec-fds -- bash

The environment and file descriptor tabs show data collected for that exec event; they do not track later changes made by the running program. If tracexec could not collect a value, the popup shows the error or an unavailable marker.

Built-in Terminal and External Terminal

By default, tracexec uses an internal & built-in terminal when performing a scoped trace of a user-specified command. The built-in terminal offers a convenient way to interact with the traced processes inside tracexec. However, the built-in terminal supports limited terminal features and may not work well for some use cases. As a workaround, you can also let tracexec trace an external terminal emulator.

Built-in terminal

tracexec tui allocates a pseudo-terminal (PTY) for the command by default. The command’s standard input, output, and error streams are connected to the Terminal pane, while the TUI itself is drawn on the surrounding terminal.

tracexec tui -- bash

Press Ctrl+S to switch between the Terminal and Events panes. If a terminal program needs to receive a literal Ctrl+S, focus the Events pane and press Alt+S.

The PTY has a 1,000-line scrollback buffer by default. Change it for one run with --scrollback-lines:

tracexec tui --scrollback-lines 10000 -- bash

TUI without a terminal pane

Use --no-tty when the command needs no interaction:

tracexec tui --no-tty -- make -j8

In this mode the command’s stdin, stdout, and stderr are redirected to /dev/null; --no-tty does not inherit the outer terminal. Use the log frontend if you want the command to keep the current terminal while tracexec prints events alongside it.

External Terminal Emulators

To use tracexec with an external terminal emulator, make tracexec trace the external terminal directly.

For example,

tracexec tui --no-tty -- konsole

launches an external konsole terminal emulator in a separate window.

You can also specify the command to run for the external terminal emulator. But the way to do it depends on which terminal emulator you are using.

# For konsole
tracexec tui --no-tty -- konsole -e "bash"
# For kitty
tracexec tui --no-tty -- kitty bash

Backtrace

The TUI can reconstruct the exec backtrace of an event. It is NOT a stack backtrace.

Traversing the Backtrace in the Events Pane

Press U when focusing the Events pane to jump to the parent event of the selected event.

View the Full Backtrace

Select an exec event in the Events pane and press T to show the full backtrace. The popup lists the oldest available ancestor first and the selected event last. Its markers distinguish two relationships:

• S (spawns): a process forked a child, and the child later executed the next program;
• B (becomes): the same process replaced its image with another program.

Incomplete backtraces

Parent links refer to earlier event IDs kept by the TUI. A popup is marked incomplete when an ancestor has already been discarded because --max-events was reached. Increase the limit, or use --max-events 0 for an unlimited list when retaining the full lineage matters.

Failed exec attempts appear in the event list but do not replace the last successful program image. They therefore do not become ancestors of later successful execs.

Breakpoints

tracexec supports setting breakpoints at exec syscall enter/exit stops, which enables you to pause programs before they start to execute user-space code.

This feature is only available for the ptrace backend.

Why

You might be wondering why such a feature is useful. It is mainly because of a limitation of the ptrace(2) API. A single tracee can only have one tracer at any time. As a result, debuggers like gdb cannot be used on processes traced by tracexec.

This feature provides a way to hand over processes traced by tracexec to other debuggers. For example, you can stop a program launched deep inside a shell script and attach gdb to it, with its environment, working directory and pipes already set up.

Breakpoint Stops

There are two places where you can set a breakpoint:

• sysenter: right before the exec syscall. The process still has its old program image.
• sysexit: right after the exec syscall. If the exec succeeded, the new program is loaded but has not started running user-space code yet.

Breakpoint Patterns

A breakpoint pattern decides which exec calls to stop at. There are three kinds of patterns:

The filename patterns match the filename recorded by tracexec. The filename is typically the exact value used in execve syscall or the resolved path of the value used in execveat syscall.

For argv-regex, the arguments include argv[0] and are joined without any quoting or escaping. For example, ["echo", "hello world"] becomes echo hello world. The regex can match anywhere in that string; use ^ and $ if you want to match the whole string.

Setting Breakpoints from the Command Line

Use -b (or --add-breakpoint) to add a breakpoint before tracing starts. The format is <breakpoint-stop>:<pattern-kind>:<pattern>.

For example, to stop whenever a program whose filename contains /my-program is executed:

tracexec tui -b 'sysexit:in-filename:/my-program' -- bash

You can now run ./my-program in the terminal pane, either directly or through a script. It will stop before it starts running, and tracexec will show a hit at the bottom of the screen.

You can use -b multiple times to add more breakpoints:

tracexec tui \
    -b 'sysexit:exact-filename:./a' \
    -b 'sysexit:exact-filename:./b' \
    -- ./shell-script

Quote the breakpoint when it contains spaces or shell special characters, especially for regex patterns.

Setting Breakpoints in the TUI

When the Events pane is focused, press B to open the Breakpoint Manager. If the Terminal pane is focused, use Ctrl+S to switch panes first.

Press N to create a new breakpoint. The editor accepts only the pattern, such as in-filename:/my-program, without the sysenter: or sysexit: prefix. Do not add a space after the colon unless you want that space to be part of the pattern.

New breakpoints are active and stop at Syscall Exit by default. While editing:

• Press Alt+S to switch between Syscall Enter and Syscall Exit.
• Press Alt+A to toggle whether the breakpoint is active.
• Press Enter to save, or Ctrl+C to cancel.

In the breakpoint list, use ↑/↓ to select a breakpoint. Press Enter or E to edit it, Space to enable or disable it, or Delete/D to delete it. Press Q to return to the events pane.

Disabling or deleting a breakpoint does not resume a process that has already hit it.

Handling Breakpoint Hits

When a process hits a breakpoint, tracexec pauses that process and shows the number of hits at the bottom of the screen. Other tracees can keep running, although they may be waiting for the stopped process.

Press Z from the Events pane to open the Hit Manager. Use ↑/↓ to select a stopped process, then:

• Press R to resume it and keep tracing it.
• Press D to detach and let it continue without tracexec tracing it.
• Press Enter to detach, leave it stopped and run the default external command.
• Press Alt+Enter to enter a command to run for this particular hit.

Press Q to close the Hit Manager. This leaves the processes stopped. You can press F1 in either manager to view its help.

Launching a Debugger

See Use tracexec as debugger launcher for a complete tutorial.

Set --default-external-command to the command you want to launch for a hit. tracexec replaces {{PID}} with the PID of the detached and stopped process. You can also set or edit this command by pressing E in the Hit Manager.

For example, if you use Konsole:

tracexec tui --seccomp-bpf=off \
    -b 'sysexit:in-filename:/my-program' \
    --default-external-command 'konsole -e gdb -p {{PID}}' \
    -- bash

Run your program in the terminal pane. When it hits the breakpoint, switch to the events pane, press Z, select the hit and press Enter. A new terminal will open with gdb attached to the process. You may need to run continue twice in gdb because of the stop signal used during detach.

Use a terminal emulator or a command such as tmux split-window for an interactive debugger: the external command’s standard input, output and error are connected to /dev/null. The command supports shell-style quoting, but is not run through a shell. If you need shell features such as pipes or redirection, invoke a shell explicitly.

The --seccomp-bpf=off option matters if the detached process or its children need to exec other programs. With the seccomp-bpf optimization enabled, those exec calls can fail with Function not implemented after detach. Set this option when starting tracexec.

Copy

When running in a desktop environment (X11 or Wayland), the TUI of tracexec supports copying details to the system clipboard.

To use it, first select an event in the event list. Then press C to open the copy popup.

After that, select an entry in the list and press Enter to copy it to the clipboard. Alternatively, press the corresponding character of the entry to directly copy it.

Available Copy targets

• (c) Command line: the reconstructed commandline
• (o) Command line with full env: the reconstructed commandline with full set of environment variables.
• Experimental. (s) Command line with Stdio: the reconstructed commandline with stdio file descriptors.
• Experimental. (f) Command line with File descriptors: the reconstructed commandline with file descriptors.
• (e) Environment Variables: environment variables in "KEY"="VALUE" format.
• (d) Diff of environment variables: diff of environment variables in the following format.

# Added:
"KEY2"="VALUE2"
# Modified: (original first)
"PATH"="OLDPATH"
"PATH"="NEWPATH"
# Removed:
"KEY1"="VALUE1"

• (a) Arguments: argv in list format, e.g. ["/usr/bin/starship", "time"].
• (w) Arguments joined by whitespace: argv joined by whitespace.
• (n) Filename: the file name of the executable.
• (r) Syscall result: the result of the exec syscall
• (l) Current Line: the current entry as displayed in the TUI.

Theme

The TUI supports custom themes. A theme is defined in a theme file and specified in the config file as follows.

[tui]
theme-file = "nord.toml"

Theme File Resolution

Absolute paths are used as-is.

Relative paths are resolved relative to the theme directories, in the following order:

1. $XDG_CONFIG_HOME/tracexec/themes/ (or $HOME/.config/tracexec/themes/)
2. $XDG_DATA_HOME/tracexec/themes/ (or $HOME/.local/share/tracexec/themes/)
3. /etc/tracexec/themes/
4. <path_to_tracexec_binary>/../share/tracexec/themes/ (usually /usr/share/tracexec/themes/)

Theme File Format

The theme file is written in TOML and the TUI theme is put under a section named tui, as demonstrated by the following example:

# Cool blue theme inspired by the Nord palette.

[tui]
inactive-border = { fg = "#4c566a" }
active-border = { fg = "#88c0d0", modifiers = ["bold"] }
app-title = { fg = "#eceff4" }
help-popup = { fg = "#2e3440", bg = "#88c0d0" }
cli-flag = { fg = "#2e3440", bg = "#81a1c1" }
help-key = { fg = "#2e3440", bg = "#88c0d0" }
help-desc = { fg = "#d8dee9", bg = "#3b4252", remove-modifiers = ["italic"] }
pid-success = { fg = "#a3be8c" }
pid-failure = { fg = "#bf616a" }
pid-enoent = { fg = "#ebcb8b" }
comm = { fg = "#88c0d0" }
tracer-info = { fg = "#81a1c1" }
tracer-warning = { fg = "#ebcb8b" }
tracer-error = { fg = "#bf616a" }
new-child-pid = { fg = "#8fbcbb" }
tracer-event = { fg = "#b48ead" }
partial-ok = { fg = "#ebcb8b", modifiers = ["italic"] }
filename = { fg = "#88c0d0" }
cwd = { fg = "#8fbcbb" }
modified-env-var = { fg = "#ebcb8b" }
added-env-var = { fg = "#a3be8c" }
query-match-current-no = { fg = "#88c0d0", modifiers = ["bold"] }
query-match-total-cnt = { fg = "#d8dee9" }
breakpoint-title-selected = { fg = "#2e3440", bg = "#81a1c1" }
breakpoint-pattern = { fg = "#88c0d0" }
breakpoint-info-value = { fg = "#2e3440", bg = "#8fbcbb" }
hit-entry-breakpoint-pattern = { fg = "#88c0d0" }
hit-manager-default-command = { fg = "#8fbcbb" }
active-tab = { fg = "#2e3440", bg = "#81a1c1" }
backtrace-parent-spawns = { content = " S ", fg = "#2e3440", bg = "#88c0d0", modifiers = ["bold"] }
backtrace-parent-becomes = { content = " B ", fg = "#eceff4", bg = "#5e81ac", modifiers = ["bold"] }

The theme file is applied as an override to the built-in theme. That is, the styles are merged with the built-in theme and unspecified entries will use the built-in theme.

A theme entry specifies the style of a UI element. It supports the following attributes.

• fg: foreground color
• bg: background color
• underline-color: color of underline decoration
• modifiers: a list of modifiers to apply.
• remove-modifiers: a list of modifiers to remove from the built-in theme.

For colors, the following formats are supported.

• An unsigned 8-bit integer representing an 8-bit color.
• A named color listed in https://docs.rs/ratatui-core/0.1.2/ratatui_core/style/enum.Color.html#variants. (Kebab-case should be used here)
• A hex color string in #RRGGBB format.
• A dict specifying the rgb color separately, like { r = 255, g = 0, b = 0 }

The following modifiers are supported:

• bold
• dim
• italic
• underlined
• slow-blink
• rapid-blink
• reversed
• hidden
• crossed-out

Some theme entries support specifying the content of the UI element.

backtrace-parent-spawns = { content = " S ", bg = "red" }

Supported Theme Entries

The supported theme entries are listed in https://github.com/kxxt/tracexec/blob/main/crates/tracexec-core/src/cli/tui_theme.rs.

Theme in Config File

The theme could also be specified directly in the config file, as shown in the following example.

[tui]
theme = { app-title = { fg = "cyan" }, active-border = { fg = "light-cyan" } }

Key Bindings

You can customize the key bindings in the config file.

The key bindings of the TUI are defined in the tui.keys section, as shown in the following example:

[tui.keys]
quit = "q"
switch_pane = "Ctrl+s"
# switch_layout = "Alt+l"
# close_popup = "q"
# help = "F1"
# page_down = ["Ctrl+Down", "Ctrl+j", "PgDn"]
# page_up = ["Ctrl+Up", "Ctrl+k", "PgUp"]

Configuration Format

For each action, you can bind a single or multiple key bindings to it.

For example, switch_pane = "Ctrl+s" binds Ctrl+S to switch_pane action. page_up = ["Ctrl+Up", "Ctrl+k", "PgUp"] binds multiple shortcuts to the page_up action.

Supported Key Bindings

The supported key bindings are listed in https://github.com/kxxt/tracexec/blob/main/crates/tracexec-core/src/cli/keys.rs.

Export Frontend

The export frontend supports exporting exec trace to various output formats.

Currently, the following formats are supported:

• JSON
• NDJSON (Newline delimited JSON)
• Perfetto trace

JSON Export

We support two formats for JSON export.

• The json format exports the whole trace as a complete JSON.
• The json-stream format exports the trace as JSON messages separated by newlines. This format is also known as NDJSON when --pretty is not used.

Use --output <OUTPUT_FILE_PATH> to keep exporter data separate from output written by the traced command.

Warning

The output JSON may contain sensitive credentials that are passed in commandline arguments or environment variables. Sharing it may leak such credentials.

JSON Format

tracexec collect --format json --output trace.json -- env -C / ls

{"version":"0.17.0","generator":"tracexec_exporter_json","baseline":{"cwd":"/home/kxxt","env":{"HOME":"/home/kxxt","PATH":"/home/kxxt/.cargo/bin:/home/kxxt/mambaforge/bin:/home/kxxt/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/opt/cuda/bin:/home/kxxt/.local/share/flatpak/exports/bin:/var/lib/flatpak/exports/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/usr/lib/rustup/bin","PWD":"/home/kxxt","SHLVL":"0","_":"/usr/bin/tracexec"},"fdinfo":{"0":{"fd":0,"path":"/dev/pts/4","pos":0,"flags":["O_RDWR"],"mnt_id":39,"ino":7,"mnt":"39 36 0:26 / /dev/pts rw,nosuid,noexec,relatime shared:4 - devpts devpts rw,gid=5,mode=600,ptmxmode=000","extra":[]},"1":{"fd":0,"path":"pipe:[202485]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":202485,"mnt":"Not found. This is probably a pipe or something else.","extra":[]},"2":{"fd":0,"path":"pipe:[202485]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":202485,"mnt":"Not found. This is probably a pipe or something else.","extra":[]}}},"events":[{"id":1,"pid":20417,"syscall":"execve","exec_pid":20417,"cwd":"/home/kxxt","comm_before_exec":"tracer","result":0,"filename":"/usr/bin/env","argv":{"result":"success","value":["env","-C","/","ls"]},"env":{"result":"success","value":{"has_added_or_modified_keys_starting_with_dash":false,"added":{},"removed":[],"modified":{}}},"fdinfo":{"0":{"fd":0,"path":"/dev/pts/4","pos":0,"flags":["O_RDWR"],"mnt_id":39,"ino":7,"mnt":"39 36 0:26 / /dev/pts rw,nosuid,noexec,relatime shared:4 - devpts devpts rw,gid=5,mode=600,ptmxmode=000","extra":[]},"1":{"fd":0,"path":"pipe:[202485]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":202485,"mnt":"Not found. This is probably a pipe or something else.","extra":[]},"2":{"fd":0,"path":"pipe:[202485]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":202485,"mnt":"Not found. This is probably a pipe or something else.","extra":[]}},"timestamp":1787669590103448326,"cred":{"result":"success","value":{"groups":[209,936,953,962,991,992,1000],"uid_real":1000,"uid_effective":1000,"uid_saved_set":1000,"uid_fs":1000,"gid_real":1000,"gid_effective":1000,"gid_saved_set":1000,"gid_fs":1000}},"cgroup":{"kind":"not-collected"}},{"id":2,"pid":20417,"syscall":"execve","exec_pid":20417,"cwd":"/","comm_before_exec":"env","result":-2,"filename":"/home/kxxt/.cargo/bin/ls","argv":{"result":"success","value":["ls"]},"env":{"result":"success","value":{"has_added_or_modified_keys_starting_with_dash":false,"added":{},"removed":[],"modified":{}}},"fdinfo":{"0":{"fd":0,"path":"/dev/pts/4","pos":0,"flags":["O_RDWR"],"mnt_id":39,"ino":7,"mnt":"39 36 0:26 / /dev/pts rw,nosuid,noexec,relatime shared:4 - devpts devpts rw,gid=5,mode=600,ptmxmode=000","extra":[]},"1":{"fd":0,"path":"pipe:[202485]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":202485,"mnt":"Not found. This is probably a pipe or something else.","extra":[]},"2":{"fd":0,"path":"pipe:[202485]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":202485,"mnt":"Not found. This is probably a pipe or something else.","extra":[]}},"timestamp":1787669590105681518,"cred":{"result":"success","value":{"groups":[209,936,953,962,991,992,1000],"uid_real":1000,"uid_effective":1000,"uid_saved_set":1000,"uid_fs":1000,"gid_real":1000,"gid_effective":1000,"gid_saved_set":1000,"gid_fs":1000}},"cgroup":{"kind":"not-collected"}},{"id":3,"pid":20417,"syscall":"execve","exec_pid":20417,"cwd":"/","comm_before_exec":"env","result":-2,"filename":"/home/kxxt/mambaforge/bin/ls","argv":{"result":"success","value":["ls"]},"env":{"result":"success","value":{"has_added_or_modified_keys_starting_with_dash":false,"added":{},"removed":[],"modified":{}}},"fdinfo":{"0":{"fd":0,"path":"/dev/pts/4","pos":0,"flags":["O_RDWR"],"mnt_id":39,"ino":7,"mnt":"39 36 0:26 / /dev/pts rw,nosuid,noexec,relatime shared:4 - devpts devpts rw,gid=5,mode=600,ptmxmode=000","extra":[]},"1":{"fd":0,"path":"pipe:[202485]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":202485,"mnt":"Not found. This is probably a pipe or something else.","extra":[]},"2":{"fd":0,"path":"pipe:[202485]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":202485,"mnt":"Not found. This is probably a pipe or something else.","extra":[]}},"timestamp":1787669590106807056,"cred":{"result":"success","value":{"groups":[209,936,953,962,991,992,1000],"uid_real":1000,"uid_effective":1000,"uid_saved_set":1000,"uid_fs":1000,"gid_real":1000,"gid_effective":1000,"gid_saved_set":1000,"gid_fs":1000}},"cgroup":{"kind":"not-collected"}},{"id":4,"pid":20417,"syscall":"execve","exec_pid":20417,"cwd":"/","comm_before_exec":"env","result":-2,"filename":"/home/kxxt/.nix-profile/bin/ls","argv":{"result":"success","value":["ls"]},"env":{"result":"success","value":{"has_added_or_modified_keys_starting_with_dash":false,"added":{},"removed":[],"modified":{}}},"fdinfo":{"0":{"fd":0,"path":"/dev/pts/4","pos":0,"flags":["O_RDWR"],"mnt_id":39,"ino":7,"mnt":"39 36 0:26 / /dev/pts rw,nosuid,noexec,relatime shared:4 - devpts devpts rw,gid=5,mode=600,ptmxmode=000","extra":[]},"1":{"fd":0,"path":"pipe:[202485]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":202485,"mnt":"Not found. This is probably a pipe or something else.","extra":[]},"2":{"fd":0,"path":"pipe:[202485]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":202485,"mnt":"Not found. This is probably a pipe or something else.","extra":[]}},"timestamp":1787669590107950785,"cred":{"result":"success","value":{"groups":[209,936,953,962,991,992,1000],"uid_real":1000,"uid_effective":1000,"uid_saved_set":1000,"uid_fs":1000,"gid_real":1000,"gid_effective":1000,"gid_saved_set":1000,"gid_fs":1000}},"cgroup":{"kind":"not-collected"}},{"id":5,"pid":20417,"syscall":"execve","exec_pid":20417,"cwd":"/","comm_before_exec":"env","result":-2,"filename":"/nix/var/nix/profiles/default/bin/ls","argv":{"result":"success","value":["ls"]},"env":{"result":"success","value":{"has_added_or_modified_keys_starting_with_dash":false,"added":{},"removed":[],"modified":{}}},"fdinfo":{"0":{"fd":0,"path":"/dev/pts/4","pos":0,"flags":["O_RDWR"],"mnt_id":39,"ino":7,"mnt":"39 36 0:26 / /dev/pts rw,nosuid,noexec,relatime shared:4 - devpts devpts rw,gid=5,mode=600,ptmxmode=000","extra":[]},"1":{"fd":0,"path":"pipe:[202485]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":202485,"mnt":"Not found. This is probably a pipe or something else.","extra":[]},"2":{"fd":0,"path":"pipe:[202485]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":202485,"mnt":"Not found. This is probably a pipe or something else.","extra":[]}},"timestamp":1787669590109016420,"cred":{"result":"success","value":{"groups":[209,936,953,962,991,992,1000],"uid_real":1000,"uid_effective":1000,"uid_saved_set":1000,"uid_fs":1000,"gid_real":1000,"gid_effective":1000,"gid_saved_set":1000,"gid_fs":1000}},"cgroup":{"kind":"not-collected"}},{"id":6,"pid":20417,"syscall":"execve","exec_pid":20417,"cwd":"/","comm_before_exec":"env","result":-2,"filename":"/usr/local/sbin/ls","argv":{"result":"success","value":["ls"]},"env":{"result":"success","value":{"has_added_or_modified_keys_starting_with_dash":false,"added":{},"removed":[],"modified":{}}},"fdinfo":{"0":{"fd":0,"path":"/dev/pts/4","pos":0,"flags":["O_RDWR"],"mnt_id":39,"ino":7,"mnt":"39 36 0:26 / /dev/pts rw,nosuid,noexec,relatime shared:4 - devpts devpts rw,gid=5,mode=600,ptmxmode=000","extra":[]},"1":{"fd":0,"path":"pipe:[202485]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":202485,"mnt":"Not found. This is probably a pipe or something else.","extra":[]},"2":{"fd":0,"path":"pipe:[202485]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":202485,"mnt":"Not found. This is probably a pipe or something else.","extra":[]}},"timestamp":1787669590110303068,"cred":{"result":"success","value":{"groups":[209,936,953,962,991,992,1000],"uid_real":1000,"uid_effective":1000,"uid_saved_set":1000,"uid_fs":1000,"gid_real":1000,"gid_effective":1000,"gid_saved_set":1000,"gid_fs":1000}},"cgroup":{"kind":"not-collected"}},{"id":7,"pid":20417,"syscall":"execve","exec_pid":20417,"cwd":"/","comm_before_exec":"env","result":-2,"filename":"/usr/local/bin/ls","argv":{"result":"success","value":["ls"]},"env":{"result":"success","value":{"has_added_or_modified_keys_starting_with_dash":false,"added":{},"removed":[],"modified":{}}},"fdinfo":{"0":{"fd":0,"path":"/dev/pts/4","pos":0,"flags":["O_RDWR"],"mnt_id":39,"ino":7,"mnt":"39 36 0:26 / /dev/pts rw,nosuid,noexec,relatime shared:4 - devpts devpts rw,gid=5,mode=600,ptmxmode=000","extra":[]},"1":{"fd":0,"path":"pipe:[202485]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":202485,"mnt":"Not found. This is probably a pipe or something else.","extra":[]},"2":{"fd":0,"path":"pipe:[202485]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":202485,"mnt":"Not found. This is probably a pipe or something else.","extra":[]}},"timestamp":1787669590111465266,"cred":{"result":"success","value":{"groups":[209,936,953,962,991,992,1000],"uid_real":1000,"uid_effective":1000,"uid_saved_set":1000,"uid_fs":1000,"gid_real":1000,"gid_effective":1000,"gid_saved_set":1000,"gid_fs":1000}},"cgroup":{"kind":"not-collected"}},{"id":8,"pid":20417,"syscall":"execve","exec_pid":20417,"cwd":"/","comm_before_exec":"env","result":0,"filename":"/usr/bin/ls","argv":{"result":"success","value":["ls"]},"env":{"result":"success","value":{"has_added_or_modified_keys_starting_with_dash":false,"added":{},"removed":[],"modified":{}}},"fdinfo":{"0":{"fd":0,"path":"/dev/pts/4","pos":0,"flags":["O_RDWR"],"mnt_id":39,"ino":7,"mnt":"39 36 0:26 / /dev/pts rw,nosuid,noexec,relatime shared:4 - devpts devpts rw,gid=5,mode=600,ptmxmode=000","extra":[]},"1":{"fd":0,"path":"pipe:[202485]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":202485,"mnt":"Not found. This is probably a pipe or something else.","extra":[]},"2":{"fd":0,"path":"pipe:[202485]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":202485,"mnt":"Not found. This is probably a pipe or something else.","extra":[]}},"timestamp":1787669590112979337,"cred":{"result":"success","value":{"groups":[209,936,953,962,991,992,1000],"uid_real":1000,"uid_effective":1000,"uid_saved_set":1000,"uid_fs":1000,"gid_real":1000,"gid_effective":1000,"gid_saved_set":1000,"gid_fs":1000}},"cgroup":{"kind":"not-collected"}}]}

NDJSON Format

tracexec collect --format json-stream --output trace.ndjson -- env -C / ls

{"version":"0.17.0","generator":"tracexec_exporter_json","baseline":{"cwd":"/home/kxxt","env":{"HOME":"/home/kxxt","PATH":"/home/kxxt/.cargo/bin:/home/kxxt/mambaforge/bin:/home/kxxt/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/opt/cuda/bin:/home/kxxt/.local/share/flatpak/exports/bin:/var/lib/flatpak/exports/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/usr/lib/rustup/bin","PWD":"/home/kxxt","SHLVL":"0","_":"/usr/bin/tracexec"},"fdinfo":{"0":{"fd":0,"path":"/dev/pts/4","pos":0,"flags":["O_RDWR"],"mnt_id":39,"ino":7,"mnt":"39 36 0:26 / /dev/pts rw,nosuid,noexec,relatime shared:4 - devpts devpts rw,gid=5,mode=600,ptmxmode=000","extra":[]},"1":{"fd":0,"path":"pipe:[205677]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":205677,"mnt":"Not found. This is probably a pipe or something else.","extra":[]},"2":{"fd":0,"path":"pipe:[205677]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":205677,"mnt":"Not found. This is probably a pipe or something else.","extra":[]}}}}
{"id":1,"pid":20782,"syscall":"execve","exec_pid":20782,"cwd":"/home/kxxt","comm_before_exec":"tracer","result":0,"filename":"/usr/bin/env","argv":{"result":"success","value":["env","-C","/","ls"]},"env":{"result":"success","value":{"has_added_or_modified_keys_starting_with_dash":false,"added":{},"removed":[],"modified":{}}},"fdinfo":{"0":{"fd":0,"path":"/dev/pts/4","pos":0,"flags":["O_RDWR"],"mnt_id":39,"ino":7,"mnt":"39 36 0:26 / /dev/pts rw,nosuid,noexec,relatime shared:4 - devpts devpts rw,gid=5,mode=600,ptmxmode=000","extra":[]},"1":{"fd":0,"path":"pipe:[205677]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":205677,"mnt":"Not found. This is probably a pipe or something else.","extra":[]},"2":{"fd":0,"path":"pipe:[205677]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":205677,"mnt":"Not found. This is probably a pipe or something else.","extra":[]}},"timestamp":1787669675400640499,"cred":{"result":"success","value":{"groups":[209,936,953,962,991,992,1000],"uid_real":1000,"uid_effective":1000,"uid_saved_set":1000,"uid_fs":1000,"gid_real":1000,"gid_effective":1000,"gid_saved_set":1000,"gid_fs":1000}},"cgroup":{"kind":"not-collected"}}
{"id":2,"pid":20782,"syscall":"execve","exec_pid":20782,"cwd":"/","comm_before_exec":"env","result":-2,"filename":"/home/kxxt/.cargo/bin/ls","argv":{"result":"success","value":["ls"]},"env":{"result":"success","value":{"has_added_or_modified_keys_starting_with_dash":false,"added":{},"removed":[],"modified":{}}},"fdinfo":{"0":{"fd":0,"path":"/dev/pts/4","pos":0,"flags":["O_RDWR"],"mnt_id":39,"ino":7,"mnt":"39 36 0:26 / /dev/pts rw,nosuid,noexec,relatime shared:4 - devpts devpts rw,gid=5,mode=600,ptmxmode=000","extra":[]},"1":{"fd":0,"path":"pipe:[205677]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":205677,"mnt":"Not found. This is probably a pipe or something else.","extra":[]},"2":{"fd":0,"path":"pipe:[205677]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":205677,"mnt":"Not found. This is probably a pipe or something else.","extra":[]}},"timestamp":1787669675402808849,"cred":{"result":"success","value":{"groups":[209,936,953,962,991,992,1000],"uid_real":1000,"uid_effective":1000,"uid_saved_set":1000,"uid_fs":1000,"gid_real":1000,"gid_effective":1000,"gid_saved_set":1000,"gid_fs":1000}},"cgroup":{"kind":"not-collected"}}
{"id":3,"pid":20782,"syscall":"execve","exec_pid":20782,"cwd":"/","comm_before_exec":"env","result":-2,"filename":"/home/kxxt/mambaforge/bin/ls","argv":{"result":"success","value":["ls"]},"env":{"result":"success","value":{"has_added_or_modified_keys_starting_with_dash":false,"added":{},"removed":[],"modified":{}}},"fdinfo":{"0":{"fd":0,"path":"/dev/pts/4","pos":0,"flags":["O_RDWR"],"mnt_id":39,"ino":7,"mnt":"39 36 0:26 / /dev/pts rw,nosuid,noexec,relatime shared:4 - devpts devpts rw,gid=5,mode=600,ptmxmode=000","extra":[]},"1":{"fd":0,"path":"pipe:[205677]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":205677,"mnt":"Not found. This is probably a pipe or something else.","extra":[]},"2":{"fd":0,"path":"pipe:[205677]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":205677,"mnt":"Not found. This is probably a pipe or something else.","extra":[]}},"timestamp":1787669675404058242,"cred":{"result":"success","value":{"groups":[209,936,953,962,991,992,1000],"uid_real":1000,"uid_effective":1000,"uid_saved_set":1000,"uid_fs":1000,"gid_real":1000,"gid_effective":1000,"gid_saved_set":1000,"gid_fs":1000}},"cgroup":{"kind":"not-collected"}}
{"id":4,"pid":20782,"syscall":"execve","exec_pid":20782,"cwd":"/","comm_before_exec":"env","result":-2,"filename":"/home/kxxt/.nix-profile/bin/ls","argv":{"result":"success","value":["ls"]},"env":{"result":"success","value":{"has_added_or_modified_keys_starting_with_dash":false,"added":{},"removed":[],"modified":{}}},"fdinfo":{"0":{"fd":0,"path":"/dev/pts/4","pos":0,"flags":["O_RDWR"],"mnt_id":39,"ino":7,"mnt":"39 36 0:26 / /dev/pts rw,nosuid,noexec,relatime shared:4 - devpts devpts rw,gid=5,mode=600,ptmxmode=000","extra":[]},"1":{"fd":0,"path":"pipe:[205677]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":205677,"mnt":"Not found. This is probably a pipe or something else.","extra":[]},"2":{"fd":0,"path":"pipe:[205677]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":205677,"mnt":"Not found. This is probably a pipe or something else.","extra":[]}},"timestamp":1787669675405266417,"cred":{"result":"success","value":{"groups":[209,936,953,962,991,992,1000],"uid_real":1000,"uid_effective":1000,"uid_saved_set":1000,"uid_fs":1000,"gid_real":1000,"gid_effective":1000,"gid_saved_set":1000,"gid_fs":1000}},"cgroup":{"kind":"not-collected"}}
{"id":5,"pid":20782,"syscall":"execve","exec_pid":20782,"cwd":"/","comm_before_exec":"env","result":-2,"filename":"/nix/var/nix/profiles/default/bin/ls","argv":{"result":"success","value":["ls"]},"env":{"result":"success","value":{"has_added_or_modified_keys_starting_with_dash":false,"added":{},"removed":[],"modified":{}}},"fdinfo":{"0":{"fd":0,"path":"/dev/pts/4","pos":0,"flags":["O_RDWR"],"mnt_id":39,"ino":7,"mnt":"39 36 0:26 / /dev/pts rw,nosuid,noexec,relatime shared:4 - devpts devpts rw,gid=5,mode=600,ptmxmode=000","extra":[]},"1":{"fd":0,"path":"pipe:[205677]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":205677,"mnt":"Not found. This is probably a pipe or something else.","extra":[]},"2":{"fd":0,"path":"pipe:[205677]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":205677,"mnt":"Not found. This is probably a pipe or something else.","extra":[]}},"timestamp":1787669675406620724,"cred":{"result":"success","value":{"groups":[209,936,953,962,991,992,1000],"uid_real":1000,"uid_effective":1000,"uid_saved_set":1000,"uid_fs":1000,"gid_real":1000,"gid_effective":1000,"gid_saved_set":1000,"gid_fs":1000}},"cgroup":{"kind":"not-collected"}}
{"id":6,"pid":20782,"syscall":"execve","exec_pid":20782,"cwd":"/","comm_before_exec":"env","result":-2,"filename":"/usr/local/sbin/ls","argv":{"result":"success","value":["ls"]},"env":{"result":"success","value":{"has_added_or_modified_keys_starting_with_dash":false,"added":{},"removed":[],"modified":{}}},"fdinfo":{"0":{"fd":0,"path":"/dev/pts/4","pos":0,"flags":["O_RDWR"],"mnt_id":39,"ino":7,"mnt":"39 36 0:26 / /dev/pts rw,nosuid,noexec,relatime shared:4 - devpts devpts rw,gid=5,mode=600,ptmxmode=000","extra":[]},"1":{"fd":0,"path":"pipe:[205677]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":205677,"mnt":"Not found. This is probably a pipe or something else.","extra":[]},"2":{"fd":0,"path":"pipe:[205677]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":205677,"mnt":"Not found. This is probably a pipe or something else.","extra":[]}},"timestamp":1787669675408335310,"cred":{"result":"success","value":{"groups":[209,936,953,962,991,992,1000],"uid_real":1000,"uid_effective":1000,"uid_saved_set":1000,"uid_fs":1000,"gid_real":1000,"gid_effective":1000,"gid_saved_set":1000,"gid_fs":1000}},"cgroup":{"kind":"not-collected"}}
{"id":7,"pid":20782,"syscall":"execve","exec_pid":20782,"cwd":"/","comm_before_exec":"env","result":-2,"filename":"/usr/local/bin/ls","argv":{"result":"success","value":["ls"]},"env":{"result":"success","value":{"has_added_or_modified_keys_starting_with_dash":false,"added":{},"removed":[],"modified":{}}},"fdinfo":{"0":{"fd":0,"path":"/dev/pts/4","pos":0,"flags":["O_RDWR"],"mnt_id":39,"ino":7,"mnt":"39 36 0:26 / /dev/pts rw,nosuid,noexec,relatime shared:4 - devpts devpts rw,gid=5,mode=600,ptmxmode=000","extra":[]},"1":{"fd":0,"path":"pipe:[205677]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":205677,"mnt":"Not found. This is probably a pipe or something else.","extra":[]},"2":{"fd":0,"path":"pipe:[205677]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":205677,"mnt":"Not found. This is probably a pipe or something else.","extra":[]}},"timestamp":1787669675409465881,"cred":{"result":"success","value":{"groups":[209,936,953,962,991,992,1000],"uid_real":1000,"uid_effective":1000,"uid_saved_set":1000,"uid_fs":1000,"gid_real":1000,"gid_effective":1000,"gid_saved_set":1000,"gid_fs":1000}},"cgroup":{"kind":"not-collected"}}
{"id":8,"pid":20782,"syscall":"execve","exec_pid":20782,"cwd":"/","comm_before_exec":"env","result":0,"filename":"/usr/bin/ls","argv":{"result":"success","value":["ls"]},"env":{"result":"success","value":{"has_added_or_modified_keys_starting_with_dash":false,"added":{},"removed":[],"modified":{}}},"fdinfo":{"0":{"fd":0,"path":"/dev/pts/4","pos":0,"flags":["O_RDWR"],"mnt_id":39,"ino":7,"mnt":"39 36 0:26 / /dev/pts rw,nosuid,noexec,relatime shared:4 - devpts devpts rw,gid=5,mode=600,ptmxmode=000","extra":[]},"1":{"fd":0,"path":"pipe:[205677]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":205677,"mnt":"Not found. This is probably a pipe or something else.","extra":[]},"2":{"fd":0,"path":"pipe:[205677]","pos":0,"flags":["O_WRONLY"],"mnt_id":16,"ino":205677,"mnt":"Not found. This is probably a pipe or something else.","extra":[]}},"timestamp":1787669675411122028,"cred":{"result":"success","value":{"groups":[209,936,953,962,991,992,1000],"uid_real":1000,"uid_effective":1000,"uid_saved_set":1000,"uid_fs":1000,"gid_real":1000,"gid_effective":1000,"gid_saved_set":1000,"gid_fs":1000}},"cgroup":{"kind":"not-collected"}}

Perfetto Trace Export

The Perfetto exporter turns an exec trace into a timeline that you can explore in the Perfetto UI. It is useful when you want to see how a build or shell script runs: which programs it starts, how long they last and which ones run in parallel.

Collecting a Trace

Use --format perfetto and give the output file a name:

tracexec collect --format perfetto --output trace.pftrace -- bash -c 'sleep 1 & sleep 2 & wait'

This example starts two sleep processes in parallel and waits for both to finish. The trace should show a shell lasting about two seconds, with two child slices lasting about one and two seconds.

Replace the command after -- with the program you want to trace. For example, to collect a parallel build:

tracexec collect --format perfetto -o build.pftrace -- make -j4

-o is the short form of --output. The trace is a binary file, so use an output file to keep it separate from the traced program’s terminal output. Wait for collection to finish before opening it.

The exporter also works with the eBPF backend:

tracexec --elevate ebpf collect --format perfetto -o build.pftrace -- make -j4

Warning

The trace file may contain sensitive credentials that are passed in commandline arguments or environment variables. Sharing the trace file may leak such credentials.

Opening the Trace

Open ui.perfetto.dev and choose Open trace file, or drag your .pftrace file into the page.

• Use W/S to zoom in/out and A/D to pan left/right.
• Click a slice to inspect it in the Current Selection panel.
• Press F to center the selected slice, then F again to fit it in the view.

See Perfetto’s UI guide for more navigation shortcuts.

Interpreting the Trace

Successful execs appear as slices, the horizontal bars in the timeline. The duration of slices represents wall time instead of CPU time. A slice starts at an exec event and ends when that program exits, is replaced by another successful exec in the same process, or is detached from tracexec. Its name comes from argv[0], falling back to the executable filename when the arguments are unavailable.

The tracks form a tree. When a process spawns a child that executes a program, the child’s slice appears on a track below its parent’s track. When the same process executes another program, the old slice ends and the new one starts on the same track.

tracexec reuses available child tracks to keep the view compact, so a row can contain different processes at different times. Check the selected slice’s pid argument when you need to identify a process.

Failed exec attempts appear as instant events instead of slices. For example, a program searching PATH may try several filenames before finding one that exists. Select an instant event and check filename and syscall_ret to see what failed. If you only want successful execs, add --successful-only when collecting the trace:

tracexec collect --format perfetto --successful-only -o build.pftrace -- make -j4

Inspecting an Event

Select a slice or instant event and expand its arguments in Current Selection. tracexec attaches the following information, when available:

To include cgroup information, add --collect-cgroup when recording.

Completed slices also carry end_reason. For example, exec means the process replaced itself with another process, exited means it exited with an exit code, and signaled means it was killed by a signal. exit_code or exit_signal provides the corresponding result when available. This lets you distinguish a successful exec followed by a program failure from an exec call that failed to start the program at all.

Example: Building tracexec

The following video uses tracexec to analyze its own build. After the build finishes, it shows the overall timeline and looks more closely at individual programs and their arguments.

To trace a Rust build in the same way, run this in the project’s directory:

tracexec collect --format perfetto -o build.pftrace -- cargo build

Filtering

In tracexec, we currently provide two mechanisms for filtering events.

Only Show Successful Execs

In many cases, you may only want to get a trace of successful exec events because the failure cases are not interesting. For example, many failure cases are just program trying every possible path in PATH environment variable until success.

To show only the successful exec events, use --successful-only option, as shown in the following example.

tracexec log --show-cmdline --successful-only -- env -u LANG A=B ls

Event filter

Under default settings, tracexec tries to output a sensible amount of details to avoid overwhelming the user.

We provide a custom event filtering system for advanced users where the default settings fall short.

tracexec produces the following types of events:

• info, warning, error: a notification to the user.
• new-child: a new traced child process is observed.
• exec: exec event.
• tracee-spawn: the root tracee spawns.
• tracee-exit: the root tracee terminates.

Please note that not all frontends display all the event types. Thus certain event types may not show in some frontends even if enabled in the filter.

By default, the filter enables warning,error,exec,tracee-exit events.

To enable additional events in the filter, use --filter-include. For example, tracexec tui --filter-include tracee-spawn -- ls shows tracee-spawn event in addition to default events.

To disable events in the filter, use --filter-exclude. For example, with tracexec tui --filter-exclude tracee-exit -- ls, the tracee-exit event is hidden.

Alternatively, instead of manipulating the default filter with --filter-include and --filter-exclude, you can also set the default filter directly with --filter. For example, tracexec tui --filter exec,error -- ... will only show exec and error events.

Convenient Privilege Elevation

When using tracexec with eBPF backend or tracing setuid/setgid binaries with ptrace backend, it usually requires running tracexec as root. However, using sudo with tracexec is a little tricky because sudo manipulates the environment variables, which might not be noticed by the user.

For example, when running sudo tracexec ebpf log -- make -j$(nproc),

• sudo resets the environment variables for tracexec and the tracee by retaining a minimal set of basic environment variables and may override some important variables for security reasons (e.g. PATH).
• sudo inserts its own environment variables like SUDO_USER, SUDO_UID and SUDO_COMMAND.
• The tracee make is ran as root, which may not be desired.

In many cases, what we want to achieve is to run tracexec with root privilege but still run the tracee in the original context as an unprivileged user. The following command almost achieves it, with the caveat that sudo -E still modifies the environment variables.

sudo -E tracexec --user $(whoami) ebpf log -- make -j$(nproc)

Starting at tracexec 1.0, we offer a new CLI flag that conveniently runs tracexec as root but runs tracee with the original user and environment variables.

For example, the following command runs tracexec as root but runs make -j$(nproc) as the original user:

tracexec --elevate ebpf log -- make -j$(nproc)

When using this feature, tracexec will internally use sudo for privilege elevation. So sudo needs to be installed on your system and you may need to authenticate yourself to sudo when tracexec executes sudo.

Experimental Features

Tutorials

Debugging a basic build problem

Sometimes a build fails even though you are sure you passed the right options. In this tutorial, we will debug a small C project that cannot find its header file, despite being given an include path.

We will use tracexec to find out what the build actually passed to the compiler, then fix the problem and run the program. You will need tracexec, GNU Make and a C compiler available as cc.

Here is a recording of the investigation and the fix:

The Example Project

Create a directory for the example:

mkdir -p basic-build-problem/include
cd basic-build-problem

It contains just three files:

basic-build-problem/
├── Makefile
├── main.c
└── include/
    └── greeting.h

The complete source is below. You can also find these files in book/tutorials/basic-build-problem in the tracexec repository.

main.c:

#include <stdio.h>
#include "greeting.h"

int main(void)
{
    puts(GREETING);
    return 0;
}

include/greeting.h:

#ifndef GREETING_H
#define GREETING_H

#define GREETING "Hello from the build tutorial!"

#endif

Makefile:

CFLAGS ?= -Wall -Wextra

.PHONY: all clean

all: hello

hello: main.c include/greeting.h
	@$(CC) $(CFLAGS) main.c -o $@

clean:
	$(RM) hello

The recipe lines must start with a tab. The @ before the compiler command tells make not to print that command, so we will only see the compiler’s output when we build. This Makefile has a small mistake that we will fix below.

Reproducing the Failure

The header is in include, so let’s pass -Iinclude through CPPFLAGS:

CPPFLAGS=-Iinclude make

With GCC, the output looks like this:

main.c:2:10: fatal error: greeting.h: No such file or directory
    2 | #include "greeting.h"
      |          ^~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:8: hello] Error 1

Clang reports the same missing header with slightly different wording. The file exists and we supplied its directory. Did that option reach the compiler?

Looking at the Compiler Invocation

Trace another build:

tracexec tui -- env CPPFLAGS=-Iinclude make

The env command sets CPPFLAGS for make and its children. The compiler error will appear in the terminal pane, while the events pane shows the programs involved in the build.

Switch to the Events pane with Ctrl+S. Select the compiler invocation with ↑/↓ and press V to open its details. In the recording, this is /usr/bin/cc, just after make. If your compiler is GCC, you may also see a later cc1 event; start with the cc invocation that make launched.

In the Info tab, press End to scroll down to Argv. The arguments look like this:

["cc", "-Wall", "-Wextra", "main.c", "-o", "hello"]

The first argument may be a full path on your system. The useful clue is that -Iinclude is missing. The compiler was never told to search our header directory.

Now press Tab to switch to the Environment tab. You should find:

+"CPPFLAGS"="-Iinclude"

The + means that the variable was added relative to tracexec’s starting environment. If you already had CPPFLAGS set before starting tracexec, it may appear as modified or unchanged instead.

So the environment variable reached the compiler, but its value did not appear in the arguments. CPPFLAGS is a convention used by build tools: setting it does not add compiler options by itself. The build recipe needs to pass those options on.

Fixing the Makefile

Press Q to close the details, then Q again to leave tracexec. Look at the compiler command in the Makefile:

	@$(CC) $(CFLAGS) main.c -o $@

It uses CFLAGS but leaves out CPPFLAGS. Change it to:

	@$(CC) $(CPPFLAGS) $(CFLAGS) main.c -o $@

If you are following along from the terminal, this is the edit shown in the recording:

sed -i 's/$(CC) $(CFLAGS)/$(CC) $(CPPFLAGS) $(CFLAGS)/' Makefile

Now build and run it:

CPPFLAGS=-Iinclude make && ./hello

Hello from the build tutorial!

The compiler now receives -Iinclude as an argument and finds greeting.h. There is no need to clean before this retry because the failed build did not produce hello. If you want to trace the successful compiler invocation too, repeat the tracing command with make -B at the end to force a rebuild.

This example is small enough to spot the mistake by reading the Makefile. In a larger build, tracexec lets you make the same check even when the compiler is launched through several scripts or nested make invocations: find the exec event, inspect its arguments, then check its environment and working directory in Event Details.

Use tracexec as debugger launcher

Without tracexec, it’s not trivial or convenient to debug a program that gets executed by other programs or debug programs with pipes:

• https://stackoverflow.com/questions/5048112/use-gdb-to-debug-a-c-program-called-from-a-shell-script
• https://stackoverflow.com/questions/1456253/gdb-debugging-with-pipe
• https://stackoverflow.com/questions/455544/how-to-load-program-reading-stdin-and-taking-parameters-in-gdb
• https://ftp.gnu.org/old-gnu/Manuals/gdb/html_node/gdb_25.html
• https://stackoverflow.com/questions/65936457/debugging-a-specific-subprocess
• https://sourceware.org/gdb/current/onlinedocs/gdb.html/Forks.html

This example demonstrates how to use tracexec as a gdb launcher to debug programs under complex setup. The following video demonstrates the whole process:

(Note: in the video the -t parameter is used, which has been removed in 1.0 release and no longer needed for this tutorial)

To run this example, first ensure that tracexec and rust is installed on your system.

Clone the tracexec repository and enter the directory for this example:

git clone https://github.com/kxxt/tracexec
cd tracexec/book/tutorials/debugger-launcher

Then run make to compile the two simple rust programs.

In order to allow gdb to attach to the detached and stopped tracees, you probably need to run:

echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope

On a machine with Wayland/X11 display, assuming you have konsole installed(if not, please change the default-external-command), run

tracexec tui \
    -b sysexit:in-filename:/a \
    -b sysexit:in-filename:/b \
    --default-external-command "konsole -e gdb -ex cont -ex cont -p {{PID}}" \
    -- ./shell-script

or on a headless server, inside a tmux session, run:

tracexec tui \
    -b sysexit:in-filename:/a \
    -b sysexit:in-filename:/b \
    --default-external-command "tmux split-window 'gdb -ex cont -ex cont -p {{PID}}'" \
    -- ./shell-script

Alternatively, launch tracexec tui with a bash session and set the breakpoints in the TUI then run ./shell-script in it.

When the breakpoint get hit, open the Hit Manager and launch the external command for the two stopped tracees. Then two gdb session will open.

To restart the tracees in gdb, Send command c twice.

Catching FD leaks

When spawning a subprocess using the fork/exec family syscalls, file descriptors can be leaked to the subprocesses if the code does not mark them as O_CLOEXEC or close them after fork. FD leaks can cause bugs or even security vulnerabilities.

In this tutorial, we will write a small launcher that accidentally passes its log file to a worker, find the descriptor with tracexec and fix the leak. You will need Linux, tracexec and a C compiler available as cc. The worker is the external printf program, which should be available in your PATH.

Here is a recording of the example, the investigation and the fix:

The Example Program

Our launcher opens launcher.log, writes a message and forks a child to run printf. The parent closes the log and waits for the worker to finish. That sounds reasonable, but there is a missing piece.

Create a directory for the example:

mkdir fd-leaks
cd fd-leaks

Save the following as launcher.c. The complete source is also available in book/tutorials/fd-leaks in the tracexec repository.

#define _POSIX_C_SOURCE 200809L

#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <sys/wait.h>
#include <unistd.h>

int main(void)
{
    int log_fd = open("launcher.log", O_WRONLY | O_CREAT | O_APPEND, 0600);
    if (log_fd == -1) {
        perror("open");
        return 1;
    }

    if (dprintf(log_fd, "Starting worker\n") < 0) {
        perror("write log");
        close(log_fd);
        return 1;
    }

    pid_t child = fork();
    if (child == -1) {
        perror("fork");
        close(log_fd);
        return 1;
    }

    if (child == 0) {
        execlp("printf", "printf", "%s\n", "Worker finished", (char *)NULL);
        perror("exec printf");
        _exit(127);
    }

    /* The parent no longer needs the log. */
    int close_failed = close(log_fd) == -1;
    if (close_failed) {
        perror("close log");
    }

    int status;
    while (waitpid(child, &status, 0) == -1) {
        if (errno != EINTR) {
            perror("waitpid");
            return 1;
        }
    }

    if (close_failed || !WIFEXITED(status)) {
        return 1;
    }
    return WEXITSTATUS(status);
}

Compile and run it:

cc -std=c11 -Wall -Wextra -o launcher launcher.c
./launcher

Worker finished

It also appends Starting worker to launcher.log. There is no error message, and the program exits successfully. The problem is what the worker inherited along the way.

Finding the Leaked Descriptor

Trace the launcher:

tracexec tui -- ./launcher

You should see an exec event for launcher, followed by one for printf. Switch to the Events pane with Ctrl+S, select the successful printf event and press V to open its details. Depending on your PATH, there may be failed attempts to find printf before the successful one.

Press → twice to switch to FdInfo, then End to scroll to the bottom. Alongside stdin, stdout and stderr, you will find another descriptor pointing to launcher.log. It is descriptor 3 in the recording, but the exact number can differ.

The entry shows the full path and flags such as O_WRONLY and O_APPEND. This is the launcher’s log, yet it appears in the worker’s exec event. printf has no reason to use this file descriptor thus it is a leak.

Fixing the Leak

Press Q to close the details, then Q again to leave tracexec. Add O_CLOEXEC to the flags passed to open:

int log_fd = open("launcher.log", O_WRONLY | O_CREAT | O_APPEND | O_CLOEXEC, 0600);

This marks the descriptor to be closed automatically when the child successfully executes the worker. It can still be used to write the log before exec.

The recording makes this edit with:

sed -i 's/O_APPEND,/O_APPEND | O_CLOEXEC,/' launcher.c

Recompile and trace the program again:

cc -std=c11 -Wall -Wextra -o launcher launcher.c
tracexec tui -- ./launcher

Open the successful printf event and check FdInfo again. The launcher.log entry is gone and thus the leak is solved.

Comparison with other tools

There are many existing tools for tracing exec syscalls when I start to create tracexec. However, none of them suit my use case well.

This chapter provides a comparison between tracexec and those tools. I hope this chapter will provide readers the knowledge about choosing the best tool for their use cases.

We can roughly divide the tools into three categories by how exec tracing is implemented. (Tracexec supports multiple ways for tracing exec)

Comparison with execsnoop(bcc)

This article compares tracexec with the latest commit of execsnoop at the time of writing. Feel free to improve it if you found anything outdated.

There are two execsnoop implementations in bcc, one implemented with Python, another one implemented with libbpf. Here we will compare with the Python implementation as it supports more features at the time of writing.

Shortcomings of execsnoop(bcc)

Default Limits are Too Limited

By default, execsnoop can only trace up to 20 arguments per exec event, which is too limited to trace complex compiler invocations by various build systems. It can be raised using --max-args argument.

And execsnoop hardcodes a very low limit(128) for the length of each argument, if any argument exceeds this limit, it is silently truncated, resulting in wrong output without any notification to the user.

Cannot Show ARGV[0]

execsnoop shows filename in the place of the first argument(argv[0]) and discards the real argv[0]. Most of the time this is not important because argv[0] is the filename or the basename of the filename.

However, sometimes argv[0] and filename are different and this difference plays an important role on how the program behaves. For example, multi-call binaries like busybox can act as different commands depending on argv[0].

Cannot Show Environment Variables

Sometimes, environment variables play a vital role in program execution. execsnoop doesn’t show them at all.

Cannot Copy-Paste-Execute

A handy feature of tracexec is to copy the shell escaped command line to clipboard, which you can directly paste into another terminal and hit enter to execute it.

But as for execsnoop. It doesn’t even quote the arguments by default, making it hard to distinguish the boundary between arguments. Even if -q/--quote is used, there is still a long way to copy-paste-execute because it does not perform shell escaping. Even if it performs shell-escaping in the future. Without the environment variables, the command may also not work.

Missing features in tracexec compared with execsnoop(bcc)

execsnoop supports tracing processes under a cgroups path and limit tracing to a specific UID.

Comparison with execsnoop(bpftrace)

bpftrace is a high-level tracing language that compiles to eBPF. An execsnoop.bt script is shipped with this package on many Linux distributions (for example, /usr/share/bpftrace/tools/execsnoop.bt on Arch Linux).

This article compares tracexec with the latest commit 93b3247 of execsnoop.bt at the time of writing. Feel free to improve it if you found anything outdated.

Shortcomings of execsnoop.bt

Missing exec result

The script is only monitoring syscall entry and thus unable to report whether or not the execs are successful.

Missing details

The script is minimalistic and cannot show the filename, environment variables and the inherited file descriptors.

Cannot Copy-Paste-Execute

A handy feature of tracexec is to copy the shell escaped command line to clipboard, which you can directly paste into another terminal and hit enter to execute it.

But as for execsnoop.bt. It doesn’t even quote the arguments, making it hard to distinguish the boundary between arguments.

Dependency Bloat

Although execsnoop.bt is minimalistic, the dependencies are not. It depends on bpftrace, which in turn depends on both clang and bcc, where the latter already includes their own implementation of execsnoop.

Comparison with strace

strace is a generic syscall tracing tool that could be used for tracing exec. This article will compare tracexec with the latest version (6.19) of strace at the time of writing. Feel free to improve it if you found anything outdated.

Shortcomings of strace

Missing a Sane Verbosity Level

To trace exec, the most simple strace command that comes to my mind is:

Default Verbosity

strace -e trace=execveat,execve -f -- bash

This produces a noisy log with lots of unrelated content that makes it hard to find the exec events:

[pid 522056] +++ exited with 0 +++
[pid 522051] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=522056, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
[pid 522055] +++ exited with 0 +++
[pid 522051] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=522055, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
[pid 522059] +++ exited with 0 +++
[pid 522058] +++ exited with 0 +++
strace: Process 522061 attached
strace: Process 522062 attached
strace: Process 522063 attached
strace: Process 522064 attached

While for actual exec events, it is not verbose because the environment variables are hidden.

[pid 522055] execve("/usr/bin/ip", ["/usr/bin/ip", "netns", "identify"], 0x7ffe989a3dc0 /* 110 vars */ <unfinished ...>
[Five lines omitted]
[pid 522055] <... execve resumed>)      = 0

Quiet

If we use -q/--quiet, that still does not fix the noisy log problem.

strace -e trace=execveat,execve -f -q -- bash

But at least it makes logs like strace: Process 522061 attached go away.

Verbose

What if we want to know the environment variables? We need to increase verbosity:

strace -e trace=execveat,execve -f -q -- bash

We still have a very noisy log but we could see the environment variables. (Well, at least the env variable names…)

[pid 571872] execve("/usr/bin/ip", ["/usr/bin/ip", "netns", "identify"], ["SHELL=/usr/bin/zsh", "SESSION_MANAGER=local/ryzen:@/tm"..., "USER_ZDOTDIR=/home/kxxt", "COLORTERM=truecolor", "XDG_CONFIG_DIRS=/home/kxxt/.conf"..., "VSCODE_DEBUGPY_ADAPTER_ENDPOINTS"..., "XDG_SESSION_PATH=/org/freedeskto"..., "XDG_MENU_PREFIX=plasma-", "TERM_PROGRAM_VERSION=1.112.01907", "ICEAUTHORITY=/run/user/1000/icea"..., "LC_ADDRESS=en_US.UTF-8", "USE_CCACHE=1", "LC_NAME=en_US.UTF-8", "SSH_AUTH_SOCK=/run/user/1000/gnu"..., "MEMORY_PRESSURE_WRITE=c29tZSAyMD"..., "PYDEVD_DISABLE_FILE_VALIDATION=1", "DESKTOP_SESSION=plasma", "LC_MONETARY=en_US.UTF-8", "__ETC_PROFILE_NIX_SOURCED=1", "GTK_RC_FILES=/etc/gtk/gtkrc:/hom"..., "NO_AT_BRIDGE=1", "EDITOR=nvim", "XDG_SEAT=seat0", "PWD=/home/kxxt/repos/tracexec", "NIX_PROFILES=/nix/var/nix/profil"..., "LOGNAME=kxxt", "XDG_SESSION_DESKTOP=KDE", "XDG_SESSION_TYPE=wayland", "SYSTEMD_EXEC_PID=2534", "BUNDLED_DEBUGPY_PATH=/home/kxxt/"..., "XAUTHORITY=/run/user/1000/xauth_"..., "VSCODE_GIT_ASKPASS_NODE=/usr/sha"..., "MOTD_SHOWN=pam", "VSCODE_INJECTION=1", "GTK2_RC_FILES=/etc/gtk-2.0/gtkrc"..., "HOME=/home/kxxt", "MCFLY_HISTORY=/tmp/mcfly.wGDRsBB"..., "SSH_ASKPASS=/run/user/1000/gnupg"..., "MCFLY_FUZZY=true", "LANG=en_US.UTF-8", "LC_PAPER=en_US.UTF-8", "MCFLY_HISTFILE=/home/kxxt/.zhist"..., "_JAVA_AWT_WM_NONREPARENTING=1", "XDG_CURRENT_DESKTOP=KDE", "PYTHONSTARTUP=/home/kxxt/.config"..., "MEMORY_PRESSURE_WATCH=/sys/fs/cg"..., "STARSHIP_SHELL=bash", "WAYLAND_DISPLAY=wayland-0", "__MISE_DIFF=eAFrXpyfk9KwOC+1vGFJ"..., "NIX_SSL_CERT_FILE=/etc/ssl/certs"..., "GIT_ASKPASS=/usr/share/vscodium/"..., "XDG_SEAT_PATH=/org/freedesktop/D"..., "INVOCATION_ID=3175cd2e73284f8aab"..., "MANAGERPID=2130", "MCFLY_SESSION_ID=ONlRzDF6foVRPQ7"..., "CHROME_DESKTOP=codium.desktop", "STARSHIP_SESSION_KEY=82575444194"..., "__MISE_ORIG_PATH=/home/kxxt/.car"..., "KDE_SESSION_UID=1000", "VSCODE_GIT_ASKPASS_EXTRA_ARGS=", "VSCODE_PYTHON_AUTOACTIVATE_GUARD"..., "XDG_SESSION_CLASS=user", "ANDROID_HOME=/opt/android-sdk", "TERM=xterm-256color", "LC_IDENTIFICATION=en_US.UTF-8", "PYTHON_BASIC_REPL=1", "__MISE_ZSH_PRECMD_RUN=1", "MCFLY_RESULTS_SORT=LAST_RUN", "ZDOTDIR=/home/kxxt", "USER=kxxt", "VSCODE_GIT_IPC_HANDLE=/run/user/"..., "CUDA_PATH=/opt/cuda", "QT_WAYLAND_RECONNECT=1", "KDE_SESSION_VERSION=6", "PAM_KWALLET5_LOGIN=/run/user/100"..., "__MISE_SESSION=eAHqWpOTn5iSmhJfk"..., "MCFLY_HISTORY_FORMAT=zsh", "MCFLY_RESULTS=20", "DISPLAY=:0", "SHLVL=3", "LC_TELEPHONE=en_US.UTF-8", "ANDROID_SDK_ROOT=/opt/android-sd"..., "CCACHE_EXEC=/usr/bin/ccache", "LC_MESSAGES=en_US.UTF-8", "LC_MEASUREMENT=en_US.UTF-8", "XDG_VTNR=2", "XDG_SESSION_ID=2", "MANAGERPIDFDID=2131", "CUDA_DISABLE_PERF_BOOST=1", "FC_FONTATIONS=1", "XDG_RUNTIME_DIR=/run/user/1000", "DEBUGINFOD_URLS=https://debuginf"..., "NVCC_CCBIN=/usr/bin/g++", "MCFLY_INTERFACE_VIEW=BOTTOM", "LC_TIME=en_US.UTF-8", "VSCODE_GIT_ASKPASS_MAIN=/usr/sha"..., "JOURNAL_STREAM=9:44333", "MISE_SHELL=bash", "XDG_DATA_DIRS=/home/kxxt/.local/"..., "GDK_BACKEND=wayland", "KDE_FULL_SESSION=true", "PATH=/home/kxxt/.local/share/mis"..., "DBUS_SESSION_BUS_ADDRESS=unix:pa"..., "KDE_APPLICATIONS_AS_SCOPE=1", "HG=/usr/bin/hg", "MAIL=/var/spool/mail/kxxt", "LC_NUMERIC=en_US.UTF-8", "OLDPWD=/home/kxxt/repos/tracexec", "TERM_PROGRAM=vscode", "_=/usr/bin/starship"]) = 0

Many variables are truncated because it exceeds the string length limit.

Showing the Full Environment Variables

To show the full environment variables, increase the string length limit with -s/--string-limit:

strace -e trace=execveat,execve -f -v -s99999 -- bash

Finally Reaching A Sane Verbosity

To silence all other noisy logs while logging all environment variables, we could use:

strace -e trace=execveat,execve -vqqq -e 'signal=!all' -f -s99999 -- bash

But that command line has become too long to type and remember. With tracexec, it is much easier to remember:

tracexec log --show-env -- bash

Cannot Diff Environment Variables

In the previous shortcoming, we can see that strace could show all the environment variables used in exec. However, showing all the environment variables is too verbose. Most of the time we are only interested in the diff of environment variables. Or to put it in another way, what environment are added and which are modified or removed.

strace has no support for doing that but tracexec by default shows diff of environment variables:

tracexec log -- bash

Cannot Copy-Paste-Execute

A handy feature of tracexec is to copy the shell escaped command line to clipboard, which you can directly paste into another terminal and hit enter to execute it.

But as for strace. It prints the arguments in an array syntax, making it impossible to directly copy and paste into shell.

Missing features in tracexec compared with strace

Tracing only a single process

strace supports tracing only a single process when -f/--follow-forks is not enabled.

In tracexec, we think this use case is too narrow to fit into a specialized exec tracing tool and didn’t implement it.

Stack trace

strace supports printing a stack trace at syscall with -k. We are working on supporting it in tracexec: https://github.com/kxxt/tracexec/issues/108.

FAQ

Developer Guide

This part of the book is for contributors changing tracexec itself.

Start with Internal Architecture for the crate layout, then read Backend Differences before changing code shared by ptrace and eBPF. Event System documents the messages passed from a backend to a frontend and the parent links used by the TUI.

The remaining chapters cover the practical work:

• Tests explains the normal, privileged, eBPF, and verifier test suites.
• Checklist for Cutting a Release lists the release steps.
• Maintaining this Book describes the local book workflow and media policy.

All crates in this workspace are implementation details. If code looks useful outside tracexec, discuss extracting a supported library before depending on an internal crate.

Internal Architecture

tracexec maintains several frontends and backends through a unified event system.

Crates

For modularity, tracexec consists of several crates. Generally speaking, most of them can be divided into two categories: frontend crates and backend crates.

Frontend crates handle presentation, while backend crates handle collection.

Currently, there are three dedicated frontend crates:

• tracexec-tui
• tracexec-exporter-json
• tracexec-exporter-perfetto

There is no separate crate for the log frontend; it lives in the tracexec-core crate.

And there are two backend crates:

• tracexec-backend-ptrace
• tracexec-backend-ebpf

Additionally, the tracexec-core crate consists of abstractions and primitives that are used throughout all the above crates.

The perfetto-trace-proto crate is an optional dependency for the tracexec-exporter-perfetto crate. We include a tiny perfetto trace protobuf binding minified by hand so perfetto-trace-proto is not used by default.

All the crates are internal implementation details even though they are published on crates.io. They shouldn’t be introduced as a dependency in other projects. If you want to reuse code from tracexec, open a discussion. We may separate the reusable parts into a supported crate.

Event System

The event system receives records from a backend and routes them to the selected frontend. TracerMessage is the channel payload. The ptrace TUI also sends PendingRequest values in the other direction for ptrace control, including breakpoint actions, seccomp-BPF suspension, and tracer termination.

See Event System for the message variants, filtering, and exec parent relationships.

Frontend Architecture

There is currently no common abstraction for frontends because the TUI, log mode, and exporters have different lifecycle and interaction requirements.

The Exporter trait covers the narrower case of converting an event stream to a structured output format.

Backend Architecture

There is no unified abstraction for backends. TracerBuilder configures the properties shared by multiple backends, but each backend owns its build and run path.

See Backend Differences before changing shared collection behavior.

Backend Differences

The ptrace and eBPF backends produce the same core event types, but they do not observe the kernel from the same place. Code shared between them must account for differences in scope, timing, data quality, and process control.

Scope and lifecycle

The ptrace backend starts a root tracee and follows forks, clones, and execs in that tree. Its completion condition is tied to that root command. The eBPF backend can do the same scoped filtering, but with no command it observes system-wide activity until interrupted.

Inspection timing

ptrace handles a syscall while the tracee is stopped. It can inspect registers and /proc/<pid> state at a defined syscall boundary. Even then, reads may fail because a process exited or procfs denied access.

eBPF programs copy data while running in kernel context. A userspace address may not be resident, and when the sleepable variant of the programs are not used, BPF helpers cannot resolve that by taking an ordinary page fault. Fields therefore use OutputMsg, Result, or another fallible wrapper. Preserve partial values instead of converting them into an empty string or empty collection.

Process and thread identity

Linux can execute a program from a non-leader thread. Exec collapses the thread group and may change the visible task ID. ExecEvent keeps both exec_pid (the task that entered exec) and pid (the process identity presented after the event). Backends must populate both according to the shared event contract. Consumers should not silently substitute one for the other.

Control path

Only the ptrace backend has a reverse control channel. RunningTracer sends PendingRequest values to resume or detach a breakpoint hit, suspend the seccomp optimization, or terminate the tracer. The TUI’s breakpoint and debugger features depend on that channel and must stay hidden in eBPF mode.

The seccomp optimization also affects detach behavior. A detached process retains the filter; without the tracer, a later exec can no longer be serviced. The UI warns users to start with --seccomp-bpf=off when a detached process needs to exec again.

Adding shared behavior

When adding a field or event:

1. define its meaning in backend-neutral terms;
2. implement and test collection in both backends, including failure cases;
3. decide whether an absent value, a partial value, and an inspection error need distinct representations;
4. check log, TUI, JSON, JSON-stream, and Perfetto consumers;
5. test scoped eBPF and system-wide eBPF separately when lifecycle matters.

If one backend cannot provide a field honestly, return an explicit unsupported or failed state. A plausible fabricated value is harder to debug than a marked gap.

Event System

Backends and frontends communicate through TracerMessage values on a Tokio unbounded MPSC channel. The channel keeps backend code independent from TUI, log, and exporter rendering while giving every frontend the same exec model.

ptrace backend ─┐
                ├─ TracerMessage channel ─┬─ log printer
eBPF backend ───┘                         ├─ TUI event list
                                          └─ JSON / Perfetto exporter

TUI ── PendingRequest channel ──> ptrace backend (ptrace control)

Message classes

TracerMessage has three variants:

• Event(TracerEvent) is a record that may get its own line in log or TUI output. Each record has a monotonically allocated EventId and a TracerEventDetails payload.
• StateUpdate(ProcessStateUpdateEvent) changes the state of existing records without adding a new line. Exit, breakpoint hit, resume, detach, and related errors use this path.
• FatalError(String) tells the frontend that the backend cannot continue.

Keeping state updates separate matters in the TUI. One process exit can update the status of several exec records for that process; rendering a second standalone line would lose that relationship.

Event payloads

TracerEventDetails currently represents:

• informational, warning, and error messages;
• discovery of a new child;
• an exec attempt and its inspected process state;
• root tracee spawn and exit lifecycle records.

ExecEvent is the main payload. It contains syscall kind and result, process identities, filename, argv, environment, working directory, credentials, interpreter chain, descriptor table, timestamp, cgroup information, and an optional parent event link.

Many fields are fallible. OutputMsg::PartialOk means some useful text was recovered but not all of it; OutputMsg::Err means the value could not be inspected. Compound fields use Result. New consumers should render or export those states, not collapse them into defaults.

Filtering

The TracerEventDetailsKind generated beside the details enum is used by --filter, --filter-include, and --filter-exclude. Backends call send_if_match before putting display events on the channel. The normal default is warning,error,exec,tracee-exit.

State updates are not ordinary display events and must not be dropped by that filter. A frontend may need an exit update even when it does not render exit events, otherwise running statuses never settle.

When adding a new display event, decide whether it belongs in the default filter. Also check help text, parser names, log formatting, TUI formatting, and tests for include/exclude combinations.

Exec parent links

An exec parent is an event relationship, not necessarily a Unix parent PID:

• ParentEvent::Spawn(id) means the process represented by id forked a child that later produced this exec event.
• ParentEvent::Become(id) means the same process represented by id replacing itself.

ParentTracker records the last successful exec. A failed exec can point to an ancestor, but it does not replace that last-successful value. The TUI uses these links for U parent navigation and the S/B markers in an exec backtrace.

Links are IDs rather than references so events can move through channels and be serialized cheaply.

Frontend consumption

Frontends consume different subsets:

• log formats display events immediately;
• the TUI stores display events and applies state updates to its event list;
• JSON exporters keep exec events and use root tracee exit to finish the file;
• Perfetto converts exec and lifecycle messages into trace packets.

Tests

tracexec currently contains two kinds of tests:

• the normal tests that are executed when running cargo test --workspace,
• tests requiring root that are excluded by default.

Running the Tests

To run the normal tests, use

cargo test --workspace

sudo is needed to run the tests that requires root:

CARGO_TARGET_<TARGET_TRIPLE>_RUNNER='sudo -E' cargo test --workspace -- --ignored

For example, if you are testing on a x86_64 linux machine, use

CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_RUNNER='sudo -E' cargo test --workspace -- --ignored

eBPF Verifier Complexity

Verifier complexity collection is available as an optional UKCI-style Nix runner. It boots the same UKCI kernels in QEMU, loads tracexec’s eBPF programs with verifier stats enabled, and writes one JSON file per kernel/LLVM combination:

nix run .#ukci-complexity

By default, results are written to verifier-complexity/. Set UKCI_COMPLEXITY_OUT_DIR to use a different output directory. This runner is intentionally separate from ukci and is not part of the required UKCI test pass.

For pull requests, a dedicated Nix workflow runs this collector on x86_64 when the compiled kernel-space eBPF sources or x86 BTF headers change. It uploads the raw JSON and rendered plots as Actions artifacts and updates a folded summary comment as soon as the complexity run finishes. The reporter also stores the PNG plots on ImgBB without an expiration and embeds them in the comment so they remain available after the artifacts expire. Configure the reporter with an IMGBB_API_KEY repository secret.

To plot the collected results from the repository root:

nix run .#plot-verifier-complexity -- verifier-complexity

The plotting script writes charts and a summary under verifier-complexity-plots/ by default. Use -o to choose another output directory, and --log-scale when comparing runs with large differences between the smallest and largest verifier counts.

Test Coverage

Most of the time you do not need to calculate the test coverage by yourself because we are tracking the test coverage continuously with CodeCov.

You will see the patch coverage and code coverage diff in a comment by CodeCov once you opened a pull request and all the tests pass.

Continue to read this section if you want to calculate the test coverage by yourself.

First, install cargo-llvm-cov if you haven’t already installed.

Then run the normal tests with coverage instrumentation to generate a coverage report named lcov.info:

cargo llvm-cov --all-features --workspace --lcov --output-path lcov.info

After that, run the root-only tests with coverage instrumentation. We use bpfcov-rs to collect coverage of eBPF code that executes in kernel-space.

export CARGO_TARGET_<TARGET_TRIPLE>_RUNNER='sudo -E env TRACEXEC_BPFCOV_OUTDIR=/tmp/bpfcov'
# Replace <TARGET_TRIPLE> with your rust target triple in uppercase and replace dash with underscore.
# For example:  export CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_RUNNER='sudo -E env TRACEXEC_BPFCOV_OUTDIR=/tmp/bpfcov'
cargo llvm-cov --all-features --workspace --lcov \
    --output-path root-lcov.info -- --ignored

After the tests finish,

• a user-space coverage report named root-lcov.info is produced,
• and kernel-space test coverage reports for each eBPF test is located in /tmp/bpfcov.

Then, combine all the kernel-space test coverage reports:

find /tmp/bpfcov -name '*.lcov' -print0 \
    | xargs -0 -I{} echo -a {} \
    | xargs lcov -o ebpf.lcov

And finally combine all three coverage reports into one:

lcov -a ebpf.lcov -a lcov.info -a root-lcov.info -o tracexec.info

Optionally you can generate an HTML report with:

genhtml tracexec.info --output-directory cov-out

Add a Test

Feel free to add new tests to cover new/modified code.

When adding a test that requires root, please mark it with

#![allow(unused)]
fn main() {
#[ignore = "root"]
}

When the test loads eBPF program, please make sure that it runs sequentially with respect to other eBPF tests by marking it with:

#![allow(unused)]
fn main() {
#[rstest]
#[file_serial(bpf)]
}

The outer rstest attribute is a workaround for getting the real test name.

Checklist for Cutting a Release

Before cutting a release, please check the following tasks.

Pre-release

• If there are changes to release pipeline in .github/workflows/release.yml, please create a pre-release to test such changes.

Documentation

• Document new features in this book.
• Update this book if some features are changed
• Update CHANGELOG.md to document
  • notable changes compared to previous stable release for a stable release or release candidate,
  • or notable changes compared to previous unstable release for an unstable alpha/beta release.
• If any CLI flags are changed/added, ensure that README.md is up-to-date by running just update-readme.

Chores

• Bump version with just bump <level>, where <level> is major, minor or patch.
• Ensure lockfile is updated after previous step.
• Commit the changes with the following commit message template: release: <VERSION>.
• Create a signed git tag named v<VERSION>.
• Push the commit and git tag to remote.
• After the release pipeline successfully finishes, edit the release in GitHub Releases to publish the draft release.

Maintaining this Book

Adding a Video

When adding a video to the book, please use the following HTML snippet. It ensures that the video is lazily loaded and has controls.

<video
  src="XXX" controls preload="none" loading="lazy"
  poster="../assets/gdb-launcher-cover.jpg"
  width="100%">
</video>

Please avoid storing videos inside the git repository unless it is below 5MiB. Currently, we post the videos in a GitHub discussion thread and then reference them by URL in the book.

Please add a cover image for the video by taking an image snapshot of the video at a suitable moment. This can be done by right clicking the video and select Take Snapshot in Firefox. The cover image should be stored in the git repository.