Introduction

Modern software systems are composed of complex chains of programs that invoke each other dynamically during execution. Shell scripts, build systems, and applications frequently spawn multiple subprocesses, each replacing itself through exec system calls. Under normal condition, the execution flow succeeds silently and few cares about it. However, when something breaks in the exec chains, many applications fail to precisely report what goes wrong and produce confusing or even misleading logs. Tracing and understanding the hidden execution flows thus are critical for debugging. Aside from that, tracing command execution is also a vital part of auditing program behavior and security monitoring.

tracexec is an exec tracer written in Rust that collects rich and accurate information and supports human-friendly output. It supports both ptrace-based backend and eBPF-based backend, where the ptrace-based backend is usually suitable for scoped tracing that does not involve setuid binaries while the eBPF-based backend could be use for system-wide tracing.

tracexec supports multiple user interfaces like logs, Terminal User Interface (TUI) and exporting structured traces.

Showcases

Installation

Different people have different opinions when it comes to how to install a program. Some may prefer using system package manager while others might like downloading a prebuilt binary. But don’t worry, tracexec supports a wide variety of installation methods.

Before installation, you should check the platform support status.

Install via Package Managers

tracexec is packaged in the following distributions.

Arch Linux (And Arch-based distributions)

tracexec is available in extra repository for Arch Linux. You can install it via

sudo pacman -S tracexec

Nix

To try tracexec without system-wide installation, running

nix-shell -p tracexec

will drop you into a shell where tracexec is available.

NixOS

If you are using NixOS, you should already have your preferred way to install packages.

e.g. by adding pkgs.tracexec to environment.systemPackages

Prebuilt Binaries

For stable versions, we release binaries in GitHub Releases.

Currently we offer two flavors of binaries

• Normal builds that dynamically links most dependencies except libbpf.
• Fully statically-linked builds which statically links all libraries including glibc.

Install from Source

Please refer to Building from Source for dependencies and feature flags.

To install the current stable version of tracexec. Run

cargo install tracexec --bin tracexec

To install the bleeding-edge development version of tracexec from git main branch. Run

cargo install --git https://github.com/kxxt/tracexec --bin tracexec

Platform Support

Currently tracexec only supports Linux operating system. Because the core of tracexec is implemented via ptrace, seccomp-bpf and eBPF, it is difficult to port to Windows, MacOS or other operating systems. (Well, technically speaking, ptrace itself is enough for initializing a port to other operating systems, but ptrace without seccomp-bpf is painfully slow.)

Architecture Support Status

Currently we support the following three architectures. You are welcome to submit PR for supporting more architectures.

*: for riscv64, some kernel versions has bugs in the ptrace implementation that would cause tracexec to display some information as errors. See this strace issue and the kernel mailing list discussion for more details if you got errors when using tracexec on riscv64.

Linux Kernel Support Status

LLVM Support Status

tracexec requires clang from LLVM for building the eBPF backend. We typically test the latest 3 versions of LLVM to ensure that the eBPF program compiled by them could be successfully loaded into the Linux kernels documented in Linux Kernel Support Status.

It is very likely that using other recent LLVM versions would work. If you encounter bugs with an LLVM version that is not covered in our CI, please open an issue and we are happy to help out.

Build from Source

To build tracexec from source, the following dependencies are needed:

• A working rust compiler and cargo.
  • Refer to package.rust-version in Cargo.toml for MSRV.
• libbpf: if not using vendored-libbpf
• zlib: if not using vendored-zlib
• libelf: if not using vendored-libbpf
• libseccomp: For seccomp-bpf.
• If any library vendoring feature is enabled:
  • build-essential autopoint gettext for Debian based distros
  • base-devel for Arch Linux
• protoc for compiling ProtoBuf proto files if protobuf-binding-from-source feature is enabled.
  • By default, protoc from PATH is used. PROTOC environment variable could be used to specify the full path to the desired protoc compiler.
• clang for compiling eBPF program.
  • By default, clang from PATH is used. CLANG environment variable could be used to specify the full path to the desired clang compiler.

Library Linkage

By default, we dynamically link to libseccomp because most distros ship it out of the box. In order to statically link to libseccomp, please set LIBSECCOMP_LINK_TYPE to static and set LIBSECCOMP_LIB_PATH to the path of the directory containing libseccomp.a.

To control whether or not to dynamically link to libbpf, libelf and zlib, consult the next Feature Flags section.

Feature Flags

• recommended: This enables the recommended functionalities of tracexec
  • ebpf(experimental): eBPF backend that doesn’t use ptrace and could be used for system wide tracing
• ebpf-debug: Not meant for end users. This flag enables debug logging to /sys/kernel/debug/tracing/trace_pipe and some debug checks.
• static: Statically link libelf, zlib and libbpf.
• vendored: Vendoring libelf, zlib and libbpf, implies static.
• vendored-libbpf: Vendoring libbpf and statically link to it.

By default, we enable the recommended and vendored-libbpf features. This means that we are dynamically linking zlib and libelf but statically linking libbpf. This choice is made because zlib and libelf are usually installed on most systems but libbpf is usually not.

To dynamically link to libbpf, turn off default features and enable recommended feature:

cargo build --release --no-default-features -F recommended

Features

tracexec supports many features, which will be explained in detail in this chapter.

At high-level, tracexec supports two backends and multiple frontends. Backend refers to how tracexec collects exec events while frontend refers to how the events are presented to the user.

Ptrace Backend

ptrace(2) is the default backend for tracexec. To use this backend, simply run tracexec with the desired frontend subcommand (log, tui and collect).

A Simple Introduction to Ptrace

ptrace(2) is the interface designed for implementing a debugger. It allows a tracer process to attach to a tracee process and do basically almost anything to it, such as reading/writing its registers and memories, intercepting its syscall and single-step debugging. A single tracer could trace multiple tracees concurrently but a single tracee could only be traced by one tracer at any given time.

strace is a generic syscall tracing tool built upon ptrace(2), while tracexec is a specialized tool for tracing exec syscall and related contexts.

But wait, isn’t ptrace slow since it is a syscall interface meant for debuggers? Would it slow down workloads significantly? It is indeed slow when used in default configuration because we need to stop/resume the program at every syscall it makes. But when combined with seccomp(2), the overhead could actually be reduced to minimal. seccomp(2) implements a fast syscall filtering interface with classic BPF, by combining ptrace(2) with a seccomp(2) filter that only notifies us when the exec syscalls happen, we avoid incurring overhead on other syscalls the tracee makes. In case you want to learn more about this optimization, read the well-written blog post from strace developer.

Strengths

• Works out of the box.
• Low overhead when combined with seccomp(2). (default in tracexec)
• The minimum required Linux kernel version is 5.3.
• Makes it possible to conveniently attach a debugger to a newly spawned process.

Weaknesses

• Cannot perform system-wide tracing.
• Does not work with setuid/setgid binaries out of the box.
• Significant overhead when seccomp(2) optimization is not used.
• ptrace(2) is a very complex interface abusing waitpid(2) and signals.

eBPF Backend

eBPF is an advanced backend and currently considered experimental.

To use this backend, run tracexec with ebpf as subcommand and the desired frontend as sub-subcommand (tracexec ebpf log or tracexec ebpf tui for example).

A Brief Introduction to eBPF

eBPF is a revolutionary technology for running sandboxed and verified programs directly in the Linux kernel. The in-kernel BPF verifier verifies the program before loading it into the kernel to ensure its safety. For tracing exec, eBPF enables us to attach tracing eBPF programs to kernel functions that handle execve and execveat syscalls and other scheduler tracepoints like sched_process_fork that fires when a process creates a new thread or a new process.

Strengths

• System-wide tracing makes the eBPF backend well-suited for system observability.
• Scoped tracing is also implemented.
• Does not use ptrace(2) so
  • Tracing setuid/setgid binaries is supported.
  • You could combine it with other tools that use ptrace(2), e.g. gdb.

Weaknesses

• Requires root privilege. (or a bunch of capabilities like CAP_SYS_ADMIN and CAP_BPF)
• Sometimes reading userspace memory will fail due to page fault, causing the trace to miss some information.
  • See also https://mozillazg.com/2024/03/ebpf-tracepoint-syscalls-sys-enter-execve-can-not-get-filename-argv-values-case-en.html
  • This could be solved once tracexec is migrated to use sleepable eBPF programs.
• Requires loading eBPF code into Linux kernel, which might be forbidden in kernel lockdown mode.
• Sometimes there are kernel eBPF bugs that could reject the eBPF program.

Required Kernel Configs for eBPF Backend

Required Config Entries

The eBPF backend of course needs a kernel with eBPF and ftrace enabled:

CONFIG_DEBUG_INFO_BTF=y
CONFIG_BPF_SYSCALL=y
CONFIG_BPF_EVENTS=y
CONFIG_FTRACE=y
CONFIG_FUNCTION_TRACER=y
CONFIG_KPROBES=y
CONFIG_KPROBE_EVENTS=y

We need the JIT of eBPF enabled and turned on because when the JIT is disabled, the verifier rejects our program.

CONFIG_BPF_JIT=y

An optional but highly recommended config entry is:

CONFIG_FUNCTION_ERROR_INJECTION=y

It enables tracexec to use sleepable eBPF programs for tracing the entry of exec syscalls. If this config is turned off, tracexec will use non-sleepable eBPF programs, which should work fine for most cases but might fail to read some data from user-space when the data is not yet loaded into the RAM. This problem is thoroughly explained in a blog post: https://mozillazg.com/2024/03/ebpf-tracepoint-syscalls-sys-enter-execve-can-not-get-filename-argv-values-case-en.html.

Example Config

The config used in our UKCI could serve as a reference for building a custom kernel that supports tracexec. It is written in Nix. To obtain a raw kernel config, build the .#ukci target and then dig /nix/store/*linux-config* out of the nix store.

Advanced Parameters for eBPF Backend

The parameters listed here are not considered a stable interface. They may be MODIFIED or completely REMOVED and it would not be considered as a breaking change.

You should only use parameters from this page if you understand it.

TRACEXEC_NO_SLEEP env var

By default, tracexec automatically detects whether the kernel supports sleepable fentry eBPF programs. If this environment variable is set to a non-empty value, tracexec will use non-sleepable eBPF programs for fentry of exec syscalls.

When fentry is disabled and kprobe is used, this setting has no effect.

TRACEXEC_USE_FENTRY/KPROBE env vars

By default, tracexec automatically detects whether the kernel supports CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS to decide whether to use fentry/fexit or kprobe/kretprobe. These two environment variables could override it.

• When TRACEXEC_USE_FENTRY is set to a non-empty value, tracexec will use fentry/fexit eBPF programs.
• When TRACEXEC_USE_KPROBE is set to a non-empty value, tracexec will use kprobe/kretprobe eBPF programs.
• Setting both variables simultaneously is not supported and may produce unpredictable results.

Log Frontend

TUI Frontend

Builtin Terminal

Backtrace

Launching External Debugger

Copy

Theme

Key Bindings

Export Frontend

Perfetto Trace Export

JSON Export

Convenient Privilege Elevation

When using tracexec with eBPF backend or tracing setuid/setgid binaries with ptrace backend, it usually requires running tracexec as root. However, using sudo with tracexec is a little tricky because sudo manipulates the environment variables, which might not be noticed by the user.

For example, when running sudo tracexec ebpf log -- make -j$(nproc),

• sudo resets the environment variables for tracexec and the tracee by retaining a minimal set of basic environment variables and may override some important variables for security reasons (e.g. PATH).
• sudo inserts its own environment variables like SUDO_USER, SUDO_UID and SUDO_COMMAND.
• The tracee make is ran as root, which may not be desired.

In many cases, what we want to achieve is to run tracexec with root privilege but still run the tracee in the original context as an unprivileged user. The following command almost achieves it, with the caveat that sudo -E still modifies the environment variables.

sudo -E tracexec --user $(whoami) ebpf log -- make -j$(nproc)

Starting at tracexec 0.18.0, we offer a new CLI flag that conveniently runs tracexec as root but runs tracee with the original user and environment variables.

For example, the following command runs tracexec as root but runs make -j$(nproc) as the original user:

tracexec --elevate ebpf log -- make -j$(nproc)

When using this feature, tracexec will internally use sudo for privilege elevation. So sudo needs to be installed on your system and you may need to authenticate yourself to sudo when tracexec executes sudo.

Experimental Features

Tutorials

Debugging a basic build problem

Use tracexec as debugger launcher

Without tracexec, it’s not trivial or convenient to debug a program that gets executed by other programs or debug programs with pipes:

• https://stackoverflow.com/questions/5048112/use-gdb-to-debug-a-c-program-called-from-a-shell-script
• https://stackoverflow.com/questions/1456253/gdb-debugging-with-pipe
• https://stackoverflow.com/questions/455544/how-to-load-program-reading-stdin-and-taking-parameters-in-gdb
• https://ftp.gnu.org/old-gnu/Manuals/gdb/html_node/gdb_25.html
• https://stackoverflow.com/questions/65936457/debugging-a-specific-subprocess
• https://sourceware.org/gdb/current/onlinedocs/gdb.html/Forks.html

This example demonstrates how to use tracexec as a gdb launcher to debug programs under complex setup. The following video demonstrates the whole process:

To run this example, first ensure that tracexec and rust is installed on your system.

Clone the tracexec repository and enter the directory for this example:

git clone https://github.com/kxxt/tracexec
cd tracexec/book/tutorials/debugger-launcher

Then run make to compile the two simple rust programs.

In order to allow gdb to attach to the detached and stopped tracees, you probably need to run:

echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope

On a machine with Wayland/X11 display, assuming you have konsole installed(if not, please change the default-external-command), run

tracexec tui -t \
    -b sysexit:in-filename:/a \
    -b sysexit:in-filename:/b \
    --default-external-command "konsole -e gdb -ex cont -ex cont -p {{PID}}" \
    -- ./shell-script

or on a headless server, inside a tmux session, run:

tracexec tui -t \
    -b sysexit:in-filename:/a \
    -b sysexit:in-filename:/b \
    --default-external-command "tmux split-window 'gdb -ex cont -ex cont -p {{PID}}'" \
    -- ./shell-script

Alternatively, launch tracexec tui with a bash session and set the breakpoints in the TUI then run ./shell-script in it.

When the breakpoint get hit, open the Hit Manager and launch the external command for the two stopped tracees. Then two gdb session will open.

To restart the tracees in gdb, Send command c twice.

Catching FD leaks

Comparison with other tools

There are many existing tools for tracing exec syscalls when I start to create tracexec. However, none of them suit my use case well.

This chapter provides a comparison between tracexec and those tools. I hope this chapter will provide readers the knowledge about choosing the best tool for their use cases.

We can roughly divide the tools into three categories by how exec tracing is implemented. (Tracexec supports multiple ways for tracing exec)

Comparison with execsnoop(bcc)

This article compares tracexec with the latest commit of execsnoop at the time of writing. Feel free to improve it if you found anything outdated.

There are two execsnoop implementations in bcc, one implemented with Python, another one implemented with libbpf. Here we will compare with the Python implementation as it supports more features at the time of writing.

Shortcomings of execsnoop(bcc)

Default Limits are Too Limited

By default, execsnoop can only trace up to 20 arguments per exec event, which is too limited to trace complex compiler invocations by various build systems. It can be raised using --max-args argument.

And execsnoop hardcodes a very low limit(128) for the length of each argument, if any argument exceeds this limit, it is silently truncated, resulting in wrong output without any notification to the user.

Cannot Show ARGV[0]

execsnoop shows filename in the place of the first argument(argv[0]) and discards the real argv[0]. Most of the time this is not important because argv[0] is the filename or the basename of the filename.

However, sometimes argv[0] and filename are different and this difference plays an important role on how the program behaves. For example, multi-call binaries like busybox can act as different commands depending on argv[0].

Cannot Show Environment Variables

Sometimes, environment variables play a vital role in program execution. execsnoop doesn’t show them at all.

Cannot Copy-Paste-Execute

A handy feature of tracexec is to copy the shell escaped command line to clipboard, which you can directly paste into another terminal and hit enter to execute it.

But as for execsnoop. It doesn’t even quote the arguments by default, making it hard to distinguish the boundary between arguments. Even if -q/--quote is used, there is still a long way to copy-paste-execute because it does not perform shell escaping. Even if it performs shell-escaping in the future. Without the environment variables, the command may also not work.

Missing features in tracexec compared with execsnoop(bcc)

execsnoop supports tracing processes under a cgroups path and limit tracing to a specific UID.

Comparison with execsnoop(bpftrace)

bpftrace is a high-level tracing language that compiles to eBPF. An execsnoop.bt script is shipped with this package on many Linux distributions (for example, /usr/share/bpftrace/tools/execsnoop.bt on Arch Linux).

This article compares tracexec with the latest commit 93b3247 of execsnoop.bt at the time of writing. Feel free to improve it if you found anything outdated.

Shortcomings of execsnoop.bt

Missing exec result

The script is only monitoring syscall entry and thus unable to report whether or not the execs are successful.

Missing details

The script is minimalistic and cannot show the filename, environment variables and the inherited file descriptors.

Cannot Copy-Paste-Execute

A handy feature of tracexec is to copy the shell escaped command line to clipboard, which you can directly paste into another terminal and hit enter to execute it.

But as for execsnoop.bt. It doesn’t even quote the arguments, making it hard to distinguish the boundary between arguments.

Dependency Bloat

Although execsnoop.bt is minimalistic, the dependencies are not. It depends on bpftrace, which in turn depends on both clang and bcc, where the latter already includes their own implementation of execsnoop.

Comparison with strace

strace is a generic syscall tracing tool that could be used for tracing exec. This article will compare tracexec with the latest version (6.19) of strace at the time of writing. Feel free to improve it if you found anything outdated.

Shortcomings of strace

Missing a Sane Verbosity Level

To trace exec, the most simple strace command that comes to my mind is:

Default Verbosity

strace -e trace=execveat,execve -f -- bash

This produces a noisy log with lots of unrelated content that makes it hard to find the exec events:

[pid 522056] +++ exited with 0 +++
[pid 522051] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=522056, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
[pid 522055] +++ exited with 0 +++
[pid 522051] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=522055, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
[pid 522059] +++ exited with 0 +++
[pid 522058] +++ exited with 0 +++
strace: Process 522061 attached
strace: Process 522062 attached
strace: Process 522063 attached
strace: Process 522064 attached

While for actual exec events, it is not verbose because the environment variables are hidden.

[pid 522055] execve("/usr/bin/ip", ["/usr/bin/ip", "netns", "identify"], 0x7ffe989a3dc0 /* 110 vars */ <unfinished ...>
[Five lines omitted]
[pid 522055] <... execve resumed>)      = 0

Quiet

If we use -q/--quiet, that still does not fix the noisy log problem.

strace -e trace=execveat,execve -f -q -- bash

But at least it makes logs like strace: Process 522061 attached go away.

Verbose

What if we want to know the environment variables? We need to increase verbosity:

strace -e trace=execveat,execve -f -q -- bash

We still have a very noisy log but we could see the environment variables. (Well, at least the env variable names…)

[pid 571872] execve("/usr/bin/ip", ["/usr/bin/ip", "netns", "identify"], ["SHELL=/usr/bin/zsh", "SESSION_MANAGER=local/ryzen:@/tm"..., "USER_ZDOTDIR=/home/kxxt", "COLORTERM=truecolor", "XDG_CONFIG_DIRS=/home/kxxt/.conf"..., "VSCODE_DEBUGPY_ADAPTER_ENDPOINTS"..., "XDG_SESSION_PATH=/org/freedeskto"..., "XDG_MENU_PREFIX=plasma-", "TERM_PROGRAM_VERSION=1.112.01907", "ICEAUTHORITY=/run/user/1000/icea"..., "LC_ADDRESS=en_US.UTF-8", "USE_CCACHE=1", "LC_NAME=en_US.UTF-8", "SSH_AUTH_SOCK=/run/user/1000/gnu"..., "MEMORY_PRESSURE_WRITE=c29tZSAyMD"..., "PYDEVD_DISABLE_FILE_VALIDATION=1", "DESKTOP_SESSION=plasma", "LC_MONETARY=en_US.UTF-8", "__ETC_PROFILE_NIX_SOURCED=1", "GTK_RC_FILES=/etc/gtk/gtkrc:/hom"..., "NO_AT_BRIDGE=1", "EDITOR=nvim", "XDG_SEAT=seat0", "PWD=/home/kxxt/repos/tracexec", "NIX_PROFILES=/nix/var/nix/profil"..., "LOGNAME=kxxt", "XDG_SESSION_DESKTOP=KDE", "XDG_SESSION_TYPE=wayland", "SYSTEMD_EXEC_PID=2534", "BUNDLED_DEBUGPY_PATH=/home/kxxt/"..., "XAUTHORITY=/run/user/1000/xauth_"..., "VSCODE_GIT_ASKPASS_NODE=/usr/sha"..., "MOTD_SHOWN=pam", "VSCODE_INJECTION=1", "GTK2_RC_FILES=/etc/gtk-2.0/gtkrc"..., "HOME=/home/kxxt", "MCFLY_HISTORY=/tmp/mcfly.wGDRsBB"..., "SSH_ASKPASS=/run/user/1000/gnupg"..., "MCFLY_FUZZY=true", "LANG=en_US.UTF-8", "LC_PAPER=en_US.UTF-8", "MCFLY_HISTFILE=/home/kxxt/.zhist"..., "_JAVA_AWT_WM_NONREPARENTING=1", "XDG_CURRENT_DESKTOP=KDE", "PYTHONSTARTUP=/home/kxxt/.config"..., "MEMORY_PRESSURE_WATCH=/sys/fs/cg"..., "STARSHIP_SHELL=bash", "WAYLAND_DISPLAY=wayland-0", "__MISE_DIFF=eAFrXpyfk9KwOC+1vGFJ"..., "NIX_SSL_CERT_FILE=/etc/ssl/certs"..., "GIT_ASKPASS=/usr/share/vscodium/"..., "XDG_SEAT_PATH=/org/freedesktop/D"..., "INVOCATION_ID=3175cd2e73284f8aab"..., "MANAGERPID=2130", "MCFLY_SESSION_ID=ONlRzDF6foVRPQ7"..., "CHROME_DESKTOP=codium.desktop", "STARSHIP_SESSION_KEY=82575444194"..., "__MISE_ORIG_PATH=/home/kxxt/.car"..., "KDE_SESSION_UID=1000", "VSCODE_GIT_ASKPASS_EXTRA_ARGS=", "VSCODE_PYTHON_AUTOACTIVATE_GUARD"..., "XDG_SESSION_CLASS=user", "ANDROID_HOME=/opt/android-sdk", "TERM=xterm-256color", "LC_IDENTIFICATION=en_US.UTF-8", "PYTHON_BASIC_REPL=1", "__MISE_ZSH_PRECMD_RUN=1", "MCFLY_RESULTS_SORT=LAST_RUN", "ZDOTDIR=/home/kxxt", "USER=kxxt", "VSCODE_GIT_IPC_HANDLE=/run/user/"..., "CUDA_PATH=/opt/cuda", "QT_WAYLAND_RECONNECT=1", "KDE_SESSION_VERSION=6", "PAM_KWALLET5_LOGIN=/run/user/100"..., "__MISE_SESSION=eAHqWpOTn5iSmhJfk"..., "MCFLY_HISTORY_FORMAT=zsh", "MCFLY_RESULTS=20", "DISPLAY=:0", "SHLVL=3", "LC_TELEPHONE=en_US.UTF-8", "ANDROID_SDK_ROOT=/opt/android-sd"..., "CCACHE_EXEC=/usr/bin/ccache", "LC_MESSAGES=en_US.UTF-8", "LC_MEASUREMENT=en_US.UTF-8", "XDG_VTNR=2", "XDG_SESSION_ID=2", "MANAGERPIDFDID=2131", "CUDA_DISABLE_PERF_BOOST=1", "FC_FONTATIONS=1", "XDG_RUNTIME_DIR=/run/user/1000", "DEBUGINFOD_URLS=https://debuginf"..., "NVCC_CCBIN=/usr/bin/g++", "MCFLY_INTERFACE_VIEW=BOTTOM", "LC_TIME=en_US.UTF-8", "VSCODE_GIT_ASKPASS_MAIN=/usr/sha"..., "JOURNAL_STREAM=9:44333", "MISE_SHELL=bash", "XDG_DATA_DIRS=/home/kxxt/.local/"..., "GDK_BACKEND=wayland", "KDE_FULL_SESSION=true", "PATH=/home/kxxt/.local/share/mis"..., "DBUS_SESSION_BUS_ADDRESS=unix:pa"..., "KDE_APPLICATIONS_AS_SCOPE=1", "HG=/usr/bin/hg", "MAIL=/var/spool/mail/kxxt", "LC_NUMERIC=en_US.UTF-8", "OLDPWD=/home/kxxt/repos/tracexec", "TERM_PROGRAM=vscode", "_=/usr/bin/starship"]) = 0

Many variables are truncated because it exceeds the string length limit.

Showing the Full Environment Variables

To show the full environment variables, increase the string length limit with -s/--string-limit:

strace -e trace=execveat,execve -f -v -s99999 -- bash

Finally Reaching A Sane Verbosity

To silence all other noisy logs while logging all environment variables, we could use:

strace -e trace=execveat,execve -vqqq -e 'signal=!all' -f -s99999 -- bash

But that command line has become too long to type and remember. With tracexec, it is much easier to remember:

tracexec log --show-env -- bash

Cannot Diff Environment Variables

In the previous shortcoming, we can see that strace could show all the environment variables used in exec. However, showing all the environment variables is too verbose. Most of the time we are only interested in the diff of environment variables. Or to put it in another way, what environment are added and which are modified or removed.

strace has no support for doing that but tracexec by default shows diff of environment variables:

tracexec log -- bash

Cannot Copy-Paste-Execute

A handy feature of tracexec is to copy the shell escaped command line to clipboard, which you can directly paste into another terminal and hit enter to execute it.

But as for strace. It prints the arguments in an array syntax, making it impossible to directly copy and paste into shell.

Missing features in tracexec compared with strace

Tracing only a single process

strace supports tracing only a single process when -f/--follow-forks is not enabled.

In tracexec, we think this use case is too narrow to fit into a specialized exec tracing tool and didn’t implement it.

Stack trace

strace supports printing a stack trace at syscall with -k. We are working on supporting it in tracexec: https://github.com/kxxt/tracexec/issues/108.

FAQ

Developer Guide

Maintaining this Book

Adding a Video

When adding a video to the book, please use the following HTML snippet. It ensures that the video is lazily loaded and has controls.

<video
  src="XXX" controls preload="none" loading="lazy"
  poster="../assets/gdb-launcher-cover.jpg"
  width="100%">
</video>

Please avoid storing videos inside the git repository unless it is below 5MiB. Currently, we post the videos in a GitHub discussion thread and then reference them by URL in the book.

Please add a cover image for the video by taking an image snapshot of the video at a suitable moment. This can be done by right clicking the video and select Take Snapshot in Firefox. The cover image should be stored in the git repository.