# cat sleepers.bt
-
-tracepoint:syscalls:sys_enter_nanosleep {
- printf("%s is sleeping.\n", comm);
-}
-When FILENAME is "-", bpftrace will read program code from stdin.
A program will continue running until Ctrl-C is hit, or an exit function is called.
+When a program exits, all populated maps are printed (more details below).
x86_64, arm64, s390x, arm32, loongarch64, mips64, ppc64, riscv64
-BPF |
-Berkeley Packet Filter: a kernel technology originally developed for optimizing the processing of packet filters (eg, tcpdump expressions). |
-
BPF map |
-A BPF memory object, which is used by bpftrace to create many higher-level objects. |
-
BTF |
-BPF Type Format: the metadata format which encodes the debug info related to BPF program/map. |
-
dynamic tracing |
-Also known as dynamic instrumentation, this is a technology that can instrument any software event, such as function calls and returns, by live modification of instruction text. Target software usually does not need special capabilities to support dynamic tracing, other than a symbol table that bpftrace can read. Since this instruments all software text, it is not considered a stable API, and the target functions may not be documented outside of their source code. |
-
eBPF |
-Enhanced BPF: a kernel technology that extends BPF so that it can execute more generic programs on any events, such as the bpftrace programs listed below. It makes use of the BPF sandboxed virtual machine environment. Also note that eBPF is often just referred to as BPF. |
-
kprobes |
-A Linux kernel technology for providing dynamic tracing of kernel functions. |
-
probe |
-An instrumentation point in software or hardware, that generates events that can execute bpftrace programs. |
-
static tracing |
-Hard-coded instrumentation points in code. Since these are fixed, they may be provided as part of a stable API, and documented. |
-
tracepoints |
-A Linux kernel technology for providing static tracing. |
-
uprobes |
-A Linux kernel technology for providing dynamic tracing of user-level functions. |
-
USDT |
-User Statically-Defined Tracing: static tracing points for user-level software. Some applications support USDT. |
-
Programs saved as files are often called scripts and can be executed by specifying their file name.
-We use a .bt file extension, short for bpftrace, but the extension is not required.
For example, listing the sleepers.bt file using cat:
# cat sleepers.bt
-
-tracepoint:syscalls:sys_enter_nanosleep {
- printf("%s is sleeping.\n", comm);
-}
-And then calling it:
-# bpftrace sleepers.bt - -Attaching 1 probe... -iscsid is sleeping. -iscsid is sleeping.-
It can also be made executable to run stand-alone.
-Start by adding an interpreter line at the top (#!) with either the path to your installed bpftrace (/usr/local/bin is the default) or the path to env (usually just /usr/bin/env) followed by bpftrace (so it will find bpftrace in your $PATH):
#!/usr/local/bin/bpftrace
-
-tracepoint:syscalls:sys_enter_nanosleep {
- printf("%s is sleeping.\n", comm);
-}
-Then make it executable:
-# chmod 755 sleepers.bt -# ./sleepers.bt - -Attaching 1 probe... -iscsid is sleeping. -iscsid is sleeping.-
The bpftrace (bt) language is inspired by the D language used by dtrace and uses the same program structure.
-Each script consists of a preamble and one or more action blocks.
preamble +config + actionblock1 actionblock2
Preprocessor and type definitions take place in the preamble:
-#include <linux/socket.h>
-#define RED "\033[31m"
-
-struct S {
- int x;
-}
-Each action block consists of three parts:
A program will continue running until Ctrl-C is hit, or an exit function is called.
-When a program exits, all populated maps are printed (this behavior and maps are explained in later sections).
A basic script that traces the open(2) and openat(2) system calls can be written as follows:
The second action block uses two probes, one for open and one for openat, and defines an action that prints the file being open ed as well as the pid and comm of the process that execute the syscall.
See the Probes section for details on the available probe types.
To improve script portability, you can set bpftrace Config Variables via the config block,
-which can only be placed at the top of the script before any probes (even BEGIN).
BEGIN).
Filters (also known as predicates) can be added after probe names. The probe still fires, but it will skip the action unless the filter is true.
@@ -1228,6 +1094,22 @@Preprocessor and type definitions take place in the preamble:
+#include <linux/socket.h>
+#define RED "\033[31m"
+
+struct S {
+ int x;
+}
+Pointers in bpftrace are similar to those found in C.
Builtins are special variables built into the language.
-Unlike scratch and map variables they don’t need a $ or @ as prefix (except for the positional parameters).
-The 'Kernel' column indicates the minimum kernel version required and the 'BPF Helper' column indicates the raw BPF helper function used for this builtin.
bpftrace supports various probe types which allow the user to attach BPF programs to different types of events.
+Each probe starts with a provider (e.g. kprobe) followed by a colon (:) separated list of options.
+The amount of options and their meaning depend on the provider and are detailed below.
+The valid values for options can depend on the system or binary being traced, e.g. for uprobes it depends on the binary.
+Also see Listing Probes.
It is possible to associate multiple probes with a single action as long as the action is valid for all specified probes.
+Multiple probes can be specified as a comma (,) separated list:
kprobe:tcp_reset,kprobe:tcp_v4_rcv {
+ printf("Entered: %s\n", probe);
+}
+Wildcards are supported too:
+kprobe:tcp_* {
+ printf("Entered: %s\n", probe);
+}
+Both can be combined:
+kprobe:tcp_reset,kprobe:*socket* {
+ printf("Entered: %s\n", probe);
+}
+Most providers also support a short name which can be used instead of the full name, e.g. kprobe:f and k:f are identical.
| Variable | -Type | -Kernel | -BPF Helper | -Description | +Probe Name |
+Short Name |
+Description |
+Kernel/User Level |
|---|---|---|---|---|---|---|---|---|
| - | int64 |
-n/a |
-n/a |
-The nth positional parameter passed to the bpftrace program.
-If less than n parameters are passed this evaluates to |
++ | - |
+Built-in events |
+Kernel/User |
|
-int64 |
-n/a |
-n/a |
-Total amount of positional parameters passed. |
++ | - |
+Built-in events |
+Kernel/User |
|
-int64 |
-n/a |
-n/a |
-nth argument passed to the function being traced. These are extracted from the CPU registers. The amount of args passed in registers depends on the CPU architecture. (kprobes, uprobes, usdt). |
-||||
|
-struct args |
-n/a |
-n/a |
-The struct of all arguments of the traced function. Available in |
-||||
cgroup |
-uint64 |
-4.18 |
-get_current_cgroup_id |
-ID of the cgroup the current process belongs to. Only works with cgroupv2. |
-||||
comm |
-string[16] |
-4.2 |
-get_current_comm |
-Name of the current thread |
-||||
cpid |
-uint32 |
-n/a |
-n/a |
-Child process ID, if bpftrace is invoked with |
-||||
cpu |
-uint32 |
-4.1 |
-raw_smp_processor_id |
-ID of the processor executing the BPF program |
-||||
curtask |
-uint64 |
-4.8 |
-get_current_task |
-Pointer to |
-||||
elapsed |
-uint64 |
-(see nsec) |
-ktime_get_ns / ktime_get_boot_ns |
-Nanoseconds elapsed since bpftrace initialization, based on |
++ |
|
+Processor-level events |
+Kernel |
func |
-string |
-n/a |
-n/a |
-Name of the current function being traced (kprobes,uprobes) |
++ |
|
+Timed output |
+Kernel/User |
gid |
-uint64 |
-4.2 |
-get_current_uid_gid |
-Group ID of the current thread, as seen from the init namespace |
++ |
|
+Iterators tracing |
+Kernel |
jiffies |
-uint64 |
-5.9 |
-get_jiffies_64 |
-Jiffies of the kernel. In 32-bit system, using this builtin might be slower. |
++ |
|
+Kernel functions tracing with BTF support |
+Kernel |
numaid |
-uint32 |
-5.8 |
-numa_node_id |
-ID of the NUMA node executing the BPF program |
++ |
|
+Kernel function start/return |
+Kernel |
pid |
-uint32 |
-4.2 |
-get_current_pid_tgid |
-Process ID of the current thread (aka thread group ID), as seen from the init namespace |
++ |
|
+Timed sampling |
+Kernel/User |
probe |
-string |
-n/na |
-n/a |
-Name of the current probe |
++ |
|
+Kernel static tracepoints with raw arguments |
+Kernel |
rand |
-uint32 |
-4.1 |
-get_prandom_u32 |
-Random number |
++ |
|
+Kernel software events |
+Kernel |
return |
-n/a |
-n/a |
-n/a |
-The return keyword is used to exit the current probe. This differs from exit() in that it doesn’t exit bpftrace. |
++ |
|
+Kernel static tracepoints |
+Kernel |
retval |
-uint64 |
-n/a |
-n/a |
-Value returned by the function being traced (kretprobe, uretprobe, fexit). For kretprobe and uretprobe, its type is |
++ |
|
+User-level function start/return |
+User |
tid |
-uint32 |
-4.2 |
-get_current_pid_tgid |
-Thread ID of the current thread, as seen from the init namespace |
++ |
|
+User-level static tracepoints |
+User |
uid |
-uint64 |
-4.2 |
-get_current_uid_gid |
-User ID of the current thread, as seen from the init namespace |
++ |
|
+Memory watchpoints |
+Kernel |
$1, $2, …, $N, $#
These are the positional parameters to the bpftrace program, also referred to as command line arguments.
-If the parameter is numeric (entirely digits), it can be used as a number.
-If it is non-numeric, it must be used as a string in the str() call.
-If a parameter is used that was not provided, it will default to zero for numeric context, and "" for string context.
-Positional parameters may also be used in probe argument and will be treated as a string parameter.
If a positional parameter is used in str(), it is interpreted as a pointer to the actual given string literal, which allows to do pointer arithmetic on it.
-Only addition of a single constant, less or equal to the length of the supplied string, is allowed.
$# returns the number of positional arguments supplied.
These are special built-in events provided by the bpftrace runtime.
+BEGIN is triggered before all other probes are attached.
+END is triggered after all other probes are detached.
This allows scripts to be written that use basic arguments to change their behavior. -If you develop a script that requires more complex argument processing, it may be better suited for bcc instead, which -supports Python’s argparse and completely custom argument processing.
+Note that specifying an END probe doesn’t override the printing of 'non-empty' maps at exit.
+To prevent printing all used maps need be cleared in the END probe:
# bpftrace -e 'BEGIN { printf("I got %d, %s (%d args)\n", $1, str($2), $#); }' 42 "hello"
-
-I got 42, hello (2 args)
-
-# bpftrace -e 'BEGIN { printf("%s\n", str($1 + 1)) }' "hello"
-
-ello
+END {
+ clear(@map1);
+ clear(@map2);
+}
self:signal:SIGUSR1
Script example, bsize.bt:
+These are special built-in events provided by the bpftrace runtime.
+The trigger function is called by the bpftrace runtime when the bpftrace process receives specific events, such as a SIGUSR1 signal.
+When multiple signal handlers are attached to the same signal, only the first one is used.
#!/usr/local/bin/bpftrace
-
-BEGIN
-{
- printf("Tracing block I/O sizes > %d bytes\n", $1);
-}
-
-tracepoint:block:block_rq_issue
-/args.bytes > $1/
-{
- @ = hist(args.bytes);
+self:signal:SIGUSR1 {
+ print("abc");
}
When run with a 65536 argument:
# ./bsize.bt 65536 - -Tracing block I/O sizes > 65536 bytes -^C - -@: -[512K, 1M) 1 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|+
hardware:event_name:
hardware:event_name:count
h
It has passed the argument in as $1 and used it as a filter.
With no arguments, $1 defaults to zero:
# ./bsize.bt -Attaching 2 probes... -Tracing block I/O sizes > 0 bytes -^C - -@: -[4K, 8K) 115 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| -[8K, 16K) 35 |@@@@@@@@@@@@@@@ | -[16K, 32K) 5 |@@ | -[32K, 64K) 3 |@ | -[64K, 128K) 1 | | -[128K, 256K) 0 | | -[256K, 512K) 0 | | -[512K, 1M) 1 | |-
These are the pre-defined hardware events provided by the Linux kernel, as commonly traced by the perf utility. +They are implemented using performance monitoring counters (PMCs): hardware resources on the processor. +There are about ten of these, and they are documented in the perf_event_open(2) man page. +The event names are:
cpu-cycles or cycles
instructions
cache-references
cache-misses
branch-instructions or branches
branch-misses
bus-cycles
frontend-stalls
backend-stalls
ref-cycles
| Name | -Description | -Sync/Async/Compile Time | -
|---|---|---|
| - | Reverse byte order |
-Sync |
-
| - | Returns a hex-formatted string of the data pointed to by d |
-Sync |
-
| - | Print file content |
-Async |
-
| - | Resolve cgroup ID |
-Compile Time |
-
| - | Convert cgroup id to cgroup path |
-Sync |
-
| - | Quit bpftrace with an optional exit code |
-Async |
-
| - | Print the array |
-Async |
-
| - | Resolve kernel symbol name |
-Compile Time |
-
| - | Annotate as kernelspace pointer |
-Sync |
-
| - | Kernel stack trace |
-Sync |
-
| - | Resolve kernel address |
-Async |
-
| - | Count ustack/kstack frames |
-Sync |
-
| - | Convert MAC address data |
-Sync |
-
| - | Timestamps and Time Deltas |
-Sync |
-
| - | Convert IP address data to text |
-Sync |
-
| - | Offset of element in structure |
-Compile Time |
-
| - | Override return value |
-Sync |
-
| - | Return full path |
-Sync |
-
| - | Resolve percpu kernel symbol name |
-Sync |
-
| - | Print a non-map value with default formatting |
-Async |
-
| - | Print formatted |
-Async |
-
| - | Convert text IP address to byte array |
-Compile Time |
-
| - | Returns the value stored in the named register |
-Sync |
-
| - | Send a signal to the current process |
-Sync |
-
| - | Return size of a type or expression |
-Sync |
-
| - | Write skb 's data section into a PCAP file |
-Async |
-
| - | Returns the string pointed to by s |
-Sync |
-
| - | Compares whether the string haystack contains the string needle. |
-Sync |
-
| - | Get error message for errno code |
-Sync |
-
| - | Return a formatted timestamp |
-Async |
-
| - | Compare first n characters of two strings |
-Sync |
-
| - | Execute shell command |
-Async |
-
| - | Print formatted time |
-Async |
-
| - | Resolve user-level symbol name |
-Compile Time |
-
| - | Annotate as userspace pointer |
-Sync |
-
| - | User stack trace |
-Sync |
-
| - | Resolve user space address |
-Async |
-
Functions that are marked async are asynchronous which can lead to unexpected behaviour, see the Invocation Mode section for more information.
+The count option specifies how many events must happen before the probe fires (sampling interval).
+If count is left unspecified a default value is used.
compile time functions are evaluated at compile time, a static value will be compiled into the program.
+This will fire once for every 1,000,000 cache misses.
+hardware:cache-misses:1e6 { @[pid] = count(); }
+unsafe functions can have dangerous side effects and should be used with care, the --unsafe flag is required for use.
uint8 bswap(uint8 n)
interval:us:count
uint16 bswap(uint16 n)
interval:ms:count
uint32 bswap(uint32 n)
interval:s:count
uint64 bswap(uint64 n)
interval:hz:rate
i
bswap reverses the order of the bytes in integer n. In case of 8 bit integers, n is returned without being modified.
-The return type is an unsigned integer of the same width as n.
The interval probe fires at a fixed interval as specified by its time spec. +Interval fires on one CPU at a time, unlike profile probes.
+This prints the rate of syscalls per second.
+tracepoint:raw_syscalls:sys_enter { @syscalls = count(); }
+interval:s:1 { print(@syscalls); clear(@syscalls); }
+buffer buf(void * data, [int64 length])
iter:task
iter:task:pin
iter:task_file
iter:task_file:pin
iter:task_vma
iter:task_vma:pin
buf reads length amount of bytes from address data.
-The maximum value of length is limited to the BPFTRACE_MAX_STRLEN variable.
-For arrays the length is optional, it is automatically inferred from the signature.
it
buf is address space aware and will call the correct helper based on the address space associated with data.
Warning this feature is experimental and may be subject to interface changes.
The buffer object returned by buf can safely be printed as a hex encoded string with the %r format specifier.
These are eBPF iterator probes that allow iteration over kernel objects. +Iterator probe can’t be mixed with any other probe, not even another iterator. +Each iterator probe provides a set of fields that could be accessed with the +ctx pointer. Users can display the set of available fields for each iterator via +-lv options as described below.
+iter:task { printf("%s:%d\n", ctx->task->comm, ctx->task->pid); }
+
+/*
+ * Sample output:
+ * systemd:1
+ * kthreadd:2
+ * rcu_gp:3
+ * rcu_par_gp:4
+ * kworker/0:0H:6
+ * mm_percpu_wq:8
+ */
+iter:task_file {
+ printf("%s:%d %d:%s\n", ctx->task->comm, ctx->task->pid, ctx->fd, path(ctx->file->f_path));
+}
+
+/*
+ * Sample output:
+ * systemd:1 1:/dev/null
+ * systemd:1 3:/dev/kmsg
+ * ...
+ * su:1622 2:/dev/pts/1
+ * ...
+ * bpftrace:1892 2:/dev/pts/1
+ * bpftrace:1892 6:anon_inode:bpf-prog
+ */
+iter:task_vma {
+ printf("%s %d %lx-%lx\n", comm, pid, ctx->vma->vm_start, ctx->vma->vm_end);
+}
+
+/*
+ * Sample output:
+ * bpftrace 119480 55b92c380000-55b92c386000
+ * ...
+ * bpftrace 119480 7ffd55dde000-7ffd55de2000
+ */
+Bytes with values >=32 and <=126 are printed using their ASCII character, other bytes are printed in hex form (e.g. \x00). The %rx format specifier can be used to print everything in hex form, including ASCII characters. The similar %rh format specifier prints everything in hex form without \x and with spaces between bytes (e.g. 0a fe).
It’s possible to pin an iterator by specifying the optional probe ':pin' part, that defines the pin file. +It can be specified as an absolute or relative path to /sys/fs/bpf.
interval:s:1 {
- printf("%r\n", buf(kaddr("avenrun"), 8));
-}
+iter:task:list { printf("%s:%d\n", ctx->task->comm, ctx->task->pid); }
+
+/*
+ * Sample output:
+ * Program pinned to /sys/fs/bpf/list
+ */
\x00\x03\x00\x00\x00\x00\x00\x00 -\xc2\x02\x00\x00\x00\x00\x00\x00+
iter:task_file:/sys/fs/bpf/files {
+ printf("%s:%d %s\n", ctx->task->comm, ctx->task->pid, path(ctx->file->f_path));
+}
+
+/*
+ * Sample output:
+ * Program pinned to /sys/fs/bpf/files
+ */
void cat(string namefmt, […args])
fentry[:module]:fn
fexit[:module]:fn
f (fentry)
fr (fexit)
--info)Kernel features:BTF
+Probe types:fentry
async
+fentry/fexit probes attach to kernel functions similar to kprobe and kretprobe.
+They make use of eBPF trampolines which allow kernel code to call into BPF programs with near zero overhead.
+Originally, these were called kfunc and kretfunc but were later renamed to fentry and fexit to match
+how these are referenced in the kernel and to prevent confusion with BPF Kernel Functions.
+The original names are still supported for backwards compatibility.
Dump the contents of the named file to stdout.
-cat supports the same format string and arguments that printf does.
-If the file cannot be opened or read an error is printed to stderr.
fentry/fexit probes make use of BTF type information to derive the type of function arguments at compile time.
+This removes the need for manual type casting and makes the code more resilient against small signature changes in the kernel.
+The function arguments are available in the args struct which can be inspected by doing verbose listing (see Listing Probes).
+These arguments are also available in the return probe (fexit), unlike kretprobe.
tracepoint:syscalls:sys_enter_execve {
- cat("/proc/%d/maps", pid);
+# bpftrace -lv 'fentry:tcp_reset'
+
+fentry:tcp_reset
+ struct sock * sk
+ struct sk_buff * skb
+fentry:x86_pmu_stop {
+ printf("pmu %s stop\n", str(args.event->pmu->name));
}
The fget function takes one argument as file descriptor and you can access it via args.fd and the return value is accessible via retval:
+55f683ebd000-55f683ec1000 r--p 00000000 08:01 1843399 /usr/bin/ls -55f683ec1000-55f683ed6000 r-xp 00004000 08:01 1843399 /usr/bin/ls -55f683ed6000-55f683edf000 r--p 00019000 08:01 1843399 /usr/bin/ls -55f683edf000-55f683ee2000 rw-p 00021000 08:01 1843399 /usr/bin/ls -55f683ee2000-55f683ee3000 rw-p 00000000 00:00 0+
fexit:fget {
+ printf("fd %d name %s\n", args.fd, str(retval->f_path.dentry->d_name.name));
+}
+
+/*
+ * Sample output:
+ * fd 3 name ld.so.cache
+ * fd 3 name libselinux.so.1
+ */
uint64 cgroupid(const string path)
kprobe[:module]:fn
kprobe[:module]:fn+offset
kretprobe[:module]:fn
compile time
+k
kr
cgroupid retrieves the cgroupv2 ID of the cgroup available at path.
kprobes allow for dynamic instrumentation of kernel functions.
+Each time the specified kernel function is executed the attached BPF programs are ran.
BEGIN {
- print(cgroupid("/sys/fs/cgroup/system.slice"));
+kprobe:tcp_reset {
+ @tcp_resets = count()
}
Function arguments are available through the argN for register args. Arguments passed on stack are available using the stack pointer, e.g. $stack_arg0 = (int64)reg("sp") + 16.
+Whether arguments passed on stack or in a register depends on the architecture and the number or arguments used, e.g. on x86_64 the first 6 non-floating point arguments are passed in registers and all following arguments are passed on the stack.
+Note that floating point arguments are typically passed in special registers which don’t count as argN arguments which can cause confusion.
+Consider a function with the following signature:
void func(int a, double d, int x)
cgroup_path_t cgroup_path(int cgroupid, string filter)
Convert cgroup id to cgroup path. -This is done asynchronously in userspace when the cgroup_path value is printed, -therefore it can resolve to a different value if the cgroup id gets reassigned. -This also means that the returned value can only be used for printing.
A string literal may be passed as an optional second argument to filter cgroup -hierarchies in which the cgroup id is looked up by a wildcard expression (cgroup2 -is always represented by "unified", regardless of where it is mounted).
+Due to d being a floating point, x is accessed through arg1 where one might expect arg2.
The currently mounted hierarchy at /sys/fs/cgroup is used to do the lookup. If -the cgroup with the given id isn’t present here (e.g. when running in a Docker -container), the cgroup path won’t be found (unlike when looking up the cgroup -path of a process via /proc/…/cgroup).
+bpftrace does not detect the function signature so it is not aware of the argument count or their type. +It is up to the user to perform Type conversion when needed, e.g.
BEGIN {
- $cgroup_path = cgroup_path(3436);
- print($cgroup_path);
- print($cgroup_path); /* This may print a different path */
- printf("%s %s", $cgroup_path, $cgroup_path); /* This may print two different paths */
+#include <linux/path.h>
+#include <linux/dcache.h>
+
+kprobe:vfs_open
+{
+ printf("open path: %s\n", str(((struct path *)arg0)->dentry->d_name.name));
}
void exit([int code])
async
+Here arg0 was cast as a (struct path *), since that is the first argument to vfs_open. +The struct support is the same as bcc and based on available kernel headers. +This means that many, but not all, structs will be available, and you may need to manually define structs.
Terminate bpftrace, as if a SIGTERM was received.
-The END probe will still trigger (if specified) and maps will be printed.
-An optional exit code can be provided.
If the kernel has BTF (BPF Type Format) data, all kernel structs are always available without defining them. For example:
BEGIN {
- exit();
+kprobe:vfs_open {
+ printf("open path: %s\n", str(((struct path *)arg0)->dentry->d_name.name));
}
Or
+You can optionally specify a kernel module, either to include BTF data from that module, or to specify that the traced function should come from that module.
BEGIN {
- exit(1);
+kprobe:kvm:x86_emulate_insn
+{
+ $ctxt = (struct x86_emulate_ctxt *) arg0;
+ printf("eip = 0x%lx\n", $ctxt->eip);
}
void join(char *arr[], [char * sep = ' '])
async
+See BTF Support for more details.
join joins all the string array arr with sep as separator into one string.
-This string will be printed to stdout directly, it cannot be used as string value.
kprobe s are not limited to function entry, they can be attached to any instruction in a function by specifying an offset from the start of the function.
The concatenation of the array members is done in BPF and the printing happens in userspace.
+kretprobe s trigger on the return from a kernel function.
+Return probes do not have access to the function (input) arguments, only to the return value (through retval).
+A common pattern to work around this is by storing the arguments in a map on function entry and retrieving in the return probe:
tracepoint:syscalls:sys_enter_execve {
- join(args.argv);
+kprobe:d_lookup
+{
+ $name = (struct qstr *)arg1;
+ @fname[tid] = $name->name;
+}
+
+kretprobe:d_lookup
+/@fname[tid]/
+{
+ printf("%-8d %-6d %-16s M %s\n", elapsed / 1e6, pid, comm,
+ str(@fname[tid]));
}
uint64 kaddr(const string name)
profile:us:count
profile:ms:count
profile:s:count
profile:hz:rate
compile time
+p
Get the address of the kernel symbol name.
Profile probes fire on each CPU on the specified interval. +These operate using perf_events (a Linux kernel facility, which is also used by the perf command).
interval:s:1 {
- $avenrun = kaddr("avenrun");
- $load1 = *$avenrun;
-}
-profile:hz:99 { @[tid] = count(); }
You can find all kernel symbols at /proc/kallsyms.
T * kptr(T * ptr)
rawtracepoint:event
Marks ptr as a kernel address space pointer.
-See the address-spaces section for more information on address-spaces.
-The pointer type is left unchanged.
kstack_t kstack([StackMode mode, ][int limit])
rt
These are implemented using BPF stack maps.
+The hook point triggered by tracepoint and rawtracepoint is the same.
+tracepoint and rawtracepoint are nearly identical in terms of functionality.
+The only difference is in the program context.
+rawtracepoint offers raw arguments to the tracepoint while tracepoint applies further processing to the raw arguments.
+The additional processing is defined inside the kernel.
kprobe:ip_output { @[kstack()] = count(); }
-
-/*
- * Sample output:
- * @[
- * ip_output+1
- * tcp_transmit_skb+1308
- * tcp_write_xmit+482
- * tcp_release_cb+225
- * release_sock+64
- * tcp_sendmsg+49
- * sock_sendmsg+48
- * sock_write_iter+135
- * __vfs_write+247
- * vfs_write+179
- * sys_write+82
- * entry_SYSCALL_64_fastpath+30
- * ]: 1708
- */
+rawtracepoint:block_rq_insert {
+ printf("%llx %llx\n", arg0, arg1);
+}
Sampling only three frames from the stack (limit = 3):
+Tracepoint arguments are available via the argN builtins.
+Each arg is a 64-bit integer.
+The available arguments can be found in the relative path of the kernel source code include/trace/events/. For example:
kprobe:ip_output { @[kstack(3)] = count(); }
-
-/*
- * Sample output:
- * @[
- * ip_output+1
- * tcp_transmit_skb+1308
- * tcp_write_xmit+482
- * ]: 1708
- */
+include/trace/events/block.h +DEFINE_EVENT(block_rq, block_rq_insert, + TP_PROTO(struct request_queue *q, struct request *rq), + TP_ARGS(q, rq) +);
You can also choose a different output format.
-Available formats are bpftrace, perf, and raw (no symbolication):
kprobe:ip_output { @[kstack(perf, 3)] = count(); }
-
-/*
- * Sample output:
- * @[
- * ffffffffb4019501 do_mmap+1
- * ffffffffb401700a sys_mmap_pgoff+266
- * ffffffffb3e334eb sys_mmap+27
- * ]: 1708
- */
+software:event:
software:event:count
s
These are the pre-defined software events provided by the Linux kernel, as commonly traced via the perf utility. +They are similar to tracepoints, but there is only about a dozen of these, and they are documented in the perf_event_open(2) man page. +If the count is not provided, a default is used.
The event names are:
ksym_t ksym(uint64 addr)
cpu-clock or cpu
task-clock
page-faults or faults
context-switches or cs
cpu-migrations
minor-faults
major-faults
alignment-faults
emulation-faults
dummy
bpf-output
async
-Retrieve the name of the function that contains address addr.
-The address to name mapping happens in user-space.
The ksym_t type can be printed with the %s format specifier.
kprobe:do_nanosleep
-{
- printf("%s\n", ksym(reg("ip")));
-}
-
-/*
- * Sample output:
- * do_nanosleep
- */
+software:faults:100 { @[comm] = count(); }
This roughly counts who is causing page faults, by sampling the process name for every one in one hundred faults.
+int64 len(ustack stack)
int64 len(kstack stack)
tracepoint:subsys:event
Retrieve the depth (measured in # of frames) of the call stack
-specified by stack.
macaddr_t macaddr(char [6] mac)
t
Create a buffer that holds a macaddress as read from mac
-This buffer can be printed in the canonical string format using the %s format specifier.
Tracepoints are hooks into events in the kernel.
+Tracepoints are defined in the kernel source and compiled into the kernel binary which makes them a form of static tracing.
+Unlike kprobe s, new tracepoints cannot be added without modifying the kernel.
The advantage of tracepoints is that they generally provide a more stable interface than kprobe s do, they do not depend on the existence of a kernel function.
kprobe:arp_create {
- $stack_arg0 = *(uint8*)(reg("sp") + 8);
- $stack_arg1 = *(uint8*)(reg("sp") + 16);
- printf("SRC %s, DST %s\n", macaddr($stack_arg0), macaddr($stack_arg1));
-}
+tracepoint:syscalls:sys_enter_openat {
+ printf("%s %s\n", comm, str(args.filename));
+}
+Tracepoint arguments are available in the args struct which can be inspected with verbose listing, see the Listing Probes section for more details.
# bpftrace -lv "tracepoint:*" -/* - * Sample output: - * SRC 18:C0:4D:08:2E:BB, DST 74:83:C2:7F:8C:FF - */+tracepoint:xhci-hcd:xhci_setup_device_slot + u32 info + u32 info2 + u32 tt_info + u32 state +... +
Alternatively members for each tracepoint can be listed from their /format file in /sys.
Apart from the filename member, we can also print flags, mode, and more. +After the "common" members listed first, the members are specific to the tracepoint.
timestamp nsecs([TimestampMode mode])
https://www.kernel.org/doc/html/latest/trace/tracepoints.html
Returns a timestamp in nanoseconds, as given by the requested kernel clock.
-Defaults to boot if no clock is explicitly requested.
nsecs(monotonic) - nanosecond timestamp since boot, exclusive of time the system spent suspended (CLOCK_MONOTONIC)
uprobe:binary:func
nsecs(boot) - nanoseconds since boot, inclusive of time the system spent suspended (CLOCK_BOOTTIME)
uprobe:binary:func+offset
nsecs(tai) - TAI timestamp in nanoseconds (CLOCK_TAI)
uprobe:binary:offset
nsecs(sw_tai) - approximation of TAI timestamp in nanoseconds, is obtained through the "triple vdso sandwich" method. For older kernels without direct TAI timestamp access in BPF.
uretprobe:binary:func
u
ur
uprobe s or user-space probes are the user-space equivalent of kprobe s.
+The same limitations that apply kprobe and kretprobe also apply to uprobe s and uretprobe s, namely: arguments are available via the argN and sargN builtins and can only be accessed with a uprobe (sargN is more common for older versions of golang).
+retval is the return value for the instrumented function and can only be accessed with a uretprobe.
interval:s:1 {
- $sw_tai1 = nsecs(sw_tai);
- $tai = nsecs(tai);
- $sw_tai2 = nsecs(sw_tai);
- printf("sw_tai precision: %lldns\n", ($sw_tai1 + $sw_tai2)/2 - $tai);
+uprobe:/bin/bash:readline { printf("arg0: %d\n", arg0); }
+What does arg0 of readline() in /bin/bash contain? +I don’t know, so I’ll need to look at the bash source code to find out what its arguments are.
+When tracing libraries, it is sufficient to specify the library name instead of
+a full path. The path will be then automatically resolved using /etc/ld.so.cache:
uprobe:libc:malloc { printf("Allocated %d bytes\n", arg0); }
+If the traced binary has DWARF included, function arguments are available in the args struct which can be inspected with verbose listing, see the Listing Probes section for more details.
# bpftrace -lv 'uprobe:/bin/bash:rl_set_prompt' + +uprobe:/bin/bash:rl_set_prompt + const char* prompt+
When tracing C++ programs, it’s possible to turn on automatic symbol demangling by using the :cpp prefix:
# bpftrace:cpp:"bpftrace::BPFtrace::add_probe" { ... }
+It is important to note that for uretprobe s to work the kernel runs a special helper on user-space function entry which overrides the return address on the stack.
+This can cause issues with languages that have their own runtime like Golang:
func myprint(s string) {
+ fmt.Printf("Input: %s\n", s)
}
-/*
- * Sample output:
- * sw_tai precision: -98ns
- * sw_tai precision: -99ns
- * ...
- */
+func main() {
+ ss := []string{"a", "b", "c"}
+ for _, s := range ss {
+ go myprint(s)
+ }
+ time.Sleep(1*time.Second)
+}
+# bpftrace -e 'uretprobe:./test:main.myprint { @=count(); }' -c ./test
+runtime: unexpected return pc for main.myprint called from 0x7fffffffe000
+stack: frame={sp:0xc00008cf60, fp:0xc00008cfd0} stack=[0xc00008c000,0xc00008d000)
+fatal error: unknown caller pc
inet ntop([int64 af, ] int addr)
usdt:binary_path:probe_name
inet ntop([int64 af, ] char addr[4])
usdt:binary_path:[probe_namespace]:probe_name
inet ntop([int64 af, ] char addr[16])
usdt:library_path:probe_name
usdt:library_path:[probe_namespace]:probe_name
ntop returns the string representation of an IPv4 or IPv6 address.
-ntop will infer the address type (IPv4 or IPv6) based on the addr type and size.
-If an integer or char[4] is given, ntop assumes IPv4, if a char[16] is given, ntop assumes IPv6.
-You can also pass the address type (e.g. AF_INET) explicitly as the first parameter.
uint64 offsetof(STRUCT, FIELD[.SUBFIELD])
uint64 offsetof(EXPRESSION, FIELD[.SUBFIELD])
U
compile time
+Where probe_namespace is optional if probe_name is unique within the binary.
Returns offset of the field offset bytes in struct.
-Similar to kernel offsetof operator.
You can target the entire host (or an entire process’s address space by using the -p arg) by using a single wildcard in place of the binary_path/library_path:
usdt:*:loop { printf("hi\n"); }
+Support any number of sub field levels, for example:
+Please note that if you use wildcards for the probe_name or probe_namespace and end up targeting multiple USDTs for the same probe you might get errors if you also utilize the USDT argument builtin (e.g. arg0) as they could be of different types.
+Arguments are available via the argN builtins:
struct Foo {
- struct {
- struct {
- struct {
- int d;
- } c;
- } b;
- } a;
-}
-BEGIN {
- @x = offsetof(struct Foo, a.b.c.d);
- exit();
-}
+usdt:/root/tick:loop { printf("%s: %d\n", str(arg0), arg1); }
bpftrace also supports USDT semaphores. +If both your environment and bpftrace support uprobe refcounts, then USDT semaphores are automatically activated for all processes upon probe attachment (and --usdt-file-activation becomes a noop). +You can check if your system supports uprobe refcounts by running:
+# bpftrace --info 2>&1 | grep "uprobe refcount" +bcc bpf_attach_uprobe refcount: yes + uprobe refcount (depends on Build:bcc bpf_attach_uprobe refcount): yes+
If your system does not support uprobe refcounts, you may activate semaphores by passing in -p $PID or --usdt-file-activation. +--usdt-file-activation looks through /proc to find processes that have your probe’s binary mapped with executable permissions into their address space and then tries to attach your probe. +Note that file activation occurs only once (during attach time). +In other words, if later during your tracing session a new process with your executable is spawned, your current tracing session will not activate the new process. +Also note that --usdt-file-activation matches based on file path. +This means that if bpftrace runs from the root host, things may not work as expected if there are processes execved from private mount namespaces or bind mounted directories. +One workaround is to run bpftrace inside the appropriate namespaces (i.e. the container).
+void override(uint64 rc)
watchpoint:absolute_address:length:mode
watchpoint:function+argN:length:mode
w
aw
unsafe
+This feature is experimental and may be subject to interface changes. +Memory watchpoints are also architecture dependent.
Kernel 4.16
+These are memory watchpoints provided by the kernel.
+Whenever a memory address is written to (w), read
+from (r), or executed (x), the kernel can generate an event.
Helper bpf_override
In the first form, an absolute address is monitored.
+If a pid (-p) or a command (-c) is provided, bpftrace takes the address as a userspace address and monitors the appropriate process.
+If not, bpftrace takes the address as a kernel space address.
kprobe
-In the second form, the address present in argN when function is entered is
+monitored.
+A pid or command must be provided for this form.
+If synchronous (watchpoint), a SIGSTOP is sent to the tracee upon function entry.
+The tracee will be SIGCONTed after the watchpoint is attached.
+This is to ensure events are not missed.
+If you want to avoid the SIGSTOP + SIGCONT use asyncwatchpoint.
When using override the probed function will not be executed and instead rc will be returned.
Note that on most architectures you may not monitor for execution while monitoring read or write.
kprobe:__x64_sys_getuid
-/comm == "id"/ {
- override(2<<21);
-}
+# bpftrace -e 'watchpoint:0x10000000:8:rw { printf("hit!\n"); }' -c ./testprogs/watchpoint
Print the call stack every time the jiffies variable is updated:
uid=4194304 gid=0(root) euid=0(root) groups=0(root)+
watchpoint:0x$(awk '$3 == "jiffies" {print $1}' /proc/kallsyms):8:w {
+ @[kstack] = count();
+}
This feature only works on kernels compiled with CONFIG_BPF_KPROBE_OVERRIDE and only works on functions tagged ALLOW_ERROR_INJECTION.
"hit" and exit when the memory pointed to by arg1 of increment is written to:
# cat wpfunc.c
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+
+__attribute__((noinline))
+void increment(__attribute__((unused)) int _, int *i)
+{
+ (*i)++;
+}
+
+int main()
+{
+ int *i = malloc(sizeof(int));
+ while (1)
+ {
+ increment(0, i);
+ (*i)++;
+ usleep(1000);
+ }
+}bpftrace does not test whether error injection is allowed for the probed function, instead if will fail to load the program into the kernel:
ioctl(PERF_EVENT_IOC_SET_BPF): Invalid argument -Error attaching probe: 'kprobe:vfs_read'+
# bpftrace -e 'watchpoint:increment+arg1:4:w { printf("hit!\n"); exit() }' -c ./wpfunc
+Builtins are special variables built into the language.
+Unlike scratch and map variables they don’t need a $ or @ as prefix (except for the positional parameters).
+The 'Kernel' column indicates the minimum kernel version required and the 'BPF Helper' column indicates the raw BPF helper function used for this builtin.
| Variable | +Type | +Kernel | +BPF Helper | +Description | +
|---|---|---|---|---|
| + | int64 |
+n/a |
+n/a |
+The nth positional parameter passed to the bpftrace program.
+If less than n parameters are passed this evaluates to |
+
|
+int64 |
+n/a |
+n/a |
+Total amount of positional parameters passed. |
+
|
+int64 |
+n/a |
+n/a |
+nth argument passed to the function being traced. These are extracted from the CPU registers. The amount of args passed in registers depends on the CPU architecture. (kprobes, uprobes, usdt). |
+
|
+struct args |
+n/a |
+n/a |
+The struct of all arguments of the traced function. Available in |
+
cgroup |
+uint64 |
+4.18 |
+get_current_cgroup_id |
+ID of the cgroup the current process belongs to. Only works with cgroupv2. |
+
comm |
+string[16] |
+4.2 |
+get_current_comm |
+Name of the current thread |
+
cpid |
+uint32 |
+n/a |
+n/a |
+Child process ID, if bpftrace is invoked with |
+
cpu |
+uint32 |
+4.1 |
+raw_smp_processor_id |
+ID of the processor executing the BPF program |
+
curtask |
+uint64 |
+4.8 |
+get_current_task |
+Pointer to |
+
elapsed |
+uint64 |
+(see nsec) |
+ktime_get_ns / ktime_get_boot_ns |
+Nanoseconds elapsed since bpftrace initialization, based on |
+
func |
+string |
+n/a |
+n/a |
+Name of the current function being traced (kprobes,uprobes) |
+
gid |
+uint64 |
+4.2 |
+get_current_uid_gid |
+Group ID of the current thread, as seen from the init namespace |
+
jiffies |
+uint64 |
+5.9 |
+get_jiffies_64 |
+Jiffies of the kernel. In 32-bit system, using this builtin might be slower. |
+
numaid |
+uint32 |
+5.8 |
+numa_node_id |
+ID of the NUMA node executing the BPF program |
+
pid |
+uint32 |
+4.2 |
+get_current_pid_tgid |
+Process ID of the current thread (aka thread group ID), as seen from the init namespace |
+
probe |
+string |
+n/na |
+n/a |
+Name of the current probe |
+
rand |
+uint32 |
+4.1 |
+get_prandom_u32 |
+Random number |
+
return |
+n/a |
+n/a |
+n/a |
+The return keyword is used to exit the current probe. This differs from exit() in that it doesn’t exit bpftrace. |
+
retval |
+uint64 |
+n/a |
+n/a |
+Value returned by the function being traced (kretprobe, uretprobe, fexit). For kretprobe and uretprobe, its type is |
+
tid |
+uint32 |
+4.2 |
+get_current_pid_tgid |
+Thread ID of the current thread, as seen from the init namespace |
+
uid |
+uint64 |
+4.2 |
+get_current_uid_gid |
+User ID of the current thread, as seen from the init namespace |
+
char * path(struct path * path [, int32 size])
$1, $2, …, $N, $#
Kernel 5.10
-Helper bpf_d_path
Return full path referenced by struct path pointer in argument. If size is set,
-the path will be clamped by size otherwise BPFTRACE_MAX_STRLEN is used.
If size is smaller than the resolved path, the resulting string will be truncated at the front rather than at the end.
This function can only be used by functions that are allowed to, these functions are contained in the btf_allowlist_d_path set in the kernel.
void *percpu_kaddr(const string name)
void *percpu_kaddr(const string name, int cpu)
These are the positional parameters to the bpftrace program, also referred to as command line arguments.
+If the parameter is numeric (entirely digits), it can be used as a number.
+If it is non-numeric, it must be used as a string in the str() call.
+If a parameter is used that was not provided, it will default to zero for numeric context, and "" for string context.
+Positional parameters may also be used in probe argument and will be treated as a string parameter.
sync
+If a positional parameter is used in str(), it is interpreted as a pointer to the actual given string literal, which allows to do pointer arithmetic on it.
+Only addition of a single constant, less or equal to the length of the supplied string, is allowed.
Get the address of the percpu kernel symbol name for CPU cpu. When cpu is
-omitted, the current CPU is used.
interval:s:1 {
- $proc_cnt = percpu_kaddr("process_counts");
- printf("% processes are running on CPU %d\n", *$proc_cnt, cpu);
-}
-$# returns the number of positional arguments supplied.
The second variant may return NULL if cpu is higher than the number of
-available CPUs. Therefore, it is necessary to perform a NULL-check on the result
-when accessing fields of the pointed structure, otherwise the BPF program will
-be rejected.
This allows scripts to be written that use basic arguments to change their behavior. +If you develop a script that requires more complex argument processing, it may be better suited for bcc instead, which +supports Python’s argparse and completely custom argument processing.
interval:s:1 {
- $runqueues = (struct rq *)percpu_kaddr("runqueues", 0);
- if ($runqueues != 0) { // The check is mandatory here
- print($runqueues->nr_running);
- }
-}
-void print(T val)
async
+# bpftrace -e 'BEGIN { printf("I got %d, %s (%d args)\n", $1, str($2), $#); }' 42 "hello"
+
+I got 42, hello (2 args)
+
+# bpftrace -e 'BEGIN { printf("%s\n", str($1 + 1)) }' "hello"
+
+ello
void print(T val)
void print(@map)
void print(@map, uint64 top)
void print(@map, uint64 top, uint64 div)
print prints a the value, which can be a map or a scalar value, with the default formatting for the type.
Script example, bsize.bt:
interval:s:1 {
- print(123);
- print("abc");
- exit();
+#!/usr/local/bin/bpftrace
+
+BEGIN
+{
+ printf("Tracing block I/O sizes > %d bytes\n", $1);
}
-/*
- * Sample output:
- * 123
- * abc
- */
-interval:ms:10 { @=hist(rand); }
-interval:s:1 {
- print(@);
- exit();
+tracepoint:block:block_rq_issue
+/args.bytes > $1/
+{
+ @ = hist(args.bytes);
}
Prints:
-@: -[16M, 32M) 3 |@@@ | -[32M, 64M) 2 |@@ | -[64M, 128M) 1 |@ | -[128M, 256M) 4 |@@@@ | -[256M, 512M) 3 |@@@ | -[512M, 1G) 14 |@@@@@@@@@@@@@@ | -[1G, 2G) 22 |@@@@@@@@@@@@@@@@@@@@@@ | -[2G, 4G) 51 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|-
Declared maps and histograms are automatically printed out on program termination.
-Note that maps are printed by reference while scalar values are copied. -This means that updating and printing maps in a fast loop will likely result in bogus map values as the map will be updated before userspace gets the time to dump and print it.
-The printing of maps supports the optional top and div arguments.
-top limits the printing to the top N entries with the highest integer values
When run with a 65536 argument:
BEGIN {
- $i = 11;
- while($i) {
- @[$i] = --$i;
- }
- print(@, 2);
- clear(@);
- exit()
-}
+# ./bsize.bt 65536
-/*
- * Sample output:
- * @[9]: 9
- * @[10]: 10
- */
-The div argument scales the values prior to printing them.
-Scaling values before storing them can result in rounding errors.
-Consider the following program:
kprobe:f {
- @[func] += arg0/10;
-}
+Tracing block I/O sizes > 65536 bytes
+^C
+
+@:
+[512K, 1M) 1 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
With the following sequence as numbers for arg0: 134, 377, 111, 99.
-The total is 721 which rounds to 72 when scaled by 10 but the program would print 70 due to the rounding of individual values.
It has passed the argument in as $1 and used it as a filter.
Changing the print call to print(@, 5, 2) will take the top 5 values and scale them by 2:
With no arguments, $1 defaults to zero:
@[6]: 3 -@[7]: 3 -@[8]: 4 -@[9]: 4 -@[10]: 5-
void printf(const string fmt, args…)
# ./bsize.bt +Attaching 2 probes... +Tracing block I/O sizes > 0 bytes +^C + +@: +[4K, 8K) 115 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| +[8K, 16K) 35 |@@@@@@@@@@@@@@@ | +[16K, 32K) 5 |@@ | +[32K, 64K) 3 |@ | +[64K, 128K) 1 | | +[128K, 256K) 0 | | +[256K, 512K) 0 | | +[512K, 1M) 1 | |
async
printf() formats and prints data.
-It behaves similar to printf() found in C and many other languages.
The format string has to be a constant, it cannot be modified at runtime. -The formatting of the string happens in user space. -Values are copied and passed by value.
bpftrace supports all the typical format specifiers like %llx and %hhu.
-The non-standard ones can be found in the table below:
| Specifier | -Type | +Name | Description | +Sync/Async/Compile Time | |
|---|---|---|---|---|---|
r |
-buffer |
-Hex-formatted string to print arbitrary binary content returned by the buf function. |
++ | Reverse byte order |
+Sync |
rh |
-buffer |
-Prints in hex-formatted string without |
++ | Returns a hex-formatted string of the data pointed to by d |
+Sync |
+
| + | Print file content |
+Async |
+|||
| + | Resolve cgroup ID |
+Compile Time |
+|||
| + | Convert cgroup id to cgroup path |
+Sync |
+|||
| + | Quit bpftrace with an optional exit code |
+Async |
+|||
| + | Print the array |
+Async |
+|||
| + | Resolve kernel symbol name |
+Compile Time |
+|||
| + | Annotate as kernelspace pointer |
+Sync |
+|||
| + | Kernel stack trace |
+Sync |
+|||
| + | Resolve kernel address |
+Async |
+|||
| + | Count ustack/kstack frames |
+Sync |
+|||
| + | Convert MAC address data |
+Sync |
+|||
| + | Timestamps and Time Deltas |
+Sync |
+|||
| + | Convert IP address data to text |
+Sync |
+|||
| + | Offset of element in structure |
+Compile Time |
+|||
| + | Override return value |
+Sync |
+|||
| + | Return full path |
+Sync |
+|||
| + | Resolve percpu kernel symbol name |
+Sync |
+|||
| + | Print a non-map value with default formatting |
+Async |
+|||
| + | Print formatted |
+Async |
+|||
| + | Convert text IP address to byte array |
+Compile Time |
+|||
| + | Returns the value stored in the named register |
+Sync |
+|||
| + | Send a signal to the current process |
+Sync |
+|||
| + | Return size of a type or expression |
+Sync |
+|||
| + | Write skb 's data section into a PCAP file |
+Async |
+|||
| + | Returns the string pointed to by s |
+Sync |
+|||
| + | Compares whether the string haystack contains the string needle. |
+Sync |
+|||
| + | Get error message for errno code |
+Sync |
+|||
| + | Return a formatted timestamp |
+Async |
+|||
| + | Compare first n characters of two strings |
+Sync |
+|||
| + | Execute shell command |
+Async |
+|||
| + | Print formatted time |
+Async |
+|||
| + | Resolve user-level symbol name |
+Compile Time |
+|||
| + | Annotate as userspace pointer |
+Sync |
+|||
| + | User stack trace |
+Sync |
+|||
| + | Resolve user space address |
+Async |
printf() can also symbolize enums as strings. User defined enums as well as enums
-defined in the kernel are supported. For example:
enum custom {
- CUSTOM_ENUM = 3,
-};
-
-BEGIN {
- $r = SKB_DROP_REASON_SOCKET_FILTER;
- printf("%d, %s, %s\n", $r, $r, CUSTOM_ENUM);
- exit();
-}
-yields:
-6, SKB_DROP_REASON_SOCKET_FILTER, CUSTOM_ENUM-
Colors are supported too, using standard terminal escape sequences:
-print("\033[31mRed\t\033[33mYellow\033[0m\n")
-char addr[4] pton(const string *addr_v4)
char addr[16] pton(const string *addr_v6)
compile time
+Functions that are marked async are asynchronous which can lead to unexpected behaviour, see the Invocation Mode section for more information.
pton converts a text representation of an IPv4 or IPv6 address to byte array.
-pton infers the address family based on . or : in the given argument.
-pton comes in handy when we need to select packets with certain IP addresses.
compile time functions are evaluated at compile time, a static value will be compiled into the program.
unsafe functions can have dangerous side effects and should be used with care, the --unsafe flag is required for use.
uint64 reg(const string name)
uint8 bswap(uint8 n)
kprobe
+uint16 bswap(uint16 n)
uprobe
+uint32 bswap(uint32 n)
uint64 bswap(uint64 n)
Get the contents of the register identified by name.
-Valid names depend on the CPU architecture.
bswap reverses the order of the bytes in integer n. In case of 8 bit integers, n is returned without being modified.
+The return type is an unsigned integer of the same width as n.
void signal(const string sig)
void signal(uint32 signum)
buffer buf(void * data, [int64 length])
unsafe
-Kernel 5.3
+buf reads length amount of bytes from address data.
+The maximum value of length is limited to the BPFTRACE_MAX_STRLEN variable.
+For arrays the length is optional, it is automatically inferred from the signature.
Helper bpf_send_signal
buf is address space aware and will call the correct helper based on the address space associated with data.
Probe types: k(ret)probe, u(ret)probe, USDT, profile
+The buffer object returned by buf can safely be printed as a hex encoded string with the %r format specifier.
Send a signal to the process being traced.
-The signal can either be identified by name, e.g. SIGSTOP or by ID, e.g. 19 as found in kill -l.
Bytes with values >=32 and <=126 are printed using their ASCII character, other bytes are printed in hex form (e.g. \x00). The %rx format specifier can be used to print everything in hex form, including ASCII characters. The similar %rh format specifier prints everything in hex form without \x and with spaces between bytes (e.g. 0a fe).
kprobe:__x64_sys_execve
-/comm == "bash"/ {
- signal(5);
+interval:s:1 {
+ printf("%r\n", buf(kaddr("avenrun"), 8));
}
$ ls -Trace/breakpoint trap (core dumped)-
uint64 sizeof(TYPE)
uint64 sizeof(EXPRESSION)
compile time
+\x00\x03\x00\x00\x00\x00\x00\x00 +\xc2\x02\x00\x00\x00\x00\x00\x00
Returns size of the argument in bytes.
-Similar to C/C++ sizeof operator.
-Note that the expression does not get evaluated.
uint32 skboutput(const string path, struct sk_buff *skb, uint64 length, const uint64 offset)
void cat(string namefmt, […args])
Kernel 5.5
-Helper bpf_skb_output
-Write sk_buff skb 's data section to a PCAP file in the path, starting from offset to offset + length.
The PCAP file is encapsulated in RAW IP, so no ethernet header is included.
-The data section in the struct skb may contain ethernet header in some kernel contexts, you may set offset to 14 bytes to exclude ethernet header.
Each packet’s timestamp is determined by adding nsecs and boot time, the accuracy varies on different kernels, see nsecs.
This function returns 0 on success, or a negative error in case of failure.
-Environment variable BPFTRACE_PERF_RB_PAGES should be increased in order to capture large packets, or else these packets will be dropped.
async
Usage
+Dump the contents of the named file to stdout.
+cat supports the same format string and arguments that printf does.
+If the file cannot be opened or read an error is printed to stderr.
# cat dump.bt
-fentry:napi_gro_receive {
- $ret = skboutput("receive.pcap", args.skb, args.skb->len, 0);
-}
-
-fentry:dev_queue_xmit {
- // setting offset to 14, to exclude ethernet header
- $ret = skboutput("output.pcap", args.skb, args.skb->len, 14);
- printf("skboutput returns %d\n", $ret);
-}
-
-# export BPFTRACE_PERF_RB_PAGES=1024
-# bpftrace dump.bt
-...
-
-# tcpdump -n -r ./receive.pcap | head -3
-reading from file ./receive.pcap, link-type RAW (Raw IP)
-dropped privs to tcpdump
-10:23:44.674087 IP 22.128.74.231.63175 > 192.168.0.23.22: Flags [.], ack 3513221061, win 14009, options [nop,nop,TS val 721277750 ecr 3115333619], length 0
-10:23:45.823194 IP 100.101.2.146.53 > 192.168.0.23.46619: 17273 0/1/0 (130)
-10:23:45.823229 IP 100.101.2.146.53 > 192.168.0.23.46158: 45799 1/0/0 A 100.100.45.106 (60)
-string str(char * data [, uint32 length)
tracepoint:syscalls:sys_enter_execve {
+ cat("/proc/%d/maps", pid);
+}
Helper probe_read_str, probe_read_{kernel,user}_str
str reads a NULL terminated (\0) string from data.
-The maximum string length is limited by the BPFTRACE_MAX_STRLEN env variable, unless length is specified and shorter than the maximum.
-In case the string is longer than the specified length only length - 1 bytes are copied and a NULL byte is appended at the end.
55f683ebd000-55f683ec1000 r--p 00000000 08:01 1843399 /usr/bin/ls +55f683ec1000-55f683ed6000 r-xp 00004000 08:01 1843399 /usr/bin/ls +55f683ed6000-55f683edf000 r--p 00019000 08:01 1843399 /usr/bin/ls +55f683edf000-55f683ee2000 rw-p 00021000 08:01 1843399 /usr/bin/ls +55f683ee2000-55f683ee3000 rw-p 00000000 00:00 0
When available (starting from kernel 5.5, see the --info flag) bpftrace will automatically use the kernel or user variant of probe_read_{kernel,user}_str based on the address space of data, see [Address-spaces] for more information.
int64 strcontains(const char *haystack, const char *needle)
uint64 cgroupid(const string path)
strcontains compares whether the string haystack contains the string needle.
-If needle is contained 1 is returned, else zero is returned.
bpftrace doesn’t read past the length of the shortest string.
-strerror_t strerror(int error)
compile time
Convert errno code to string. -This is done asynchronously in userspace when the strerror value is printed, hence the returned value can only be used for printing.
+cgroupid retrieves the cgroupv2 ID of the cgroup available at path.
#include <errno.h>
-BEGIN {
- print(strerror(EPERM));
+BEGIN {
+ print(cgroupid("/sys/fs/cgroup/system.slice"));
}
timestamp strftime(const string fmt, int64 timestamp_ns)
cgroup_path_t cgroup_path(int cgroupid, string filter)
async
+Convert cgroup id to cgroup path. +This is done asynchronously in userspace when the cgroup_path value is printed, +therefore it can resolve to a different value if the cgroup id gets reassigned. +This also means that the returned value can only be used for printing.
Format the nanoseconds since boot timestamp timestamp_ns according to the format specified by fmt.
-The time conversion and formatting happens in user space, therefore the timestamp value returned can only be used for printing using the %s format specifier.
A string literal may be passed as an optional second argument to filter cgroup +hierarchies in which the cgroup id is looked up by a wildcard expression (cgroup2 +is always represented by "unified", regardless of where it is mounted).
bpftrace uses the strftime(3) function for formatting time and supports the same format specifiers.
The currently mounted hierarchy at /sys/fs/cgroup is used to do the lookup. If +the cgroup with the given id isn’t present here (e.g. when running in a Docker +container), the cgroup path won’t be found (unlike when looking up the cgroup +path of a process via /proc/…/cgroup).
interval:s:1 {
- printf("%s\n", strftime("%H:%M:%S", nsecs));
+BEGIN {
+ $cgroup_path = cgroup_path(3436);
+ print($cgroup_path);
+ print($cgroup_path); /* This may print a different path */
+ printf("%s %s", $cgroup_path, $cgroup_path); /* This may print two different paths */
}
bpftrace also supports the following format string extensions:
-| Specifier | -Description | -
|---|---|
|
-Microsecond as a decimal number, zero-padded on the left |
-
int64 strncmp(char * s1, char * s2, int64 n)
strncmp compares up to n characters string s1 and string s2.
-If they’re equal 0 is returned, else a non-zero value is returned.
bpftrace doesn’t read past the length of the shortest string.
-The use of the == and != operators is recommended over calling strncmp directly.
void system(string namefmt [, …args])
void exit([int code])
unsafe -async
+async
system lets bpftrace run the specified command (fork and exec) until it completes and print its stdout.
-The command is run with the same privileges as bpftrace and it blocks execution of the processing threads which can lead to missed events and delays processing of async events.
Terminate bpftrace, as if a SIGTERM was received.
+The END probe will still trigger (if specified) and maps will be printed.
+An optional exit code can be provided.
interval:s:1 {
- time("%H:%M:%S: ");
- printf("%d\n", @++);
-}
-interval:s:10 {
- system("/bin/sleep 10");
-}
-interval:s:30 {
+BEGIN {
exit();
}
Note how the async time and printf first print every second until the interval:s:10 probe hits, then they print every 10 seconds due to bpftrace blocking on sleep.
Or
Attaching 3 probes... -08:50:37: 0 -08:50:38: 1 -08:50:39: 2 -08:50:40: 3 -08:50:41: 4 -08:50:42: 5 -08:50:43: 6 -08:50:44: 7 -08:50:45: 8 -08:50:46: 9 -08:50:56: 10 -08:50:56: 11 -08:50:56: 12 -08:50:56: 13 -08:50:56: 14 -08:50:56: 15 -08:50:56: 16 -08:50:56: 17 -08:50:56: 18 -08:50:56: 19+
BEGIN {
+ exit(1);
+}
void join(char *arr[], [char * sep = ' '])
system supports the same format string and arguments that printf does.
async
+join joins all the string array arr with sep as separator into one string.
+This string will be printed to stdout directly, it cannot be used as string value.
The concatenation of the array members is done in BPF and the printing happens in userspace.
tracepoint:syscalls:sys_enter_execve {
- system("/bin/grep %s /proc/%d/status", "vmswap", pid);
+ join(args.argv);
}
void time(const string fmt)
uint64 kaddr(const string name)
async
+compile time
Format the current wall time according to the format specifier fmt and print it to stdout.
-Unlike strftime() time() doesn’t send a timestamp from the probe, instead it is the time at which user-space processes the event.
Get the address of the kernel symbol name.
interval:s:1 {
+ $avenrun = kaddr("avenrun");
+ $load1 = *$avenrun;
+}
+bpftrace uses the strftime(3) function for formatting time and supports the same format specifiers.
You can find all kernel symbols at /proc/kallsyms.
T * uaddr(const string sym)
T * kptr(T * ptr)
Marks ptr as a kernel address space pointer.
+See the address-spaces section for more information on address-spaces.
+The pointer type is left unchanged.
uprobes
-uretprobes
-USDT
+kstack_t kstack([StackMode mode, ][int limit])
Does not work with ASLR, see issue #75
+These are implemented using BPF stack maps.
+kprobe:ip_output { @[kstack()] = count(); }
+
+/*
+ * Sample output:
+ * @[
+ * ip_output+1
+ * tcp_transmit_skb+1308
+ * tcp_write_xmit+482
+ * tcp_release_cb+225
+ * release_sock+64
+ * tcp_sendmsg+49
+ * sock_sendmsg+48
+ * sock_write_iter+135
+ * __vfs_write+247
+ * vfs_write+179
+ * sys_write+82
+ * entry_SYSCALL_64_fastpath+30
+ * ]: 1708
+ */
+The uaddr function returns the address of the specified symbol.
-This lookup happens during program compilation and cannot be used dynamically.
Sampling only three frames from the stack (limit = 3):
+kprobe:ip_output { @[kstack(3)] = count(); }
+
+/*
+ * Sample output:
+ * @[
+ * ip_output+1
+ * tcp_transmit_skb+1308
+ * tcp_write_xmit+482
+ * ]: 1708
+ */
+The default return type is uint64*.
-If the ELF object size matches a known integer size (1, 2, 4 or 8 bytes) the return type is modified to match the width (uint8*, uint16*, uint32* or uint64* resp.).
-As ELF does not contain type info the type is always assumed to be unsigned.
You can also choose a different output format.
+Available formats are bpftrace, perf, and raw (no symbolication):
uprobe:/bin/bash:readline {
- printf("PS1: %s\n", str(*uaddr("ps1_prompt")));
-}
+kprobe:ip_output { @[kstack(perf, 3)] = count(); }
+
+/*
+ * Sample output:
+ * @[
+ * ffffffffb4019501 do_mmap+1
+ * ffffffffb401700a sys_mmap_pgoff+266
+ * ffffffffb3e334eb sys_mmap+27
+ * ]: 1708
+ */
T * uptr(T * ptr)
ksym_t ksym(uint64 addr)
Marks ptr as a user address space pointer.
-See the address-spaces section for more information on address-spaces.
-The pointer type is left unchanged.
async
+Retrieve the name of the function that contains address addr.
+The address to name mapping happens in user-space.
The ksym_t type can be printed with the %s format specifier.
kprobe:do_nanosleep
+{
+ printf("%s\n", ksym(reg("ip")));
+}
+
+/*
+ * Sample output:
+ * do_nanosleep
+ */
+ustack_t ustack([StackMode mode, ][int limit])
int64 len(ustack stack)
int64 len(kstack stack)
These are implemented using BPF stack maps.
+Retrieve the depth (measured in # of frames) of the call stack
+specified by stack.
kprobe:do_sys_open /comm == "bash"/ { @[ustack()] = count(); }
-
-/*
- * Sample output:
- * @[
- * __open_nocancel+65
- * command_word_completion_function+3604
- * rl_completion_matches+370
- * bash_default_completion+540
- * attempt_shell_completion+2092
- * gen_completion_matches+82
- * rl_complete_internal+288
- * rl_complete+145
- * _rl_dispatch_subseq+647
- * _rl_dispatch+44
- * readline_internal_char+479
- * readline_internal_charloop+22
- * readline_internal+23
- * readline+91
- * yy_readline_get+152
- * yy_readline_get+429
- * yy_getc+13
- * shell_getc+469
- * read_token+251
- * yylex+192
- * yyparse+777
- * parse_command+126
- * read_command+207
- * reader_loop+391
- * main+2409
- * __libc_start_main+231
- * 0x61ce258d4c544155
- * ]: 9
- */
macaddr_t macaddr(char [6] mac)
Sampling only three frames from the stack (limit = 3):
+Create a buffer that holds a macaddress as read from mac
+This buffer can be printed in the canonical string format using the %s format specifier.
kprobe:ip_output { @[ustack(3)] = count(); }
+kprobe:arp_create {
+ $stack_arg0 = *(uint8*)(reg("sp") + 8);
+ $stack_arg1 = *(uint8*)(reg("sp") + 16);
+ printf("SRC %s, DST %s\n", macaddr($stack_arg0), macaddr($stack_arg1));
+}
/*
* Sample output:
- * @[
- * __open_nocancel+65
- * command_word_completion_function+3604
- * rl_completion_matches+370
- * ]: 20
+ * SRC 18:C0:4D:08:2E:BB, DST 74:83:C2:7F:8C:FF
*/
timestamp nsecs([TimestampMode mode])
You can also choose a different output format.
-Available formats are bpftrace, perf, and raw (no symbolication):
Returns a timestamp in nanoseconds, as given by the requested kernel clock.
+Defaults to boot if no clock is explicitly requested.
nsecs(monotonic) - nanosecond timestamp since boot, exclusive of time the system spent suspended (CLOCK_MONOTONIC)
nsecs(boot) - nanoseconds since boot, inclusive of time the system spent suspended (CLOCK_BOOTTIME)
nsecs(tai) - TAI timestamp in nanoseconds (CLOCK_TAI)
nsecs(sw_tai) - approximation of TAI timestamp in nanoseconds, is obtained through the "triple vdso sandwich" method. For older kernels without direct TAI timestamp access in BPF.
kprobe:ip_output { @[ustack(perf, 3)] = count(); }
+interval:s:1 {
+ $sw_tai1 = nsecs(sw_tai);
+ $tai = nsecs(tai);
+ $sw_tai2 = nsecs(sw_tai);
+ printf("sw_tai precision: %lldns\n", ($sw_tai1 + $sw_tai2)/2 - $tai);
+}
/*
* Sample output:
- * @[
- * 5649feec4090 readline+0 (/home/mmarchini/bash/bash/bash)
- * 5649fee2bfa6 yy_readline_get+451 (/home/mmarchini/bash/bash/bash)
- * 5649fee2bdc6 yy_getc+13 (/home/mmarchini/bash/bash/bash)
- * ]: 20
+ * sw_tai precision: -98ns
+ * sw_tai precision: -99ns
+ * ...
*/
Note that for these examples to work, bash had to be recompiled with frame pointers.
-usym_t usym(uint64 * addr)
inet ntop([int64 af, ] int addr)
inet ntop([int64 af, ] char addr[4])
inet ntop([int64 af, ] char addr[16])
async
+ntop returns the string representation of an IPv4 or IPv6 address.
+ntop will infer the address type (IPv4 or IPv6) based on the addr type and size.
+If an integer or char[4] is given, ntop assumes IPv4, if a char[16] is given, ntop assumes IPv6.
+You can also pass the address type (e.g. AF_INET) explicitly as the first parameter.
uprobes
+uint64 offsetof(STRUCT, FIELD[.SUBFIELD])
uretprobes
+uint64 offsetof(EXPRESSION, FIELD[.SUBFIELD])
Equal to ksym but resolves user space symbols.
+compile time
If ASLR is enabled, user space symbolication only works when the process is running at either the time of the symbol resolution or the time of the probe attachment. The latter requires BPFTRACE_CACHE_USER_SYMBOLS to be set to PER_PID, and might not work with older versions of BCC. A similar limitation also applies to dynamically loaded symbols.
Returns offset of the field offset bytes in struct.
+Similar to kernel offsetof operator.
Support any number of sub field levels, for example:
uprobe:/bin/bash:readline
-{
- printf("%s\n", usym(reg("ip")));
+struct Foo {
+ struct {
+ struct {
+ struct {
+ int d;
+ } c;
+ } b;
+ } a;
}
-
-/*
- * Sample output:
- * readline
- */
+BEGIN {
+ @x = offsetof(struct Foo, a.b.c.d);
+ exit();
+}
void unwatch(void * addr)
void override(uint64 rc)
async
+unsafe
Removes a watchpoint
+Kernel 4.16
+Helper bpf_override
kprobe
+When using override the probed function will not be executed and instead rc will be returned.
kprobe:__x64_sys_getuid
+/comm == "id"/ {
+ override(2<<21);
+}
+uid=4194304 gid=0(root) euid=0(root) groups=0(root)
Map functions are built-in functions who’s return value can only be assigned to maps. -The data type associated with these functions are only for internal use and are not compatible with the (integer) operators.
+This feature only works on kernels compiled with CONFIG_BPF_KPROBE_OVERRIDE and only works on functions tagged ALLOW_ERROR_INJECTION.
Functions that are marked async are asynchronous which can lead to unexpected behavior, see the Invocation Mode section for more information.
+bpftrace does not test whether error injection is allowed for the probed function, instead if will fail to load the program into the kernel:
+ioctl(PERF_EVENT_IOC_SET_BPF): Invalid argument +Error attaching probe: 'kprobe:vfs_read'+
char * path(struct path * path [, int32 size])
See Advanced Topics for more information on Map Printing.
+Kernel 5.10
+Helper bpf_d_path
Return full path referenced by struct path pointer in argument. If size is set,
+the path will be clamped by size otherwise BPFTRACE_MAX_STRLEN is used.
If size is smaller than the resolved path, the resulting string will be truncated at the front rather than at the end.
This function can only be used by functions that are allowed to, these functions are contained in the btf_allowlist_d_path set in the kernel.
| Name | -Description | -Sync/async | -
|---|---|---|
| - | Calculate the running average of |
-Sync |
-
| - | Clear all keys/values from a map. |
-Async |
-
| - | Count how often this function is called. |
-Sync |
-
| - | Delete a single key from a map. |
-Sync |
-
| - | Return true (1) if the key exists in this map. Otherwise return false (0). |
-Sync |
-
| - | Create a log2 histogram of n using buckets per power of 2, 0 ⇐ k ⇐ 5, defaults to 0. |
-Sync |
-
| - | Return the number of elements in a map. |
-Sync |
-
| - | Create a linear histogram of n. lhist creates M ((max - min) / step) buckets in the range [min,max) where each bucket is step in size. |
-Sync |
-
| - | Update the map with n if n is bigger than the current value held. |
-Sync |
-
| - | Update the map with n if n is smaller than the current value held. |
-Sync |
-
| - | Combines the count, avg and sum calls into one. |
-Sync |
-
| - | Calculate the sum of all n passed. |
-Sync |
-
| - | Set all values for all keys to zero. |
-Async |
-
avg_t avg(int64 n)
void *percpu_kaddr(const string name)
void *percpu_kaddr(const string name, int cpu)
Calculate the running average of n between consecutive calls.
sync
+Get the address of the percpu kernel symbol name for CPU cpu. When cpu is
+omitted, the current CPU is used.
interval:s:1 {
- @x++;
- @y = avg(@x);
- print(@x);
- print(@y);
+ $proc_cnt = percpu_kaddr("process_counts");
+ printf("% processes are running on CPU %d\n", *$proc_cnt, cpu);
}
Internally this keeps two values in the map: value count and running total.
-The average is computed in user-space when printing by dividing the total by the
-count. However, you can get the average in kernel space in expressions like
-if (@y == 5) but this is expensive as bpftrace needs to iterate over all the
-cpus to collect and sum BOTH count and total.
The second variant may return NULL if cpu is higher than the number of
+available CPUs. Therefore, it is necessary to perform a NULL-check on the result
+when accessing fields of the pointed structure, otherwise the BPF program will
+be rejected.
interval:s:1 {
+ $runqueues = (struct rq *)percpu_kaddr("runqueues", 0);
+ if ($runqueues != 0) { // The check is mandatory here
+ print($runqueues->nr_running);
+ }
+}
+void clear(map m)
void print(T val)
async
void print(T val)
void print(@map)
void print(@map, uint64 top)
void print(@map, uint64 top, uint64 div)
Clear all keys/values from map m.
print prints a the value, which can be a map or a scalar value, with the default formatting for the type.
interval:ms:100 {
- @[rand % 10] = count();
+interval:s:1 {
+ print(123);
+ print("abc");
+ exit();
}
-interval:s:10 {
+/*
+ * Sample output:
+ * 123
+ * abc
+ */
+interval:ms:10 { @=hist(rand); }
+interval:s:1 {
print(@);
+ exit();
+}
+Prints:
+@: +[16M, 32M) 3 |@@@ | +[32M, 64M) 2 |@@ | +[64M, 128M) 1 |@ | +[128M, 256M) 4 |@@@@ | +[256M, 512M) 3 |@@@ | +[512M, 1G) 14 |@@@@@@@@@@@@@@ | +[1G, 2G) 22 |@@@@@@@@@@@@@@@@@@@@@@ | +[2G, 4G) 51 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|+
Declared maps and histograms are automatically printed out on program termination.
+Note that maps are printed by reference while scalar values are copied. +This means that updating and printing maps in a fast loop will likely result in bogus map values as the map will be updated before userspace gets the time to dump and print it.
+The printing of maps supports the optional top and div arguments.
+top limits the printing to the top N entries with the highest integer values
BEGIN {
+ $i = 11;
+ while($i) {
+ @[$i] = --$i;
+ }
+ print(@, 2);
clear(@);
+ exit()
+}
+
+/*
+ * Sample output:
+ * @[9]: 9
+ * @[10]: 10
+ */
+The div argument scales the values prior to printing them.
+Scaling values before storing them can result in rounding errors.
+Consider the following program:
kprobe:f {
+ @[func] += arg0/10;
}
With the following sequence as numbers for arg0: 134, 377, 111, 99.
+The total is 721 which rounds to 72 when scaled by 10 but the program would print 70 due to the rounding of individual values.
Changing the print call to print(@, 5, 2) will take the top 5 values and scale them by 2:
@[6]: 3 +@[7]: 3 +@[8]: 4 +@[9]: 4 +@[10]: 5+
count_t count()
void printf(const string fmt, args…)
Count how often this function is called.
+async
Using @=count() is conceptually similar to @++.
-The difference is that the count() function uses a map type optimized for
-performance and correctness using cheap, thread-safe writes (PER_CPU). However, sync reads
-can be expensive as bpftrace needs to iterate over all the cpus to collect and
-sum these values.
printf() formats and prints data.
+It behaves similar to printf() found in C and many other languages.
Note: This differs from "raw" writes (e.g. @++) where multiple writers to a
-shared location might lose updates, as bpftrace does not generate any atomic instructions
-for ++.
The format string has to be a constant, it cannot be modified at runtime. +The formatting of the string happens in user space. +Values are copied and passed by value.
Example one:
+bpftrace supports all the typical format specifiers like %llx and %hhu.
+The non-standard ones can be found in the table below:
| Specifier | +Type | +Description | +
|---|---|---|
r |
+buffer |
+Hex-formatted string to print arbitrary binary content returned by the buf function. |
+
rh |
+buffer |
+Prints in hex-formatted string without |
+
printf() can also symbolize enums as strings. User defined enums as well as enums
+defined in the kernel are supported. For example:
BEGIN {
- @ = count();
- @ = count();
- printf("%d\n", (int64)@); // prints 2
+enum custom {
+ CUSTOM_ENUM = 3,
+};
+
+BEGIN {
+ $r = SKB_DROP_REASON_SOCKET_FILTER;
+ printf("%d, %s, %s\n", $r, $r, CUSTOM_ENUM);
exit();
}
Example two:
+yields:
interval:ms:100 {
- @ = count();
-}
-
-interval:s:10 {
- // async read
- print(@);
- // sync read
- if (@ > 10) {
- print(("hello"));
- }
- clear(@);
-}
+6, SKB_DROP_REASON_SOCKET_FILTER, CUSTOM_ENUM+
Colors are supported too, using standard terminal escape sequences:
+print("\033[31mRed\t\033[33mYellow\033[0m\n")
void delete(map m, mapkey k)
char addr[4] pton(const string *addr_v4)
deprecated void delete(mapkey k)
char addr[16] pton(const string *addr_v6)
Delete a single key from a map.
-For scalar maps (e.g. no explicit keys), the key is omitted and is equivalent to calling clear.
-For map keys that are composed of multiple values (e.g. @mymap[3, "hello"] = 1 - remember these values are represented as a tuple) the syntax would be: delete(@mymap, (3, "hello"));
compile time
The, now deprecated, API (supported in version ⇐ 0.21.x) of passing map arguments with the key is still supported:
-e.g. delete(@mymap[3, "hello"]);.
kprobe:dummy {
- @scalar = 1;
- delete(@scalar); // ok
- @single["hello"] = 1;
- delete(@single, "hello"); // ok
- @associative[1,2] = 1;
- delete(@associative, (1,2)); // ok
- delete(@associative); // error
- delete(@associative, 1); // error
-
- // deprecated but ok
- delete(@single["hello"]);
- delete(@associative[1, 2]);
-}pton converts a text representation of an IPv4 or IPv6 address to byte array.
+pton infers the address family based on . or : in the given argument.
+pton comes in handy when we need to select packets with certain IP addresses.
int has_key(map m, mapkey k)
uint64 reg(const string name)
Return true (1) if the key exists in this map. -Otherwise return false (0). -Error if called with a map that has no keys (aka scalar map). -Return value can also be used for scratch variables and map keys/values.
-kprobe:dummy {
- @associative[1,2] = 1;
- if (!has_key(@associative, (1,3))) { // ok
- print(("bye"));
- }
-
- @scalar = 1;
- if (has_key(@scalar)) { // error
- print(("hello"));
- }
-
- $a = has_key(@associative, (1,2)); // ok
- @b[has_key(@associative, (1,2))] = has_key(@associative, (1,2)); // ok
-}kprobe
+uprobe
+Get the contents of the register identified by name.
+Valid names depend on the CPU architecture.
hist_t hist(int64 n[, int k])
void signal(const string sig)
void signal(uint32 signum)
Create a log2 histogram of n using $2^k$ buckets per power of 2,
-0 ⇐ k ⇐ 5, defaults to 0.
unsafe
+Kernel 5.3
+Helper bpf_send_signal
Probe types: k(ret)probe, u(ret)probe, USDT, profile
+Send a signal to the process being traced.
+The signal can either be identified by name, e.g. SIGSTOP or by ID, e.g. 19 as found in kill -l.
kretprobe:vfs_read {
- @bytes = hist(retval);
+kprobe:__x64_sys_execve
+/comm == "bash"/ {
+ signal(5);
}
Prints:
-@: -[1M, 2M) 3 | | -[2M, 4M) 2 | | -[4M, 8M) 2 | | -[8M, 16M) 6 | | -[16M, 32M) 16 | | -[32M, 64M) 27 | | -[64M, 128M) 48 |@ | -[128M, 256M) 98 |@@@ | -[256M, 512M) 191 |@@@@@@ | -[512M, 1G) 394 |@@@@@@@@@@@@@ | -[1G, 2G) 820 |@@@@@@@@@@@@@@@@@@@@@@@@@@@ |+
$ ls +Trace/breakpoint trap (core dumped)
int64 len(map m)
uint64 sizeof(TYPE)
uint64 sizeof(EXPRESSION)
Return the number of elements in the map.
+compile time
+Returns size of the argument in bytes.
+Similar to C/C++ sizeof operator.
+Note that the expression does not get evaluated.
lhist_t lhist(int64 n, int64 min, int64 max, int64 step)
uint32 skboutput(const string path, struct sk_buff *skb, uint64 length, const uint64 offset)
Create a linear histogram of n.
-lhist creates M ((max - min) / step) buckets in the range [min,max) where each bucket is step in size.
-Values in the range (-inf, min) and (max, inf) get their get their own bucket too, bringing the total amount of buckets created to M+2.
Kernel 5.5
interval:ms:1 {
- @ = lhist(rand %10, 0, 10, 1);
-}
-
-interval:s:5 {
- exit();
-}
+Helper bpf_skb_output
Write sk_buff skb 's data section to a PCAP file in the path, starting from offset to offset + length.
Prints:
+The PCAP file is encapsulated in RAW IP, so no ethernet header is included.
+The data section in the struct skb may contain ethernet header in some kernel contexts, you may set offset to 14 bytes to exclude ethernet header.
Each packet’s timestamp is determined by adding nsecs and boot time, the accuracy varies on different kernels, see nsecs.
This function returns 0 on success, or a negative error in case of failure.
+Environment variable BPFTRACE_PERF_RB_PAGES should be increased in order to capture large packets, or else these packets will be dropped.
Usage
@: -[0, 1) 306 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | -[1, 2) 284 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | -[2, 3) 294 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | -[3, 4) 318 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | -[4, 5) 311 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | -[5, 6) 362 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| -[6, 7) 336 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | -[7, 8) 326 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | -[8, 9) 328 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | -[9, 10) 318 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |+
# cat dump.bt
+fentry:napi_gro_receive {
+ $ret = skboutput("receive.pcap", args.skb, args.skb->len, 0);
+}
+
+fentry:dev_queue_xmit {
+ // setting offset to 14, to exclude ethernet header
+ $ret = skboutput("output.pcap", args.skb, args.skb->len, 14);
+ printf("skboutput returns %d\n", $ret);
+}
+
+# export BPFTRACE_PERF_RB_PAGES=1024
+# bpftrace dump.bt
+...
+
+# tcpdump -n -r ./receive.pcap | head -3
+reading from file ./receive.pcap, link-type RAW (Raw IP)
+dropped privs to tcpdump
+10:23:44.674087 IP 22.128.74.231.63175 > 192.168.0.23.22: Flags [.], ack 3513221061, win 14009, options [nop,nop,TS val 721277750 ecr 3115333619], length 0
+10:23:45.823194 IP 100.101.2.146.53 > 192.168.0.23.46619: 17273 0/1/0 (130)
+10:23:45.823229 IP 100.101.2.146.53 > 192.168.0.23.46158: 45799 1/0/0 A 100.100.45.106 (60)
max_t max(int64 n)
string str(char * data [, uint32 length)
Update the map with n if n is bigger than the current value held.
-Similar to count this uses a PER_CPU map (thread-safe, fast writes, slow reads).
Note: this is different than the typical userspace max() in that bpftrace’s max()
-only takes a single argument. The logical "other" argument to compare to is the value
-in the map the "result" is being assigned to.
For example, compare the two logically equivalent samples (C++ vs bpftrace):
-In C++:
-int x = std::max(3, 33); // x contains 33-
Helper probe_read_str, probe_read_{kernel,user}_str
In bpftrace:
-@x = max(3); -@x = max(33); // @x contains 33-
str reads a NULL terminated (\0) string from data.
+The maximum string length is limited by the BPFTRACE_MAX_STRLEN env variable, unless length is specified and shorter than the maximum.
+In case the string is longer than the specified length only length - 1 bytes are copied and a NULL byte is appended at the end.
Also note that bpftrace takes care to handle the unset case. In other words,
-there is no default value. The first value you pass to max() will always
-be returned.
When available (starting from kernel 5.5, see the --info flag) bpftrace will automatically use the kernel or user variant of probe_read_{kernel,user}_str based on the address space of data, see [Address-spaces] for more information.
min_t min(int64 n)
int64 strcontains(const char *haystack, const char *needle)
Update the map with n if n is smaller than the current value held.
-Similar to count this uses a PER_CPU map (thread-safe, fast writes, slow reads).
strcontains compares whether the string haystack contains the string needle.
+If needle is contained 1 is returned, else zero is returned.
See max() above for how this differs from the typical userspace min().
bpftrace doesn’t read past the length of the shortest string.
stats_t stats(int64 n)
strerror_t strerror(int error)
stats combines the count, avg and sum calls into one.
kprobe:vfs_read {
- @bytes[comm] = stats(arg2);
-}
+
Convert errno code to string. +This is done asynchronously in userspace when the strerror value is printed, hence the returned value can only be used for printing.
@bytes[bash]: count 7, average 1, total 7 -@bytes[sleep]: count 5, average 832, total 4160 -@bytes[ls]: count 7, average 886, total 6208 -@+
#include <errno.h>
+BEGIN {
+ print(strerror(EPERM));
+}
sum_t sum(int64 n)
timestamp strftime(const string fmt, int64 timestamp_ns)
Calculate the sum of all n passed.
Using @=sum(5) is conceptually similar to @+=5.
-The difference is that the sum() function uses a map type optimized for
-performance and correctness using cheap, thread-safe writes (PER_CPU). However, sync reads
-can be expensive as bpftrace needs to iterate over all the cpus to collect and
-sum these values.
async
Note: This differs from "raw" writes (e.g. @+=5) where multiple writers to a
-shared location might lose updates, as bpftrace does not generate any implicit
-atomic operations.
Format the nanoseconds since boot timestamp timestamp_ns according to the format specified by fmt.
+The time conversion and formatting happens in user space, therefore the timestamp value returned can only be used for printing using the %s format specifier.
Example one:
+bpftrace uses the strftime(3) function for formatting time and supports the same format specifiers.
BEGIN {
- @ = sum(5);
- @ = sum(6);
- printf("%d\n", (int64)@); // prints 11
- clear(@);
- exit();
+interval:s:1 {
+ printf("%s\n", strftime("%H:%M:%S", nsecs));
}
Example two:
-interval:ms:100 {
- @ = sum(5);
-}
-
-interval:s:10 {
- // async read
- print(@);
- // sync read
- if (@ > 10) {
- print(("hello"));
- }
- clear(@);
-}
-bpftrace also supports the following format string extensions:
| Specifier | +Description | +
|---|---|
|
+Microsecond as a decimal number, zero-padded on the left |
+
void zero(map m)
int64 strncmp(char * s1, char * s2, int64 n)
async
+strncmp compares up to n characters string s1 and string s2.
+If they’re equal 0 is returned, else a non-zero value is returned.
Set all values for all keys to zero.
-bpftrace doesn’t read past the length of the shortest string.
bpftrace supports various probe types which allow the user to attach BPF programs to different types of events.
-Each probe starts with a provider (e.g. kprobe) followed by a colon (:) separated list of options.
-The amount of options and their meaning depend on the provider and are detailed below.
-The valid values for options can depend on the system or binary being traced, e.g. for uprobes it depends on the binary.
-Also see Listing Probes.
The use of the == and != operators is recommended over calling strncmp directly.
It is possible to associate multiple probes with a single action as long as the action is valid for all specified probes.
-Multiple probes can be specified as a comma (,) separated list:
kprobe:tcp_reset,kprobe:tcp_v4_rcv {
- printf("Entered: %s\n", probe);
-}
+void system(string namefmt [, …args])
unsafe +async
Wildcards are supported too:
+system lets bpftrace run the specified command (fork and exec) until it completes and print its stdout.
+The command is run with the same privileges as bpftrace and it blocks execution of the processing threads which can lead to missed events and delays processing of async events.
kprobe:tcp_* {
- printf("Entered: %s\n", probe);
+interval:s:1 {
+ time("%H:%M:%S: ");
+ printf("%d\n", @++);
+}
+interval:s:10 {
+ system("/bin/sleep 10");
+}
+interval:s:30 {
+ exit();
}
Both can be combined:
+Note how the async time and printf first print every second until the interval:s:10 probe hits, then they print every 10 seconds due to bpftrace blocking on sleep.
kprobe:tcp_reset,kprobe:*socket* {
- printf("Entered: %s\n", probe);
-}
-Most providers also support a short name which can be used instead of the full name, e.g. kprobe:f and k:f are identical.
Probe Name |
-Short Name |
-Description |
-Kernel/User Level |
-
| - | - |
-Built-in events |
-Kernel/User |
-
| - | - |
-Built-in events |
-Kernel/User |
-
| - |
|
-Processor-level events |
-Kernel |
-
| - |
|
-Timed output |
-Kernel/User |
-
| - |
|
-Iterators tracing |
-Kernel |
-
| - |
|
-Kernel functions tracing with BTF support |
-Kernel |
-
| - |
|
-Kernel function start/return |
-Kernel |
-
| - |
|
-Timed sampling |
-Kernel/User |
-
| - |
|
-Kernel static tracepoints with raw arguments |
-Kernel |
-
| - |
|
-Kernel software events |
-Kernel |
-
| - |
|
-Kernel static tracepoints |
-Kernel |
-
| - |
|
-User-level function start/return |
-User |
-
| - |
|
-User-level static tracepoints |
-User |
-
| - |
|
-Memory watchpoints |
-Kernel |
-
These are special built-in events provided by the bpftrace runtime.
-BEGIN is triggered before all other probes are attached.
-END is triggered after all other probes are detached.
Attaching 3 probes... +08:50:37: 0 +08:50:38: 1 +08:50:39: 2 +08:50:40: 3 +08:50:41: 4 +08:50:42: 5 +08:50:43: 6 +08:50:44: 7 +08:50:45: 8 +08:50:46: 9 +08:50:56: 10 +08:50:56: 11 +08:50:56: 12 +08:50:56: 13 +08:50:56: 14 +08:50:56: 15 +08:50:56: 16 +08:50:56: 17 +08:50:56: 18 +08:50:56: 19+
Note that specifying an END probe doesn’t override the printing of 'non-empty' maps at exit.
-To prevent printing all used maps need be cleared in the END probe:
system supports the same format string and arguments that printf does.
END {
- clear(@map1);
- clear(@map2);
+tracepoint:syscalls:sys_enter_execve {
+ system("/bin/grep %s /proc/%d/status", "vmswap", pid);
}
self:signal:SIGUSR1
void time(const string fmt)
These are special built-in events provided by the bpftrace runtime.
-The trigger function is called by the bpftrace runtime when the bpftrace process receives specific events, such as a SIGUSR1 signal.
-When multiple signal handlers are attached to the same signal, only the first one is used.
async
self:signal:SIGUSR1 {
- print("abc");
-}
+Format the current wall time according to the format specifier fmt and print it to stdout.
+Unlike strftime() time() doesn’t send a timestamp from the probe, instead it is the time at which user-space processes the event.
bpftrace uses the strftime(3) function for formatting time and supports the same format specifiers.
hardware:event_name:
hardware:event_name:count
h
T * uaddr(const string sym)
These are the pre-defined hardware events provided by the Linux kernel, as commonly traced by the perf utility. -They are implemented using performance monitoring counters (PMCs): hardware resources on the processor. -There are about ten of these, and they are documented in the perf_event_open(2) man page. -The event names are:
-cpu-cycles or cycles
instructions
cache-references
cache-misses
branch-instructions or branches
branch-misses
bus-cycles
frontend-stalls
uprobes
backend-stalls
uretprobes
ref-cycles
USDT
The count option specifies how many events must happen before the probe fires (sampling interval).
-If count is left unspecified a default value is used.
Does not work with ASLR, see issue #75
This will fire once for every 1,000,000 cache misses.
+The uaddr function returns the address of the specified symbol.
+This lookup happens during program compilation and cannot be used dynamically.
The default return type is uint64*.
+If the ELF object size matches a known integer size (1, 2, 4 or 8 bytes) the return type is modified to match the width (uint8*, uint16*, uint32* or uint64* resp.).
+As ELF does not contain type info the type is always assumed to be unsigned.
hardware:cache-misses:1e6 { @[pid] = count(); }
+uprobe:/bin/bash:readline {
+ printf("PS1: %s\n", str(*uaddr("ps1_prompt")));
+}
interval:us:count
interval:ms:count
interval:s:count
interval:hz:rate
i
T * uptr(T * ptr)
The interval probe fires at a fixed interval as specified by its time spec. -Interval fires on one CPU at a time, unlike profile probes.
-This prints the rate of syscalls per second.
-tracepoint:raw_syscalls:sys_enter { @syscalls = count(); }
-interval:s:1 { print(@syscalls); clear(@syscalls); }
-Marks ptr as a user address space pointer.
+See the address-spaces section for more information on address-spaces.
+The pointer type is left unchanged.
iter:task
iter:task:pin
iter:task_file
iter:task_file:pin
iter:task_vma
iter:task_vma:pin
it
ustack_t ustack([StackMode mode, ][int limit])
Warning this feature is experimental and may be subject to interface changes.
-These are eBPF iterator probes that allow iteration over kernel objects. -Iterator probe can’t be mixed with any other probe, not even another iterator. -Each iterator probe provides a set of fields that could be accessed with the -ctx pointer. Users can display the set of available fields for each iterator via --lv options as described below.
-iter:task { printf("%s:%d\n", ctx->task->comm, ctx->task->pid); }
-
-/*
- * Sample output:
- * systemd:1
- * kthreadd:2
- * rcu_gp:3
- * rcu_par_gp:4
- * kworker/0:0H:6
- * mm_percpu_wq:8
- */
-iter:task_file {
- printf("%s:%d %d:%s\n", ctx->task->comm, ctx->task->pid, ctx->fd, path(ctx->file->f_path));
-}
-
-/*
- * Sample output:
- * systemd:1 1:/dev/null
- * systemd:1 3:/dev/kmsg
- * ...
- * su:1622 2:/dev/pts/1
- * ...
- * bpftrace:1892 2:/dev/pts/1
- * bpftrace:1892 6:anon_inode:bpf-prog
- */
-These are implemented using BPF stack maps.
iter:task_vma {
- printf("%s %d %lx-%lx\n", comm, pid, ctx->vma->vm_start, ctx->vma->vm_end);
-}
+kprobe:do_sys_open /comm == "bash"/ { @[ustack()] = count(); }
/*
* Sample output:
- * bpftrace 119480 55b92c380000-55b92c386000
- * ...
- * bpftrace 119480 7ffd55dde000-7ffd55de2000
+ * @[
+ * __open_nocancel+65
+ * command_word_completion_function+3604
+ * rl_completion_matches+370
+ * bash_default_completion+540
+ * attempt_shell_completion+2092
+ * gen_completion_matches+82
+ * rl_complete_internal+288
+ * rl_complete+145
+ * _rl_dispatch_subseq+647
+ * _rl_dispatch+44
+ * readline_internal_char+479
+ * readline_internal_charloop+22
+ * readline_internal+23
+ * readline+91
+ * yy_readline_get+152
+ * yy_readline_get+429
+ * yy_getc+13
+ * shell_getc+469
+ * read_token+251
+ * yylex+192
+ * yyparse+777
+ * parse_command+126
+ * read_command+207
+ * reader_loop+391
+ * main+2409
+ * __libc_start_main+231
+ * 0x61ce258d4c544155
+ * ]: 9
*/
It’s possible to pin an iterator by specifying the optional probe ':pin' part, that defines the pin file. -It can be specified as an absolute or relative path to /sys/fs/bpf.
+Sampling only three frames from the stack (limit = 3):
iter:task:list { printf("%s:%d\n", ctx->task->comm, ctx->task->pid); }
+kprobe:ip_output { @[ustack(3)] = count(); }
/*
* Sample output:
- * Program pinned to /sys/fs/bpf/list
+ * @[
+ * __open_nocancel+65
+ * command_word_completion_function+3604
+ * rl_completion_matches+370
+ * ]: 20
*/
You can also choose a different output format.
+Available formats are bpftrace, perf, and raw (no symbolication):
iter:task_file:/sys/fs/bpf/files {
- printf("%s:%d %s\n", ctx->task->comm, ctx->task->pid, path(ctx->file->f_path));
-}
+kprobe:ip_output { @[ustack(perf, 3)] = count(); }
/*
* Sample output:
- * Program pinned to /sys/fs/bpf/files
+ * @[
+ * 5649feec4090 readline+0 (/home/mmarchini/bash/bash/bash)
+ * 5649fee2bfa6 yy_readline_get+451 (/home/mmarchini/bash/bash/bash)
+ * 5649fee2bdc6 yy_getc+13 (/home/mmarchini/bash/bash/bash)
+ * ]: 20
*/
Note that for these examples to work, bash had to be recompiled with frame pointers.
+fentry[:module]:fn
fexit[:module]:fn
usym_t usym(uint64 * addr)
f (fentry)
fr (fexit)
async
--info)Kernel features:BTF
+uprobes
Probe types:fentry
+uretprobes
fentry/fexit probes attach to kernel functions similar to kprobe and kretprobe.
-They make use of eBPF trampolines which allow kernel code to call into BPF programs with near zero overhead.
-Originally, these were called kfunc and kretfunc but were later renamed to fentry and fexit to match
-how these are referenced in the kernel and to prevent confusion with BPF Kernel Functions.
-The original names are still supported for backwards compatibility.
fentry/fexit probes make use of BTF type information to derive the type of function arguments at compile time.
-This removes the need for manual type casting and makes the code more resilient against small signature changes in the kernel.
-The function arguments are available in the args struct which can be inspected by doing verbose listing (see Listing Probes).
-These arguments are also available in the return probe (fexit), unlike kretprobe.
# bpftrace -lv 'fentry:tcp_reset' - -fentry:tcp_reset - struct sock * sk - struct sk_buff * skb-
fentry:x86_pmu_stop {
- printf("pmu %s stop\n", str(args.event->pmu->name));
-}
-Equal to ksym but resolves user space symbols.
The fget function takes one argument as file descriptor and you can access it via args.fd and the return value is accessible via retval:
+If ASLR is enabled, user space symbolication only works when the process is running at either the time of the symbol resolution or the time of the probe attachment. The latter requires BPFTRACE_CACHE_USER_SYMBOLS to be set to PER_PID, and might not work with older versions of BCC. A similar limitation also applies to dynamically loaded symbols.
fexit:fget {
- printf("fd %d name %s\n", args.fd, str(retval->f_path.dentry->d_name.name));
+uprobe:/bin/bash:readline
+{
+ printf("%s\n", usym(reg("ip")));
}
/*
* Sample output:
- * fd 3 name ld.so.cache
- * fd 3 name libselinux.so.1
+ * readline
*/
kprobe[:module]:fn
kprobe[:module]:fn+offset
kretprobe[:module]:fn
k
kr
void unwatch(void * addr)
kprobes allow for dynamic instrumentation of kernel functions.
-Each time the specified kernel function is executed the attached BPF programs are ran.
kprobe:tcp_reset {
- @tcp_resets = count()
-}
-Function arguments are available through the argN for register args. Arguments passed on stack are available using the stack pointer, e.g. $stack_arg0 = (int64)reg("sp") + 16.
-Whether arguments passed on stack or in a register depends on the architecture and the number or arguments used, e.g. on x86_64 the first 6 non-floating point arguments are passed in registers and all following arguments are passed on the stack.
-Note that floating point arguments are typically passed in special registers which don’t count as argN arguments which can cause confusion.
-Consider a function with the following signature:
void func(int a, double d, int x)-
Due to d being a floating point, x is accessed through arg1 where one might expect arg2.
bpftrace does not detect the function signature so it is not aware of the argument count or their type. -It is up to the user to perform Type conversion when needed, e.g.
-#include <linux/path.h>
-#include <linux/dcache.h>
-
-kprobe:vfs_open
-{
- printf("open path: %s\n", str(((struct path *)arg0)->dentry->d_name.name));
-}
-Here arg0 was cast as a (struct path *), since that is the first argument to vfs_open. -The struct support is the same as bcc and based on available kernel headers. -This means that many, but not all, structs will be available, and you may need to manually define structs.
+async
If the kernel has BTF (BPF Type Format) data, all kernel structs are always available without defining them. For example:
-kprobe:vfs_open {
- printf("open path: %s\n", str(((struct path *)arg0)->dentry->d_name.name));
-}
-Removes a watchpoint
You can optionally specify a kernel module, either to include BTF data from that module, or to specify that the traced function should come from that module.
kprobe:kvm:x86_emulate_insn
-{
- $ctxt = (struct x86_emulate_ctxt *) arg0;
- printf("eip = 0x%lx\n", $ctxt->eip);
-}
See BTF Support for more details.
+Map functions are built-in functions who’s return value can only be assigned to maps. +The data type associated with these functions are only for internal use and are not compatible with the (integer) operators.
kprobe s are not limited to function entry, they can be attached to any instruction in a function by specifying an offset from the start of the function.
Functions that are marked async are asynchronous which can lead to unexpected behavior, see the Invocation Mode section for more information.
kretprobe s trigger on the return from a kernel function.
-Return probes do not have access to the function (input) arguments, only to the return value (through retval).
-A common pattern to work around this is by storing the arguments in a map on function entry and retrieving in the return probe:
kprobe:d_lookup
-{
- $name = (struct qstr *)arg1;
- @fname[tid] = $name->name;
-}
-
-kretprobe:d_lookup
-/@fname[tid]/
-{
- printf("%-8d %-6d %-16s M %s\n", elapsed / 1e6, pid, comm,
- str(@fname[tid]));
-}
-See Advanced Topics for more information on Map Printing.
| Name | +Description | +Sync/async | +
|---|---|---|
| + | Calculate the running average of |
+Sync |
+
| + | Clear all keys/values from a map. |
+Async |
+
| + | Count how often this function is called. |
+Sync |
+
| + | Delete a single key from a map. |
+Sync |
+
| + | Return true (1) if the key exists in this map. Otherwise return false (0). |
+Sync |
+
| + | Create a log2 histogram of n using buckets per power of 2, 0 ⇐ k ⇐ 5, defaults to 0. |
+Sync |
+
| + | Return the number of elements in a map. |
+Sync |
+
| + | Create a linear histogram of n. lhist creates M ((max - min) / step) buckets in the range [min,max) where each bucket is step in size. |
+Sync |
+
| + | Update the map with n if n is bigger than the current value held. |
+Sync |
+
| + | Update the map with n if n is smaller than the current value held. |
+Sync |
+
| + | Combines the count, avg and sum calls into one. |
+Sync |
+
| + | Calculate the sum of all n passed. |
+Sync |
+
| + | Set all values for all keys to zero. |
+Async |
+
profile:us:count
profile:ms:count
profile:s:count
profile:hz:rate
p
avg_t avg(int64 n)
Profile probes fire on each CPU on the specified interval. -These operate using perf_events (a Linux kernel facility, which is also used by the perf command).
+Calculate the running average of n between consecutive calls.
profile:hz:99 { @[tid] = count(); }
+interval:s:1 {
+ @x++;
+ @y = avg(@x);
+ print(@x);
+ print(@y);
+}
+Internally this keeps two values in the map: value count and running total.
+The average is computed in user-space when printing by dividing the total by the
+count. However, you can get the average in kernel space in expressions like
+if (@y == 5) but this is expensive as bpftrace needs to iterate over all the
+cpus to collect and sum BOTH count and total.
rawtracepoint:event
rt
void clear(map m)
The hook point triggered by tracepoint and rawtracepoint is the same.
-tracepoint and rawtracepoint are nearly identical in terms of functionality.
-The only difference is in the program context.
-rawtracepoint offers raw arguments to the tracepoint while tracepoint applies further processing to the raw arguments.
-The additional processing is defined inside the kernel.
rawtracepoint:block_rq_insert {
- printf("%llx %llx\n", arg0, arg1);
-}
-async
Tracepoint arguments are available via the argN builtins.
-Each arg is a 64-bit integer.
-The available arguments can be found in the relative path of the kernel source code include/trace/events/. For example:
Clear all keys/values from map m.
include/trace/events/block.h -DEFINE_EVENT(block_rq, block_rq_insert, - TP_PROTO(struct request_queue *q, struct request *rq), - TP_ARGS(q, rq) -);+
interval:ms:100 {
+ @[rand % 10] = count();
+}
+
+interval:s:10 {
+ print(@);
+ clear(@);
+}
software:event:
software:event:count
count_t count()
s
Count how often this function is called.
These are the pre-defined software events provided by the Linux kernel, as commonly traced via the perf utility. -They are similar to tracepoints, but there is only about a dozen of these, and they are documented in the perf_event_open(2) man page. -If the count is not provided, a default is used.
+Using @=count() is conceptually similar to @++.
+The difference is that the count() function uses a map type optimized for
+performance and correctness using cheap, thread-safe writes (PER_CPU). However, sync reads
+can be expensive as bpftrace needs to iterate over all the cpus to collect and
+sum these values.
The event names are:
+Note: This differs from "raw" writes (e.g. @++) where multiple writers to a
+shared location might lose updates, as bpftrace does not generate any atomic instructions
+for ++.
cpu-clock or cpu
task-clock
page-faults or faults
context-switches or cs
cpu-migrations
minor-faults
major-faults
alignment-faults
emulation-faults
dummy
bpf-output
Example one:
software:faults:100 { @[comm] = count(); }
+BEGIN {
+ @ = count();
+ @ = count();
+ printf("%d\n", (int64)@); // prints 2
+ exit();
+}
This roughly counts who is causing page faults, by sampling the process name for every one in one hundred faults.
+Example two:
+interval:ms:100 {
+ @ = count();
+}
+
+interval:s:10 {
+ // async read
+ print(@);
+ // sync read
+ if (@ > 10) {
+ print(("hello"));
+ }
+ clear(@);
+}
+tracepoint:subsys:event
void delete(map m, mapkey k)
t
deprecated void delete(mapkey k)
Tracepoints are hooks into events in the kernel.
-Tracepoints are defined in the kernel source and compiled into the kernel binary which makes them a form of static tracing.
-Unlike kprobe s, new tracepoints cannot be added without modifying the kernel.
The advantage of tracepoints is that they generally provide a more stable interface than kprobe s do, they do not depend on the existence of a kernel function.
tracepoint:syscalls:sys_enter_openat {
- printf("%s %s\n", comm, str(args.filename));
-}
-Delete a single key from a map.
+For scalar maps (e.g. no explicit keys), the key is omitted and is equivalent to calling clear.
+For map keys that are composed of multiple values (e.g. @mymap[3, "hello"] = 1 - remember these values are represented as a tuple) the syntax would be: delete(@mymap, (3, "hello"));
Tracepoint arguments are available in the args struct which can be inspected with verbose listing, see the Listing Probes section for more details.
The, now deprecated, API (supported in version ⇐ 0.21.x) of passing map arguments with the key is still supported:
+e.g. delete(@mymap[3, "hello"]);.
# bpftrace -lv "tracepoint:*" +-kprobe:dummy { + @scalar = 1; + delete(@scalar); // ok + @single["hello"] = 1; + delete(@single, "hello"); // ok + @associative[1,2] = 1; + delete(@associative, (1,2)); // ok + delete(@associative); // error + delete(@associative, 1); // error -tracepoint:xhci-hcd:xhci_setup_device_slot - u32 info - u32 info2 - u32 tt_info - u32 state -...
Alternatively members for each tracepoint can be listed from their /format file in /sys.
-Apart from the filename member, we can also print flags, mode, and more. -After the "common" members listed first, the members are specific to the tracepoint.
-uprobe:binary:func
uprobe:binary:func+offset
uprobe:binary:offset
uretprobe:binary:func
int has_key(map m, mapkey k)
Return true (1) if the key exists in this map. +Otherwise return false (0). +Error if called with a map that has no keys (aka scalar map). +Return value can also be used for scratch variables and map keys/values.
+kprobe:dummy {
+ @associative[1,2] = 1;
+ if (!has_key(@associative, (1,3))) { // ok
+ print(("bye"));
+ }
+
+ @scalar = 1;
+ if (has_key(@scalar)) { // error
+ print(("hello"));
+ }
+
+ $a = has_key(@associative, (1,2)); // ok
+ @b[has_key(@associative, (1,2))] = has_key(@associative, (1,2)); // ok
+}u
ur
hist_t hist(int64 n[, int k])
uprobe s or user-space probes are the user-space equivalent of kprobe s.
-The same limitations that apply kprobe and kretprobe also apply to uprobe s and uretprobe s, namely: arguments are available via the argN and sargN builtins and can only be accessed with a uprobe (sargN is more common for older versions of golang).
-retval is the return value for the instrumented function and can only be accessed with a uretprobe.
Create a log2 histogram of n using $2^k$ buckets per power of 2,
+0 ⇐ k ⇐ 5, defaults to 0.
uprobe:/bin/bash:readline { printf("arg0: %d\n", arg0); }
-kretprobe:vfs_read {
+ @bytes = hist(retval);
+}
What does arg0 of readline() in /bin/bash contain? -I don’t know, so I’ll need to look at the bash source code to find out what its arguments are.
When tracing libraries, it is sufficient to specify the library name instead of
-a full path. The path will be then automatically resolved using /etc/ld.so.cache:
Prints:
uprobe:libc:malloc { printf("Allocated %d bytes\n", arg0); }
-@: +[1M, 2M) 3 | | +[2M, 4M) 2 | | +[4M, 8M) 2 | | +[8M, 16M) 6 | | +[16M, 32M) 16 | | +[32M, 64M) 27 | | +[64M, 128M) 48 |@ | +[128M, 256M) 98 |@@@ | +[256M, 512M) 191 |@@@@@@ | +[512M, 1G) 394 |@@@@@@@@@@@@@ | +[1G, 2G) 820 |@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
If the traced binary has DWARF included, function arguments are available in the args struct which can be inspected with verbose listing, see the Listing Probes section for more details.
# bpftrace -lv 'uprobe:/bin/bash:rl_set_prompt' - -uprobe:/bin/bash:rl_set_prompt - const char* prompt
int64 len(map m)
When tracing C++ programs, it’s possible to turn on automatic symbol demangling by using the :cpp prefix:
Return the number of elements in the map.
# bpftrace:cpp:"bpftrace::BPFtrace::add_probe" { ... }
lhist_t lhist(int64 n, int64 min, int64 max, int64 step)
It is important to note that for uretprobe s to work the kernel runs a special helper on user-space function entry which overrides the return address on the stack.
-This can cause issues with languages that have their own runtime like Golang:
Create a linear histogram of n.
+lhist creates M ((max - min) / step) buckets in the range [min,max) where each bucket is step in size.
+Values in the range (-inf, min) and (max, inf) get their get their own bucket too, bringing the total amount of buckets created to M+2.
func myprint(s string) {
- fmt.Printf("Input: %s\n", s)
+interval:ms:1 {
+ @ = lhist(rand %10, 0, 10, 1);
}
-func main() {
- ss := []string{"a", "b", "c"}
- for _, s := range ss {
- go myprint(s)
- }
- time.Sleep(1*time.Second)
+interval:s:5 {
+ exit();
}
Prints:
+# bpftrace -e 'uretprobe:./test:main.myprint { @=count(); }' -c ./test
-runtime: unexpected return pc for main.myprint called from 0x7fffffffe000
-stack: frame={sp:0xc00008cf60, fp:0xc00008cfd0} stack=[0xc00008c000,0xc00008d000)
-fatal error: unknown caller pc
+@: +[0, 1) 306 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | +[1, 2) 284 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | +[2, 3) 294 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | +[3, 4) 318 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | +[4, 5) 311 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | +[5, 6) 362 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| +[6, 7) 336 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | +[7, 8) 326 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | +[8, 9) 328 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ | +[9, 10) 318 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ |
usdt:binary_path:probe_name
usdt:binary_path:[probe_namespace]:probe_name
usdt:library_path:probe_name
usdt:library_path:[probe_namespace]:probe_name
U
max_t max(int64 n)
Where probe_namespace is optional if probe_name is unique within the binary.
+Update the map with n if n is bigger than the current value held.
+Similar to count this uses a PER_CPU map (thread-safe, fast writes, slow reads).
You can target the entire host (or an entire process’s address space by using the -p arg) by using a single wildcard in place of the binary_path/library_path:
usdt:*:loop { printf("hi\n"); }
-Note: this is different than the typical userspace max() in that bpftrace’s max()
+only takes a single argument. The logical "other" argument to compare to is the value
+in the map the "result" is being assigned to.
Please note that if you use wildcards for the probe_name or probe_namespace and end up targeting multiple USDTs for the same probe you might get errors if you also utilize the USDT argument builtin (e.g. arg0) as they could be of different types.
+For example, compare the two logically equivalent samples (C++ vs bpftrace):
Arguments are available via the argN builtins:
In C++:
usdt:/root/tick:loop { printf("%s: %d\n", str(arg0), arg1); }
+int x = std::max(3, 33); // x contains 33
bpftrace also supports USDT semaphores. -If both your environment and bpftrace support uprobe refcounts, then USDT semaphores are automatically activated for all processes upon probe attachment (and --usdt-file-activation becomes a noop). -You can check if your system supports uprobe refcounts by running:
+In bpftrace:
# bpftrace --info 2>&1 | grep "uprobe refcount" -bcc bpf_attach_uprobe refcount: yes - uprobe refcount (depends on Build:bcc bpf_attach_uprobe refcount): yes+
@x = max(3); +@x = max(33); // @x contains 33
If your system does not support uprobe refcounts, you may activate semaphores by passing in -p $PID or --usdt-file-activation. ---usdt-file-activation looks through /proc to find processes that have your probe’s binary mapped with executable permissions into their address space and then tries to attach your probe. -Note that file activation occurs only once (during attach time). -In other words, if later during your tracing session a new process with your executable is spawned, your current tracing session will not activate the new process. -Also note that --usdt-file-activation matches based on file path. -This means that if bpftrace runs from the root host, things may not work as expected if there are processes execved from private mount namespaces or bind mounted directories. -One workaround is to run bpftrace inside the appropriate namespaces (i.e. the container).
+Also note that bpftrace takes care to handle the unset case. In other words,
+there is no default value. The first value you pass to max() will always
+be returned.
watchpoint:absolute_address:length:mode
watchpoint:function+argN:length:mode
min_t min(int64 n)
Update the map with n if n is smaller than the current value held.
+Similar to count this uses a PER_CPU map (thread-safe, fast writes, slow reads).
See max() above for how this differs from the typical userspace min().
w
aw
stats_t stats(int64 n)
This feature is experimental and may be subject to interface changes. -Memory watchpoints are also architecture dependent.
+stats combines the count, avg and sum calls into one.
These are memory watchpoints provided by the kernel.
-Whenever a memory address is written to (w), read
-from (r), or executed (x), the kernel can generate an event.
kprobe:vfs_read {
+ @bytes[comm] = stats(arg2);
+}
+@bytes[bash]: count 7, average 1, total 7 +@bytes[sleep]: count 5, average 832, total 4160 +@bytes[ls]: count 7, average 886, total 6208 +@+
sum_t sum(int64 n)
In the first form, an absolute address is monitored.
-If a pid (-p) or a command (-c) is provided, bpftrace takes the address as a userspace address and monitors the appropriate process.
-If not, bpftrace takes the address as a kernel space address.
Calculate the sum of all n passed.
In the second form, the address present in argN when function is entered is
-monitored.
-A pid or command must be provided for this form.
-If synchronous (watchpoint), a SIGSTOP is sent to the tracee upon function entry.
-The tracee will be SIGCONTed after the watchpoint is attached.
-This is to ensure events are not missed.
-If you want to avoid the SIGSTOP + SIGCONT use asyncwatchpoint.
Using @=sum(5) is conceptually similar to @+=5.
+The difference is that the sum() function uses a map type optimized for
+performance and correctness using cheap, thread-safe writes (PER_CPU). However, sync reads
+can be expensive as bpftrace needs to iterate over all the cpus to collect and
+sum these values.
Note that on most architectures you may not monitor for execution while monitoring read or write.
-# bpftrace -e 'watchpoint:0x10000000:8:rw { printf("hit!\n"); }' -c ./testprogs/watchpoint
-Note: This differs from "raw" writes (e.g. @+=5) where multiple writers to a
+shared location might lose updates, as bpftrace does not generate any implicit
+atomic operations.
Print the call stack every time the jiffies variable is updated:
Example one:
watchpoint:0x$(awk '$3 == "jiffies" {print $1}' /proc/kallsyms):8:w {
- @[kstack] = count();
+BEGIN {
+ @ = sum(5);
+ @ = sum(6);
+ printf("%d\n", (int64)@); // prints 11
+ clear(@);
+ exit();
}
"hit" and exit when the memory pointed to by arg1 of increment is written to:
Example two:
# cat wpfunc.c
-#include <stdio.h>
-#include <stdlib.h>
-#include <unistd.h>
-
-__attribute__((noinline))
-void increment(__attribute__((unused)) int _, int *i)
-{
- (*i)++;
+interval:ms:100 {
+ @ = sum(5);
}
-int main()
-{
- int *i = malloc(sizeof(int));
- while (1)
- {
- increment(0, i);
- (*i)++;
- usleep(1000);
+interval:s:10 {
+ // async read
+ print(@);
+ // sync read
+ if (@ > 10) {
+ print(("hello"));
}
-}# bpftrace -e 'watchpoint:increment+arg1:4:w { printf("hit!\n"); exit() }' -c ./wpfunc
void zero(map m)
async
+Set all values for all keys to zero.
Some behavior can only be controlled through config variables, which are listed here.
These can be set via the Config Block directly in a script (before any probes) or via their environment variable equivalent, which is upper case and includes the BPFTRACE_ prefix e.g. stack_mode's environment variable would be BPFTRACE_STACK_MODE.
Default: PER_PROGRAM if ASLR disabled or -c option given, PER_PID otherwise.
Default: 1
This feature can be turned off by setting the value of this environment variable to 0.
Default: 0
For user space symbols, symbolicate lazily/on-demand (1) or symbolicate everything ahead of time (0).
Default: 1000000
Log size in bytes.
Default: 512
+Default: 1024
This is the maximum number of BPF programs (functions) that bpftrace can generate. @@ -5243,8 +5137,8 @@
Default: 10000
Maximum bytes read by cat builtin.
Default: 4096
Default: 512
+Default: 1024
This is the maximum number of probes that bpftrace can attach to. @@ -5274,8 +5168,8 @@
Default: 1024
Default: 0
Default: warn
ignore - silently ignore missing probes
Default: 32
This exists because the BPF stack is limited to 512 bytes and large objects make it more likely that we’ll run out of space. bpftrace can store objects that are larger than the on_stack_limit in pre-allocated memory to prevent this stack error. However, storing in pre-allocated memory may be less memory efficient. Lower this default number if you are still seeing a stack memory error or increase it if you’re worried about memory consumption.
Default: 64
Default: bpftrace
This can be overwritten at the call site.
Default: ..
Default: 1
Controls whether maps are printed on exit. Set to 0 in order to change the default behavior and not automatically print maps at program exit.
Default: dwarf if bpftrace is compiled with LLDB, symbol_table otherwise
dwarf - locate uprobes using DebugInfo, which yields more accurate stack traces (ustack). Fall back to the Symbol Table if it can’t locate the probe using DebugInfo.
symbol_table - don’t use DebugInfo and rely on the ELF Symbol Table instead.
If the DebugInfo was rewritten by a post-linkage optimisation tool (like BOLT or AutoFDO), it might yield an incorrect address for a probe location. -This config can force using the Symbol Table, for when the DebugInfo returns invalid addresses.
-These are not available as part of the standard set of Config Variables and can only be set as environment variables.
-Default: None
-The path to a BTF file. By default, bpftrace searches several locations to find a BTF file. -See src/btf.cpp for the details.
-Default: 0
-Outputs bpftrace’s runtime debug messages to the trace_pipe. This feature can be turned on by setting
-the value of this environment variable to 1.
Default: /lib/modules/$(uname -r)
Only used with BPFTRACE_KERNEL_SOURCE if it is out-of-tree Linux kernel build.
Default: /lib/modules/$(uname -r)
bpftrace requires kernel headers for certain features, which are searched for in this directory.
-Default: None
-This specifies the vmlinux path used for kernel symbol resolution when attaching kprobe to offset. -If this value is not given, bpftrace searches vmlinux from pre defined locations. -See src/attached_probe.cpp:find_vmlinux() for details.
-Default: auto
-Colorize the bpftrace log output message. Valid values are auto, always and never.
-The -d STAGE option produces debug output. It prints the output of the
-bpftrace execution stage given by the STAGE argument. The option can be used
-multiple times (with different stage names) and the special value all prints
-the output of all the supported stages. The option also takes multiple stages
-in one invocation as comma separated values.
Note: This is primarily used for bpftrace developers.
-The supported options are:
-
|
-Prints the Abstract Syntax Tree (AST) after every pass. |
-
|
-Prints the unoptimized LLVM IR as produced by |
-
|
-Prints the optimized LLVM IR, i.e. the code which will be compiled into BPF -bytecode. |
-
|
-Disassembles and prints out the generated bytecode that |
-
|
-Captures and prints libbpf log for all libbpf operations that bpftrace uses. |
-
|
-Captures and prints the BPF verifier log. |
-
|
-Prints the output of all of the above stages. |
-
symbol_table - don’t use DebugInfo and rely on the ELF Symbol Table instead.
If the DebugInfo was rewritten by a post-linkage optimisation tool (like BOLT or AutoFDO), it might yield an incorrect address for a probe location. +This config can force using the Symbol Table, for when the DebugInfo returns invalid addresses.
+Probe listing is the method to discover which probes are supported by the current system.
-Listing supports the same syntax as normal attachment does and alternatively can be
-combined with -e or filename args to see all the probes that a program would attach to.
These are not available as part of the standard set of Config Variables and can only be set as environment variables.
# bpftrace -l 'kprobe:*'
-# bpftrace -l 't:syscalls:*openat*
-# bpftrace -l 'kprobe:tcp*,trace
-# bpftrace -l 'k:*socket*,tracepoint:syscalls:*tcp*'
-# bpftrace -l -e 'tracepoint:xdp:mem_* { exit(); }'
-# bpftrace -l my_script.bt
-# bpftrace -lv 'enum cpu_usage_stat'
+Default: None
+The path to a BTF file. By default, bpftrace searches several locations to find a BTF file. +See src/btf.cpp for the details.
The verbose flag (-v) can be specified to inspect arguments (args) for providers that support it:
Default: 0
# bpftrace -l 'fexit:tcp_reset,tracepoint:syscalls:sys_enter_openat' -v
-fexit:tcp_reset
- struct sock * sk
- struct sk_buff * skb
-tracepoint:syscalls:sys_enter_openat
- int __syscall_nr
- int dfd
- const char * filename
- int flags
- umode_t mode
-
-# bpftrace -l 'uprobe:/bin/bash:rl_set_prompt' -v # works only if /bin/bash has DWARF
-uprobe:/bin/bash:rl_set_prompt
- const char *prompt
-
-# bpftrace -lv 'struct css_task_iter'
-struct css_task_iter {
- struct cgroup_subsys *ss;
- unsigned int flags;
- struct list_head *cset_pos;
- struct list_head *cset_head;
- struct list_head *tcset_pos;
- struct list_head *tcset_head;
- struct list_head *task_pos;
- struct list_head *cur_tasks_head;
- struct css_set *cur_cset;
- struct css_set *cur_dcset;
- struct task_struct *cur_task;
- struct list_head iters_node;
-};
+Outputs bpftrace’s runtime debug messages to the trace_pipe. This feature can be turned on by setting
+the value of this environment variable to 1.
Default: /lib/modules/$(uname -r)
The -I option can be used to add directories to the list of directories that bpftrace uses to look for headers.
-Can be defined multiple times.
Only used with BPFTRACE_KERNEL_SOURCE if it is out-of-tree Linux kernel build.
# cat program.bt
-#include <foo.h>
-
-BEGIN { @ = FOO }
-
-# bpftrace program.bt
-
-definitions.h:1:10: fatal error: 'foo.h' file not found
-
-# /tmp/include
-foo.h
-
-# bpftrace -I /tmp/include program.bt
-
-Attaching 1 probe...
Default: /lib/modules/$(uname -r)
The --include option can be used to include headers by default.
-Can be defined multiple times.
-Headers are included in the order they are defined, and they are included before any other include in the program being executed.
bpftrace requires kernel headers for certain features, which are searched for in this directory.
# bpftrace --include linux/path.h --include linux/dcache.h \
- -e 'kprobe:vfs_open { printf("open path: %s\n", str(((struct path *)arg0)->dentry->d_name.name)); }'
-
-Attaching 1 probe...
-open path: .com.google.Chrome.ASsbu2
-open path: .com.google.Chrome.gimc10
-open path: .com.google.Chrome.R1234s
Default: None
+This specifies the vmlinux path used for kernel symbol resolution when attaching kprobe to offset. +If this value is not given, bpftrace searches vmlinux from pre defined locations. +See src/attached_probe.cpp:find_vmlinux() for details.
The -v option prints more information about the program as it is run:
Default: auto
# bpftrace -v -e 'tracepoint:syscalls:sys_enter_nanosleep { printf("%s is sleeping.\n", comm); }'
-AST node count: 7
-Attaching 1 probe...
-
-load tracepoint:syscalls:sys_enter_nanosleep, with BTF, with func_infos: Success
-
-Program ID: 111
-Attaching tracepoint:syscalls:sys_enter_nanosleep
-iscsid is sleeping.
-iscsid is sleeping.
-[...]
+Colorize the bpftrace log output message. Valid values are auto, always and never.
One example is updating a map value in a tight loop:
+One example is updating a map value in a tight loop:
+BEGIN {
+ @=0;
+ unroll(10) {
+ print(@);
+ @++;
+ }
+ exit()
+}
+Maps are printed by reference not by value and as the value gets updated right after the print user-space will likely only see the final value once it processes the event:
+@: 10 +@: 10 +@: 10 +@: 10 +@: 10 +@: 10 +@: 10 +@: 10 +@: 10 +@: 10+
Therefore, when you need precise event statistics, it is recommended to use synchronous functions (e.g. count() and hist()) to ensure more reliable and accurate results.
+By default when a bpftrace program exits it will print all maps to stdout.
+If you don’t want this, you can either override the print_maps_on_exit configuration option or you can specify an END probe and clear the maps you don’t want printed.
For example, these two scripts are equivalent and will print nothing on exit:
+config = {
+ print_maps_on_exit=0
+}
+
+BEGIN {
+ @a = 1;
+ @b[1] = 1;
+}BEGIN {
+ @a = 1;
+ @b[1] = 1;
+}
+
+END {
+ clear(@a);
+ clear(@b);
+}The -d STAGE option produces debug output. It prints the output of the
+bpftrace execution stage given by the STAGE argument. The option can be used
+multiple times (with different stage names) and the special value all prints
+the output of all the supported stages. The option also takes multiple stages
+in one invocation as comma separated values.
Note: This is primarily used for bpftrace developers.
+The supported options are:
+
|
+Prints the Abstract Syntax Tree (AST) after every pass. |
+
|
+Prints the unoptimized LLVM IR as produced by |
+
|
+Prints the optimized LLVM IR, i.e. the code which will be compiled into BPF +bytecode. |
+
|
+Disassembles and prints out the generated bytecode that |
+
|
+Captures and prints libbpf log for all libbpf operations that bpftrace uses. |
+
|
+Captures and prints the BPF verifier log. |
+
|
+Prints the output of all of the above stages. |
+
Probe listing is the method to discover which probes are supported by the current system.
+Listing supports the same syntax as normal attachment does and alternatively can be
+combined with -e or filename args to see all the probes that a program would attach to.
BEGIN {
- @=0;
- unroll(10) {
- print(@);
- @++;
- }
- exit()
-}
+# bpftrace -l 'kprobe:*'
+# bpftrace -l 't:syscalls:*openat*
+# bpftrace -l 'kprobe:tcp*,trace
+# bpftrace -l 'k:*socket*,tracepoint:syscalls:*tcp*'
+# bpftrace -l -e 'tracepoint:xdp:mem_* { exit(); }'
+# bpftrace -l my_script.bt
+# bpftrace -lv 'enum cpu_usage_stat'
Maps are printed by reference not by value and as the value gets updated right after the print user-space will likely only see the final value once it processes the event:
+The verbose flag (-v) can be specified to inspect arguments (args) for providers that support it:
@: 10 -@: 10 -@: 10 -@: 10 -@: 10 -@: 10 -@: 10 -@: 10 -@: 10 -@: 10+
# bpftrace -l 'fexit:tcp_reset,tracepoint:syscalls:sys_enter_openat' -v
+fexit:tcp_reset
+ struct sock * sk
+ struct sk_buff * skb
+tracepoint:syscalls:sys_enter_openat
+ int __syscall_nr
+ int dfd
+ const char * filename
+ int flags
+ umode_t mode
+
+# bpftrace -l 'uprobe:/bin/bash:rl_set_prompt' -v # works only if /bin/bash has DWARF
+uprobe:/bin/bash:rl_set_prompt
+ const char *prompt
+
+# bpftrace -lv 'struct css_task_iter'
+struct css_task_iter {
+ struct cgroup_subsys *ss;
+ unsigned int flags;
+ struct list_head *cset_pos;
+ struct list_head *cset_head;
+ struct list_head *tcset_pos;
+ struct list_head *tcset_head;
+ struct list_head *task_pos;
+ struct list_head *cur_tasks_head;
+ struct css_set *cur_cset;
+ struct css_set *cur_dcset;
+ struct task_struct *cur_task;
+ struct list_head iters_node;
+};
+Therefore, when you need precise event statistics, it is recommended to use synchronous functions (e.g. count() and hist()) to ensure more reliable and accurate results.
+The -I option can be used to add directories to the list of directories that bpftrace uses to look for headers.
+Can be defined multiple times.
# cat program.bt
+#include <foo.h>
+
+BEGIN { @ = FOO }
+
+# bpftrace program.bt
+
+definitions.h:1:10: fatal error: 'foo.h' file not found
+
+# /tmp/include
+foo.h
+
+# bpftrace -I /tmp/include program.bt
+
+Attaching 1 probe...
By default when a bpftrace program exits it will print all maps to stdout.
-If you don’t want this, you can either override the print_maps_on_exit configuration option or you can specify an END probe and clear the maps you don’t want printed.
For example, these two scripts are equivalent and will print nothing on exit:
+The --include option can be used to include headers by default.
+Can be defined multiple times.
+Headers are included in the order they are defined, and they are included before any other include in the program being executed.
config = {
- print_maps_on_exit=0
-}
+# bpftrace --include linux/path.h --include linux/dcache.h \
+ -e 'kprobe:vfs_open { printf("open path: %s\n", str(((struct path *)arg0)->dentry->d_name.name)); }'
-BEGIN {
- @a = 1;
- @b[1] = 1;
-}The -v option prints more information about the program as it is run:
BEGIN {
- @a = 1;
- @b[1] = 1;
-}
+# bpftrace -v -e 'tracepoint:syscalls:sys_enter_nanosleep { printf("%s is sleeping.\n", comm); }'
+AST node count: 7
+Attaching 1 probe...
-END {
- clear(@a);
- clear(@b);
-}BPF |
+Berkeley Packet Filter: a kernel technology originally developed for optimizing the processing of packet filters (eg, tcpdump expressions). |
+
BPF map |
+A BPF memory object, which is used by bpftrace to create many higher-level objects. |
+
BTF |
+BPF Type Format: the metadata format which encodes the debug info related to BPF program/map. |
+
dynamic tracing |
+Also known as dynamic instrumentation, this is a technology that can instrument any software event, such as function calls and returns, by live modification of instruction text. Target software usually does not need special capabilities to support dynamic tracing, other than a symbol table that bpftrace can read. Since this instruments all software text, it is not considered a stable API, and the target functions may not be documented outside of their source code. |
+
eBPF |
+Enhanced BPF: a kernel technology that extends BPF so that it can execute more generic programs on any events, such as the bpftrace programs listed below. It makes use of the BPF sandboxed virtual machine environment. Also note that eBPF is often just referred to as BPF. |
+
kprobes |
+A Linux kernel technology for providing dynamic tracing of kernel functions. |
+
probe |
+An instrumentation point in software or hardware, that generates events that can execute bpftrace programs. |
+
static tracing |
+Hard-coded instrumentation points in code. Since these are fixed, they may be provided as part of a stable API, and documented. |
+
tracepoints |
+A Linux kernel technology for providing static tracing. |
+
uprobes |
+A Linux kernel technology for providing dynamic tracing of user-level functions. |
+
USDT |
+User Statically-Defined Tracing: static tracing points for user-level software. Some applications support USDT. |
+
x86_64, arm64, s390x, arm32, loongarch64, mips64, ppc64, riscv64
+Programs saved as files are often called scripts and can be executed by specifying their file name.
+It is convention to use the .bt file extension but it is not required.
For example, listing the sleepers.bt file using cat:
# cat sleepers.bt
+
+tracepoint:syscalls:sys_enter_nanosleep {
+ printf("%s is sleeping.\n", comm);
+}
+And then calling it:
+# bpftrace sleepers.bt + +Attaching 1 probe... +iscsid is sleeping. +iscsid is sleeping.+
It can also be made executable to run stand-alone.
+Start by adding an interpreter line at the top (#!) with either the path to your installed bpftrace (/usr/local/bin is the default) or the path to env (usually just /usr/bin/env) followed by bpftrace (so it will find bpftrace in your $PATH):
#!/usr/local/bin/bpftrace
+
+tracepoint:syscalls:sys_enter_nanosleep {
+ printf("%s is sleeping.\n", comm);
+}
+Then make it executable:
+# chmod 755 sleepers.bt +# ./sleepers.bt + +Attaching 1 probe... +iscsid is sleeping. +iscsid is sleeping.+