BPFtrace in Hosting: Detect Server Issues Faster

I'll show how bpftrace hosting in Linux environments drastically reduces the time it takes to identify the cause of an error, and in doing so Kernel-signals. Instead of guessing, I measure system calls, I/O latencies, and network events in real time in the eBPF-Context – without stopping services.

Key points

The following bullet points provide a quick overview of the main points of this article.

• In-Depth Insight in system calls, I/O, and networking directly from the kernel
• Low overhead thanks to secure eBPF programs in the kernel
• Quick Narrowing Down from process, I/O, and database bottlenecks
• Flexible Tracing with filters, histograms, and stack traces
• Practice Workflow for urgent incidents within minutes

Why bpftrace Makes Hosting Issues More Visible Faster

In modern hosting stacks, many services compete for Resources, whereas traditional dashboards often show only surface-level metrics. I dig a level deeper: bpftrace hooks into system calls, tracepoints, and function hooks to show me what’s really slowing things down. Timeouts accompanied by unremarkable CPU usage often indicate I/O latencies or blocking calls. This is exactly where bpftrace shines, providing counts, latency histograms, and stack traces directly from the Kernel. This way, I can identify the sources of the load—whether they are specific processes, containers, or queries—and take targeted action.

How eBPF and bpftrace Work Together

eBPF runs small, verified programs in the Kernel and provides first-hand event data. bpftrace compiles scripts at runtime into eBPF bytecode and attaches them to probes, filters, and actions. For example, I select a tracepoint for file reads, filter by a process name, and aggregate latencies in a histogram. The „probe–filter–action“ pattern remains manageable, even when I’m measuring multiple signals simultaneously. This way, I can set up an observation in minutes that gives me the crucial Indicators supplies.

Quick One-Liners for When It Counts

In incidents, speed is key. I use concise one-liners that reveal a pattern in seconds. Here are some of my tried-and-true openers:

# Count „loud“ file accesses by process name (clear every 5 seconds)
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { @[comm] = count(); }
interval:s:5 { clear(@); }'

# Latency Histogram for File Reads (per Process)
bpftrace -e '
kprobe:vfs_read { @ts[tid] = nsecs; }
kretprobe:vfs_read /@ts[tid]/ {
  @lat[comm] = hist(nsecs - @ts[tid]);
  delete(@ts[tid]);
}'

# Bundling TCP Retransmissions with Kernel Stacks
bpftrace -e 'tracepoint:tcp:tcp_retransmit_skb { @[kstack] = count(); }'

# Sum SoftIRQ Times (10-second window)
bpftrace -e '
tracepoint:irq:softirq_entry { @t[args->vec] = nsecs; }
tracepoint:irq:softirq_exit /@t[args->vec]/ {
  @soft[args->vec] = sum(nsecs - @t[args->vec]);
  delete(@t[args->vec]);
}
interval:s:10 { print(@soft); clear(@soft); }'

# Make the accept() load visible on the database or web server
bpftrace -e 'tracepoint:syscalls:sys_enter_accept4 /comm=="mysqld" || comm=="nginx"/ { @[comm] = count(); }'

Using these „probes,“ I can quickly determine whether a service is opening an abnormally high number of files, causing I/O bottlenecks, or causing network congestion. I then refine the filters to PID, process names, or paths.

Process and Resource Diagnostics on Live Servers

If a single account or container is slowing down a shared server, I count the system calls per Process and identify „noisy“ sources. A strikingly high number of execve calls indicates excessive process launches, which may point to faulty cron jobs, for example. If a service opens countless files, I see it immediately and use filters to narrow the scan down to specific paths. For heavily trafficked web servers, this is invaluable because it allows me to quickly isolate the culprits. Anyone who wants to delve deeper into tooling ideas should also check out approaches to eBPF analysis tools and applies the principle to its own hosts.

Container and Kubernetes Perspective with cgroups

In multi-tenant or Kubernetes hosts, I need clear tenant isolation. bpftrace provides me with the cgroup-Perspective as the key:

# Group system calls by cgroup (container) and process name
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { @[cgroup, comm] = count(); }'

This is how I identify which container is noisy without having to manually collect individual PIDs. For more targeted analyses, I apply additional filters:

# Consider only PHP-FPM (e.g., in an app container)
bpftrace -e 'tracepoint:syscalls:sys_enter_execve /comm=="php-fpm"/ { @[pid] = count(); }'

In Kubernetes, I often measure the Node and group by cgroup. I document the mapping of cgroup IDs to pod/container names in my runbook (kubectl/CRI) so that I can clearly reference the measurement results.

Reliably Measuring I/O and File System Latencies

Slow pages despite a „decent“ CPU often indicate I/O-bottlenecks. I measure read and write operations per process, log slow paths, and build latency histograms. In WordPress environments, this helps me determine whether many small PHP files or large media files are throttling throughput. Based on that, I decide whether caching, the PHP opcode cache, or file system tuning should be applied first. For those who want to dive deeper, you’ll find background information on Disk Latencies in Storage and can adjust measurement points as needed.

Visualizing Off-CPU and Lock Wait Times

Not all wait times are I/O: Threads can off-CPU block—for example, on locks. I use Futex and scheduler events for this.

# Futex Wait Times (Lock Contention) as a Histogram
bpftrace -e '
tracepoint:syscalls:sys_enter_futex { @ts[tid] = nsecs; }
tracepoint:syscalls:sys_exit_futex /@ts[tid]/ {
  @futex[comm] = hist(nsecs - @ts[tid]);
  delete(@ts[tid]);
}'

With these profiles, I can see whether PHP-FPM workers or DB threads are waiting for locks. Combined with I/O histograms, I disconnect Storage- from Concurrency-Problems.

Display Network Errors, SoftIRQs, and Retransmissions

I often attribute complaints about occasional timeouts to Network-signals. I monitor TCP retransmissions, RST events, and connection drops directly in the kernel. In addition, I take a look at SoftIRQs, because overloaded network queues leave traces there. A pattern of retransmissions combined with increasing softIRQ times indicates packet drops, buffer bottlenecks, or QoS issues. Background articles on the following topics are a good supplement for troubleshooting: SoftIRQ and network throughput, which I link to bpftrace measurements.

Case Study: WordPress Host with Occasional 504 Timeouts

A shared server is returning 504 errors, and the CPU usage is only at 35%. Here's what I'm doing:

• „Network or I/O“ hypothesis. I start retransmissions and SoftIRQ timing. Result: few retransmissions, SoftIRQs stable.
• Switch to I/O: The vfs_read latency histogram shows a long tail up to 80 ms for php-fpm. Many openat calls per request.
• Filter for paths under wp-content and wp-includes: countless small file reads dominate.
• Cross-check of locks: Futex-Histo shows no anomalies—no lock contention.
• Action: Adjust the OPCache configuration to cache static assets more aggressively. This will reduce openat counts and latency.

With less than 15 minutes of active tracing, it's clear: it's not the network, but File I/O and Lack of Caching cause the timeouts.

Databases and PHP-FPM: Quickly Pinpoint Bottlenecks

For MySQL/MariaDB, I look at system calls, locks, and I/O latencies of the DB Processes I monitor accept/connect phases to see if connections are stalling or TLS handshakes are getting stuck. For PHP-FPM, I check to see if execve and file accesses are unusually high, which indicates a lack of caching. Using stack traces for specific system calls, I identify where in the code requests are waiting. This allows me to systematically rule out the network, the app, and the database one by one and pinpoint the narrowest Position.

Best Practices for High-Performance Servers

I start every tracing session with a clear Research Question and narrow down probes using filters. Time limits or intervals keep the data volume manageable. For recurring analyses, I save scripts with useful default filters such as PID, cgroup, or process name. Before deploying them on customer hosts, I test complex scripts on staging systems. This keeps overhead low and prevents unnecessary Side effects.

Measurement Quality, Overhead, and Limits in Practice

bpftrace remains within the single-digit percentage range for CPU overhead when I use targeted probes, provided I follow these guidelines:

• Filtering at the Inlet: I filter early (e.g., by comm/PID) instead of waiting to filter in Maps.
• Sampling: For very hot samples, I use sampling, e.g., 1% of the events:

  tracepoint:syscalls:sys_enter_openat
  / rand() % 100 == 0 / { @[comm] = count(); }

• Use stack traces sparingly: kstack/ustack only when necessary—count first, then go deeper.
• Buffer size: During event peaks, I increase the ring buffer:

  export BPFTRACE_PERF_RB_PAGES=4096

• Keep the window open briefly: Intervals (5–30 s) and a clear end prevent data corruption.

If I see „dropped events,“ I increase the buffer size, reduce the stack depth, or tighten the filters. For accuracy, I prefer Tracepoints (stable ABI) versus kprobes (kernel function names may vary).

Security, Governance, and Multi-Tenant Rules

When it comes to shared hosting, I strictly pay attention to Data protection and clear scopes. I trace technical signals, not customer data, and document the reason, scope, and duration. For multi-tenant environments, I establish fixed guidelines: who is authorized to start tracing, which filters are required, and when I stop tracing. I minimize or pseudonymize logs containing sensitive paths. This allows me to obtain usable technical data without violating tenant boundaries, and I maintain the Compliance in.

Installation and Prerequisites on Modern Linux Servers

I use Linux 5.x for bpftrace because of its features and Stability are noticeably better there, even though 4.9 is considered the lower limit. I install bpftrace via apt or dnf and add kernel headers as soon as more complex probes are needed. Next, I check cgroup configurations, container runtimes, and security modules that control access to probes. A quick test with simple tracepoints ensures that signatures and symbols match. This way, nothing stands in the way of a structured start, and I can begin Measurements drive.

Portability: BTF, symbol resolution, and stable probes

For robust scripts, I rely on BTF-Type information (vmlinux) that helps bpftrace with field resolution. If this information is missing, I prefer to use tracepoints instead of kprobes. For uprobes (Userland) I need unstripped binaries or separate debug symbols—this is especially worthwhile for PHP-FPM or mysqld. I check versions using „bpftrace –info“ and include a small compatibility block in my scripts in case event names vary depending on the kernel.

Clinical Workflow: From Symptom to Cause in 15 Minutes

First, I'll formulate the Hypothesis: Network, I/O, CPU, or DB? Then I set up a quick trace at the most likely level—for example, retransmissions or file latencies. If the first few minutes reveal a pattern, I refine the filters, add stack traces, and limit the runtime. If my suspicion is confirmed, I drill down deeper into the affected service and capture only the relevant paths. With this narrowed-down focus, I avoid shooting in the dark and quickly narrow it down to the most likely Cause.

Runbook: 15-Minute Initial Response

• Minutes 0–2: Select a hypothesis (Network/I/O/CPU/DB). Run the Baseline One-Liner.
• Minutes 3–5: Identify the first outlier (e.g., high openat count, retransmissions, Futex histograms).
• Minutes 6–8: Sharpen filters (comm/PID/cgroup, paths) and add latency histograms.
• Minutes 9–12: Enable stack traces only at the hotspot to highlight specific sections of code.
• Minutes 13–15: Determine the appropriate action (caching, limits, configuration changes) and test it briefly.

Comparison Chart: Probes and Everyday Uses

The following table shows typical rehearsals, their areas of application, and a key benefit in the context of hosting. I use them as a cheat sheet when I need to quickly choose the right measurement point.

Sample Type	Use	Example	Benefit
tracepoint:syscalls	Count/Filter System Calls	sys_enter_openat, execve	„Sounds“ Processes Find
kprobe/kretprobe	Measuring Kernel Functions	vfs_read, tcp_retransmit	I/O and Network-Latencies visible
uprobes/uretprobes	Tracing Userland Functions	mysqld, php-fpm symbols	Locate DB/App Hotspots
tracepoint:net/*	Identifying Networking Events	TCP Retransmissions, RST	Timeout-Causes narrow down
perf events	CPU and Scheduler View	On-CPU/Off-CPU Profiles	Identify scheduling bottlenecks

Summary for Admins and DevOps

bpftrace gives me a sharp Lens on kernel and application signals that traditional monitoring often overlooks. I start small, apply targeted filtering, and control execution times to ensure the measurement results remain clear. With just a few lines of script, I can detect process noise, file latencies, network retransmissions, and database wait times. This approach noticeably reduces the mean time to resolution on production hosts. Those who integrate bpftrace into their workflow can resolve hosting incidents in a targeted manner and keep websites and APIs noticeably responsive.