Skip to content 
I'll show you how to work with bpftool can specifically examine running Linux systems, control eBPF programs, and obtain meaningful telemetry without having to rebuild the kernel. This article provides a step-by-step guide to installation, core concepts, typical use cases, and useful routines so that you can Kernel Analysis uses it safely in operations and development.
Key points
To start, I'll summarize the most important points so you can put the following chapters into context and Priorities can set.
- Proximity to the core: Direct access to eBPF programs, maps, and statistics
- Transparency: Verifier logs, bytecode, and JIT dumps for debugging
- Production-ready: JSON output, scriptability, reproducible workflows
- Width: Network, system calls, scheduler, cgroups, perf_events
- Ecosystem: Complements high-level tools such as BCC and bpftrace
I use the points mentioned to illustrate practical steps and Decisions to make things easier. This way, you can quickly see where bpftool offers direct benefits and where other tools are a better fit. The list serves as a guide for the examples in the following chapters and keeps the focus on Measurability. Keep your target system in mind as you read, because the configuration and kernel version determine the available options. The more clearly you define your purpose, the faster eBPF and bpftool will deliver results Signal instead of noise.
eBPF as a secure runtime in the kernel
eBPF provides a secure runtime in the Kernel that binds small programs to defined events and rigorously checks them before execution. The verifier prevents invalid memory accesses and loops, thereby keeping systems under control and operational. I attach programs to Kprobes, tracepoints, XDP, or cgroups and obtain precise context data. This proximity provides measurement values without expensive syscall transitions and without having to build modules. This creates a flexible telemetry layer that I can use with bpftool make it visible, testable, and controllable.
Installation and requirements
I check the kernel version and features first, because many functions become available starting with 5.x . On distributions, I install bpftool as a package or build it from the kernel sources in tools/bpf/bpftool, depending on the system's maintenance status. For the build, I need Clang/LLVM, libelf, make, and the appropriate headers so that the toolchain works with the kernel fits. After installation, I use “bpftool version” to check availability and verify that it meets my requirements. If the kernel capabilities are correct, I run tests on a separate system before deploying to production hosts follow.
bpffs and Pinning: A Look at the Object Lifecycle
To ensure reproducible workflows, I first mount the BPF filesystem under “/sys/fs/bpf.” If it’s missing, I set it up with “mount -t bpf bpf /sys/fs/bpf” and check the namespaces if containers are involved. Next, I pin loaded objects to stable paths, for example, “bpftool prog pin id X /sys/fs/bpf/myapp/xdp_ingress” or “bpftool map pin id M /sys/fs/bpf/myapp/counters”. This way, programs, links, and maps survive process restarts, remain discoverable, and are uniquely identified. addressable.
I organize the pinning hierarchy by service, hook, and version, such as “/sys/fs/bpf/”service/hook/version”. This makes rollbacks and side-by-side testing easier. For attachments, I prefer the link approach: “`bpftool link list` shows me stable handles, and `bpftool link pin id L /sys/fs/bpf/myapp/link_xdp` captures the link. When cleaning up, I first remove the pins (`rm`), which then frees up the objects. This way, I avoid Orphans-Programs that continue to run unnoticed.
Central Subcommands and Concepts
bpftool organizes commands by object type, such as prog, map, cgroup, or feature, which allow workflows to be logically structured. I use “prog list” and “prog show” for an overview, “dump xlated/jited” for in-depth insights, and “map dump/lookup” for data flow analysis. The “feature” subcommand displays the enabled helper and map types, which helps prevent errors later on. JSON output facilitates automation in CI/CD and configuration management. The following table summarizes typical tasks and examples compact together.
| Object | Task | Example |
|---|---|---|
| prog | List and describe the programs | bpftool prog list | bpftool prog show id X |
| prog | View Bytecode/JIT | bpftool prog dump xlated id X | dump jited id X |
| prog | Loading and Attaching | bpftool prog load file.o /sys/fs/bpf/p && … attach |
| map | Check Content and Keys | bpftool map dump id M | map lookup id M key HEX |
| feature | View Kernel Capabilities | bpftool feature probe |
BTF, CO-RE, and Skeletons in Everyday Life
I make sure that BTF is available in the kernel because it enables CO-RE (Compile Once – Run Everywhere) and provides convenient debug output. I use “bpftool feature probe” to check if BTF is active, and inspect the type information as needed with “bpftool btf dump file /sys/kernel/btf/vmlinux”. For development, I generate a suitable header file from the kernel types using “bpftool gen vmlinux,” which allows me to safely reference structures. This significantly reduces breakpoints during kernel updates.
When it comes to packaging, I rely on skeletons: “bpftool gen skeleton obj.o” generates a C wrapper that encapsulates loading, appending, map accesses, and cleanups. This reduces my glue code and keeps the interaction between user space and the eBPF program robust. CO-RE helps me use the same artifacts on different kernels, as long as helpers and hooks are available—I check this early on with “feature probe.”.
Performance Analysis with bpftool
For performance-related questions, I use bpftool to count the calls I analyze individual programs, measure execution times, and compare them to workload peaks. This helps me identify which traces are running heavily or whether an XDP filter is consuming too much CPU on hot paths. I then assess whether sampling or tighter filters make more sense. If I find anomalies, I examine JIT dumps to understand code paths and avoid unnecessary instructions. Ultimately, the data is fed into dashboards so that operators can maintain Transparency retain.
Counter Values, Per-CPU Maps, and Statistics
I analyze “bpftool prog show id X” to check “run_time_ns” and “run_cnt.” Their ratio gives me average runtimes; I interpret outliers using workload metrics. For map counters, I pay attention to per-CPU variations: Some dumps show per-CPU values, while others are aggregated. For precise analyses, I use machine-readable output and deliberately calculate aggregations so that spikes on individual CPUs do not to go under.
To get a quick look at trace output, I run “bpftool prog tracelog.” This lets me read print output from the tracing buffer without having to use separate tools. In production environments, I strictly limit such print statements and replace them with counters in maps or ring buffer events to avoid overhead and noise.
Network Observability: Packets, Flows, Errors
In a network environment, I test XDP and TC programs, read meter readings from maps, and identify Hotspots along the data paths. I use bpftool to visualize dropped rules and characterize flows. If filters make incorrect decisions, map dumps reveal the actual keys and values. This allows me to quickly identify discrepancies between expected and actual processing. This overview helps me with in-depth tool selection by eBPF Analysis Tools, which is used in hosting environments concrete classifies.
XDP/TC Variants and Visibility with bpftool net
For the network path, I use “bpftool net” to check the programs attached to the interfaces. This lets me determine whether XDP is running in Generic, Native, or Offload mode and which TC hooks (ingress/egress) are in use. If the modes are incorrect, I correct the attach options or driver parameters. I regularly document the output as an artifact so that changes to network paths comprehensible remain.
For hotpaths, I aim for short paths: XDP programs should make decisions early (pass/drop/redirect), while TC programs consolidate rules and avoid redundant lookups. I use map statistics to evaluate hit quality, and JIT dumps reveal whether jump patterns are suboptimal. If queuing or checksum costs become apparent, I adjust filters and reconsider the placement between XDP and TC.
Security Monitoring and Compliance
I use eBPF programs for Process-Starts, file accesses, and network events to identify security-related patterns. Using bpftool, I check which programs are active, where they attach, and whether rules are being enforced. If the attachment points are correct, I review map contents to document policy compliance. In suspicious cases, I refer to Verifier logs and bytecode to verify the logic. This insight speeds up audits and makes agent behavior transparent to teams understandable.
Permissions, Isolation, and Security Models
In production, I make sure to use clear permissions. On many systems, unprivileged eBPF functions are disabled; therefore, I plan to use dedicated service accounts and specific capabilities. Depending on the kernel version, CAP_BPF, CAP_PERFMON, and CAP_NET_ADMIN are used, while CAP_SYS_ADMIN is only used when absolutely necessary. I isolate bpffs by namespace when containers require their own traces, and I configure cgroups so that attachments targeted have an effect.
For compliance purposes, I freeze sensitive maps after populating them using “bpftool map freeze.” This makes the policies read-only, while programs can still read them. During audits, I document the program tag and the attach points so that decisions remain reproducible, even if artifacts are rebuilt.
Custom eBPF Programs: Loading, Attaching, and Debugging
During development, I compile C source files with Clang into eBPF objects, load them with bpftool, and connect them to Hooks. If the verifier triggers an error, I save the log and gradually eliminate risky paths. I inspect compiled bytecode and JIT output to evaluate instruction sequences. If the results are correct, I write to and read from test data via maps and check edge cases. This shortens feedback loops and keeps my toolchain ready for both experimentation and production. uniform.
CO-RE Strategy and Stable Artifacts
To make builds last longer, I rely on CO-RE. I incorporate BTF information, use “gen vmlinux,” and check the relocations during loading. If discrepancies occur in kernel structures, the verifier log identifies the locations. I keep the programs as generic as possible and store policies in maps. The advantage: When schema changes occur, I only update the data, not the Code. I use Skeletons to automate setup, pinning, and cleanup, which significantly reduces error rates, especially in CI/CD pipelines.
Integration with high-level tools
To achieve quick results, I start by focusing on BCC-scripts and use them as a starting point for more in-depth analyses. As soon as a script provides useful signals, I use bpftool to inspect the underlying programs and maps. This approach shows me what’s actually loaded into the kernel and which data structures are in use. This allows me to clearly separate the abstraction layer from the actual objects. For an overview, it’s worth taking a look at these concise BCC Tools, which answers common questions with just a few commands cover.
Best practices for operation
I strictly separate test and production environments, collect Verifier logs early on, and maintain Rollbacks Ready. Before every rollout, I check “bpftool feature” to ensure that the program type, helper, and map variants match the target. I integrate program statistics into existing monitoring to keep overhead visible. I continuously document all attachment points, as this is the only way for teams to maintain an overview. If you’d like to dive deeper, you’ll find the eBPF Analysis Tools further impetus for Workflows.
Resource Management, Cleanup, and Rollback
I use pins to create defined states and actively clean them up. For rollbacks, I keep the previous version available in the same namespace (e.g., “/sys/fs/bpf/myapp/v1” and “/sys/fs/bpf/myapp/v2”). Switching is done by re-attaching or changing links with minimal downtime. Afterward, I remove old links and maps so that no resources lick. Before deleting, I check to see if any references still exist (“prog show,” “link list,” “map show”).
To prevent configuration drift, I freeze maps that contain policies and make changes exclusively through defined deployments. I schedule batch updates outside of peak load times, monitor runtime and error counts, and confirm successful updates with a second “map dump.”.
Automation and JSON Output
The JSON flag and machine-readable formats make bpftool great scriptable for CI/CD, CMDBs, and audits. I seal builds in a reproducible manner, document the hashes of the object files, and store bpffs paths. This is how I link deployments to specific programs and maps. Simple wrapper scripts write status reports to the console and to artifacts after every change. This ensures that the eBPF landscape remains consistently testable.
Building Trust: Tags, Hashes, and Artifacts
After loading, I read the program tag (“bpftool prog show id X”), which is derived from the bytecode. I link this tag to the build number and commit hash in my CMDB. During subsequent checks, I compare the expected tag with the current one—this allows me to detect discrepancies without accessing the original binaries. For maps, I log the type, key/value sizes, and flags so that structural changes during updates in a timely manner show.
bpftrace in Practice
For ad hoc traces, I use bpftrace, when a few lines of syntax are needed to quickly provide answers. I then check the resulting output with bpftool to see exactly which programs, attachment points, and maps are involved. This way, I combine expressiveness with a focus on the core and keep both perspectives in sync. This brief overview is a good starting point for bpftrace, which handles typical queries well frames. Once a pattern is in place, I migrate it to compact C programs as needed.
Error Analysis Using Verifier Logs
If the verifier rejects a submission, I first look for potential Zero-dereferences, missing bounds checks, or paths that are too long. I simplify the logic, isolate suspect helper calls, and validate offsets. It helps to break down large maps into smaller ones and divide hot paths into clearly defined blocks. JIT dumps show me whether loops are expanding unintentionally or jumps are inefficient. With each step, error messages become fewer and fewer until the program runs reliably loads.
Quickly Identify Common Fault Patterns
When I see messages like “invalid mem access” or “R.. unbounded loop,” I check array bounds, pointer validation, and loop boundaries. For CO-RE issues, the messages suggest missing or incorrect BTF data; I verify “/sys/kernel/btf/vmlinux” and adjust the target structures. If loading fails due to missing helpers, “feature probe” displays the available helpers and map types. If JIT issues occur during dumping, I check whether JIT is enabled and whether hardening options are affecting the output prevent.
If attachments get stuck, it’s often because a link is still pinned. I list the links, unpin them one by one, and then remove the pins. If the status is “EBUSY,” I check whether another instance of the service is keeping objects open and plan a brief, coordinated switchover.
Outlook: bpftool and Modern Kernel Analysis
With each new kernel release, the number of program types, helpers, and Statistics, and bpftool quickly reflects these advancements. I therefore set aside time for regular updates to ensure that the tooling and documentation remain up to date. JSON improvements and new subcommands open up additional avenues for automation. At the same time, integration with high-level stacks is maturing, which simplifies onboarding. Those who actively follow these developments will benefit when it comes to diagnostics, tuning, and Security Speed.
Briefly summarized
bpftool gives me direct Access It focuses on eBPF programs and their data structures and makes kernel processes visible. I can identify bottlenecks, verify security rules, and develop my own traces without modifying the kernel. With a clean installation, clear tests, and scripting, the process remains reproducible. High-level tools speed up the onboarding process, while bpftool reliably documents the actual objects. This allows me to bring observability and diagnostics to a robust level that supports daily operations carries.
