Detecting and Investigating Kernel Memory Leaks with kmemleak

What kmemleak detects in the kernel

Kmemleak Supports Linux kernel debugging by searching for potentially orphaned memory objects. To do this, it tracks supported allocations and checks whether references to these objects can be identified in the scanned memory regions. An object that is not recognized as accessible is flagged as a potential leak, not as a conclusively proven error.

Imagine that a kernel module reserves an object and then loses the last reference to it without freeing the memory. If kmemleak finds, during its Reference Search If there are no longer any references to the object, it can flag it as suspicious. The report provides a starting point for investigating the object's allocation and intended lifecycle in the code.

The process is similar to a garbage collector's reachability search, but it does not perform the cleanup: Kmemleak does not free reported objects and does not fix the code. Whether an object is actually lost must be determined based on its usage, ownership, and the intended release process.

A kernel leak involves memory managed by the kernel code or a module. A leak in a PHP service or another user-space process is a different class of error; kmemleak does not examine its heap. For the broader Memory Leak Detection in Hosting Operations Therefore, process data and other diagnostic tools are also worth considering.

Rising RAM usage alone does not prove a Memory Leak nor its location. Changes in the scope of work or data retained for longer periods can also affect the requirements. A sensible starting point, therefore, is a process that can be delimited in terms of time and function, allowing a suspected kernel issue to be investigated in a targeted manner.

Why Reports Remain Suspected Cases

The search for pointer values cannot detect every meaning that a program assigns to a reference. If a valid reference lies outside the scanned ranges or is stored in a way that kmemleak cannot detect it, an object that is still needed may appear to be orphaned. This is a possible False alarm: The message alone does not prove that authorization is lacking.

The opposite is also possible. A numeric value may happen to look like a pointer to an object, even though the code does not use it as a reference. Kmemleak may then classify the object as reachable and overlook an actual leak. An empty output is therefore not proof that the code under examination is error-free.

Added to this is the scope of the allocations being tracked: direct page allocations and ioremap According to the kernel documentation, these are not tracked. Anyone who suspects a memory increase in these areas cannot conclude from an unremarkable kmemleak report that the cause in question has been ruled out. A different type of investigation is required for that.

The Accessibility It is therefore a technical heuristic and not a complete semantic analysis of the program. It does not reliably determine whether a detected reference is used appropriately or whether an overlooked reference is actually missing. A single finding constitutes a test case; the absence of findings merely limits the conclusions that can be drawn from the respective scan.

To evaluate the issue, it is worth recreating the suspected behavior under controlled conditions and then examining the corresponding code path. In particular, it is necessary to determine who owns the object and when it should be released. Only this investigation can provide a plausible explanation for the findings.

Check the test environment and prerequisites

The versioned documentation for Linux 6.6 serves as the technical basis here. However, whether kmemleak is actually available on your system depends on the kernel build you are using and its configuration, not solely on the version number. Therefore, check the specific distribution kernel that is installed and make a note of the build and configuration for later analysis.

One prerequisite is the compiled-in kernel option CONFIG_DEBUG_KMEMLEAK. Separate from that is the Runtime Activation: Is the kernel configured with CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF Once configured, the boot parameter kmemleak=on may be necessary. Be sure to plan for changes to the boot process specifically for the approved test configuration.

To access the diagnostic interface, you must also debugfs be available and properly integrated. Kernel configuration, activation, and file system access are different points of inspection: A missing interface alone does not reveal which of these is the cause. Privileged rights may be required for access.

If possible, use an isolated VM or a dedicated test system, and obtain approval for changes in advance. Kmemleak causes additional overhead during allocation and deallocation; the impact depends on the environment and workload. Without appropriate measurements, it is not possible to specify a universal performance penalty.

In addition to the kernel, document the module under investigation, the planned sequence of events, and the time of the output. Diagnostic reports may contain internal addresses and other sensitive information. Store them in a secure, restricted-access location, and before sharing them, verify which details are actually needed for the error analysis.

Interface and Diagnostic Commands

On the shared test system, start by accessing /sys/kernel/debug/kmemleak. The file shows currently reported suspected cases. Access may require privileged rights; an empty output merely describes the visible diagnostic state and does not prove that the kernel is free of memory leaks.

Run the following command in a shell with sufficient read permissions. It does not change the diagnostic status and is also useful for saving an initial assessment before taking further steps.

cat /sys/kernel/debug/kmemleak

If the file is missing, there are three possible causes: The kernel may have been compiled without CONFIG_DEBUG_KMEMLEAK Built, kmemleak is not enabled at runtime, or debugfs is not mounted in the expected location. The missing file alone does not indicate which of these is the case. Therefore, check the build configuration, boot configuration, and mount point separately.

The table deliberately distinguishes between reading and Write access: Running a scan and cleaning up the visible list changes the diagnostic status. Mounting `debugfs`, if necessary, also alters the system. Perform such steps only with authorization and the necessary permissions; redirecting write commands also requires sufficient permissions.

Reproducibly Narrow Down Suspected Cases

First, save the complete output of the interface along with the timestamp, kernel build, and test configuration. Any existing messages are part of the Initial Findings and must not be attributed to the following test without comparison. Keep the output in a secure place, as it may contain internal addresses and other diagnostic information.

On a shared test system, you can then remove the currently visible messages from the suspect list using the following shell command. The shell requires write access to the interface. The command does not free up memory blocks or resolve the underlying cause; without a prior backup, important information may be lost.

echo clear > /sys/kernel/debug/kmemleak

Next, initiate a procedure that is as narrowly defined and well-documented as possible—such as initializing and performing a controlled cleanup of a module under investigation. Record which steps were performed and when they were completed. This allows any findings discovered later to be compared more specifically with the procedure, without the mere temporal proximity proving a causal relationship on its own.

After the timer expires, trigger a scan in an appropriately authorized shell, and then read the interface again. Both commands require that kmemleak be active and that the debugfs interface be available.

echo scan > /sys/kernel/debug/kmemleak
cat /sys/kernel/debug/kmemleak

Repeat the process systematically and allow sufficient time. A scan does not necessarily have to yield reproducible results immediately. For regression testing, compare the same test before and after the code change, keeping the kernel configuration and load as consistent as possible; document any remaining differences. A new suspicion supports a hypothesis but does not replace testing the object’s lifecycle.

Module Testing and Analysis of Results

During a module test, for example, you can execute an initialization path and then run through the intended cleanup path. If a new entry appears afterward, first note the object’s size, age, and process context, as well as the associated test steps. This information helps in assigning findings; on its own, it does not yet indicate whether the object should actually have been released.

The Allocation Stack Trace leads to the point where memory was allocated. From there, you can trace through the code to see who takes ownership of the object and which path is supposed to free it. The trace does not necessarily have to include the point where the object was not freed: allocation and failure to free the object can occur in different functions.

Also consider the Allocation Order: The documentation describes the output in this order. An earlier orphaned object may trigger additional messages. Therefore, check whether multiple entries belong to the same object group or error path before counting them as separate defects.

For an optional functional test, the current kernel documentation describes the module kmemleak-test, which intentionally creates memory leaks. Use it only in an isolated test environment, and check beforehand to see if it is available in the specific kernel and configured accordingly. The current documentation does not guarantee its availability for every older or distribution-specific kernel.

Even with this diagnostic module, an immediate notification or a match on the first scan is not guaranteed. Record the time of the test and the scan sequence, and repeat the test in a controlled manner. If the output remains empty at first, check the prerequisites and the procedure instead of concluding solely that the diagnostic environment is either functioning or defective.

Check the cause instead of hiding the message

When dealing with a suspected issue, start by examining the lifecycle of the reported object: Which function reserves it, who subsequently takes responsibility for it, and when should it be released? The allocation stack trace leads you to the source of the reservation. Whether an error lies there or at a later point can only be determined by examining the code path.

Be sure to check the Ownership Structure during handoffs between functions and modules. When a reference is passed on, it must be clear which side retains it and which side performs the release. Also, compare the normal cleanup path with termination and error paths—for example, when an initialization is only partially completed.

An error message may also appear even though a valid reference exists that kmemleak does not recognize. Before you suppress it, check where and in what form the pointer is stored. Document why the object remains accessible during its intended lifetime and how it is eventually released.

For verified special cases, developers have Annotations Available: kmemleak_not_leak marks a known object as not being a leak, but continues to scan it for references. kmemleak_ignore Excludes the object from both the report and the scan. As a result, references contained within it may be missing from the analysis.

Check the availability and semantics of the API in the kernel actually in use; the current API documentation does not automatically guarantee compatibility with every older build. An annotation does not resolve a missing authorization or an incorrect ownership change. Therefore, do not use it solely to make disruptive output disappear.

Troubleshooting and Safe Operation

If the file is missing /sys/kernel/debug/kmemleak, first check the kernel build that was actually started and its configuration. CONFIG_DEBUG_KMEMLEAK must be available; the version number of a distribution alone is not sufficient. Record the build and configuration so that future comparisons are based on the same criteria.

Separately, check the Runtime Activation and the inclusion of debugfs. In CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF the boot parameter can kmemleak=on may be required. A missing file does not distinguish between these possible causes on its own; changes to the boot process or mounting debugfs should be part of an approved test configuration.

If the output is empty after a test, check whether the process under investigation was actually executed and whether there was sufficient time between the process and the scan. Repeat the test under documented conditions and compare the results with the saved initial findings. Even then, an empty report has its limitations: kmemleak does not detect every allocation or every relevant reference.

Back up existing findings before making any changes to the diagnostic status. The output may contain internal addresses and other sensitive information; handle it in accordance with your environment’s access and disclosure policies. Preferably, perform state-changing steps on an isolated, approved test system, and take into account the additional diagnostic effort without assuming an unspecified percentage.

High memory requirements or Memory Pressure is a standalone operational issue and, in and of itself, does not constitute evidence of a kernel leak. The article on [...], helps in classifying wait times and memory shortages. Memory Pressure in the Linux Kernel. Combine such operational observations with a targeted review of the code and findings, rather than drawing immediate conclusions about the cause based solely on a RAM curve.

Choosing the Right Diagnostic Tool

Choose the tool based on the presumed Error Class, not solely based on the general symptom of increasing memory usage. kmemleak searches for potentially orphaned, supported kernel objects. For invalid accesses or memory issues in a user process, a different set of tools is more appropriate; these tools address different issues.

A suspected lost kernel reference initially points to a kmemleak. If, on the other hand, an error pattern indicates an access outside an object or to memory that has already been freed, KASAN is suitable for investigating this class of errors. KFENCE also tracks certain memory errors but uses a different instrumentation approach; its availability and suitability depend on the specific kernel and the test environment.

If a user-space process is suspected, examine its memory using a tool designed for that purpose, such as Valgrind Memcheck. A kmemleak report on kernel objects is no substitute for an analysis of an application’s heap. Conversely, a warning from a user-space process does not automatically indicate whether a kernel module is leaking memory.

To continue, select one reproducible test and document the kernel build, configuration, and execution. Examine the reported object lifecycle in the code, correct any identified cause, and repeat the same process under conditions that are as similar as possible. Compare the results again; the absence of a report alone does not prove that the system is completely error-free.