Backing Up and Analyzing Linux Crash Dumps with kdump

What kdump Captures During a Kernel Crash

kdump prepares to save a Linux kernel memory dump before a crash occurs. To do this, a memory region is reserved when the regular kernel starts. A separate Capture Kernel loaded, which can be launched via kexec in the event of a suitable crash.

The capture kernel runs independently of the crashed system kernel and gains access to its memory image via /proc/vmcore. It can write this image to a configured destination. The saved file is usually called vmcore; whether it contains all of the original memory regions depends, among other things, on the selected filtering options.

The tasks are separate: kdump captures and saves the state, while the program crash an existing dump is then analyzed. This allows you to examine kernel messages and task stacks, for example. A dump is a snapshot and does not automatically provide the complete history leading up to the error.

A kernel oops is not the same as a kernel panic: An oops can occur without the system immediately going into a panic. Therefore, not every kernel failure results in a kdump. An application core dump, on the other hand, reflects the state of a process and is not a kernel vmcore.

This preparation is crucial for the subsequent investigation. If no capture kernel was prepared at the time of the crash or if no image was saved, kdump will not be able to reconstruct the lost memory state later. In that case, existing logs and other diagnostic data retained after the reboot will serve as a starting point.

Requirements and Limitations of Data Collection

First, determine the server's distribution, version, and architecture. Package names, service management, boot configuration, and supported storage targets all depend on these factors. When setting up the server, use the documentation specific to this platform instead of combining commands from different distributions; hardware and installed drivers also affect the requirements.

The boot parameter crashkernel specifies the Storage Reservation for crash capture. This area is not available for normal operation. A one-size-fits-all value is not reliable: Base your decision on the specifications for distribution, architecture, and specific hardware, and take into account the storage requirements of the capture system.

Reserved memory alone is not enough. The capture kernel must be loaded and able to reach the configured destination. To do this, the necessary drivers and components must be available in its environment; for a network destination, the connection and an accessible remote host are also prerequisites.

Plan for sufficient capacity at the destination to accommodate the expected images and their storage. The size of a dump depends, among other things, on the contents of the storage and the filtering. A full or unreachable destination can prevent the backup from completing, even if the reservation is set correctly.

Even a prepared installation is no guarantee of a usable dump. An “oops” does not necessarily trigger a “panic”; in the event of a complete power loss, the capture kernel can no longer function. Similarly, a severe hardware failure can prevent it from starting or from writing the image. The capture process and the target must therefore be considered separately from the actual cause of the error.

Which files are included in the analysis

The dump file isn't the only thing that matters for the investigation. If possible, also save the associated kernel files and logs so that you can compare the system state at the time of the crash with the messages and changes. The following overview distinguishes the actual memory dump from the auxiliary files that enable its analysis.

This is particularly important appropriate vmlinux: The architecture and exact kernel build must match those of the crashed system kernel. The kernel running after the reboot may be a different one. The output of uname -r on the newly started server is therefore not enough to select the correct file for the old dump.

Also, use a version of `crash` that is compatible with the kernel and dump format. If external or additional drivers are relevant, have their corresponding module files and debug information ready as well. If symbols are missing or required memory pages have been filtered out, the analysis may be limited; such gaps should be documented in the findings.

A vmcore may contain confidential information from memory. Therefore, restrict access, secure the transmission, and establish retention and deletion policies. Before you forward a dump or supplementary logs to a support service, clarify the necessary authorization and the permissible scope.

Preparing kdump on the server

The starting point is the server's distribution, version, and architecture. Check the documentation for your specific distribution to see which packages are required for kdump and loading the capture kernel, and how to set up the service. Instructions for different distributions are not interchangeable: package names, boot configuration, and verification tools may vary.

Part of the preparation is finding the right Storage Reservation about the boot parameter crashkernel. Select the size and, if necessary, other settings according to the specifications for the specific platform and hardware. The reserved memory is not available to the system running normally; therefore, a universal size specification would be just as unhelpful as blindly copying settings from another server.

After completing the platform-specific boot configuration and, if necessary, restarting the system, the following shell command displays the parameters of the currently running kernel. Look for the following in the output: crashkernel. An entry shows the setting that was passed, but does not confirm that a capture kernel has been loaded or that a backup was successful.

cat /proc/cmdline

On RHEL 9, you can also check the status using the following command in the shell. On other distributions, use the verification method documented there. A positive Status Message is an important indicator of readiness, but it does not replace a controlled test of the entire safety system.

kdumpctl status

Also, verify that the intended target and the necessary drivers are available in the capture system. Record the selected settings and the time of the check so that you can reassess them after kernel or infrastructure changes. You should verify specific settings for the actual target platform before deployment.

Select Backup Target and Dump Scope

The Hedging Objective must be accessible during the capture process and have sufficient free space. A local destination avoids the need for a network connection, but requires usable storage space on the server. Also keep in mind that multiple saved dumps and accompanying diagnostic files can take up space.

In addition to local targets, RHEL 9 also supports network targets via NFS or SSH. This allows the file to be stored separately from the affected server; however, the network, the remote system, and the required components in the capture system must be functioning properly. Check the specific limitations of the actual RHEL configuration rather than assuming that accessibility during normal operation is the same as in the capture system.

With makedumpfile The image can be compressed and filtered by selected memory areas. Filtering This can save storage space and reduce data transfer volume, but it removes pages that may be needed for a later investigation. Therefore, tailor the scope of the dump to the expected diagnosis and document the filters used; the smallest file is not necessarily the most useful.

A vmcore may contain confidential information from memory. Therefore, restrict access, secure the transmission, and specify how long the files will be retained and when they will be deleted. Before sharing a vmcore with a support service, verify that you have the necessary authorization; this also applies to separately stored copies.

Open an existing vmcore file with crash

For post-incident analysis, you'll need the existing vmcore, one that matches the crashed kernel build and architecture vmlinux with debug information as well as a version of `crash` that is compatible with the kernel and dump format. The kernel running after a reboot may be a different one; therefore, the version information alone does not reliably identify the crashed build.

The following command should be entered in the standard shell on the analysis system. Replace both paths with the actual file locations. The command opens an existing image for analysis; it neither creates a dump nor sets up kdump.

crash /pfad/zum/passenden/vmlinux /pfad/zum/vmcore

After a successful start, you'll work on crash prompt, no longer in the normal shell. There, it displays log the available kernel messages. Start by looking for the panic message and the messages surrounding it; messages can provide a starting point, but they do not necessarily describe the complete cause.

With bt you examine the stack of the current analysis context; bt -a Displays the stacks of the active tasks on the CPUs. Compare the call chains with the messages instead of treating a single frame as proof of fault. Depending on the state and scope of the dump, some information may be missing.

In the same prompt, try ps in the task overview; kmem -i Provides general memory information. This snapshot alone does not prove the existence of a memory leak or identify its cause. If a module involved needs to be examined more closely, the corresponding module files and debug information may also be required.

Organizing Stacks and Storage Notes

If a panic occurs after a driver change, first note the time, the kernel build that crashed, and the modules involved. Compare this information with the change log: Was only the driver updated, or was a new kernel installed at the same time? This will help you narrow down which changes are relevant to the investigation.

Then read along with log the available kernel messages and examine them using bt the stack of the current analysis context. bt -a supplements the view of the CPUs' active tasks. In addition to the panic message, make a note of any messages that preceded the panic; they may provide context for an error that becomes apparent later.

A Driver Frame A stack trace provides a starting point, but it is not proof of fault. For example, a previous memory corruption might not become apparent until it reaches a driver function. Therefore, check whether the messages, the modules involved, and the sequence of events all align before concluding that a change is the cause.

If storage is suspected, it provides kmem -i An overview of memory information, to the extent that the dump and the tool provide it. Consider these values in conjunction with the kernel messages. A single value does not prove the existence of a memory leak or identify its cause; furthermore, memory pages may be missing due to filtering.

To determine whether usage builds up over time, you'll need additional time series data. If you analyze CPU load separately, you can CPU Profiling with the Linux Perf Tool help. Such measurements supplement the crash dump, but do not replace the state recorded in it at the time of the crash.

If there is no usable vmcore

After a system freeze followed by a reboot, you should first determine whether a panic actually occurred and whether a vmcore file was saved. A reboot alone does not confirm either of these. Look for messages from the relevant time period and check the configured storage location before concluding that a specific error occurred simply because a file is missing.

Examine the Traceability Chain Separately: Was a capture kernel loaded for the affected boot, and was the event suitable for triggering a kdump? Not every "oops" leads to a panic. A complete power loss or a severe hardware failure can also prevent the capture kernel from starting or writing data.

Next, check the backup destination: Was it accessible, was there enough space available, and did the capture system include the necessary drivers and components? For a network destination, the connection and the remote host are additional dependencies. Under RHEL, the file kexec-dmesg.log Provide information about the data collection phase, if it has been created and saved.

If there is no usable dump, previous logs or persistent console and kernel messages are important alternatives. However, these sources do not provide a complete memory image; the lost state cannot be reconstructed later using kdump.

If reports or other observations indicate network problems, a supplementary Analysis of Linux SoftIRQ Utilization help in classifying packet processing. It addresses a different issue than checking the dump destination and, on its own, does not indicate either a panic or the cause of a failed backup.

Test Data Collection and Validate Results

Choose the scope of the dump based on the expected diagnostic question and the available resources. Compression and filtering can save storage space and reduce the volume of data transferred; however, removed memory pages will no longer be available for later analysis. Document this decision so that it remains clear during the analysis which data may be missing.

A vmcore may contain confidential memory data. Therefore, establish policies for access, secure transmission, and retention periods, as well as for the storage of related debug files. Before sharing the data with a support service, clarify the required authorization and the intended handling of the data.

Check the setup again if the kernel, drivers, boot configuration, or backup infrastructure change. This includes the loaded capture kernel and the accessibility of the selected target. A positive status message is helpful, but it does not in itself indicate whether an analyzable dump will actually be generated under the current conditions.

Warning: A controlled end-to-end test intentionally triggers a crash and may cause system restarts and data loss. Perform this test only in an approved test environment or during a scheduled maintenance window with data backup in place. Clarify responsibilities, impacts, and recovery procedures beforehand; a trigger command is intentionally not provided for this purpose.

A meaningful End-to-End Test Don't stop at the status check: Verify that the file has arrived at its intended destination and can be opened with the files corresponding to the recorded build. When debugging the server, document your findings, missing data, uncertainties, and the next steps in the verification process. Be sure to clearly distinguish between observed indications and causes that have not yet been confirmed.