How to Properly Analyze Linux NUMA Statistics

Linux NUMA Statistics show me how well processes manage their local memory and where remote accesses increase latency. I'll explain how I interpret these numbers in a targeted way, evaluate trends over time, and use that information to develop clear tuning steps for Performance derived.

Key points

• Understanding Meters: numa_hit, numa_miss, numa_foreign, local_node, other_node, interleave_hit
• Evaluate the context: Load profile, topology, workload type
• Measuring Trends: Before/After and by Intervals
• Review Processes: System-wide vs. per process
• Apply Tuning: Affinity, Policies, Placement

What NUMA Statistics Really Show

I interpret NUMA numbers as a map for Storage location and data paths. A high numa_hit means that allocations were placed on the desired node. In contrast, `numa_miss` indicates that the kernel had to fall back to another node. The `numa_foreign` counter shows the counterpart on the target node and completes the picture. Using `local_node` and `other_node`, I can determine whether accesses remained local or utilized remote memory.

These values should never be interpreted in isolation because Workloads react very differently. Short processes occasionally result in misses without noticeably affecting overall performance. Interleave policies, on the other hand, intentionally create distributed allocations, which causes interleave_hit to increase. I therefore always check the intended policy and the current load. Only then do I decide whether a value warrants action or is consistent with the design.

For me, the core idea is: Counter They provide signals, not judgments. I look for patterns over time, not individual numbers. That’s how I determine whether a change to the system shifts the locality. Only once I have this trend picture do I evaluate whether to move processes, adjust policies, or set CPU bindings. Every NUMA analysis therefore begins with a clear research question and repeatable measurement points.

Interpreting Kern Counters in Context

I'm always comparing numa_hit and compare numa_miss with each other, rather than evaluating absolute values. If misses increase, I simultaneously check the trend of numa_foreign on the potential target node. If both match, this indicates a genuine shift and not merely a read-out artifact. local_node and other_node complement this picture by showing the actual accesses. This allows me to determine whether an allocation started locally but the operation later read from more distant memory.

A single tall other_node It doesn't bother me if the workload is intentionally distributed. Web servers with many workers, on the other hand, benefit from consistent locality. That’s why I look at individual processes, not just the big picture. As soon as individual services start to stand out, I address their placement. Only when system-wide issues arise do I look for reasons related to topology or load.

Comparisons Over Time: A Measurement Routine That Pays Off

I take meter readings at the beginning and end of a Final phase and calculate the difference. Individual values obscure trends; differences reveal movement. Repeated intervals of, for example, 30 to 60 seconds are often sufficient to identify trends. After deployments, kernel updates, or hardware changes, I compare the same intervals again. If there are more misses or if `local_node` shifts, a real change has occurred.

Such time series cover Placement error faster than snapshots. I correlate the lines with CPU utilization, context switches, and memory usage per node. This lets me see whether bottlenecks in a node's RAM lead to workarounds. Or whether new processes tip the balance of a node's resources. I always present the raw ratio of hits to misses as a trend, not as a single number.

Check the entire system, then zoom in on processes

I'll start with the big picture from numastat and only then do I examine individual processes. This order saves time because many effects become visible on a global scale. For the process view, I use the process-specific output to isolate services that stand out. Once candidates have been identified, I adjust the Placement on CPU and memory allocation. The article on this topic provides practical tips CPU and Memory Affinity.

Especially with Java services, PHP-FPM, or databases, a clean Affinity, to significantly reduce errors. Container orchestration tends to mask these problems because schedulers distribute workloads without NUMA awareness. I therefore monitor node allocation on a per-pod or per-VM basis. When CPU sets and RAM allocations match, `local_node` increases measurably. Some issues resolve themselves as soon as the process runs close to the required data set.

NUMA Counters at a Glance (Table)

I use the following table as a guide when evaluating a new system so that I can consider every Key figure I can quickly assess the situation. It outlines the significance, typical interpretation, and possible actions. I don't view it as a rigid framework, but rather as a checklist. The key is to compare it with the load profile and server topology. Only with this context can I make a sound decision.

Counter	Meaning	interpretation	Approach
numa_hit	Allocation at the desired node	A high value is positive	Maintain ranking
numa_miss	Allocation was redirected to other nodes	Increased Risk of Latencies	Check Affinity/Policy
numa_foreign	External allocation on this node	Counterpart to numa_miss	Analyze Destination Nodes
local_node	Accesses to Local Storage	The Higher It Is, the Cheaper It Is	Process closer to RAM
other_node	Access to Remote Storage	Just a bit concerning, but unintentional	Check Topology/Load
interleave_hit	Hits in Interleaved Distribution	Expected for Interleave Policy	Evaluate Uniformity

With this Overview I can decide more quickly when to intervene. An increase in `numa_miss` without an explainable change triggers a root cause analysis. If interleave_hit remains high, I check whether the policy is intentionally active. If other_node shows an increase without a rise in load, I check for competing workloads. This way, the table becomes the starting point for targeted actions.

Understanding and Using NUMA Topology

Before I tune, I check the Topology Server specifications: sockets, cores, memory channels, latency paths. If a process runs on Socket 0 but its work sets are stored on Socket 1, access time increases. This reduces throughput and causes response times to fluctuate. Services that are particularly memory-intensive are especially affected by any unnecessary long-distance access. I therefore assign data-intensive processes to nodes with sufficient free RAM.

Asymmetric Connections This amplifies certain effects, such as when a node uses fewer channels. In such cases, I selectively move cached data rather than distributing the process. I configure VM and container hosts so that each instance is assigned a consistent node binding. This allows me to reduce remote data traffic without throttling quotas. The machine’s physical limitations set the boundaries, and I stick to them.

Understanding Interleaving and Balancing

Interleave policies intentionally distribute memory across nodes so that Throughput per process increases or hotspots decrease. In this configuration, high interleave_hit values are considered desirable. I then focus primarily on uniformity, not absolute locality. AutoNUMA or NUMA balancing can help, but not in every situation.

I decide on a case-by-case basis whether to use automatic Balancing remains active. For consistent, long-running services, I tend to use fixed bindings. AutoNUMA can respond effectively to fluctuating workloads. The article provides a good overview of the benefits and risks NUMA Balancing. Only once the goal and scope are clear do I choose the appropriate mode.

Workload Patterns: Databases, VMs, Web Services

Databases are sensitive to Latency between the CPU and RAM. Therefore, I keep the instance, buffer cache, and active shards on the same node. Virtual machines benefit from clearly defined CPU sets plus node RAM, so that guest operating systems see consistent paths. Web services with many workers perform best when worker groups remain bound to a single node. For the storage strategy, I use targeted approaches depending on the specific case NUMA Memory Policies.

On the other hand, I sometimes work on analytical tasks and large scans distributed . Here, interleaving often yields better throughput than strict locality. It’s important to take an honest look at the workload’s I/O patterns. Writes behave differently than reads, and random accesses differ from sequential ones. I choose the policy that fits the access pattern, not the one that sounds good in a textbook.

Practical Measurement Routine and Tools

To get started, all I need is numastat and the process view. I log meter readings along with the date, PID, and load indicators. It’s important to take measurements over identical time windows. This allows for a clear comparison of before-and-after differences. During production windows, I record the deltas and correlate them with release times.

If there are any noticeable Services I also check CPU affinity and node visibility using tools like lscpu, numactl, and perf to monitor remote load. I document the selected policy for each round of measurements. After making a change, I measure again. Only when the trend lines shift steadily do I consider the effect a success. Switching settings blindly can easily lead to false improvements.

Avoiding Common Misinterpretations

A high interleave_hit It is not an error if interleave is intentionally enabled. Similarly, a single miss is negligible over a long runtime. I always check density and distribution across the interval, not just the peaks. Some people interpret “other_node” as universally negative and overlook the nature of the workload. That’s why I first look at the design goal and then evaluate the numbers.

Another misconception: Overall view Well, that takes care of everything. Often, outliers are hidden in just a few PIDs. Or container schedulers distribute pods across nodes, even though a local group would make more sense. I don't notice these effects until I measure on a per-process basis. Without this level of detail, the analysis remains incomplete.

Tuning Steps That Make a Difference

I start with Placement: Processes on the nodes where data is located or should be located. Then I set CPU affinity so that threads don’t jump across sockets. Memory binding follows, so that the kernel allocates memory at the desired location. For variable workloads, I check policies and, if appropriate, use AutoNUMA.

After that, I'll make sure to Consistency In the lifecycle: Restarts, deployments, and scaling must not randomly change node references. I document bindings as code so that they remain reproducible. Then I measure again, evaluate the changes, and decide on fine-tuning. Every change deserves clear, measurable evidence.

Real-World Example: From Flop to Hit

Suppose a Database Under load, it shows more `numa_miss` and an increasing `other_node` value. Query latency fluctuates more significantly. I first check the process binding and find that, following a rollout, the service is running on Node A, but the cache was allocated to Node B. After permanently binding the CPU and memory to Node B, the ratio shifts: numa_hit increases, and misses decrease. Response times become more consistent, and the CPU load drops slightly because remote accesses are eliminated.

At the same time, I check the Policy. Interleave was unintentionally enabled and distributed allocations. After switching to the preferred node, the cache remains closed and local. After an hour of measurement, the deltas confirm the improvement. Only then do I consider the tuning a success. Without this cross-check, a snapshot would have been misleading.

Metrics and Benchmarks for Practical Use

When making decisions, I rely on reliable Odds instead of individual raw values. For each process, I calculate the allocation ratio `local_alloc = numa_hit / (numa_hit + numa_miss)`. In addition, I evaluate the Access Rate local_access = local_node / (local_node + other_node). Together, these metrics indicate whether memory remains in local use even after allocation. My rough guidelines are as follows: For latency-critical services, I aim for remote accesses below 5–10 %. For analytical bandwidth workloads, I tolerate 20–30 %, provided that throughput increases. The decisive factor is the Stability Over time. I prefer a reading that remains consistently stable under load to a brief spike with perfect values. I document these target ranges for each service so that later measurements can be clearly contextualized.

Cgroups, Containers, and Scheduler Pitfalls

In container environments, I first check the cpuset-Mapping: CPU sets and cpuset.mems must map to the same node space; otherwise, conflicts are inevitable. I ensure that pods with fixed CPU requests are not spread across multiple NUMA nodes, and that the scheduler does not distribute workers from the same application across different nodes. For burst types, I limit the maximum number of threads per pod so that they remain within a single node. I document the NUMA domain per deployment and require consistent replicas (one worker group per node, not half-groups spanning two nodes). If I calculate the storage size per pod too tightly, I create unnecessary pressure: A small amount of headroom per node prevents the kernel from having to fail over to other nodes prematurely. If containers are restarted frequently, I ensure deterministic binding so that Cold starts not end up with a worse location by chance.

Aligning Virtual Machines and vNUMA Consistently

When it comes to VMs, my focus is on vNUMA: The virtual topology must match the physical topology. I allocate vCPUs so that each vNUMA node maps to exactly one physical NUMA node. On the host side, I pin the QEMU/hypervisor threads to this domain and ensure that the allocated RAM is provided entirely from this node. Ballooning I use overcommit sparingly for latency-critical VMs; aggressive ballooning can push hotsets out of the node and drive up remote access. For live migration, I re-verify the bindings after the move—some environments lose their finely tuned CPU and memory bindings during this process. Only once the vNUMA alignment is correct do I run `numastat` system-wide; otherwise, I’m fixing the symptoms, not the cause.

THP, Hugepages, and Page Migration

Transparent Huge Pages (THP) can be both helpful and disruptive. Larger pages reduce TLB misses and improve bandwidth, but if the kernel enables huge pages too late collapsed or migrated, inappropriate Long-distance travel arise. I follow two rules: First, the Policy Clearly define (e.g., preferred node) and, whenever possible, allocate large amounts of memory locally from the start. Second, for workloads with fixed, large caches, I use static hugepages (hugetlb) whenever possible, which I explicitly reserve on a single node. This reduces fragmentation and compensation-related swapping. If I see in time series that, after a longer runtime, the other_node—As the share increases, I'll check whether Page Migration or whether compaction is occurring and whether the THP settings match the pattern. What’s important to me is not to disable or enable them globally—I make decisions on a per-service basis and measure the effect on locality and latency.

Understanding Reclaim, Swap, and Memory Pressure Correctly

If the scores go up without any obvious change in ranking, I look for storage pressure per node. Full nodes force the kernel to reclaim and compact, partly triggered by kswapd on another node—this causes side effects with the counters. I check the free memory and page cache utilization for each node. Enabled Swap can displace hot sets and cause latencies to skyrocket; for particularly sensitive services, I disable swap or strictly limit it. In logs and perf views, I look for reclaim peaks during peak load times. The goal is to have enough free, local Keep RAM on the target node so that allocations don't drift away. If this requires reducing caches, I prioritize the service's working set over the generic page cache.

Practical Commands and Analysis

For the process view, I use numastat -p and add cat /proc//numa_maps, to view allocations by area (anonymous, file-backed) and by node. numactl --hardware gives me latency matrices and node sizes, lscpu --extended shows the CPU assignment to nodes. For memory accesses focused on long-distance paths, I set perf mem to verify load patterns. I collect deltas in a reproducible manner, for example:

Measuring routine

• t0: Save numastat (total) and numastat -p for top PIDs
• 30–60 s of exercise, identical workload phase
• t1: Reread numastat, calculate deltas per counter
• Log parallel CPU, context switch, and node memory values

Afterward, I calculate the Odds and identify which processes deviate significantly from system-wide trends. Where uncertainty remains, I repeat the measurement at least three times. I consider only consistent deltas to be reliable. For persistent monitoring, I map the counters to time series and link them to release metadata—this allows me to identify Regression points immediately.

Checklist for Systematic NUMA Tuning

• Define the objective: latency vs. throughput, fixed load vs. variable load
• Capturing Topology: Nodes, Latenacies, Available RAM Reserves per Node
• Measure baseline: total numastat and per process; calculate ratios
• Correct Placement: CPU Affinity, Memory Binding, Policies
• Configure containers/VMs: `cpuset.cpus = node`, `cpuset.mems` as appropriate; map vNUMA correctly
• Choose THP/Hugepages Carefully, Keeping an Eye on Fragmentation
• Reducing Memory Pressure: Headroom per Node, Check Swap Strategy
• Follow-up Measurements: Compare Deltas, Ensure Stability Over Time
• Documentation: Bindings as Code, Release Notes with NUMA Context

Briefly summarized

I analyze NUMA metrics by Signals See also: counter pairs, time series, and process view. The key metrics—numa_hit, numa_miss, numa_foreign, local_node, other_node, and interleave_hit—show me locality, evasion patterns, and distribution strategies. I make decisions based on topology and workload, not on rigid thresholds. Tuning begins with placement, affinity, an appropriate policy, and a clean measurement routine. This is how I deliver consistent Performance, because the CPU and RAM are well-suited to the application, and long-distance data transfers are rare.