I’ll demonstrate in a practical way how IRQ affinity on multiprocessor systems specifically binds network interrupts to CPU cores, reduces latency, and increases throughput. Using clear steps, examples, and a table, I’ll outline how to select hex masks, account for NUMA, and properly pin processes.
Key points
- IRQ Affinity directs hardware interrupts to specific CPUs and reduces overhead.
- CPU Affinity Pinning services to the same cores keeps caches local.
- NUMA Note: Use adapters and cores from the same node.
- irqbalance Weigh the options: automatic distribution or manual fine-tuning.
- Monitoring and iterative adjustment ensure stable latencies.
Understanding IRQ Affinity: Basics and Effects
On Linux hosts, incoming packets, disk I/O, and timers IRQs that the kernel distributes among the CPU cores. I specify this using the files /proc/irq//smp_affinity and .../smp_affinity_list, which CPUs are allowed to access a source. A broad default mask may seem flexible at first, but it results in load cache misses, costly context switches, and scattered soft IRQs across many CPUs. I map critical queues from individual NICs to specific cores, relieve congestion at hotspots, and keep data paths short. This routing noticeably improves stability when many flows are active at the same time.
Tuning IRQ and CPU Affinity: Keeping Data Local
I successfully connected the IRQ-Mapping NIC queues to the CPUs associated with relevant workers. To do this, I pin web server or proxy threads using taskset or CPUAffinity= in systemd to those cores that also handle the RX/TX interrupts. This keeps cache lines local and minimizes inter-CPU communication, which improves the Latency smooths out. API backends, real-time services, and virtualized stacks, in particular, benefit from this consistency. I test the binding under production load until flows, soft IRQs, and user space align seamlessly.
Automation vs. Fine-Tuning: Understanding irqbalance
The service irqbalance automatically distributes interrupts across available cores, which works well on general-purpose servers. In heavily loaded network setups, however, this distribution hinders cache locality and makes targeted pinning more difficult. I limit or selectively disable `irqbalance` when certain queues require specific cores. This guide helps me understand the basics and find suitable profiles: Configure irqbalance. As a result, the automatic system handles the non-critical IRQs, while I assign the sensitive IRQs manually.
Step: Identify the relevant IRQs
I'll start by taking a look at /proc/interrupts and filter by device name, such as ens192, eno1 or eth0. Modern adapters create multiple RX and TX queues, so I identify a group of IRQ numbers assigned to the same NIC. I monitor counter values to quickly identify hotspots and prioritize the binding of heavily loaded queues. I check this view regularly during load tests to ensure the mapping remains effective over time. Additionally, I review driver names because they provide clues about RSS capabilities and offloading.
# View all interrupts
cat /proc/interrupts
# Display only NIC-related lines (example: ens192)
grep -i ens192 /proc/interrupts
Making Smart Use of NUMA Topology
On hosts with multiple nodes, I prefer to map IRQs to the cores of those NUMA-The node to which the NIC is physically connected. I'll check that with lscpu and numactl --hardware and mark the appropriate CPU sets for later mask creation. I also bind processes that use these network paths to the same node and use memory policies to ensure local Memory-Assignments. This is how I avoid costly remote access across QPI/UPI links. This approach quickly yields measurable benefits in latency tests.
Choosing Bitmasks Safely: Hex Logic at a Glance
The file smp_affinity accepts hex bitmasks that map directly to core IDs and can also cover large systems. I often start with simple patterns: CPU0 is 0x1, CPU1 is 0x2, CPU2 is 0x4, CPU3 is 0x8, and so on, while 0xF covers cores 0–3. On machines with many cores, I write multiple 32-bit blocks, separated by commas, so that the Mask accurately maps all IDs. This overview helps me make error-free assignments and avoid unintended reassignments. I often use the following table as a reference.
| CPU Number | Bit (binary) | Hex Mask | Note |
|---|---|---|---|
| 0 | …0001 | 0x1 | CPU0 Unload it frequently and use it sparingly. |
| 1 | …0010 | 0x2 | IRQ set to CPU1 pin. |
| 2 | …0100 | 0x4 | Pin the IRQ to CPU2. |
| 3 | …1,000 | 0x8 | Connect the IRQ to CPU3. |
| 0–3 | …1111 | 0xF | Enable all four cores; latency often increases slightly. |
| 0–7 | 11111111 | 0xFF | Wide distribution; cache locality suffers. |
Setting IRQ Affinity: How to Bind Queues to Cores
After identifying the NIC's queue IRQs, I assign them to dedicated Cores like 1, 2, and 3, to scale parallel processing efficiently. This keeps each RX/TX queue on its own core, which prevents crosstalk and ensures consistent SoftIRQ handling. During fine-tuning, I compare throughput and latency until the distribution performs reliably. For a more in-depth look, a brief Practical guide with variations depending on the adapter. I deliberately run the commands during maintenance windows and ensure they are executed via a boot script.
# Example: Bind three queue IRQs to CPUs 1–3
echo 2 > /proc/irq/181/smp_affinity # CPU1 (0x2)
echo 4 > /proc/irq/182/smp_affinity # CPU2 (0x4)
echo 8 > /proc/irq/183/smp_affinity # CPU3 (0x8)
Process Pinning: Assigning Services to the Same Cores
I pin application workers to those CPUs, which handle the corresponding IRQs, so that data paths remain short. With systemd, I use CPUAffinity=1 2 3 or get started with a one-time payment of taskset -c 1-3. For multi-worker servers, I assign fixed sets of cores per worker group to prevent contention. This configuration results in consistently lower Latencies, because CPU caches remain effectively filled. After making changes, I monitor threads, sockets, and soft IRQs using htop, ss, and perf.
Network Tuning: Combining RSS, RPS/RFS, and Kernel Parameters
Many NICs forward packets via RSS about queues, which I then bind to cores using IRQ affinity. I summarize the details and how this works under Receive-Side Scaling compactly. In addition, I control RPS/RFS so that soft IRQs do not conflict with the hard pinning logic. At the same time, I adjust buffers via net.core.rmem_max and net.core.wmem_max and check TCP options such as tcp_timestamps. These building blocks all contribute to the same goal: low latency with high Bitrate.
Interpreting MSI-X and Queue Layout Correctly
Modern NICs use MSI-X and create separate IRQs for each RX/TX queue. First, I check how many queues and channels the driver currently has enabled, and adjust this to match the core budget of the corresponding NUMA node. This prevents too many queues from crowding onto too few cores or, conversely, capacity from going unused.
Check and adjust the number of # queues and channels
ethtool -l ens192 # current limits (RX/TX/combined)
ethtool -L ens192 combined 4 # e.g., enable 4 queues
# Review RSS indirection and hash settings
ethtool -x ens192 # display indirection table and hash key
Many drivers give IRQs descriptive names (e.g.,. ens192-TxRx-0). I keep the Indirection Table consistent with the core assignment so that flows reliably end up in „their“ queue. If the hardware distribution differs, this results in unnecessary soft IRQ migrations.
Using RPS/RFS and XPS Effectively
RPS/RFS can distribute packets across cores via software—which is good for NICs without many queues, but counterproductive if I’m already using RSS and IRQ affinity for precise binding. I therefore make a conscious decision: Either hard binding via RSS+IRQ affinity and RPS off, or a small number of hardware queues and Targeted RPS. I also configure XPS for the TX direction so that outgoing packets are sent by the „correct“ cores.
IF=ens192
# Completely disable RPS (with proper IRQ pinning via RSS)
for q in /sys/class/net/$IF/queues/rx-*/rps_cpus; do echo 0 > "$q"; done
echo 0 > /proc/sys/net/core/rps_sock_flow_entries
# Alternative: Enable RPS selectively (example: CPUs 1–3)
for q in /sys/class/net/$IF/queues/rx-*/rps_cpus; do echo 0-0,0-0,0-0,0-0 > /dev/null; done
# Better: Use smp_affinity_list-like syntax:
for q in /sys/class/net/$IF/queues/rx-*/rps_cpus; do echo 1-3 > "$q"; done
echo 32768 > /proc/sys/net/core/rps_sock_flow_entries
echo 32768 > /proc/sys/net/ipv4/tcp_rfs_sock_flow_entries
Set # XPS to match the selected worker CPUs (TX paths)
for q in /sys/class/net/$IF/queues/tx-*; do echo 1-3 > "$q"/xps_cpus; done
Consistency is key: the RX-IRQ core, ksoftirqd load, and worker thread should all be on the same core (or pair of cores). This eliminates many cross-core bounce effects.
Take SMT/Hyper-Threading and core pairs into account
On systems with SMT, it often makes sense to reserve a physical core for an RX-IRQ and assign the corresponding worker to the Sibling Thread to assign—or to deliberately separate them if the workload is computationally intensive. I determine the thread pairs based on the topology and then make a clear decision rather than relying on random distribution.
Identify # sibling pairs
for c in /sys/devices/system/cpu/cpu*/topology/thread_siblings_list; do
echo "$(basename "$(dirname "$c")") : $(cat "$c")"
done
If I distribute RX-IRQ and user-space workers across the same physical core (different SMT threads), I get good L1/L2 locality, but bottlenecks can occur with CPU-bound workloads. Alternatively, I distribute the IRQ to Core X and the worker to Core Y on the same NUMA node to achieve true parallelism. I A/B test both variants and choose the one that delivers more stable latency.
Use smp_affinity_list, effective_affinity, and defaults
In addition to hex masks, I like to write in smp_affinity_list, since this allows for areas such as 1-3,6,8-9 can be set comfortably. To check, I verify effective_affinity respectively effective_affinity_list, because the kernel or drivers can exclude certain CPUs (e.g., offline cores or „managed interrupts“).
# Human-readable assignment
echo 1-3 > /proc/irq/181/smp_affinity_list
# Verify effective affinity
cat /proc/irq/181/effective_affinity_list
To prevent new IRQs—or those added after a driver reload—from being widely scattered again, I set the following when necessary: /proc/irq/default_smp_affinity to a reasonable default value (e.g., all cores of the relevant NUMA node, but excluding CPU0). I then specifically override individual critical IRQs.
Ensuring Persistence Through Reboots and Reloads
Affinity settings are volatile. I save them using a systemd oneshot unit that runs after the network initialization target, or via a small script that dynamically determines and maps IRQ lists. This ensures that assignments are preserved even after kernel updates and link resets.
# /usr/local/sbin/net-irq-pin.sh (example)
#!/bin/bash
set -euo pipefail
IF=${1:-ens192}
CPUS="1-3" # Target CPUs (select NUMA-consistent ones)
for irq in $(grep -i "$IF" /proc/interrupts | awk '{print $1}' | tr -d ':'); do
echo "$CPUS" > /proc/irq/$irq/smp_affinity_list || true
done
# systemd unit (outline)
# /etc/systemd/system/net-irq-pin.service
[Unit]
Description=Pin NIC IRQs
After=network-online.target
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/net-irq-pin.sh ens192
[Install]
WantedBy=multi-user.target
Important: I run the script again if the driver is reloaded or the number of queues is changed, because the IRQ numbers shift in those cases.
Virtualization and Containers: Thinking About Host and Guest Together
In KVM environments, I mount on the Host the physical NIC IRQs to cores on the appropriate NUMA node. At the same time, I pin vhost-net‑Threads and the QEMU process (or individual vCPUs) are also located there, so that the data paths on the host side remain short. In the Guest I map the IRQ affinity of the vNICs to those vCPUs that I have associated with physical cores on the host. Container workloads (cgroups/cpuset) benefit when the containers’ allowed CPUs overlap with the host’s RX/TX cores—otherwise, avoidable remote access occurs.
Delve Deeper into the Analysis: Identifying SoftIRQs, NAPI, and Congestion
Besides /proc/interrupts I look into /proc/softirqs, to see if there is a lot of work in the context of ksoftirqd instead of being handled directly in the IRQ handler—an indication of a persistently high load. With napi_defer_hard_irqs (Kernel-dependent) and by managing queue allocations, I control how aggressively NAPI batches data. ethtool -S It provides me with statistics for each queue on drops, busy states, and packet rates; this allows me to identify unbalanced queues and adjust affinity or RSS indirection accordingly.
# Quick Overview of SoftIRQ Distribution
cat /proc/softirqs | egrep 'NET_RX|NET_TX'
# View driver/queue statistics
ethtool -S ens192 | egrep -i 'rx|tx|drop|busy'
Practical Example: Mapping for 4 Queues on a Single Node
A typical setup I often use: NIC on NUMA Node 0 with 4 RSS queues. I avoid CPU0 and bind the queues to CPU1–4. I also assign the corresponding web or proxy workers to 1–4, assign XPS identically, and leave RPS disabled. This ensures short, consistent paths in both directions.
IF=ens192
QUEUES=(181 182 183 184) # example IRQs (determine beforehand)
CPUS="1-4"
Set # IRQ affinity and XPS
for i in ${!QUEUES[@]}; do
echo "$CPUS" > /proc/irq/${QUEUES[$i]}/smp_affinity_list
done
for q in /sys/class/net/$IF/queues/tx-*; do echo "$CPUS" > "$q"/xps_cpus; done
# Pin workers (systemd or taskset)
# systemd: CPUAffinity=1 2 3 4
# One-time: taskset -c 1-4
If the load continues to increase, I'll increase the number of queues (ethtool -L) up to the number of relevant cores on the node, and distribute them strictly according to a recognizable pattern (e.g., Queue ID → Core ID) so that flows do not „migrate“ over time.
Best Practices for Multi-Core Hosts
I relieve the pressure CPU0, since timers and kernel-level services often run there, which can cause interference under heavy load. I therefore prefer to assign critical IRQs to other cores and have CPU0 handle only a few non-critical sources. On NUMA systems, I remain consistent and keep adapters, IRQs, processes, and memory accesses on the same Node. In high-load environments, I separate I/O cores from application cores and isolate them as needed. I monitor all changes continuously and adjust the allocations iteratively.
A Measurable Approach: Analysis, Scripts, and a Fallback Plan
Before making any changes, I document the current state using mpstat, htop, /proc/interrupts and latency measurements using iperf3. I set up scripts that automatically apply affinity settings after a reboot or driver reload. For rollbacks, I keep neutral templates on hand so I can revert immediately in case of issues. In staging, I test load profiles that closely resemble my production environment and repeat the Measurement after each adjustment. Only then do I permanently enable the profile on the target host.
Avoid common stumbling blocks cleanly
Masks that are too wide spread the work across too many people CPUs and slow down caches, while masks that are too tight clog up queues. Overlooked NUMA-specific behaviors generate remote memory accesses that cause response times to fluctuate. Hard pinning occasionally conflicts with RPS/RFS settings, so I explicitly check the SoftIRQ load balancing. After kernel updates or driver changes, I revalidate all IRQ numbers because mappings can change. With careful steps and clear Documentary I remain capable of acting.
Briefly summarized
Targeted IRQ pinning maps network interrupts to a small number of suitable Cores, reduces overhead, and stabilizes response times. To achieve this, I fine-tune IRQ and CPU affinity, take NUMA into account, and verify the results with a series of measurements. Where automatic tuning suffices, I let `irqbalance` handle it; I assign critical queues to specific cores. With RSS, RPS/RFS, and customized kernel parameters, the tuning achieves its full Effect. Anyone who follows these steps diligently will see a noticeable improvement in network performance on multi-core Linux servers.
