Visualizing Linux PSI Monitoring with Grafana: Understanding and Monitoring Resource Pressure Effectively

I show how I Linux Psi collect data with Prometheus and visualize it in Grafana to measure resource pressure on the CPU, memory, and I/O as actual latency. This allows me to identify Bottlenecks Early on, assign them to cgroups or containers and trigger automated countermeasures as needed.

Key points

• PSI Metrics: some/full for CPU, memory, I/O, and moving averages
• cgroup Focus: A container- and service-specific view rather than just a global one
• Alert Trigger: Using poll/epoll for events at thresholds
• Grafana: Panels for Trends, Peaks, and Top-N Contributors
• Best Practices: Thresholds, Time Windows, Correlation with System Metrics

PSI in a Nutshell: Understanding Pressure as Actual Waiting Time

PSI answers the question of how much Wall-clock time Tasks wait in vain for CPU, RAM, or I/O. The files under /proc/pressure/{cpu,memory,io,irq} offer two perspectives: some shows phases during which some tasks are waiting, full indicates moments when all non-idle tasks are blocked. I evaluate both values separately because some starts earlier and full highlights actual downtime. In addition, I use avg10, avg60 and avg300, to distinguish between short-term fluctuations and longer-term trends. Regarding the growing total-I realize how much pressure has built up since the boat trip and where Hotspots lie.

System-wide vs. cgroup PSI: Choosing the Right Level

I deliberately distinguish between global pressure values and the view per cgroup. The files under /proc/pressure/ show the system as a whole, while cgroup v2 also cpu.pressure, memory.pressure and io.pressure per group. In container environments, I use this to neatly organize pods, services, or dumpster diving This assignment prevents blind flights: Instead of having to guess, I can see the source directly within the group. On multi-tenant hosts, this allows me to separate shared and dedicated workloads and manage Limits targeted.

Check Prerequisites: Enable the kernel, PSI, and cgroup v2

Before I start collecting metrics, I make sure the platform is a good fit:

• Kernel version: PSI is available starting with Linux 4.20. I'm checking with uname -r and check via zcat /proc/config.gz | grep CONFIG_PSI, whether the support is compiled in.
• PSI Runtime Flag: Some distributions allow the optional boot flag psi=1, to fully activate PSI. I use it as needed and verify that /proc/pressure/* Provides content.
• cgroup v2: For the per-service/container view, I use the unified hierarchy. I'll check with mount | grep cgroup2 and expect a cgroup2-Mount (often /sys/fs/cgroup). If it's missing, I enable it using the kernel parameter systemd.unified_cgroup_hierarchy=1 (Restart required).
• Authorizations: Exporters on the host need read permissions for /proc/pressure/* and, if applicable, to /sys/fs/cgroup/*/*.pressure. In containers, I mount these paths as read-only.

Using PSI Triggers: Respond Automatically Instead of Just Observing

In addition to time series, I also present events via poll or epoll by writing thresholds and time windows into the PSI files. As soon as the resource load exceeds the threshold within the window, an event is triggered and I initiate countermeasures. This could be an additional pod, a cache flush, or the temporary throttling of a Batch Jobs . In Systemd units, I tie this response directly to services and keep latency low. This way, monitoring becomes a control mechanism, not just a Display.

In practice, I use a compact watcher program that monitors the relevant *pressure*-files, via write() a trigger (some or full (including threshold value and window in µs) and then processed using epoll waits for events in a blocking manner. This allows me to save poll cycles and respond deterministically. I intentionally keep the windows a little longer (e.g., 10–30 s) to filter out transients, and differentiate based on the resource: memory is more sensitive than io, CPU It needs to be clearer in order to fire.

Exporting PSI to Prometheus: Agents, Metrics, Labels

For the time series, I collect PSI data using a dedicated exporter or integrate the values into existing agents such as the Node Exporter. It's crucial to use consistent labels for hosts, cgroups, and containers so that queries in Grafana filter properly. In Kubernetes, I also use cAdvisor and Kubelet metrics for *_pressure_*_waiting_seconds_total, so that the node, pod, and container levels remain aligned. For traditional hosts, I read /proc/pressure/* direct and folder some and full on separate metric names. A guide to agent integration can help you get started; for example, see Set Up Node Exporter.

Exporter Variants in Detail: Node, cgroup, and Kubernetes

I use different approaches depending on the situation:

• Node Exporter (Host level): I'm enabling the pressure-Collector (if not enabled by default), e.g., via --collector.pressure. It provides metrics such as node_pressure_cpu_some_avg10, node_pressure_memory_full_avg60 and node_pressure_io_waiting_seconds_total{state="some|full"}. The latter are suitable for rate()-Analyses and Top-N.
• Custom cgroup exporter (Service/Container Level): For a fine-grained view, I read /sys/fs/cgroup//{cpu,memory,io}.pressure and generate metrics such as cgroup_pressure_memory_waiting_seconds_total{state="full",cgroup="..."} and the avg10/60/300‑Gauges. I normalize the cgroup path as a label (cgroup) or open the folder service/container Labels.
• Kubernetes: Prometheus scrapes at the node level node-exporter. For the container view, I use a DaemonSet exporter with hostPID:true and read-only mounts from /sys/fs/cgroup and /proc, so that I can see the host's cgroup files. I also use Kubelet/cAdvisor metrics, provided they return the PSI totals; the labels namespace, pod and container I remain consistent in this regard.

A clear... helps me Label Strategy: instance (Host or NodeName), cgroup (path), namespace/pod/container (for K8s) as well as state (some/full) and resource (CPU/memory/io/irq). This allows me to aggregate data at a high level and zoom in at the same time.

Prometheus Jobs, Recording Rules, and Sample Queries

To ensure accurate analyses, I use two patterns: percentage gauges (avg10/60/300) and derived rate()-Values on the *_total_seconds_waited_-meters.

• Scrape: 15 seconds is a good starting point. Shorter durations increase the load, but avg10 but rarely adds value.
• Recording Rules: I calculate derived time series to simplify dashboards and alerts:
  • record: psi:node_memory_full:avg60 = avg_over_time(node_pressure_memory_full_avg10[60s])
  • record: psi:node_io_full:rate5m = rate(node_pressure_io_waiting_seconds_total{state="full"}[5m])
  • record: psi:cgroup_memory_full:rate5m = sum by (cgroup) (rate(cgroup_pressure_memory_waiting_seconds_total{state="full"}[5m]))

I use PromQL to build typical views:

• Host Trend: node_pressure_memory_full_avg60 over time, broken down by node.
• Top N Polluters: topk(5, psi:cgroup_memory_full:rate5m) Shows the loudest cgroups.
• Impact on Latency: (increase(http_request_duration_seconds_sum[5m]) / increase(http_request_duration_seconds_count[5m])) against node_pressure_io_full_avg60 to identify correlations.
• Recognizing Plateaus: clamp_min(psi:node_io_full:rate5m, 0.0) as a heat map for each node.

Grafana Dashboards: Visualizing Trends and Identifying Hotspots

In Grafana, I display CPU, memory, and I/O load separately, for each of some and full as separate charts. Bar gauges show me the current status, while time series panels reveal peaks and plateaus. For root cause analysis, I use Top-N views organized by cgroup, container, or pod, and from there I jump to detailed panels. The key is the combination of avg10, avg60 and avg300, to prevent overloads caused by brief spikes. Anyone looking to plan ahead for dashboards will find helpful ideas related to the Grafana and Prometheus Stack.

Alerts in Practice: Rules, Thresholds, Escalation

I follow a two-step model: warning for early signs, Critical for a persistent bottleneck. As an example, I'll use:

• Memory
  • Warning: node_pressure_memory_full_avg60 > 0.01 for 10–30 seconds
  • Critical: node_pressure_memory_full_avg60 > 0.05 for ≥60 s
• I/O
  • Warning: rate(node_pressure_io_waiting_seconds_total{state="some"}[5m]) > 0.02
  • Critical: node_pressure_io_full_avg60 > 0.02 for ≥120 s
• CPU
  • Warning: node_pressure_cpu_some_avg60 > 0.05
  • Critical: node_pressure_cpu_full_avg60 > 0.01 (because full (which really hurts here)

In annotations, I link contexts (top cgroups, throughput, latency) and trigger playbooks: Scale-Up, Adjust Limits, Cache Actions, Batch Throttling. When events recur, I prioritize capacity decisions.

Thresholds and Alerts: Choosing the Right Time Window

I define clear thresholds for each resource and distinguish between hard failures and brief load spikes. For example, I evaluate Memory full I consider 5 % lasting longer than 60 seconds to be critical, while 1–2 % lasting more than ten seconds only trigger a warning. For the CPU, I set stricter limits full, since widespread waiting times there noticeably slow things down. Linking this to throughput and latency metrics adds value: When pressure increases and requests slow down, the urgency rises. I always set alerts based on time windows, not on individual values, to avoid false positives caused by Bursts to avoid.

Kubernetes: Scrape Setup, Permissions, and Correlation

In the cluster, I collect PSI in such a way that the levels align:

• Node Exporter as a DaemonSet: Standard Scrape per node provides global PSI values.
• cgroup Exporter as a Sidecar/DaemonSet: Reads the host's cgroup v2 files and applies labels based on them namespace/pod/container. I use only the minimum necessary permissions and RO mounts.
• Kubelet/cAdvisor: I enable the output of relevant container metrics and scrape the Kubelet endpoint. I keep the label join keys (e.g.,. container vs. container_name) consistent so that PromQL joins work seamlessly.
• JOIN with Workload Metrics: I correlate Pod-PSI with app latencies (e.g., HTTP metrics), CPU limit hits, and memory errors. This helps me determine whether limits, scheduling, or storage bottlenecks are the cause.

PSI Files, Key Figures, and Interpretation: A Concise Overview

The following table summarizes the most important files, key metrics, and use cases so that I can interpret them more quickly and build appropriate Grafana dashboards. I use it during analysis to plan my next step: tuning, scaling, or troubleshooting. The differences between some and full as well as the three average windows. This is how I organize symptoms chronologically and check whether the pressure is localized or widespread. The „Use“ column helps with quickly Classification.

Resource	File	Key figures	Meaning	Use
CPU	/proc/pressure/cpu	some, full; avg10/60/300; total	Waiting time for available computing time	Overloaded hosts, CPUs that are too constrained—Limits
Memory	/proc/pressure/memory	some, full; avg10/60/300; total	Wait time due to reclaim, swap, or near OOM	RAM bottlenecks, cache pressure, faulty Requests
I/O	/proc/pressure/io	some, full; avg10/60/300; total	Wait time for storage devices/file system	Slow storage devices, sync storms, Flush-phases
IRQ	/proc/pressure/irq	some, full; avg10/60/300; total	Pressure from Interrupt Handling	Network load, driver tuning, Affinity
cgroup	*/{cpu,memory,io}.pressure	some, full; avg10/60/300; total	Pressure per Service/Container	Root-cause analysis, targeted Limits

Practical Guide: Securing Hosting and WordPress Stacks Efficiently

On heavily used WordPress hosts, PHP-FPM, the database, and the cache layer regularly compete for RAM and I/O, which I can monitor via memory and io I see it right away. It rises full For memory, I optimize the OpCache, increase pool sizes gradually, or reduce the number of resource-intensive plugins. When there’s I/O pressure, I check query plans, journaling settings, and asynchronous writes. PSI per cgroup shows whether the web server, worker, or database is causing the bottleneck. If you want to dig deeper, you’ll find guidance in the Linux-PSI Guide, which summarizes the introduction and evaluation.

Capacity Planning and Tuning: From Numbers to Action

I correlate PSI with CPU utilization, page faults, I/O throughput, and latencies to identify the root causes. If the problem persists, Memory full I scale RAM, optimize reclaim parameters, or split up workloads. Shows io full For long plateaus, I increase queue depths, enable write-back strategies, or use faster storage devices. For CPU pressure, I measure runqueue lengths in parallel, adjust scheduling classes, and distribute hot threads. I make decisions only when trends in the avg60 and avg300 stay consistent and not just a Spike is available.

Troubleshooting and Validation: From the Host to the Container

If the PSI values are missing, I check the kernel version, CONFIG_PSI and, optionally, the boot parameter psi=1. Then I verify the output of the files at /proc/pressure/* manually and compare them with the Exporter metrics. In cgroup v2, I also monitor the *.pressure-files within the group directories. I test alerts using load generators and monitor the response logic via epoll, to detect configuration errors early on. Finally, I cross-check Grafana dashboards with logs, traces, and profiler results to ensure that diagnostics and corrective actions are accurate fit.

Typical obstacles I take into account:

• Swap Interaction: Light memory some-These values are normal during aggressive reclaim. The situation becomes critical when full increases while latencies rise at the same time.
• CPU Isolation and Affinity: Pinned/isolated cores can be locally CPU at full capacity generate, even though the host still has some capacity left. I'm checking irq-PSI additionally, if the network/storage has a high interrupt load.
• Virtual Environments: In VMs, PSI values also reflect hypervisor-related factors. I measure host- and guest-level metrics separately to clearly identify overcommitment.
• Scrape Overhead: PSI itself is efficient, but scrape intervals that are too short increase the load on Prometheus. 15 seconds is often the sweet spot.
• Label cardinality: cgroup paths can explode. I manage them using labeldrop/keep and map only the layers I'm analyzing (e.g., "Service" instead of every short-lived task cgroup).

Briefly summarized

PSI measures actual waiting time on CPU, RAM, and I/O, thereby providing a clear indication of resource constraints. Using Prometheus exports and Grafana dashboards, I build a view that isolates root causes and quickly identifies hotspots. The separation of some and full plus the windows avg10/60/300 makes decision-making more reliable. I set alerts for specific time periods, link them to latency thresholds, and control automatic responses via triggers. This allows me to make informed capacity decisions, resolve bottlenecks in a timely manner, and keep services running smoothly on a day-to-day basis responsive.