XFS Allocation Groups: Maximizing Performance on NVMe Systems

XFS NVMe It only truly demonstrates its full potential when I consistently tailor allocation groups, block sizes, mount options, and the I/O scheduler to the characteristics of modern NVMe SSDs. This post demonstrates specifically how I plan, format, and operate an XFS file system on NVMe so that AG parallelism, log tuning, and hardware queue depth deliver measurable throughput and low latency.

Key points

• AG Design: Choose enough allocation groups to support parallelism, but without causing excessive CPU overhead.
• Block size: Map file system blocks to physical 4K sectors to avoid multiple accesses.
• Mount Tuning: Combine `noatime`, `allocsize`, and `logbufs/logbsize` as needed instead of using the default settings.
• scheduler: Test and set „none“ or „mq-deadline“ depending on your latency goals.
• Workloads: Configure the database, streaming, and AI-Scratch with the appropriate number of AGs and readahead.

Why Allocation Groups Speed Up NVMe

Allocation groups separate free blocks, inodes, and B+ trees into independent regions so that multiple threads can work simultaneously and Locks access them less frequently. This exact distribution is well-suited to NVMe, which handles requests simultaneously using many queues and deep parallelism, thereby reducing block conflicts. In practice, I translate hardware parallelism into a sufficient number of I/O operations, resulting in parallel allocations and fast metadata updates, which smooths out latency spikes. A Performance comparison File system benchmarks often show how XFS scales under parallel access, while sequential workloads continue to run reliably. However, striking the right balance remains crucial: Too few AGs limit parallel allocations, while too many result in noticeable performance overhead. CPU time.

Determine the number and size of workgroups

When formatting, I deliberately set the number of AGs—typically ranging from a few dozen to 64–128 AGs per terabyte—to achieve sufficient concurrency without excessive administrative overhead and to Parallelism to take full advantage of. With mkfs.xfs -f -d agcount=64 /dev/nvme0n1 I explicitly define the distribution; via -d size= Alternatively, you can adjust the AG size. For workloads with many small files, I tend to use more AGs; for large sequential streams, I use slightly fewer AGs to keep CPU load under control. I avoid extremes because a large number of very small AGs cause a lot of overhead when filling up the file system. The key factor remains: I base my decision on capacity, RAM configuration, and typical I/O characteristics to ensure that allocations are distributed evenly across Clubs sprinkle.

Correctly map block sizes to the hardware

Many NVMe SSDs use 4K sectors internally, even though they provide 512 bytes externally, so I set the file system block size to 4096 bytes, thereby reducing internal read-modify-write cycles for Write Accesses. For formatting, for example, I use mkfs.xfs -f -b size=4096 /dev/nvme0n1, when the physical sector size is 4K. A misaligned file system generates unnecessary additional I/O operations, which noticeably slows things down—especially with small, random writes. The correct block size makes accesses consistent, smooths out latency, and delivers better IOPS for short requests. For special cases involving very large sequential jobs, I combine 4K blocks with a larger read-ahead so that the Throughput rate increases.

Mount Options for NVMe Volumes

Even without tuning, XFS already runs quickly, but specific mount options can squeeze out even more performance and eliminate unnecessary metadata updates Reading load. I activate noatime,nodiratime, set a larger one depending on the workload allocsize (e.g., 64M) and increase the log buffer by logbufs=8,logbsize=256k for higher metadata throughput. Instead of discard At Mount, I lead fstrim periodically so that TRIM commands run in batches. An example line in /etc/fstab looks like this: /dev/nvme0n1 /data xfs noatime,nodiratime,allocsize=64m,logbufs=8,logbsize=256k 0 0. The following table lists common options along with their effects and typical use cases, so that I can make decisions more quickly and the Configuration document.

Option	Effect	When to use
noatime,nodiratime	Reduces metadata writes during accesses	Many reads, web and analytics workloads
allocsize=64m	Consolidates allocations, reduces fragmentation	Large sequential write streams
logbufs=8	More parallel log buffers for metadata	Transaction load, many small updates
logbsize=256k	Larger log pads	Higher metadata throughput
none discard	Avoids synchronous TRIM costs	Instead, regular fstrim

I/O Schedulers: none, mq-deadline, and others.

NVMe controllers efficiently manage requests on their own, so I often use none best and keep the Overhead low. For workloads with strict latency requirements, I'm testing mq-deadline, because it can stabilize response times, even if the maximum throughput decreases slightly. While bfq While it scores well in terms of interactivity, it’s rarely the first choice for server NVMe. I don’t make that decision until after running tests with fio, which measure IOPS, throughput, and latency separately for read/write and random/sequential operations. I'll go into more detail on how to weigh the options in this concise I/O Scheduler Guide, before I set the configuration to production.

Tailor Workloads to Specific Needs

Databases with many commits benefit from a moderate number of AGs, aligned 4K blocks, noatime and raised logbsize, so that metadata transactions run quickly. I set up Analytics jobs and streaming pipelines with a higher allocsize and more read-ahead for high sequential throughput. For AI/ML scratch data and many parallel workers, I tend to choose more AGs, noatime, bundled allocations, and none as a scheduler. Backups and archiving jobs also benefit from periodic fstrim, to reduce the load on SSD garbage collection. I validate each adjustment with a series of reproducible measurements before I Defaults permanently replace.

Quickly Recognizing Common Symptoms

If XFS reports „No space left on device“ despite there obviously being free space, it is often because a single AG is full, which is why I adjust the data distribution, agcount and check for free metadata spaces. I usually interpret unexpectedly high latency during small random writes as a sign of improper block alignment or block sizes that are too small allocsize or excessive metadata updates. In such cases, 4K blocks, larger allocation chunks, and noatime, to consolidate write operations. If the CPU load on the file system increases noticeably, the number of AGs may have been set too high, especially if the file system is nearly full. In that case, I reduce the number of AGs when reformatting or expand the partition to Administration to reduce.

Parameter Overview: A Quick Check

For recurring setups, I keep a short checklist on hand that I go through before every format, and that way Constance to optimize performance. First, I check the physical sector size, queue depth, and controller features of the NVMe devices. Then I set the number of AGs or the AG size and set the block size to 4K. Next, I define mount options that are appropriate for the workload and schedule a periodic fstrim. Finally, I test different I/O scheduler options and document the fastest combination for each specific use case.

Step-by-Step Planning for a New XFS on NVMe

To start, I determine the capacity, physical sector size, typical file sizes, and number of parallel threads so that the AG Planning starts properly. Then I format the drive with a customized AG count, a 4K block size, and optional inode parameters if I expect a large number of small files. In the next step, I mount the drive using noatime, more appropriate allocsize as well as optimized log parameters, and check the results using fio. Next comes the choice of scheduler, where I none and mq-deadline compare them while keeping an eye on both IOPS and latency. Finally, I set up monitoring and scheduled fstrim, so that performance remains strong in the long term constant remains the same and there are no surprises.

Integration into Hosting Environments

In hosting scenarios involving containers, web stacks, and databases, a well-planned XFS configuration directly translates into faster response times and Throughput . In doing so, I take into account the queue depth and the number of parallel workers to optimally combine the number of threads and the scheduler. I provided a detailed explanation of why the queue length on NVMe sets the pace in the post on Queue Depth developed. For data-intensive microservices, I often increase readahead, bundle allocations, and measure performance iteratively after each change. Those who run their applications on high-performance managed or root servers benefit from low latency, high parallelism, and predictable operations on XFS.

Choose Reflink, inodes, and metadata features carefully

When formatting, I decide whether CoW/Reflink makes sense for my use case. With mkfs.xfs -m reflink=1 I enable Copy-on-Write and fast clones, which saves space and time when creating many copies, VM images, or build artifacts. For databases with heavy write loads, I disable Reflink (reflink=0), to reduce metadata overhead and minimize log output. In addition, I check finobt (Free-Inode-B-Tree), which speeds up allocation decisions for many inodes and is typically enabled by default in current tools anyway.

The Inode Size I decide on -i size=. For workloads with many extended attributes (ACLs, SELinux, application metadata), I choose 512 or 1024 bytes so that attributes are more likely to fit within the inode and do not end up in separate blocks. Example: mkfs.xfs -f -b size=4096 -i size=512 /dev/nvme0n1. Larger inodes take up a little space, but reduce the number of accesses when metadata is read or written frequently. Features such as bigtime extend the usable timestamp range of modern systems and are useful in new installations without any measurable performance penalty. For optional structures such as rmapbt I usually skip them for pure performance volumes, since they primarily improve manageability and auditability but also create additional work.

External Log, Log Size, and Stripe Alignment

For metadata-intensive workloads, it’s worth using a separate log device (journal) on a second, very low-latency NVMe drive to minimize contention between user data and log writes. I set this up during formatting with -l logdev=/dev/nvme1n1,size= and set the log size so that burst phases do not constantly trigger log forces (often 1–4 GiB, depending on the transaction pattern). Together with logbufs/logbsize On Mount, an external log noticeably stabilizes transaction times when there are many small files or metadata updates.

If the NVMe is behind a RAID or Device Mapper, I align XFS to the stripe sizes so that write operations fall neatly on stripe boundaries. This is done during formatting using -d su=,sw=. Then I check the values using xfs_info /mount. Important: These parameters cannot be changed later without reformatting. For single-NVMe drives without underlying striping, I let XFS handle the autotuning.

Ensure Partition and Block Alignment

Before I format, I create 1 MiB-aligned partitions so that file system blocks align neatly with physical 4K boundaries. With parted -a optimal or the corresponding GPT configuration to prevent incorrect offsets. I verify the effective physical and logical sector sizes using cat /sys/block/nvme0n1/queue/physical_block_size and logical_block_size. Only when this foundation is in place can 4K blocks and allocation chunks achieve their full potential.

Controlling Direct I/O, Page Cache, and Writeback

For databases and log streams that manage their own cache, I specifically set O_DIRECT, to avoid duplicate caching in the page cache. XFS scales very well here, as long as I don't simultaneously buffer the same files in a mixed manner and write to them directly. For streaming workloads, a higher readahead the throughput: blockdev --setra 4096 /dev/nvme0n1 (equivalent to 2 MiB) is a practical starting value that I measure and fine-tune as needed.

To maintain system balance, I carefully adjust the writeback thresholds. Instead of using percentages, I rely on absolute values to avoid accumulating too much dirty data in systems with large amounts of RAM. Example (test with caution):

sysctl -w vm.dirty_background_bytes=268435456
sysctl -w vm.dirty_bytes=2147483648

This is how I prevent long flush waves that increase latency. I document these settings for each host so that they remain reproducible and aren't overridden by distribution defaults without my noticing.

Leveraging Queue and CPU Topology

NVMe uses multi-queue I/O: Each CPU core typically has its own hardware queues, so I don't leave IRQ distribution and CPU affinity to chance. A running irqbalance is the baseline; for special latency cases, I pin the NVMe IRQs using /proc/irq/*/smp_affinity specifically targeting NUMA-adjacent cores. cat /sys/block/nvme0n1/queue/scheduler shows me the active scheduler, nr_requests and rq_affinity influence how requests are distributed across queues. For highly parallelized workers, I'm increasing this as a test /sys/block/nvme0n1/queue/nr_requests moderate, to better cushion peaks without overloading the driver.

In addition, I can fine-tune interrupt coalescing for the NVMe devices (a controller feature). A moderate increase in the coalescing parameters smooths out the IRQ load, but must not compromise the latency targets. I always document such adjustments with fio‑Latency percentiles before they go into production.

Quotas, Projects, and Isolation

In multi-tenant environments, I rely on Project Quotas, so that loads and space usage remain clearly separated. I mount it with prjquota and manage borders via xfs_quota velvet /etc/projects and /etc/projid. For example, this allows you to set strict limits on build directories, database instances, or client directories without restricting transaction concurrency.

For ingestion-heavy directory trees that write many large files sequentially, the filestreams‑Allocator can be useful. It keeps files in a directory closer together and reduces fragmentation. I enable it specifically via a mount option for volumes that are clearly streaming-oriented, and measure the effect on throughput and CPU load.

Growth, Snapshots, and Lifecycle

XFS can grow online, but not shrink. That's why I plan capacity and the virtual machine layout so that future expansions via LVM/VMDK are possible without any problems. With xfs_growfs /mount When I expand the file system upward, the AG structure grows along with it. Parameters such as sunit and swidth are fixed—so anyone who changes RAID geometries should plan for reformatting and restoration.

For consistent Snapshots When using LVM or storage backends, I briefly freeze the file system: xfs_freeze -f /mount, Create a snapshot, xfs_freeze -u /mount. This minimizes log replays and ensures clean restores. To maintain runtime health, I plan to perform regular xfs_scrub (where available) and hold xfs_repair available as an offline tool. SMART data, nvme smart log and iostat -x are on my watchlist so I can detect degradation early on.

Test Methodology and Reliable Baselines

Before I replace the default settings, I take measurements that are reproducible. I start with clear fio- Profiles that consider IOPS, throughput, and latency separately and include warm-up phases:

[global]
ioengine=io_uring
direct=1
runtime=60
time_based=1
group_reporting=1
randrepeat=0

[randread-4k]
filename=/data/testfile
rw=randread
bs=4k
iodepth=64
numjobs=8

[randwrite-4k]
filename=/data/testfile
rw=randwrite
bs=4k
iodepth=64
numjobs=8

[seqread-1m]
filename=/data/testfile
rw=read
bs=1m
iodepth=32
numjobs=4

[seqwrite-1m]
filename=/data/testfile
rw=write
bs=1m
iodepth=32
numjobs=4

Depending on the goal, I adapt numjobs to the CPU cores and iodepth to the desired queue depth. Consistent boundary conditions are important (same fill level, identical mount options, cleanly trimmed volume). I filter out outliers by comparing the mean and 99th percentile across multiple runs. This allows me to make informed decisions between none and mq-deadline, between smaller and larger allocsize or when considering whether an external log is actually helpful.

Fine-Tuning allocsize, Reflink, and Others.

allocsize is a useful tool, but not a panacea. With purely random small writes, allocation chunks that are too large create unnecessary write overhead. I therefore choose conservative values for each workload and monitor fragmentation and latency. With Reflink enabled, I avoid continuous small updates in the same file regions because CoW involves additional metadata overhead. If I need fast clones, I keep the log buffer large and ensure there is plenty of free, contiguous space across multiple AGs.

Safe Defaults: Barriers, Discard, and Consistency

Writing Barriers (Write Barriers) and FUA are enabled by default on modern stacks—I don't change that setting to avoid the risk of data loss. nobarrier That's out of the question for me, even if individual benchmarks show a short-term increase. discard at Mount is not available; the system-wide fstrim‑Timer efficiently performs TRIM during idle periods. This combination provides me with reliably low latency and consistent SSD performance over the long term.

Briefly summarized

XFS scales horizontally through allocation groups, thereby leveraging the inherent parallelism of NVMe effectively. I don't determine the number of AGs, block sizes, mount options, and schedulers based on gut feeling, but rather on the workload profile and measurement data. For small random writes, proper alignment and a lean scheduler are key; for large streams, more generous allocsize and readahead. I resolve typical issues, such as unbalanced AGs or synchronous discards, by redistributing and periodically fstrim. By systematically adjusting these parameters, you can keep latency low, boost IOPS, and ensure long-term Performance.