Skip to content 
TCP Small Queues limits the number of pending bytes in the Linux outbound path per TCP flow, thereby reducing Latency including buffer bloat, in a targeted manner. I'll show how this mechanism works in the Linux networking Stack shows how I set reasonable limits and what interactions arise with pacing, QDiscs, and congestion control.
Key points
- Per-Flow Limit: TSQ caps the number of outstanding bytes per TCP socket.
- Less Bufferbloat: Shorter queues reduce RTT.
- Backpressure: Applications write more slowly when the limit is reached.
- Fairness: No single flow occupies an entire queue.
- Adaptive Control: The limit is based on the rate and segment size.
How TCP Small Queues Works
TSQ picks up where TCP segments leave off QDisc and passes it to the driver. When I write data to a socket, the kernel checks the bytes already allocated for that flow before each enqueue operation. If the flow reaches the limit, the logic marks the socket as throttled and stops further enqueue operations. Only when the network card frees up buffers is the socket allowed to send again, and I can push data back into the stack. This tight feedback loop keeps the Queues It's short and makes response times more predictable.
Why Long Queues Drive Up Response Times
Create large driver and QDisc queues Buffer bloat, especially with TSO/GSO and high outbound traffic volumes. A large download can fill up the outbound queues, while interactive flows such as SSH, API calls, or VoIP are put on hold. The overflowing queue then dominates the RTT instead of the actual link time. Congestion control reacts slowly because ACKs arrive late, and makes poorer CWND decisions. TSQ limits the number of pre-buffered bytes per flow so that small, time-critical packets are sent over the link quickly.
A Look Under the Hood: What the Kernel Does
Under the hood, the kernel does not count „packets,“ but rather bytes—more precisely, the memory bytes that the socket has already placed in the queue. What matters is what the stack contains in terms of skbuff structures, including truesize has been allocated but has not yet been processed by the NIC. TSQ links this to a Throttle/Unthrottle-Path: If a socket reaches the credit limit, the stack sets a throttling flag and does not resume until TX operations are complete (NAPI/IRQ) write_space() so that the application can send again. This feedback is faster than purely loss-based signals from congestion control and takes effect before the QDisc. With TSO/GSO, the mechanism remains effective because the limit at the before applies the byte budget associated with the segmentation: Large superframes are only admitted into the QDisc if there is sufficient credit available, which helps contain bursts.
Dynamic Limits and Pacing
I benefit from TSQ because the limit doesn't remain strictly static, but rather adjusts to Rate and takes segment size into account. The goal is about one millisecond of data in the transmission path per flow, regardless of whether the rate is 100 Mbit, 1 Gbit, or 10 Gbit. With a fast connection, the allowed byte credit increases; with a slow connection, it decreases. In combination with TCP pacing, bursts remain small and ACKs return faster. This way, I achieve noticeably lower Latency peaks, without unnecessarily reducing throughput.
Per-Socket and App Interaction
TSQ is only effective if the application also senses the backpressure. Therefore, I take into account settings such as SO_SNDBUF, TCP_NOTSENT_LOWAT and autocorking. A send buffer window that is too large can push many bytes onto the stack in a short period of time; TSQ does slow things down, but the app doesn't notice it until send() is blocked or returns EAGAIN. With TCP_NOTSENT_LOWAT I adjust the „unsent“ portion in userland and use it to supplement TSQ on the kernel side. Autocorking (or explicitly TCP_CORK/MSG_MORE) helps bundle small writes without causing latency spikes. Pacing limits per socket (e.g., via SO_MAX_PACING_RATE) are compatible with TSQ: The rate smooths out over time, and the byte limit restricts the data in space. Important: TCP_NODELAY It disables Nagle and can increase interactivity, but without TSQ, the risk of bursts increases; with TSQ, I have both under control.
Practical Guide: Useful TSQ Values
I establish the global framework with net.ipv4.tcp_limit_output_bytes (Sysctl). Typical defaults range from 128 to 262 KB per flow. For many web and API workloads, I choose lower values to keep interactive responses snappy. For backups or replication, I raise the limit moderately, as long as the RTT remains stable. If you want to dive deeper into the queue page, you’ll find the basics at Packet Queues on the Server, which help with classification.
| Scenario | Link Rate | Guideline: tcp_limit_output_bytes | Goal |
|---|---|---|---|
| API/HTTP: Highly Interactive | 100 Mbit – 1 Gbit | 64–128 KB | low RTT, short spikes |
| Mixed Load: Web + Downloads | 1–10 Gbit | 128–256 KB | Balance off Throughput and latency |
| Replication/Backups | 1–10 Gbit | 256–512 KB | Constant bulk flow, acceptable Latency |
| WAN with high RTT | 10–100 Mbit | 96–192 KB | shorter bursts, fairer Cues |
QDisc and Congestion Control Working Together
TSQ operates at the entrance to the QDisc, while algorithms such as fq_codel manage congestion on the link. Together, they reduce queues and ensure fair distribution. With TCP BBR I also benefit because more realistic RTT measurements lead to better pacing and cwnd control. CUBIC also responds more smoothly when I eliminate excessive queueing times. This allows throughput to grow organically, while the Response time remains under control.
Virtualization and Cloud Stacks
In VMs, multiple buffer levels add up: guest QDisc, virtio/vhost queues, host QDisc, and the physical NIC. I keep TSQ active in the guest and set a conservative limit there to prevent large bursts from reaching the host. On the hypervisor, I ensure short latency chains by using fair QDiscs, moderate TX rings, and proper IRQ pinning. SR-IOV can reduce latency but shifts responsibility to the guests: without TSQ in the guest, long VF queues are a risk. In containers, TSQ per NetNS As usual, I use cgroup pacing and CPU limits to prevent a noisy neighbor from indirectly increasing latency. It’s also important to look at coalescing and offloads in the virtio path: excessive coalescing lengthens ACKs, while too little reduces efficiency—I adjust based on the latency target, not dogmatically.
Wi-Fi and Embedded Systems: Handling Special Cases Correctly
On Wi-Fi connections, the Aggregation at the MAC layer. If I set the number of bytes in the transmission path too low, the driver can’t bundle as many frames, which reduces efficiency. In such setups, I carefully increase the limit and check the aggregation level. OpenWrt and embedded platforms also benefit from streamlined driver paths and fewer atomic operations. I test every adjustment under real-world wireless load before I Profile roll out widely.
Monitoring and metrics that really count
I observe the RTT—Distribution per socket, and I look for outliers, not just averages. Using ss, tc, and exporters, I read queue lengths, retransmissions, and pacing_rate. eBPF programs send me events when sockets are throttled and become available again. Time-to-First-Byte and the 95th/99th percentiles show whether TSQ is having the desired effect. Without measurement data, any Optimization a flight by blind flight.
Meaningful A/B and load tests
I measure TSQ effects in a reproducible manner: First, a baseline without any changes, then isolated parameter sweeps (e.g., 64, 96, 128, 192 KB). For mixed workloads, I run parallel streams (bulk + many short requests) and compare the 95th and 99th percentiles of latencies, not just the median. Even when I clearly segment test runs (warm-up, measurement window, cool-down), artifacts remain detectable. I ensure consistency: the same payload patterns, identical route/MTU, and identical server and client CPU frequencies. On WAN links, I simulate delay, jitter, and loss using tc netem, to check whether TSQ limits cap too early at high BDP. Only when the percentiles narrow and retransmits/losses remain stable do I implement the values into production.
Hardware Tuning and Driver Details
I'm checking the TSO/GSO settings, the NIC's ring buffer, and IRQ control so that TSQ works smoothly. TX rings that are too large extend the queue at the device; those that are too small reduce utilization. Coarse interrupt bundling delays ACKs, while fine bundling increases the CPU load. I adapt the presentation to real-world scenarios and, to get started, refer to Interrupt coalescing. The goal remains a reliable Latency with a viable throughput.
NUMA, RSS, and CPU Affinity
Short queues aren't very helpful if packets are constantly crossing NUMA boundaries. I bind RX/TX queues to cores in the same NUMA domain where the app is running using RSS/irqbalance. Using XPS/RPS, I control which CPUs handle TX work, thereby avoiding cross-socket hopping. Fewer cache misses and less lock contention indirectly benefit TSQ: Completions return faster, the socket is „unthrottled“ sooner, and latency spikes are avoided. When there are a very large number of flows per host, I allocate sufficient queues and prevent multiple high-traffic flows from colliding on the same TX ring.
Step-by-Step: Checking TSQ Status
I start with a look at Sysctl: `sysctl net.ipv4.tcp_limit_output_bytes` displays the current limit. I then use `ss -tin` to examine individual sockets, paying attention to `send-q` and `rtt`, and compare load phases with and without limit adjustments. Using iperf3, I generate background load and measure API response times in parallel to highlight priorities. tc -s qdisc provides me with packet and drop counts for the output discipline. If the 95th and 99th percentiles remain close together and the CPU-Load within the frame; adjust the limit accordingly.
Common Mistakes and Anti-Patterns
- „More buffer = more performance“: This holds true for throughput tests without a latency target, but fails for interactive services. TSQ replaces oversized queues with demand-based credit per flow.
- „TSQ costs throughput“: When set correctly, TSQ limits bursts, not the average rate. For bulk workloads, I scale the limit moderately high and measure the percentiles instead of just the peak Mbit/s.
- „Pacing alone is enough“: Time smoothing is important, but without a byte cap, large GSO frames will still slip into the QDisc. TSQ and pacing complement each other.
- „One value for all“: Workloads, links, and NICs differ. I work with range values and validate them on a per-environment basis.
- „Only TCP is affected“: The focus is on TCP, but there are other configuration options in the system (e.g., for UDP load). I prevent parallel protocols from clogging up the same queues in an uncontrolled manner.
Conclusion: Keeping Latency Under Control
TSQ shifts control of driver queues to the socket and thus reduces congestion right at the source. I limit the number of pre-buffered bytes per flow, ensuring fast ACKs, lower RTT, and fairly shared queues. Combined with fq_codel and modern congestion control, the response time remains reliable even under load. I handle special cases involving Wi-Fi and embedded systems with customized limits and tests under real-world conditions. By monitoring the metrics and gradually adjusting the limits, you can maintain the Latency consistently low, without losing unnecessary throughput.
