TCP SYN Cookies in the Linux Kernel: Protection Against SYN Floods

TCP SYN Cookies In the Linux kernel, they keep the handshake load low by cryptographically encoding state information into the initial sequence number and establishing a connection fully only after receiving a valid ACK. This prevents SYN floods from clogging the queue of half-open connections and blocking legitimate clients.

Key points

• Functionality: Cookie in the ISN; state determined only after ACK
• Linux Control System: net.ipv4.tcp_syncookies with modes 0/1/2
• Benefit: Low memory usage during attack load
• Boundaries: Does not help against bandwidth or app attacks
• Tuning: Set backlog and retry values carefully

How SYN Floods Slow Down the TCP Handshake

An attacker floods the server with SYN packets and ignores the subsequent SYN/ACK responses, causing half-open entries to occupy the SYN queue. I then find that new, legitimate requests can't get through and timeouts start to pile up. This is exactly where Syncookies To: The kernel does not initially store any connection state and incorporates the necessary data into the sequence number. Only a valid ACK confirms the existence of a genuine peer, allowing the connection establishment to proceed normally. LWN.net and the TUM documents describe this principle as an established, effective handshake protection mechanism that does not consume a lot of memory. This architecture keeps the server responsive even under a flood of traffic because it delays the creation of resource-intensive states until very late in the process.

Technical Process: Cookies Instead of the Previous State Management System

The kernel responds to a SYN using a specially encoded SYN/ACK whose ISN is derived from a secret key, TCP options, and time slices. If an ACK with the matching number arrives, I reconstruct the session parameters from the ISN and open the socket normally. If no response is received, there is no occupied half-open state, which conserves memory and CPU resources. This approach drastically reduces the vulnerability of the acceptance phase without permanently altering the regular path. According to Ubuntu and Red Hat documentation, this technique has been working reliably for many kernel generations and only kicks in when the queue threatens to overflow.

Enabling and Testing: tcp_syncookies in Practice

About the sysctl switch net.ipv4.tcp_syncookies I control the behavior: 0 = off, 1 = only during overload, 2 = permanently. In production environments, I usually set the mode to 1 so that the standard handshake remains intact and protection is activated only when needed. I can quickly check the status in the shell; I apply changes via sysctl or permanently in /etc/sysctl.d/. A relevant background article on socket behavior and attack patterns helps with planning the whole setup; I go into more detail in the post SYN flood protection. I use the following commands regularly:

View # status
sysctl net.ipv4.tcp_syncookies

Temporarily enable # (until reboot)
sudo sysctl -w net.ipv4.tcp_syncookies=1

Set # permanently
echo "net.ipv4.tcp_syncookies = 1" | sudo tee /etc/sysctl.d/60-syncookies.conf
sudo sysctl --system

Limitations: What SYN Cookies Cannot Do

SYN cookies primarily address the Syn-Queue and prevent semi-open states from tying up memory. However, they are ineffective against an overloaded line, overwhelmed application logic, or CPU saturation. For volumetric attacks, I need upstream filters, QoS, and, if necessary, scrubbing. Application-layer attacks, such as HTTP GET floods, also require additional controls, limits, and caches. I therefore always incorporate syncookies into a multi-layered defense strategy that combines the network, kernel, and service layers.

Tuning: Backlogs, Queues, and Retries

Before an emergency strikes, I'll vote backlogs and retries so that legitimate spikes do not trigger protection mode unnecessarily. `tcp_max_syn_backlog` affects the queue of half-open connections, while `somaxconn` determines the maximum length of the acceptance queue for connections waiting for `accept()`. With `tcp_synack_retries`, I determine how many times the kernel will attempt to resend a SYN/ACK before giving up. Higher backlogs handle short-term traffic spikes but consume memory; fewer retries free up slots sooner but carry the risk of hitting disconnected clients too hard. I test these trade-offs under realistic load using tools like hping3 or tcp_syn_flooder in an isolated network.

# Candidates for Peak Loads
sudo sysctl -w net.core.somaxconn=4096
sudo sysctl -w net.ipv4.tcp_max_syn_backlog=8192
sudo sysctl -w net.ipv4.tcp_synack_retries=3

Comparing Operating Modes: Effects and Applications

For everyday use, I choose the Modes Be aware of this, as they influence diagnosis, metrics, and behavior under pressure. Persistent cookies (2) prevent any early state establishment, but they alter metrics for retries and can affect rare edge cases involving TCP options. Adaptive mode (1) allows the stack to run normally and intervenes when an overrun is imminent. Off (0) is only useful in laboratory situations or closed networks. The following table summarizes this concisely:

Mode	Description	Advantage	Potential side effect	Example
0	Disabled, no cookies	Clear Baseline Behavior	Attacker fills the Syn queue	Isolated Test Network
1	Adaptive, only in case of overflow	Normal TCP in Idle State	Calibrate the switching point	Public Services
2	Forced, always active	Early Relief	Analytical values are shifting	Tough offensive situation

Measurable Effects: Latencies and Success Rate

Under pressure, the Memory requirement This is clearly evident at the start of each connection because no half-open state is created. As a result, SYN cookies keep the acceptance rate high, and short bursts cause fewer dropouts. Under flood traffic, I observe faster recovery as soon as the source dries up. Ubuntu and Tenable guidelines recommend adaptive deployment so that normal clients continue to operate as usual. For regression testing, I check for retransmissions, drop rates, and server latency during the transition to cookie mode.

Additional layers of protection: Firewall and limits

I clear syncookies using Filter Rules and rate limits so that the load doesn't even reach the TCP stack. On Linux, I prefer to use nftables rules to throttle or drop offenders early based on connection rates. The guide provides a concise overview of modern packet filters nftables vs. Netfilter. In addition, SYNPROXY scenarios on edge firewalls help by terminating the handshake and passing through only valid connections. For exposed ports, I define strict access rules, logging thresholds, and a maximum number of connection attempts per source address.

High-Performance Approaches: XDP and Others.

When volume attacks target the PPS rate To offload processing, I move the filtering logic to the network edge of the NIC via XDP. This allows me to drop suspicious SYN packets before they reach the socket layer, which reduces CPU load and alleviates pressure on the receive queue. An introduction to this technique makes it easier to get started with XDP Package Processing. When used in conjunction with SYN cookies, this creates a two-stage system: first, a rough selection based on the map, then a reliable handshake check in the kernel. This chain significantly reduces the attack surface and keeps services accessible.

Diagnosis: Interpreting Metrics and Log Entries Correctly

If there are any noticeable Timeouts I check netstat/ss statistics, dmesg messages, and Grafana dashboards showing connection rates. An increasing proportion of SYN-RECV, high retransmissions, and drops indicate a transition to protection mode. I look for SYN backlog overflow messages and correlate them with CPU and IRQ load. Packet captures with tcpdump verify the sequence number logic and help identify false positives. I also use iptables/nftables counters to measure the hit rate of rate-limiting rules.

Compatibility: TCP Options and Edge Cases

Coding Modern Kernels Options such as MSS, SACK, or timestamps, so that cookies are transmitted in a way that allows them to be reconstructed. Older or less common stacks may exhibit peculiarities, so I check critical paths before rollout. I monitor behavior particularly closely in the case of proxies, NAT, and anycast topologies. LWN.net discusses design details that explain why today’s implementations operate reliably. In very specific scenarios, forced operating mode (2) remains a tool that I use only when absolutely necessary.

Common Misconceptions: Things I Often Have to Correct

Syncookies are not a substitute for DDoS defense at the periphery; they primarily protect the handshake phase. A high somaxconn value alone does not prevent overflows if SYN/ACK is never acknowledged. Likewise, the assumption that persistent cookies (2) are always the best choice is misleading; diagnostics and special cases suffer as a result. Without monitoring, I lack the signals needed to adjust switching points and limits. Load testing remains essential to ensure that the configuration and hardware are suited to real-world access patterns.

Practical Check: Steps Toward a Reliable Assumption

I start with Mode 1 for `tcp_syncookies` and verify the intervention point under load. Afterward, I moderately increase `tcp_max_syn_backlog` and `somaxconn` while lowering `tcp_synack_retries` and measuring success rates. Firewall rate limits and geo/ASN filters filter out noise before it reaches the stack. I reserve XDP or SmartNIC filters for high PPS scenarios so that I can allocate resources in a targeted manner. Finally, I document the metrics so that future adjustments are data-driven.

IPv6 and Dual-Stack: Same Switch, Same Logic

In dual-stack environments, the following behavior occurs: IPv4 and IPv6 consistent in the context of cookies. The toggle net.ipv4.tcp_syncookies controls protection globally for TCP, including v6 sockets. I therefore test the transition to cookie mode on both protocols—especially when upstream devices use different filter paths in IPv6. Important: SYN cookies protect TCP only. UDP services or QUIC require their own rate limits and edge policies to prevent high-volume traffic from exhausting the CPU.

Kernel Metrics: Reliable Indicators

For reliable monitoring, I use kernel counters that explicitly track cookies. In addition to ss -s To monitor state distributions, I track the counts of sent, accepted, and failed cookies. This allows me to determine whether the protection is working, whether legitimate clients are getting through, and whether there are any misconfigurations.

# Overview
ss -s
ss -ant state syn-recv | wc -l

# Cookie Counter (Kernel: /proc/net/netstat)
grep -E 'Syncookies|ListenOverflows|ListenDrops' /proc/net/netstat

# Live View
watch -n1 'grep -E "Syncookies(Sent|Recv|Failed)|Listen(Overflows|Drops)" /proc/net/netstat'

# Log Entries (Sample Message)
# dmesg displays, among other things:
# TCP: Possible SYN flooding on port 443. Sending cookies. Check SNMP counters.

Rise List Overflows and ListenDrops in parallel with SyncookiesSent I adjust backlogs, retries, and upstream filters. Remain SyncookiesRecv ...this suggests a pure bot flood; on the other hand, if SyncookiesFailed, I check NAT/proxy paths and look for any potential tampering along the way.

Proxies, Load Balancers, and Kubernetes

At Proxy and Load Balancing Chains determines where cookie protection is placed. If an L4/L7 load balancer terminates the TCP handshake, a SYN flood never reaches the backends; in that case, I enable cookies at the edge. If the load balancer operates only passively (DSR, ECMP), backend nodes must protect themselves independently. In Kubernetes, I adjust sysctls on the worker nodes, especially for NodePort or HostNetwork workloads. For Ingress controllers with their own SYN defense (SYNPROXY, eBPF), I adjust the policies so they don’t slow each other down. I factor in the load balancer’s health checks during testing, since otherwise short sampling windows with low retry rates can falsely indicate instability.

Edge Cases in Detail: Options, Time Slices, NAT

Cookies only encode limited parameters. Modern Linux implementations generally reconstruct MSS, SACK, and window scaling reliably; however, timestamps and rare options may have limitations depending on the kernel version. I therefore prefer operating mode (1), so that the standard path takes precedence and cookies only take effect in the event of an overflow. The validity of a cookie is tied to time slices—in cases of highly asymmetric paths or delay spikes, a legitimate ACK may fall just outside the window. In WAN and satellite scenarios, I therefore measure round-trip variance before reducing the number of retries. NAT and middleboxes that modify sequence numbers or options are additional candidates for edge cases; I use targeted captures to identify where bits are lost.

ACK/RST Floods and Variants Beyond the SYN Storm

Not every Transport Attack is a pure SYN flood. ACK or RST floods target the CPU and packet paths without triggering the handshake—cookies are of little help here. In such cases, I use early filters (nftables/XDP) with stateful logic or minimal ACK rate limiting. I specifically terminate RST waves targeting established connections using a set of rules that discards unexpected RSTs without a matching window. I also cover semi-open repeat attacks (SYN with spoofing plus late ACKs) using rate limits per source network segment.

Further Tuning: Listen/Accept Queues and Fast Errors

In addition to the standard parameters, I use supplementary switches that shape the behavior at the limits:

• Backlog vs. somaxconn: The value in lists (backlog) per process is determined by net.core.somaxconn capped. I make sure that the server software and the kernel work together smoothly; otherwise, optimizations go to waste.
• tcp_abort_on_overflow: Whether requests are silently dropped when the accept queue is full or actively responded to with an RST. In high-volume APIs, a quick error can allow the client to retry quickly; for TLS or legacy clients, I usually prefer the default dropping behavior.
• Port and TIME-WAIT Management: Cookies don't prevent Ephemeral Port Bottleneck. I'm planning to ip_local_port_range Be generous and use TIME-WAIT optimizations judiciously so that reuse doesn't lead to Heisenbugs.
• SO_REUSEPORT: Having multiple accept queues per port distributes the load across worker processes and reduces overflows on individual CPUs.

Test methods: reproducible and meaningful

I simulate load conditions that closely resemble real-world scenarios and measure the transition point to cookie mode, the success rate of legitimate connections, and the recovery time after the load peak. To do this, I combine synthetic SYN floods with real application requests.

Generate # flood (Lab!)
sudo hping3 -S -p 443 --flood --rand-source 

Vary # network conditions
sudo tc qdisc add dev eth0 root netem delay 80ms 30ms loss 1%

# Mix in legitimate traffic
wrk -t8 -c512 -d60s https:///

# Parallel monitoring
watch -n1 'ss -s; echo; grep -E "Syncookies|Listen(Overflows|Drops)" /proc/net/netstat'

These steps help me determine whether the retry rate is dropping too aggressively, whether upstream firewalls are incorrectly filtering timestamps, or whether the accept queues of individual workers are overflowing disproportionately. I document the key metrics (success rate of legitimate connections approaching 100%, cookie hit rate, latency behavior) so that I can make data-driven adjustments later.

Operation and Maintenance: Ensuring Stability Over the System's Lifespan

In continuous operation, I plan to Secret rotation (automatically on the kernel side) and observe whether time-slot changes have any visible effects on very long RTT paths. I keep the kernel and drivers up to date so that improvements in the cookie implementation (better option encoding, robust time slots) take effect. For audits, I note when protection mode was triggered, how many connections it allowed through, and whether additional filters were enabled. When making changes to MTU, offloading, or NF stacks (e.g., new nftables sets), I repeat quick tests so I can detect interactions causing issues early on.

Abridged version for those in a hurry

SYN cookies store the Handshake load small by creating states only after a confirmed ACK, thereby protecting the SYN queue from flooding. I enable Mode 1, carefully tune backlogs and retries, and measure the effects using clear metrics. Additional layers such as nftables rate limits, SYNPROXY, and XDP throttle traffic even before it reaches the TCP stack. Overall, this allows me to protect web, email, VPN, and API services against SYN floods without disadvantaging regular clients. Those who rigorously implement these steps will significantly improve availability and reduce outages during attack loads.