What Is a TCP Out-of-State Flood Attack? | Stateful Firewall and Conntrack Exhaustion

A TCP Out-of-State Flood is a Layer 4 DDoS attack that sends TCP packets with flag combinations that don’t correspond to any valid transition in the TCP state machine for an existing or expected connection, forcing stateful firewalls and connection-tracking systems (conntrack) to classify and process large volumes of packets that match no legitimate session. The attack targets the CPU and table capacity of the stateful inspection layer itself, rather than the connection queue or the application behind it.

TL;DR: TCP connections follow a defined state machine (RFC 793/RFC 9293): SYN_SENT, SYN_RECV, ESTABLISHED, FIN_WAIT, CLOSE_WAIT, and so on, each with a specific set of valid incoming flag combinations. A TCP Out-of-State Flood sends packets — ACK, FIN, RST, PSH, or combinations of these — that don’t match any tracked connection or any valid state transition for one that exists. Stateful firewalls and conntrack must evaluate and classify every packet as INVALID, consuming CPU and table capacity proportional to volume. Unlike SYN Flood, it doesn’t fill the SYN backlog queue; unlike a pure ACK Flood, it isn’t limited to a single flag pattern. Mitigation relies on dropping INVALID-state packets as early as possible, tuning conntrack table size and timeouts, and rate limiting by flag pattern at the edge.

Last updated: 2026-08-27

How a TCP Out-of-State Flood Works

1. Every TCP connection tracked by a stateful firewall or the kernel’s connection tracking system moves through a defined sequence of states: NONE → SYN_SENT/SYN_RECV → ESTABLISHED → FIN_WAIT/CLOSE_WAIT → TIME_WAIT/CLOSED.
2. Each state only accepts specific incoming flag combinations as valid next steps. An ACK is only expected after a SYN-ACK; a FIN is only expected from an established connection winding down; a RST can arrive at almost any point to abort a connection.
3. An out-of-state packet is any TCP segment whose flags don’t correspond to a legal transition for the connection’s current tracked state — or that references a connection tuple with no tracked state at all.
4. The attacker generates a stream of such packets: ACKs with no prior SYN, FINs for connections not in an established state, RSTs for untracked tuples, or PSH+ACK combinations directed at closed or nonexistent sessions — often mixing several flag patterns in the same flood rather than sticking to one.
5. Netfilter’s conntrack module (or an equivalent stateful engine on any firewall/load balancer) evaluates each packet against its state table. Anything that fails to match a tracked connection or violates the expected transition gets classified INVALID.
6. At flood volume, the sheer number of classification decisions — plus, in some configurations, conntrack table churn from attempts to create new tracking entries for the mismatched traffic — consumes CPU and memory on the firewall or kernel network stack.

Normal TCP state transitions:
Client ──SYN──▶ Server              [state: NONE → SYN_RECV]
Client ◀─SYN-ACK── Server
Client ──ACK──▶ Server              [state: SYN_RECV → ESTABLISHED]
Client ──PSH,ACK (data)──▶ Server   [valid: ESTABLISHED accepts PSH,ACK]
Client ──FIN,ACK──▶ Server          [valid: ESTABLISHED → FIN_WAIT]
Client ──ACK──▶ Server              [valid: closes cleanly]

TCP Out-of-State Flood:
Bot ──FIN,ACK (no tracked connection)──▶ Server   [no matching state → INVALID]
Bot ──RST (no tracked connection)──▶ Server        [no matching state → INVALID]
Bot ──ACK (no prior SYN)──▶ Server                 [no matching state → INVALID]
Bot ──PSH,ACK (tuple never established)──▶ Server  [no matching state → INVALID]
... [millions of mismatched packets, mixed flag patterns]

Result: conntrack/firewall CPU spent classifying INVALID packets;
        SYN backlog queue and application layer are never directly touched

Which TCP Flag Attack Is This? Comparison Table

The distinction from ACK Flood: Out-of-State Flood is the broader category — it includes ACK Flood as one instance but also uses FIN, RST, and combined-flag traffic to maximize the variety of mismatches a defender must classify, which can complicate signature-based filtering that only watches for one flag pattern. The distinction from TCP Invalid Packet: out-of-state packets typically have structurally legal flag combinations (a lone ACK, a lone FIN) that are simply inconsistent with the tracked connection’s state, whereas invalid packets are illegal at the protocol level regardless of any tracked state (such as SYN+FIN set simultaneously).

Attack Variations

Mixed-flag rotation. The attacker rotates between ACK, FIN-ACK, RST, and PSH-ACK packets across the flood to prevent any single-flag detection rule from catching the full attack, forcing defenders to monitor the aggregate INVALID counter rather than a specific flag signature.

Out-of-state flood targeting mid-session churn. Rather than only targeting nonexistent connections, some variants send out-of-state packets referencing tuples that were recently torn down (in TIME_WAIT), attempting to interact with a connection tracking entry that’s in the process of expiring, adding churn to the conntrack table’s cleanup logic.

Combined with legitimate low-and-slow traffic. Attackers sometimes blend out-of-state packets with genuine low-volume traffic to make the malicious flow harder to isolate from background noise in coarse-grained monitoring.

Detection Signals and Telemetry

# Socket summary — unaffected in a pure out-of-state flood since SYN_RECV isn't the target
ss -s

# Full connection state breakdown
ss -tan state all | awk '{print $1}' | sort | uniq -c

# Netfilter conntrack invalid packet counter — the primary signal
conntrack -S

# Inspect current tracked connections and table pressure
conntrack -L | wc -l
cat /proc/sys/net/netfilter/nf_conntrack_max
cat /proc/sys/net/netfilter/nf_conntrack_count

# TCP kernel counters for anomalies not caught by conntrack alone
nstat -az

# nftables counting/dropping invalid-state traffic, broken out by flag for diagnosis
nft add rule inet filter input ct state invalid counter drop
nft add rule inet filter input tcp flags fin,ack / fin,ack ct state invalid counter
nft add rule inet filter input tcp flags rst / rst ct state invalid counter

# iptables equivalents
iptables -A INPUT -m conntrack --ctstate INVALID -j DROP
iptables -A INPUT -p tcp ! --syn -m conntrack --ctstate NEW -j DROP

Mitigation Techniques

Common Mistakes and Fixes

How to Implement on Azion

Azion’s distributed network can absorb mixed-flag out-of-state traffic before it reaches origin infrastructure:

• DDoS Protection is designed to detect Layer 3/4 anomalies, including traffic that doesn’t match any valid TCP state transition, at the edge.
• Network Shield applies network-level filtering that can classify and drop packets inconsistent with tracked connection state.
• Firewall supports custom rules for state- and rate-based blocking.
• WAAP combines network- and application-layer protection for teams needing broader coverage.

Because Azion can terminate and track TCP state at the edge, an origin server behind it generally only receives traffic belonging to sessions the edge itself validated — out-of-state packets are filtered upstream rather than exhausting origin conntrack or firewall resources. Actual protection depends on your specific configuration; validate behavior against your traffic profile.

Related Resources

• What Is a DDoS Attack?
• DDoS Attack Types
• What Is a SYN Flood Attack?
• What Is DDoS Protection and Mitigation?
• What Is a TCP SYN-ACK Flood Attack?
• What Is a TCP Invalid Packet Attack?

Frequently Asked Questions

What is a TCP Out-of-State Flood attack? It’s a DDoS attack that sends TCP packets whose flags don’t correspond to any valid transition in the TCP state machine for a tracked connection — such as an ACK with no prior SYN, or a FIN for a connection that isn’t established. Stateful firewalls and conntrack must classify each one, consuming CPU and table capacity.

How is Out-of-State Flood different from a TCP ACK Flood? TCP ACK Flood uses a single fixed flag pattern (ACK only). Out-of-State Flood is the broader category and typically mixes multiple flag patterns — ACK, FIN, RST, PSH combinations — specifically to defeat single-signature detection and stress the conntrack classification logic more broadly.

How is this different from a SYN Flood? SYN Flood exhausts the connection queue by allocating half-open state for every SYN packet received. Out-of-State Flood doesn’t touch the SYN backlog at all — it targets the stateful firewall’s classification and table-management logic with packets that don’t fit any legitimate state transition.

What does “out of state” mean in TCP terms? It means a packet’s flags don’t match a legal next step for the connection’s currently tracked state in the state machine defined by RFC 793/RFC 9293 — for example, receiving a FIN for a connection the firewall never saw a SYN or ESTABLISHED entry for.

Can conntrack table size increases alone stop this attack? No. Increasing nf_conntrack_max delays table exhaustion under moderate attack volume but doesn’t address the CPU cost of classifying each out-of-state packet, and a sufficiently large flood will still overwhelm a bigger table. It should be combined with early INVALID-state drops and upstream filtering.

Is IP spoofing required for a TCP Out-of-State Flood? No. The attack works whether source IPs are spoofed or real, since the mechanism relies on flag/state mismatches rather than exploiting a return-path requirement. Spoofing can still be used to complicate attribution and evade IP-based blocking.

How do I tell a TCP Out-of-State Flood apart from a TCP Invalid Packet attack? Out-of-state packets are usually protocol-legal on their own (a lone ACK, a lone FIN) but inconsistent with the tracked connection’s state. Invalid packets are illegal at the protocol level regardless of state — such as SYN and FIN set in the same packet, or a corrupted checksum — and are typically caught by parsing/validation logic rather than state-transition logic.

What Linux command shows out-of-state packet counts? conntrack -S shows the invalid counter maintained by Netfilter’s connection tracking subsystem, which increments whenever a packet is classified as not matching any tracked connection or valid state transition.

Does this attack affect the application layer directly? Not directly. The primary impact is on the stateful inspection layer — firewall, load balancer, or kernel conntrack — before traffic reaches the application. If that layer saturates, legitimate traffic (including to the application) can be dropped as collateral damage.

Can this attack be combined with a SYN Flood? Yes. Attackers sometimes combine SYN Flood traffic with out-of-state ACK/FIN/RST traffic to pressure both the SYN backlog queue and the stateful firewall’s classification logic at the same time, requiring defenders to apply SYN Cookies and INVALID-state filtering simultaneously.

Sources

• IETF. “Transmission Control Protocol.” RFC 793. 1981.
• IETF. “Transmission Control Protocol (TCP) Specification.” RFC 9293. 2022.
• NIST. “Guide to Intrusion Detection and Prevention Systems.” SP 800-94.
• CISA. “Understanding and Responding to Distributed Denial-of-Service Attacks.”