What Is a TCP Invalid Packet Attack? | Malformed Flags, Bad Checksums, and Parser Exhaustion

A TCP Invalid Packet attack is a Layer 4 DDoS and evasion technique that sends TCP segments violating the protocol specification itself — illegal flag combinations such as SYN+FIN or SYN+RST, corrupted checksums, malformed header fields, or packets with no flags set at all — to exhaust packet parsing and validation resources on the target, or to slip past inspection devices that don’t handle malformed input consistently.

TL;DR: RFC 793/RFC 9293 define which TCP flag combinations are meaningful; some combinations, like SYN and FIN set simultaneously, have no valid interpretation in the protocol and should never occur in legitimate traffic. A TCP Invalid Packet attack deliberately generates these malformed segments — illegal flag pairs, bad checksums, reserved-bit misuse, or zero-flag “null” packets — at volume. The goal is either denial of service, by forcing every device in the path to spend CPU validating and rejecting malformed input, or reconnaissance/evasion, since different operating systems and middleboxes handle malformed packets inconsistently. Mitigation is comparatively direct: enforce strict header and flag-combination validation as early as possible in the packet-processing pipeline and drop anything that fails RFC-defined legality checks.

Last updated: 2026-08-27

How a TCP Invalid Packet Attack Works

1. TCP defines six primary control flags: URG, ACK, PSH, RST, SYN, FIN. Only certain combinations correspond to a meaningful protocol event. SYN initiates a connection; FIN closes one gracefully; RST aborts one. SYN and FIN set together contradicts itself — “open a new connection” and “close a connection” in the same segment — and has no defined meaning in the specification.
2. An attacker crafts packets with combinations the specification doesn’t define as valid: SYN+FIN, SYN+RST, all flags set simultaneously (“Christmas tree” or Xmas packets), or no flags set at all (“null” packets).
3. Separately, or in combination, the attacker can corrupt the TCP checksum field, misuse reserved header bits, or truncate/malform header length fields so the segment doesn’t parse cleanly according to the protocol’s own framing rules.
4. Every device in the path — NIC offload engines, the kernel TCP/IP stack, stateful firewalls, IDS/IPS sensors — must parse and evaluate the packet against protocol rules before deciding to accept, reject, or flag it. Malformed input is often more expensive to evaluate than well-formed input, since exception paths and validation logic run before the packet can be discarded.
5. At flood volume, this validation cost accumulates across every hop, and inconsistent handling between devices (a firewall might drop a packet an application-layer parser would otherwise choke on) can be exploited to build reconnaissance or IDS-evasion techniques on top of the same core technique.
6. Historically, tools like Nmap use exactly these malformed patterns (Null, FIN, and Xmas scans) for OS fingerprinting and firewall-rule reconnaissance — because different TCP/IP stack implementations respond differently to input the specification doesn’t define, the response itself leaks information.

Normal traffic (legal flag combinations only):
Client ──SYN──▶ Server                  [valid: connection request]
Client ◀─SYN-ACK── Server               [valid: connection response]
Client ──ACK──▶ Server                  [valid: handshake completion]
Client ──FIN,ACK──▶ Server              [valid: graceful close]
Client ──RST──▶ Server                  [valid: abort]

TCP Invalid Packet attack:
Bot ──SYN,FIN (illegal combination)──▶ Server     [contradictory: open + close]
Bot ──SYN,RST (illegal combination)──▶ Server     [contradictory: open + abort]
Bot ──URG,ACK,PSH,RST,SYN,FIN (all flags/"Xmas")──▶ Server   [undefined by spec]
Bot ──(no flags set/"null")──▶ Server             [undefined by spec]
Bot ──(valid flags, corrupted checksum)──▶ Server [fails integrity check]
... [millions of malformed packets]

Result: every parsing/validation layer in the path spends cycles
        evaluating and rejecting packets that never should have been sent

Which TCP Flag Attack Is This? Comparison Table

The key distinction from Out-of-State Flood: out-of-state packets are individually legal TCP segments (a lone ACK, a lone FIN) that are simply inconsistent with a specific connection’s tracked state — evaluating them requires stateful context. Invalid packets are illegal by the protocol specification itself, independent of any tracked state — a SYN+FIN packet is malformed whether or not any connection with that tuple exists, so it can often be identified and dropped by stateless header inspection alone.

Attack Variations

Xmas scan flood. All six flags (URG, ACK, PSH, RST, SYN, FIN) set simultaneously. Originally an Nmap reconnaissance technique for OS fingerprinting and firewall-rule discovery, at volume it becomes a resource-exhaustion vector, since some legacy stacks and inspection devices handle the “everything set” case inefficiently.

Null scan flood. Packets with no flags set at all. Like the Xmas scan, this pattern is undefined by the specification and used historically for stealth reconnaissance; at flood volume it forces every receiving device to run through validation logic for input matching no expected pattern.

Checksum-corruption flood. Otherwise well-formed packets with deliberately invalid TCP checksums. Depending on where checksum validation occurs (NIC offload, kernel, or application), this can either be cheaply dropped or, on misconfigured or older hardware/software paths, consume more cycles than expected.

Reserved-bit and header-length manipulation. Setting reserved header bits or manipulating the data offset field to produce a header that doesn’t cleanly delineate where TCP options end and payload begins, targeting parsers with less rigorous bounds checking.

Detection Signals and Telemetry

# Kernel TCP counters — malformed segments increment error/discard counters
nstat -az | grep -iE "invalid|csum|error"

# Interface-level checksum error counters (driver/NIC dependent)
ethtool -S eth0 | grep -i err

# nftables rule to explicitly match and count illegal flag combinations
nft add rule inet filter input tcp flags syn,fin / syn,fin counter drop
nft add rule inet filter input tcp flags syn,rst / syn,rst counter drop
nft add rule inet filter input tcp flags fin,syn,rst,psh,ack,urg / fin,syn,rst,psh,ack,urg counter drop
nft add rule inet filter input tcp flags == 0x0 counter drop

# iptables equivalents (illegal combination checks)
iptables -A INPUT -p tcp --tcp-flags SYN,FIN SYN,FIN -j DROP
iptables -A INPUT -p tcp --tcp-flags SYN,RST SYN,RST -j DROP
iptables -A INPUT -p tcp --tcp-flags ALL ALL -j DROP
iptables -A INPUT -p tcp --tcp-flags ALL NONE -j DROP

# Netfilter's built-in --tcp-flags with the standard "INVALID" state check as a backstop
iptables -A INPUT -m conntrack --ctstate INVALID -j DROP

# Packet capture for classification during investigation (use sampling at high PPS)
tcpdump -ni eth0 'tcp[13] & 0x03 == 0x03' -c 1000

Any nonzero count of SYN+FIN, SYN+RST, all-flags, or no-flags packets in production traffic is inherently suspicious — legitimate TCP/IP stacks never generate these combinations, so detection thresholds can be stricter than for attacks built from individually legal packets.

Mitigation Techniques

Common Mistakes and Fixes

How to Implement on Azion

Azion’s distributed network can filter malformed TCP traffic before it reaches origin infrastructure:

• DDoS Protection is designed to detect and help mitigate Layer 3/4 anomalies, including malformed and illegal-flag-combination traffic, at the edge.
• Network Shield applies filtering rules that can identify and drop packets violating protocol-level flag and header rules.
• Firewall supports custom rules for packet-level validation and blocking.
• WAAP combines network- and application-layer protection for teams needing broader coverage.

Because Azion terminates TCP connections at the edge, malformed and illegal-flag traffic aimed at an origin behind Azion is generally filtered upstream rather than reaching origin infrastructure. Actual protection depends on your specific configuration; validate behavior against your traffic profile.

Related Resources

• What Is a DDoS Attack?
• DDoS Attack Types
• What Is a SYN Flood Attack?
• What Is DDoS Protection and Mitigation?
• What Is a TCP ACK Flood Attack?
• What Is a TCP SYN-ACK Flood Attack?
• What Is a TCP Out-of-State Flood Attack?

Frequently Asked Questions

What is a TCP Invalid Packet attack? It’s an attack that sends TCP segments violating the protocol specification itself — illegal flag combinations like SYN+FIN or SYN+RST, all flags set, no flags set, or corrupted checksums — to exhaust packet-parsing resources or evade inspection devices that handle malformed input inconsistently.

Why would SYN and FIN ever be set in the same packet? They wouldn’t, in legitimate traffic. SYN signals “open a new connection” and FIN signals “close a connection gracefully” — the combination is self-contradictory and has no defined meaning in RFC 793/RFC 9293. Any SYN+FIN packet observed in production traffic indicates malformed or malicious traffic.

How is a TCP Invalid Packet attack different from a TCP Out-of-State Flood? Out-of-state packets are individually legal TCP segments (a lone ACK or FIN) that are inconsistent with a specific connection’s tracked state, requiring stateful context to identify. Invalid packets are illegal at the protocol level regardless of any tracked state — a SYN+FIN packet is malformed whether or not a matching connection exists, so it can often be dropped with stateless header inspection alone.

What is a Christmas tree (Xmas) packet? An Xmas packet has all six TCP flags (URG, ACK, PSH, RST, SYN, FIN) set simultaneously — named for lighting up like a Christmas tree in packet analyzers. It’s undefined by the TCP specification and historically used for OS fingerprinting and firewall reconnaissance via tools like Nmap.

What is a Null scan and why is it dangerous? A Null scan sends TCP packets with no flags set at all. Like the Xmas scan, this is undefined behavior in the specification, and different operating systems respond differently to it — which lets an attacker fingerprint the target OS or firewall ruleset. At volume, it also forces validation overhead on every receiving device.

Can a firewall drop invalid TCP packets without stateful inspection? Yes, for most illegal flag combinations. Since SYN+FIN, SYN+RST, all-flags, and no-flags patterns are never legitimate regardless of connection state, they can be matched and dropped with stateless header rules, which is generally cheaper than running them through full stateful conntrack evaluation.

Does a bad TCP checksum always mean an attack? Not necessarily — occasional checksum errors can result from real network corruption or faulty hardware. A sustained, high-volume pattern of checksum failures correlated with a specific source or traffic pattern is the signal that distinguishes an attack from incidental corruption.

How is this related to IDS/IPS evasion techniques? Different TCP/IP stack implementations and inspection devices handle malformed packets inconsistently. Attackers can exploit these inconsistencies to craft traffic that one device drops but another processes differently, potentially slipping malicious payloads past an inspection point that doesn’t reject the malformed framing the way the ultimate target’s stack does.

What TCP flag combinations should always be blocked? SYN+FIN, SYN+RST, all six flags set simultaneously, and no flags set at all have no legitimate use in standard TCP traffic and are safe to block unconditionally at the firewall in virtually all production environments.

Is IP spoofing required for a TCP Invalid Packet attack? No. The technique relies on malforming the packet’s flags, checksum, or header fields rather than exploiting a return-path requirement, so it works with either spoofed or real source IP addresses.

Sources

• IETF. “Transmission Control Protocol.” RFC 793. 1981.
• IETF. “Transmission Control Protocol (TCP) Specification.” RFC 9293. 2022.
• NIST. “Guide to Intrusion Detection and Prevention Systems.” SP 800-94.
• CISA. “Understanding and Responding to Distributed Denial-of-Service Attacks.”