TLS/SSL Handshake Exhaustion | Cryptographic Attacks and Edge Mitigation

TLS/SSL handshake exhaustion is a category of DDoS attack that exploits this cost asymmetry. Instead of generating network volume, the attacker maximizes server CPU consumption by forcing repeated TLS handshake processing — either through renegotiation on a single connection or by opening a large number of new connections.

The mathematical cost asymmetry

The TLS handshake consists of cryptographic operations with radically different cost profiles:

Cheap operations (symmetric, executed by dedicated hardware): after the handshake, data transmission uses AES-GCM with AES-NI instruction support. Marginal cost per byte is negligible.

Expensive operations (asymmetric, executed by software or specialized accelerators):

• In protocols up to TLS 1.2 with RSA: the server performs modular exponentiation with the private exponent d, typically 1024, 2048, or 4096 bits. Computational complexity grows with the square of the key size. An RSA-2048 operation requires hundreds of arbitrary-precision modular multiplications — something the attacker doesn’t need to perform, since RSA encryption with the public key is orders of magnitude faster.
• In TLS 1.3 with ECDHE: the server performs scalar multiplication on an elliptic curve to generate the ephemeral Diffie-Hellman key pair, then generates an ECDSA or Ed25519 signature to authenticate the key share. The client only verifies that signature — an operation significantly cheaper than generating it.

In benchmarks with openssl speed, the difference between symmetric encryption and asymmetric signature operations spans several orders of magnitude. Adam Langley documented in 2010 that Gmail processed SSL handshakes with less than 1% additional CPU overhead after optimizations — demonstrating that the raw handshake cost, without those optimizations, is the real bottleneck.

The asymmetry is structural: the client validates the server, but the server does the heavy lifting of proving its identity.

Attack vectors

THC-SSL-DOS — renegotiation on a single connection

THC-SSL-DOS does not open and close multiple TCP connections. Its mechanism is different and more efficient for the attacker:

1. The attacker establishes a single persistent TCP connection to the server.
2. Completes the initial TLS handshake normally — the server processes one full handshake.
3. Immediately sends a new renegotiation ClientHello on the same connection, requesting renegotiation of TLS parameters.
4. The server, as required by the protocol, executes a new full handshake over the same TCP session.
5. The attacker repeats step 3 hundreds of times per second on the same connection.

The result is that a single TCP connection forces N complete cryptographic handshakes on the server, with no additional TCP connection cost for the attacker. RFC 5746 (Renegotiation Indication Extension) mitigated the data injection dimension of the attack, but did not eliminate the CPU cost of renegotiation itself — until TLS 1.3 removed the feature entirely.

TLS Handshake Flood — multiple TCP connections

This vector is distinct from THC-SSL-DOS and operates the way an “obvious attack” intuitively works:

1. The attacker simultaneously opens a large number of new TCP connections.
2. Initiates the TLS handshake on each one.
3. May complete the handshake and immediately close, or abandon before completion.
4. Repeats continuously to maintain pressure on the server.

The key difference from THC-SSL-DOS: each handshake here consumes a separate socket and TCP connection. TLS Handshake Flood is constrained by TCP connection establishment overhead and operating system file descriptor limits, making it more expensive for the attacker per handshake induced on the server.

TLS Session Ticket Flood and the STEK risk

Session Tickets allow the client to resume a TLS session without a full handshake, presenting a ticket encrypted by the server’s STEK (Session Ticket Encryption Key). An attacker can exploit this mechanism in two ways:

The first is sending invalid or fabricated session tickets in bulk. The server must attempt to decrypt each ticket with its STEK before confirming it is invalid — a non-trivial AES-CBC or AES-GCM operation per ticket.

The second, more sophisticated approach, arises when the STEK is compromised or is weak. If the attacker obtains the STEK (for example, in implementations using the same key for extended periods without rotation), they can fabricate valid tickets that force the server to process malicious resumptions at scale. RFC 8446 recommends frequent STEK rotation and per-instance STEKs to limit the impact of compromise.

Cipher suite downgrade attack

The attacker deliberately negotiates cipher suites with more expensive asymmetric operations — such as RSA 4096 bits instead of ECDSA P-256 — maximizing CPU cost per handshake on the server. This vector complements TLS Handshake Flood: a smaller number of simultaneous connections may be sufficient to saturate CPU if each handshake is artificially expensive.

Encrypted Client Hello (ECH) and new challenges

Encrypted Client Hello (ECH), a TLS 1.3 extension, encrypts the inner ClientHello — including the Server Name Indication (SNI) — using a server public key distributed via DNS (HTTPS/SVCB records). ECH improves user privacy but introduces a new exhaustion vector: the server must attempt to decapsulate the outer ClientHello with its private ECH key before determining whether the client is legitimate. An attacker sending fabricated ECH ClientHellos forces the server to perform HPKE (Hybrid Public Key Encryption) decapsulation operations in bulk — a computationally non-trivial cost that did not exist before ECH.

The primary mitigation is rate limiting by source IP before ECH processing, and monitoring decapsulation failures as an attack signal.

Cost profile and attack surface by TLS version

Hardware acceleration and HPS metrics

A server’s resistance to TLS exhaustion attacks depends directly on its cryptographic processing capacity, measured in HPS (Handshakes Per Second):

AES-NI instructions (available on Intel and AMD processors since 2010): accelerate AES-GCM operations used in post-handshake data transmission and session key derivation. They do not reduce the cost of asymmetric handshake operations themselves.

Elliptic curve hardware support (MULX/ADCX/ADOX instructions on Intel Broadwell+): accelerate scalar multiplication used in ECDHE and ECDSA, reducing the cost of the most expensive TLS 1.3 handshake operations.

Intel QuickAssist Technology (QAT): a dedicated co-processor for asymmetric cryptographic operations including RSA, ECDSA, and public key operations.

The openssl speed command measures actual server HPS by running openssl speed ecdsa ecdhx25519 rsa2048 to compare algorithm throughput. In general, ECDSA P-256 and X25519 (Curve25519) offer substantially higher HPS than RSA 2048 on the same hardware.

Mitigation techniques

TLS termination at the edge

The most effective mitigation offloads TLS handshake processing from origin servers to edge nodes with hardware optimized for cryptography. The flow works as follows:

Without edge termination, each client accessing the application forces the origin server to execute a full cryptographic handshake. Under an exhaustion attack, the origin server directly absorbs all cryptographic pressure.

With edge termination, handshakes are completed at the edge network nodes. The origin server maintains only one persistent TLS connection with the edge — established once, not per client. The per-user handshake CPU cost is eliminated entirely from the origin.

Rate limiting by IP — before and after the handshake

Rate limiting new TLS handshakes per source IP interrupts TLS Handshake Floods. For THC-SSL-DOS (which uses a single connection), the relevant control is the renegotiation limit per session — configurable in OpenSSL (SSL_CTX_set_max_send_fragment) and in server configurations such as nginx’s ssl_renegotiate_size.

Session resumption with STEK rotation

Session Tickets allow legitimate clients to avoid full handshakes on reconnection, reducing CPU load. The security of the mechanism depends on proper STEK management:

• Rotate STEKs frequently (every 24 hours or less in high-sensitivity environments).
• Use per-instance STEKs — not shared across nodes — to limit the blast radius of a compromise.
• Apply rate limiting on invalid tickets per IP to mitigate Session Ticket Flood.

Disabling TLS renegotiation (TLS 1.2 and earlier)

For servers still running TLS 1.2, disabling client-initiated renegotiation eliminates the THC-SSL-DOS vector without impacting legitimate users:

In nginx: ssl_renegotiation off; In Apache: SSLRenegotiate off In OpenSSL directly: SSL_set_options(ssl, SSL_OP_NO_RENEGOTIATION);

Migrating to TLS 1.3

TLS 1.3 (RFC 8446) eliminates renegotiation by design, reduces RTTs from 2 to 1, and standardizes ECDHE as the sole key exchange mechanism. Supported by more than 96% of modern browsers, the migration eliminates the largest historical surface for TLS exhaustion. See also TLS ciphers and how they affect handshake security and performance.

Detection signals

Common mistakes and solutions

Mistake: Treating TLS exhaustion as a network volume problem. Solution: TLS exhaustion attacks consume CPU, not bandwidth. Volume-based protection tools (bps or pps-based) neither detect nor mitigate this vector. Monitoring must focus on HPS and cryptographic CPU usage.

Mistake: Assuming that HTTPS is automatically “protected” because it uses encryption. Solution: Asymmetric encryption is precisely the source of the vulnerability. Protecting HTTPS against exhaustion requires edge TLS termination and connection and renegotiation rate limiting.

Mistake: Not migrating from TLS 1.2 to TLS 1.3 due to compatibility concerns. Solution: TLS 1.3 is supported by more than 96% of modern browsers. The compatibility gap is minimal; the security and efficiency gains are substantial.

Mistake: Using RSA 2048 or 4096 when ECDSA is available. Solution: ECDSA P-256 offers security equivalent to RSA 3072 with much lower per-signature computational cost. The migration raises server HPS and reduces per-handshake cost.

Mistake: Not rotating STEKs periodically. Solution: Static STEKs amplify the impact of a potential compromise and facilitate Session Ticket Flood. Rotate STEKs at least every 24 hours.

Frequently asked questions

What is the exact difference between THC-SSL-DOS and TLS Handshake Flood? THC-SSL-DOS uses a single persistent TCP connection and forces multiple renegotiations on it — each renegotiation is a full handshake with no additional TCP cost for the attacker. TLS Handshake Flood opens multiple new TCP connections, one per handshake. THC-SSL-DOS is more efficient for the attacker per handshake induced; TLS Handshake Flood is easier to scale with a botnet. TLS 1.3 eliminates the THC-SSL-DOS vector by removing renegotiation.

Why does the server execute the most expensive operation, not the client? In the TLS handshake, the server must prove its identity by generating a cryptographic signature with its private key — an operation involving modular exponentiation (RSA) or elliptic curve scalar multiplication followed by signature generation (ECDSA/Ed25519). The client only verifies the signature using the server’s public key, which is orders of magnitude faster. This asymmetry is intrinsic to the design of public key infrastructure (PKI).

What is HPS and how do you measure it? HPS (Handshakes Per Second) is the cryptographic throughput metric of a server — how many TLS handshakes it can process per second before saturating the CPU. Measure with openssl speed ecdsa ecdhx25519 rsa2048 to compare algorithms, or use tools like wrk and h2load to measure real HPS with simulated HTTPS traffic. Servers with Intel QAT have substantially higher HPS for RSA workloads.

What is ECH and why does it create a new attack surface? Encrypted Client Hello (ECH) encrypts the TLS 1.3 inner ClientHello using HPKE (Hybrid Public Key Encryption). This improves user privacy by hiding the SNI from network observers. The new attack vector: an attacker can send fabricated ECH ClientHellos forcing the server to perform HPKE decapsulation operations in bulk — a computationally non-trivial operation — before determining the ticket is invalid.

Can Session Tickets be exploited even without compromising the STEK? Yes. Even without knowing the STEK, an attacker can send fabricated tickets (cryptographic garbage) in bulk. The server must attempt to decrypt them before confirming they are invalid — AES-CBC/GCM cost per ticket. Rate limiting invalid tickets per IP is the primary mitigation.

Does mTLS significantly increase the exhaustion surface? Yes, substantially. In mTLS, the server executes additional operations per handshake: verification of the client certificate signature, validation of the entire client certificate chain (which may include intermediate CAs), and — depending on configuration — OCSP query or CRL verification to confirm the client certificate has not been revoked. Each of these steps adds cryptographic cost and, in the case of OCSP, network latency. In high-scale mTLS environments, hardware acceleration and aggressive caching of OCSP responses are essential to maintain adequate HPS levels.

Does TLS 1.3 completely eliminate exhaustion risk? It eliminates the renegotiation vector (THC-SSL-DOS). But the initial handshake still requires ECDHE and ECDSA/Ed25519 signature generation — operations with non-negligible cost. TLS Handshake Flood remains possible against TLS 1.3, but the per-handshake cost is lower than with RSA-2048 or RSA-4096, raising the saturation threshold. Eliminating renegotiation is the most significant gain.

How to implement on Azion

Azion protects against TLS handshake exhaustion with distributed edge TLS termination:

1. TLS termination at Azion’s data centers: TLS handshakes are terminated at Azion’s data centers, not at client origin servers. The origin never processes handshakes with end users — it only maintains a secure connection with the edge.
2. TLS 1.3 by default: Azion supports and recommends TLS 1.3, eliminating renegotiation and reducing per-handshake cost and attack surface compared to TLS 1.2.
3. Session resumption with STEK management: Azion implements session tickets with key rotation, reducing legitimate client reconnection costs and mitigating Session Ticket Flood by volume and per IP.
4. TLS connection rate limiting: Azion’s edge applies new TLS handshake limits per source IP, blocking both TLS Handshake Floods and mass renegotiation attempts before they consume cryptographic processing resources.

Learn more in the Azion Certificate Manager documentation.