Network Handshaking

Introduction

Network handshaking is the foundational negotiation process that two endpoints perform before they can exchange application data reliably. It establishes connection parameters such as sequence numbers, encryption keys, protocol versions, and flow-control windows. Understanding handshakes — from the classic TCP three-way handshake to TLS 1.3 and WebSocket upgrades — is essential for diagnosing latency, hardening security, and designing low-latency distributed systems.

Core Concepts

What Is a Handshake?

A handshake is a structured sequence of messages exchanged between a client (initiator) and a server (responder) to mutually agree on communication parameters before data transfer begins. Every handshake has three responsibilities:

1. Capability advertisement — each side declares what it supports.
2. Parameter negotiation — both sides converge on a shared set of options.
3. State synchronization — each side transitions into a "connected" state only after verifying the other's readiness.

Why Handshakes Matter

• Reliability: TCP's handshake guarantees both endpoints are alive and ready.
• Security: TLS handshakes authenticate identities and derive encryption keys.
• Performance: Every round-trip in a handshake adds latency; modern protocols like TLS 1.3 and QUIC minimize round-trips.
• Correctness: Without sequence number synchronization (TCP) or version negotiation (HTTP/2), data corruption or protocol mismatch occurs.

---

TCP Three-Way Handshake

The Transmission Control Protocol uses a three-way handshake (SYN → SYN-ACK → ACK) to establish a reliable, bidirectional byte stream.

Step-by-Step

Step	Sender	Flag(s)	Key Fields	Purpose
1	Client	SYN	seq=x	Client picks initial sequence number
2	Server	SYN+ACK	seq=y, ack=x+1	Server picks its own ISN, acknowledges client
3	Client	ACK	seq=x+1, ack=y+1	Client acknowledges server

After step 3, both sides are in the ESTABLISHED state and may send data.

TCP State Machine During Handshake

TCP Four-Way Teardown

Connection termination is also a handshake — a four-way exchange using FIN and ACK flags.

---

TLS Handshake

The TLS handshake runs on top of a TCP connection and negotiates cryptographic parameters. TLS 1.2 requires two round-trips; TLS 1.3 reduces this to one.

TLS 1.2 Full Handshake

TLS 1.3 Handshake (1-RTT)

TLS 1.3 merges key exchange into the first flight, eliminating one full round-trip.

Combined TCP + TLS Latency

---

WebSocket Upgrade Handshake

WebSocket piggybacks on HTTP/1.1's Upgrade mechanism to transition a request-response connection into a full-duplex message stream.

---

Implementation — TCP Handshake in Java

Java's Socket and ServerSocket abstract the TCP handshake, but we can observe it programmatically.

Basic TCP Server and Client

import java.io.*;
import java.net.*;

public class TcpHandshakeDemo {

    // Server thread — binds, listens, accepts (SYN-ACK happens inside accept())
    static class TcpServer implements Runnable {
        private final int port;

        TcpServer(int port) {
            this.port = port;
        }

        @Override
        public void run() {
            try (ServerSocket serverSocket = new ServerSocket(port)) {
                System.out.println("[Server] LISTEN on port " + port);
                Socket client = serverSocket.accept(); // TCP 3-way handshake completes here
                System.out.println("[Server] ESTABLISHED with " + client.getRemoteSocketAddress());

                BufferedReader in = new BufferedReader(
                        new InputStreamReader(client.getInputStream()));
                PrintWriter out = new PrintWriter(client.getOutputStream(), true);

                String message = in.readLine();
                System.out.println("[Server] Received: " + message);
                out.println("ACK: " + message);

                client.close(); // FIN handshake begins
                System.out.println("[Server] Connection closed");
            } catch (IOException e) {
                System.err.println("[Server] Error: " + e.getMessage());
            }
        }
    }

    public static void main(String[] args) throws Exception {
        int port = 9090;

        // Start server in background
        Thread serverThread = new Thread(new TcpServer(port));
        serverThread.start();
        Thread.sleep(500); // Give server time to bind

        // Client — connect() triggers SYN → SYN-ACK → ACK
        try (Socket socket = new Socket()) {
            System.out.println("[Client] Initiating handshake (SYN)...");
            socket.connect(new InetSocketAddress("localhost", port), 5000);
            System.out.println("[Client] ESTABLISHED with " + socket.getRemoteSocketAddress());

            PrintWriter out = new PrintWriter(socket.getOutputStream(), true);
            BufferedReader in = new BufferedReader(
                    new InputStreamReader(socket.getInputStream()));

            out.println("Hello from client");
            String response = in.readLine();
            System.out.println("[Client] Server responded: " + response);
        } catch (ConnectException e) {
            System.err.println("[Client] Handshake failed (connection refused): " + e.getMessage());
        } catch (SocketTimeoutException e) {
            System.err.println("[Client] Handshake timed out: " + e.getMessage());
        }

        serverThread.join();
    }
}

TLS Handshake in Java with SSLSocket

import javax.net.ssl.*;
import java.io.*;
import java.security.cert.X509Certificate;

public class TlsHandshakeDemo {

    public static void main(String[] args) {
        String host = "www.google.com";
        int port = 443;

        try {
            SSLSocketFactory factory = (SSLSocketFactory) SSLSocketFactory.getDefault();

            System.out.println("[TLS] Starting TCP + TLS handshake to " + host + ":" + port);
            long start = System.nanoTime();

            SSLSocket socket = (SSLSocket) factory.createSocket(host, port);

            // Force TLS 1.3 if available
            socket.setEnabledProtocols(new String[]{"TLSv1.3", "TLSv1.2"});

            // This triggers the TLS handshake explicitly
            socket.startHandshake();

            long elapsed = (System.nanoTime() - start) / 1_000_000;
            System.out.println("[TLS] Handshake completed in " + elapsed + " ms");

            SSLSession session = socket.getSession();
            System.out.println("[TLS] Protocol : " + session.getProtocol());
            System.out.println("[TLS] Cipher   : " + session.getCipherSuite());

            // Print server certificate chain
            java.security.cert.Certificate[] certs = session.getPeerCertificates();
            for (int i = 0; i < certs.length; i++) {
                if (certs[i] instanceof X509Certificate x509) {
                    System.out.println("[TLS] Cert[" + i + "] Subject: " + x509.getSubjectX500Principal());
                    System.out.println("[TLS] Cert[" + i + "] Issuer : " + x509.getIssuerX500Principal());
                }
            }

            // Send an HTTP request over the secure channel
            PrintWriter out = new PrintWriter(new OutputStreamWriter(socket.getOutputStream()));
            out.println("GET / HTTP/1.1");
            out.println("Host: " + host);
            out.println("Connection: close");
            out.println();
            out.flush();

            BufferedReader in = new BufferedReader(new InputStreamReader(socket.getInputStream()));
            String line = in.readLine();
            System.out.println("[TLS] Response: " + line);

            socket.close();
        } catch (SSLHandshakeException e) {
            System.err.println("[TLS] Handshake failed — certificate error: " + e.getMessage());
        } catch (IOException e) {
            System.err.println("[TLS] I/O error: " + e.getMessage());
        }
    }
}

WebSocket Handshake with Java's HttpClient

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.WebSocket;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.CompletionStage;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;

public class WebSocketHandshakeDemo {

    public static void main(String[] args) throws Exception {
        HttpClient client = HttpClient.newHttpClient();
        CountDownLatch latch = new CountDownLatch(1);

        System.out.println("[WS] Initiating HTTP Upgrade handshake...");
        long start = System.nanoTime();

        CompletableFuture<WebSocket> wsFuture = client.newWebSocketBuilder()
                .buildAsync(URI.create("wss://echo.websocket.events"), new WebSocket.Listener() {

                    @Override
                    public void onOpen(WebSocket webSocket) {
                        long elapsed = (System.nanoTime() - start) / 1_000_000;
                        System.out.println("[WS] Handshake complete in " + elapsed + " ms");
                        System.out.println("[WS] Subprotocol: " + webSocket.getSubprotocol());
                        webSocket.sendText("Hello from Java!", true);
                        WebSocket.Listener.super.onOpen(webSocket);
                    }

                    @Override
                    public CompletionStage<?> onText(WebSocket webSocket, CharSequence data, boolean last) {
                        System.out.println("[WS] Received: " + data);
                        latch.countDown();
                        return WebSocket.Listener.super.onText(webSocket, data, last);
                    }

                    @Override
                    public CompletionStage<?> onClose(WebSocket webSocket, int statusCode, String reason) {
                        System.out.println("[WS] Closed: " + statusCode + " " + reason);
                        return WebSocket.Listener.super.onClose(webSocket, statusCode, reason);
                    }

                    @Override
                    public void onError(WebSocket webSocket, Throwable error) {
                        System.err.println("[WS] Error: " + error.getMessage());
                        latch.countDown();
                    }
                });

        try {
            wsFuture.join(); // wait for handshake
        } catch (Exception e) {
            System.err.println("[WS] Handshake failed: " + e.getCause().getMessage());
            return;
        }

        latch.await(10, TimeUnit.SECONDS);
        wsFuture.join().sendClose(WebSocket.NORMAL_CLOSURE, "bye").join();
    }
}

---

QUIC and 0-RTT Handshake

QUIC (used by HTTP/3) combines transport and cryptographic handshakes into a single flight, achieving 1-RTT for new connections and 0-RTT for resumed connections.

---

Handshake Latency Comparison

---

Handshake Failures and Attacks

Understanding how handshakes fail is critical for debugging and security.

Common Failure Modes

SYN Flood Attack

The SYN flood exploits the TCP handshake by sending many SYN packets without completing the three-way handshake, exhausting the server's SYN backlog queue.

Mitigation: SYN cookies — the server encodes state into the SYN-ACK sequence number instead of storing it in a queue, eliminating the backlog problem.

---

Implementation — Handshake Timing Analyzer

This utility measures and reports the duration of each handshake phase.

import javax.net.ssl.*;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.Socket;

public class HandshakeTimingAnalyzer {

    public record TimingResult(
            String host,
            int port,
            long tcpHandshakeMs,
            long tlsHandshakeMs,
            String protocol,
            String cipherSuite
    ) {
        public long totalMs() {
            return tcpHandshakeMs + tlsHandshakeMs;
        }

        @Override
        public String toString() {
            return String.format(
                    "%s:%d | TCP: %dms | TLS: %dms | Total: %dms | %s | %s",
                    host, port, tcpHandshakeMs, tlsHandshakeMs, totalMs(), protocol, cipherSuite
            );
        }
    }

    public static TimingResult analyze(String host, int port, int timeoutMs) throws IOException {
        // Phase 1: TCP handshake
        long tcpStart = System.nanoTime();
        Socket rawSocket = new Socket();
        rawSocket.connect(new InetSocketAddress(host, port), timeoutMs);
        long tcpElapsed = (System.nanoTime() - tcpStart) / 1_000_000;

        // Phase 2: TLS handshake
        SSLSocketFactory factory = (SSLSocketFactory) SSLSocketFactory.getDefault();
        long tlsStart = System.nanoTime();
        SSLSocket sslSocket = (SSLSocket) factory.createSocket(
                rawSocket, host, port, true);
        sslSocket.setEnabledProtocols(new String[]{"TLSv1.3", "TLSv1.2"});
        sslSocket.startHandshake();
        long tlsElapsed = (System.nanoTime() - tlsStart) / 1_000_000;

        SSLSession session = sslSocket.getSession();
        TimingResult result = new TimingResult(
                host, port, tcpElapsed, tlsElapsed,
                session.getProtocol(), session.getCipherSuite()
        );

        sslSocket.close();
        return result;
    }

    public static void main(String[] args) {
        String[] hosts = {"www.google.com", "www.github.com", "aws.amazon.com"};

        System.out.println("=== Handshake Timing Analysis ===");
        System.out.println();

        for (String host : hosts) {
            try {
                TimingResult result = analyze(host, 443, 5000);
                System.out.println(result);
            } catch (SSLHandshakeException e) {
                System.err.println(host + " — TLS handshake failed: " + e.getMessage());
            } catch (IOException e) {
                System.err.println(host + " — Connection failed: " + e.getMessage());
            }
        }
    }
}

---

Protocol Handshake Comparison

Protocol	Layers	RTTs	Encrypted	Resumable	Multiplexed
TCP	Transport	1	No	N/A	No
TCP + TLS 1.2	Transport + Security	3	Yes	Session tickets (1-RTT)	No
TCP + TLS 1.3	Transport + Security	2	Yes	0-RTT	No
QUIC	Transport + Security	1	Yes	0-RTT	Yes
WebSocket	Application	1 (HTTP Upgrade)	Via WSS	No	No
HTTP/2	Application	0 (ALPN in TLS)	Via TLS	N/A	Yes

---

Best Practices

1. Use TLS 1.3 wherever possible: It reduces handshake latency by one full RTT compared to TLS 1.2 and removes insecure cipher suites from negotiation.

2. Enable TCP Fast Open (TFO): TFO allows data in the SYN packet for subsequent connections, saving one RTT on the transport layer for repeat visitors.

3. Implement connection pooling: Reusing established connections eliminates handshake overhead entirely; use HTTP keep-alive or dedicated connection pools in database drivers.

4. Set appropriate timeouts: Configure connect timeouts (TCP handshake) and handshake timeouts (TLS) separately — a missing TLS timeout can cause threads to hang indefinitely.

5. Enable SYN cookies on servers: This kernel-level defense prevents SYN flood attacks from exhausting the connection backlog without affecting legitimate clients.

6. Pin certificates carefully: Certificate pinning prevents man-in-the-middle attacks during TLS handshake but requires a robust rotation strategy to avoid outages.

7. Monitor handshake metrics: Track TCP and TLS handshake durations in your observability stack — spikes indicate network congestion, certificate chain problems, or OCSP stapling failures.

8. Prefer ALPN over NPN: Application-Layer Protocol Negotiation is the standard mechanism for HTTP/2 discovery during TLS handshake; NPN is deprecated.

9. Pre-warm connections for latency-sensitive paths: Establish TCP+TLS connections before they're needed to avoid cold-start latency on critical request paths.

10. Handle handshake failures gracefully: Distinguish between connection refused (port closed), timeout (network issue), and TLS errors (certificate/cipher mismatch) to provide actionable error messages.

---

Related Concepts

• REST HTTP Verbs and Status Codes — HTTP methods and status codes that flow over connections established by TCP and TLS handshakes.
• OAuth — OAuth token exchanges that rely on TLS handshakes for transport security.
• Cryptography — The cryptographic primitives (key exchange, digital signatures) used during TLS handshakes.
• Asynchronous Programming — Non-blocking I/O patterns for handling many concurrent handshakes efficiently.
• Eventual Consistency — Distributed systems where connection establishment and handshake retries affect consistency guarantees.