Vitaliy Leschenko’s blog
ru
Виталий Лещенко

HTTP/1.1, HTTP/2 and HTTP/3: how they differ, how they work and how to enable them in Traefik

This week I enabled HTTP/3 on Traefik in my cluster, and along the way I had to sort out again in my head how the three versions of HTTP that live side by side differ from each other, and why a third one is needed at all when the second appeared ten years ago. This is that breakdown: how each version works, which problem of its predecessor it solves, when one is faster than another, how it's all configured in Traefik, and the pitfalls I either hit or managed to spot in advance.

The article stands on its own, but it builds on two parts of the series "Networking for beginners": the ones on UDP and on TCP. They cover connections, losses and windows in detail; here we'll refer back to them.

What stays the same

HTTP is a request-response protocol: the client asks for a resource, the server answers with a status, headers and a body. Methods (GET, POST), codes (200, 404), headers (Host, Content-Type, Cookie), URLs: all of this is HTTP semantics, and it's the same in all three versions. An application that deals with requests at the level of "method, path, headers, body" doesn't care about the version: the web server or proxy accepts a request over any of them and hands the application the same thing.

What differs is how the semantics is packed into bytes and how those bytes travel across the network. That's the whole story: three attempts to deliver the same requests faster.

HTTP/1.1: text over TCP

HTTP/1.1 (1997, now RFC 9112) is a text protocol. A request is lines separated by \r\n, an empty line ends the headers, and the body follows:

GET /index.html HTTP/1.1
Host: www.example.com
User-Agent: curl/8.21.0
Accept: */*

The response looks the same:

HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 12345
Cache-Control: max-age=3600

<!doctype html>...

All of this travels inside a TCP connection: a reliable byte stream in which the stack itself takes care of delivery and ordering. HTTP/1.1 relies on that completely: it has no checksums or numbering of its own, it simply writes text into the stream and reads from it. The stream is delimited with Content-Length or chunked encoding (the body in pieces, each preceded by its length), because TCP doesn't preserve message boundaries.

What version 1.1 added to the very first, 1.0:

  • Persistent connections. In HTTP/1.0 every request opened a new TCP connection with a handshake. In 1.1 the connection stays alive by default (keep-alive), and the next request goes over the same one.
  • The Host header. So that hundreds of sites can live on one address, the request says which of them it's for.
  • Chunked transfer, ranges (Range), caching (ETag, Cache-Control), 100 Continue.

The problem: a single queue

Within one HTTP/1.1 connection, requests go strictly in turn: send a request, wait for the response, and only then the next one. The specification allowed pipelining (sending several requests without waiting for responses), but the responses still had to arrive in request order, intermediate proxies broke pipelining, and browsers turned it off for good.

A modern page is a hundred resources: HTML, a dozen scripts, styles, fonts, images. Over a single connection with a 50 ms RTT, a hundred requests would take five seconds of pure waiting. So browsers open six connections per host and run requests in parallel. That helps, but:

  • six connections mean six TCP handshakes and six TLS handshakes (more on TLS below), six TCP slow starts that ramp up independently and compete with each other;
  • a large resource on one connection blocks every request queued behind it on that connection: this is head-of-line blocking at the application level;
  • the headers of every request are repeated in full as text: a kilobyte of Cookie, User-Agent, Accept-*, in each of the hundred requests.

A generation of frontend developers grew up on workarounds for these limits: concatenating scripts into one file, image sprites, inlining small resources into HTML, serving static files from several domains (domain sharding) to get more than six connections. HTTP/2 made these tricks unnecessary, and some of them harmful.

The easiest way to see HTTP/1.1 live is this:

$ curl -v --http1.1 https://www.example.com/ -o /dev/null
> GET / HTTP/1.1
> Host: www.example.com
> User-Agent: curl/8.21.0
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: text/html; charset=utf-8
...

HTTP/2: frames and streams in one connection

HTTP/2 (2015, RFC 9113) grew out of Google's SPDY and solves exactly the problems of the previous section without changing the semantics: the same methods, headers and codes, different packaging.

Binary frames

Text is replaced by frames: binary blocks with a nine-byte header:

 bytes:   3         1       1        4
        ┌─────────┬───────┬───────┬─────────────┬──────────────┐
        │ Length  │ Type  │ Flags │ Stream ID   │ payload      │
        │         │       │       │             │              │
        └─────────┴───────┴───────┴─────────────┴──────────────┘

Frame types: HEADERS (request or response headers), DATA (body), SETTINGS (connection parameters), WINDOW_UPDATE (flow control), RST_STREAM (cancel a single request), PING, GOAWAY (we're closing), PRIORITY, PUSH_PROMISE. A binary format is simpler and faster to parse than text, and above all, every frame has a Stream ID.

Streams: multiplexing

A stream is one request-response exchange within a connection. The client opens odd-numbered streams, the server even-numbered ones; frames of different streams are interleaved in one TCP connection in any order:

TCP connection ───►  HEADERS(1)  HEADERS(3)  DATA(1)  HEADERS(5)  DATA(3)  DATA(1)  DATA(5) …
                     request 1   request 3   reply 1  request 5   reply 3  reply 1  reply 5

A page's hundred resources are a hundred streams in one connection: all requests are sent at once, and responses arrive in whatever order they're ready. One handshake, one slow start, one congestion window for all of them. Browsers keep one connection per host (and reuse it for other hosts with the same certificate and address: coalescing).

Within a connection there's flow control per stream and for the connection as a whole (WINDOW_UPDATE, like the TCP window but at the application level): a slow receiver of one large response doesn't stall the rest. There are priorities: the browser hints that CSS matters more than an image. In practice servers barely take them into account, and HTTP/3 simplified the scheme.

HPACK: header compression

Headers are compressed with HPACK: a static table of the 61 most common headers (:method: GET, :status: 200, accept-encoding: gzip, deflate each take one byte) and a dynamic table into which both sides put the headers seen on this connection. A second request with the same kilobyte of Cookie sends it as a single index. Plain gzip didn't work for headers: it's vulnerable to the CRIME attack, in which the attacker guesses a secret from the compressed size, while HPACK only compresses repeats of whole values.

Headers became lowercase pseudo-headers: :method, :path, :authority (formerly Host), :scheme, :status. That's why curl -v shows lowercase headers for HTTP/2.

What else

  • Server push: the server could send a resource before the client asked for it. It looked good on slides, but in real life it mostly got in the way (the client already had the resource in its cache), and Chrome dropped support in 2022. The replacement is 103 Early Hints with preload hints.
  • TLS only in browsers. The standard allows HTTP/2 in clear text (h2c), but browsers don't implement it: middleboxes that parse HTTP/1.1 would break an unfamiliar binary protocol. The version is negotiated inside the TLS handshake with the ALPN (Application-Layer Protocol Negotiation) extension: the client lists h2, http/1.1, and the server picks. Without ALPN it's HTTP/1.1. h2c lives between proxies and backends, where TLS isn't needed, and in gRPC inside clusters.

TLS here and below is the encryption protocol inside the TCP connection that HTTPS runs on: it protects the data of a single connection and, importantly for us, uses ALPN to say which HTTP version will run inside.

The problem that remained: one TCP for everyone

HTTP/2 removed the application-level queue, but all streams still travel in one TCP connection, and TCP hands data to the application strictly in order. Lose one segment, and until it's retransmitted all hundred streams are stuck, even though their data may already have arrived and be sitting in the buffer. This is head-of-line blocking one layer down, and on a lossy network HTTP/2 over one connection can turn out slower than HTTP/1.1 over six: there, a loss on one connection slowed down only that connection.

Second: TCP and TLS are two separate handshakes before the first byte. TCP takes one round trip (SYN, SYN-ACK), TLS 1.3 another. That's two RTTs before the first request; on a mobile network with a 100 ms RTT, that's 200 ms of nothing.

Third: TCP lives in the kernel. To change the congestion control algorithm or add an extension, you have to update the kernels on billions of devices and teach middleboxes not to break the new thing. Over thirty years TCP has "ossified": any deviation from the familiar breaks on some firewall along the way.

To see HTTP/2:

$ curl -v --http2 https://www.example.com/ -o /dev/null
* ALPN: curl offers h2,http/1.1
* ALPN: server accepted h2
...
> GET / HTTP/2
> Host: www.example.com
...
< HTTP/2 200
< content-type: text/html; charset=utf-8

Frames and streams are visible in Wireshark if you give it the TLS keys via SSLKEYLOGFILE; in the browser's DevTools the Protocol column shows h2.

HTTP/3: QUIC instead of TCP

HTTP/3 (2022, RFC 9114) has the same semantics and almost the same stream model as HTTP/2, but underneath it, instead of TCP+TLS, is QUIC (RFC 9000). QUIC is a transport protocol that redoes what TCP and TLS did together, and does it on top of UDP.

Why UDP: UDP adds nothing to IP except ports (part nine of the series), and every NAT and firewall lets it through. A new protocol number on top of IP wouldn't get through half the internet. The QUIC implementation lives in user space (in the browser, in curl, in Traefik) and gets updated with the program, not with the kernel.

What QUIC has:

  • Reliability and congestion control of its own, as in TCP: packet numbers, acknowledgements, retransmissions, a window, the same algorithms (CUBIC, BBR). But packet numbers never repeat: lost data is retransmitted in a new packet with a new number, which makes RTT measurement unambiguous.
  • Streams at the transport level. Losses are handled per stream. A packet with data for stream 5 is lost, and only stream 5 stalls; the others keep going. Transport-level head-of-line blocking is gone. This is the main thing.
  • Built-in TLS 1.3. The QUIC handshake and the TLS handshake are a single exchange: one RTT to the first request instead of two, and on reconnection 0-RTT: the request goes in the very first packet. Almost everything is encrypted, including packet numbers and most QUIC headers; middleboxes see only addresses, ports and the connection ID, so there's nothing for them to break. This is deliberate protection against ossification.
  • A connection ID instead of the four-tuple of addresses and ports. A phone switches from Wi-Fi to cellular, the address changes, and the QUIC connection carries on with the new address (connection migration). In TCP that's impossible by definition: the connection is the four-tuple.
  • QPACK: header compression like HPACK, but tolerant of frames arriving out of order: the dynamic table is updated over a separate stream with acknowledgements.

What QUIC costs: encryption and all the transport logic run on the application's CPU, not in the kernel with hardware offloads. An HTTP/3 server spends one and a half to two times more CPU on the same traffic than HTTP/2. For CDNs that's a solved problem; for a single proxy it's an expense you should be aware of.

How the browser learns HTTP/3 is available

You can't probe a UDP port "just in case": that would be a delay on every connection to a server without HTTP/3. So the first connection always goes over TCP (HTTP/2 or 1.1), and the server sends a header in its response:

Alt-Svc: h3=":443"; ma=2592000

"This same host has HTTP/3 on UDP port 443, remember it for 30 days." The browser opens subsequent connections over QUIC, and DevTools shows the protocol changing to h3. If UDP doesn't get through (a firewall, a corporate network), the browser silently falls back to TCP and remembers the failure: HTTP/3 fundamentally can't break a site, it can only fail to kick in. That's both good and bad: "works for me" doesn't mean HTTP/3 works.

To skip the first TCP connection there's the HTTPS DNS record type (RFC 9460): example.com. HTTPS 1 . alpn="h3,h2" tells the client about HTTP/3 support before first contact. Browsers query it, but for now it's mostly CDNs that publish it.

Seeing HTTP/3

The regular curl in distributions is built without HTTP/3; you need a build with quiche or ngtcp2, for example the ymuski/curl-http3 image:

$ docker run --rm ymuski/curl-http3 curl -sv --http3-only https://www.example.com/ -o /dev/null
* QUIC connection established
* using HTTP/3
> GET / HTTP/3
...
< HTTP/3 200

And in tcpdump on the server it looks like this:

$ sudo tcpdump -i eth0 -n udp port 443 -c 3
IP 198.51.100.7.52133 > 203.0.113.10.443: UDP, length 1250
IP 203.0.113.10.443 > 198.51.100.7.52133: UDP, length 1250
IP 198.51.100.7.52133 > 203.0.113.10.443: UDP, length 1200

Three UDP datagrams and nothing else: no flags, no numbers, no headers. Diagnosing QUIC takes either the keys (SSLKEYLOGFILE and Wireshark) or the server's own logs.

Comparison

HTTP/1.1 HTTP/2 HTTP/3
Transport TCP TCP QUIC over UDP
Format text binary frames binary frames
Connections per host in a browser 6 1 1
Concurrent requests per connection 1 many (streams) many (streams)
Queue blocked on packet loss this connection only all streams this stream only
Header compression none HPACK QPACK
Handshakes before the first request TCP + TLS: 2 RTT 2 RTT 1 RTT, 0 on reconnect
Encryption optional mandatory in browsers built into the transport
Survives a client address change no no yes
Visible to middleboxes everything, without TLS TCP headers addresses and ports
Where the transport is implemented kernel kernel application
Server CPU load low low noticeably higher

When which is faster

  • Many small resources, a decent network: HTTP/2 and HTTP/3 beat 1.1 several times over: one connection, a hundred streams, compressed headers. On a lossless network they're nearly equal to each other.
  • Mobile and Wi-Fi networks with losses and high RTT: HTTP/3 is noticeably better than HTTP/2: a loss doesn't stall everything, the handshake is shorter, and switching networks doesn't drop the connection. That's exactly what it was built for: according to Google and Cloudflare, the gain for "bad" clients is tens of percent of load time.
  • One large file: almost no difference: one stream, and everything comes down to congestion control, which is the same.
  • Inside a data center, proxy → backend: HTTP/1.1 with keep-alive or h2c; HTTP/3 between your own servers isn't needed: the network is lossless, the RTT is a fraction of a millisecond, and CPU costs more.
  • gRPC: a remote procedure call protocol on top of HTTP/2 (bidirectional streams, binary messages); it needs HTTP/2 all the way to the backend, HTTP/1.1 won't do.
  • WebSocket: a bidirectional channel inside an HTTP connection for chats and live updates; historically it runs over HTTP/1.1 via Upgrade, and many browsers and proxies still open a separate HTTP/1.1 connection for it even on an HTTP/2 site.

In short: HTTP/2 has been the mandatory minimum for any public site since 2016; HTTP/3 is a free improvement for mobile clients if your infrastructure lets UDP through; HTTP/1.1 is for internal connections and old clients.

Configuring Traefik

Traefik 3 supports all three versions, and they're configured at the entrypoint level, not on routes: IngressRoute, Ingress or Gateway API on top stay the same. The examples are values for the traefik/traefik Helm chart 41.x (the same syntax works in a static config file or as flags).

HTTP/1.1 and HTTP/2

Enabled by default. On a TLS entrypoint Traefik offers h2 and http/1.1 in ALPN, and the client picks the version; on an entrypoint without TLS it serves HTTP/1.1 and, if the client asks, h2c. There's almost nothing to tune:

ports:
  websecure:
    http2:
      maxConcurrentStreams: 250   # 250 streams per connection by default

Between Traefik and the backend it's HTTP/1.1 with keep-alive by default. If the backend needs HTTP/2 (gRPC), that's set on the service: Traefik switches to h2c by itself when the service scheme is h2c:

# Kubernetes Service of a gRPC backend
metadata:
  annotations:
    traefik.ingress.kubernetes.io/service.serversscheme: h2c

For Gateway API the same is done with an annotation on the service or appProtocol: kubernetes.io/h2c on the service port. If the backend is behind TLS and speaks HTTP/2, Traefik negotiates with it via ALPN on its own.

HTTP/3

Three conditions: TLS on the entrypoint (QUIC doesn't exist without TLS), a UDP port, and a flag:

ports:
  websecure:
    http3:
      enabled: true
      advertisedPort: 443

The chart turns this into --entryPoints.websecure.http3 and --entryPoints.websecure.http3.advertisedPort=443, adds port 8443/UDP next to 8443/TCP to the pod, and 443/UDP next to 443/TCP to the Service. advertisedPort is needed because the container listens on 8443, while Alt-Svc has to carry the port the client sees. Traefik adds the header to all of the entrypoint's responses by itself.

Without Kubernetes, in a static file:

entryPoints:
  websecure:
    address: ":443"
    http3:
      advertisedPort: 443
    http:
      tls: {}

Checks after the rollout, in order:

$ kubectl -n traefik get svc traefik          # 443/UDP must be there
$ curl -sI https://www.example.com/ | grep -i alt-svc
alt-svc: h3=":443"; ma=2592000
$ docker run --rm ymuski/curl-http3 curl -s --http3-only -o /dev/null -w '%{http_version} %{http_code}\n' https://www.example.com/
3 200

And in the browser: DevTools → Network → the Protocol column, or performance.getEntriesByType('navigation')[0].nextHopProtocol in the console: it should be h3 from the second load on. In Traefik's access log the version is written to the RequestProtocol field: HTTP/1.1, HTTP/2.0, HTTP/3.0; the Prometheus metrics have no breakdown by version, only by entrypoint.

Pitfalls

Collected from my own experience and from what others run into.

UDP 443 has to get through the whole path

HTTP/3 is UDP, and it has to be allowed everywhere TCP used to be enough: the firewall in front of the server, port forwarding on the router, the Security Group in the cloud, NetworkPolicy in the cluster. Forget one hop, and HTTP/3 simply won't kick in, silently: the browser will fall back to TCP, and you'll only find out from DevTools. A separate note on NAT: UDP entries live on a timer (in Linux, 30 seconds without replies, 180 with replies); QUIC keeps them alive with its PING packets, but on exotic NATs with short timeouts connections will drop, and also silently reopen.

Helm and ports with the same number

The most unexpected thing I ran into. After a helm upgrade with HTTP/3 enabled, Traefik rolled out, the flags appeared, but there were no UDP ports in either the Service or the pod, even though helm get manifest showed them. Helm applies changes with a Kubernetes strategic merge patch, and in it Service ports are merged by the port field and container ports by containerPort, ignoring the protocol. The new 443/UDP entry merged with the existing 443/TCP and vanished. The fix is to apply the Service and Deployment from the release manifest with kubectl apply --server-side --force-conflicts: server-side apply distinguishes ports by the port+protocol pair. Subsequent helm upgrade runs that don't change the ports don't bring the problem back, but checking get svc after an upgrade is worth it.

MTU and black holes

QUIC starts with 1200-byte packets, which are guaranteed to get through anywhere, and then probes the path MTU itself, increasing the size and watching whether larger packets arrive. Unlike TCP, it gets no help from MSS clamping on routers (rewriting the maximum segment size in the TCP handshake): UDP has no MSS. If the path includes a tunnel with an MTU of 1400 and a firewall that drops ICMP, an HTTP/3 connection will live until QUIC tries a bigger packet, and then hang; HTTP/2 over the same path works thanks to clamping. The symptom: "over the VPN the site loads slowly and in fits and starts, without the VPN it's fine". Diagnosis: DevTools and comparing h2/h3; the cure: fix ICMP along the path, because there's no way to disable HTTP/3 for such clients other than on the server as a whole.

UDP buffers in the kernel

Traefik's QUIC implementation is the quic-go library, and on startup it asks the kernel for large receive and send buffers for the UDP socket. By default in Linux net.core.rmem_max and wmem_max are about 200 KB, quic-go wants 7 MB, and if refused it works with what it gets: under load packets are dropped in the socket buffer, and QUIC, taking that for network congestion, slows down. On a bare server it's two lines in sysctl.conf:

net.core.rmem_max = 7500000
net.core.wmem_max = 7500000

In Kubernetes these are node parameters, not pod ones: set them on every node where Traefik may end up. To check whether you need it: kubectl logs of Traefik at the DEBUG level will show the warning failed to sufficiently increase receive buffer size if the buffers weren't big enough.

Load balancing and QUIC sessions

QUIC goes by the connection ID, while a load balancer in front of several Traefik replicas goes by addresses and ports (ECMP on the router, externalTrafficPolicy: Local in the cluster). The client switches networks or NAT changes its port, and the packets go to another replica that doesn't have this connection. It replies that it doesn't know the connection, and the client opens a new one. For the web that's unnoticeable; for long downloads it's a break and a retry. Same with 0-RTT: a resumption ticket issued by one replica won't be accepted by another, which will do a full handshake: it works, but without the gain. The solution for large installations is load balancing by connection ID; for a few Traefik replicas behind ECMP, just be aware of it.

Alt-Svc lives for 30 days

ma=2592000 is how long the browser remembers that HTTP/3 is available here. Turn HTTP/3 off on the server, and clients will keep trying UDP first for another month, falling back to TCP only on timeout; the delay is small, but it hits every new connection. Servers where the lifetime is configurable lower ma to minutes before turning it off; Traefik sends a fixed 30 days, so you either turn it off and wait, or put up with a month of extra attempts.

HTTP/2: losses and Rapid Reset

Two things worth remembering even without HTTP/3. The first is behavior on a lossy network: one TCP for a hundred streams, and every loss slows down everything; for clients on bad Wi-Fi, HTTP/2 can feel worse than HTTP/1.1, and that's a perfectly good reason to enable HTTP/3. The second is the Rapid Reset attack (CVE-2023-44487): the client opens streams and immediately resets them with an RST_STREAM frame, never hitting the maxConcurrentStreams limit, and the server burns resources processing them. The protection lives in the implementations themselves; Traefik fixed it in 2.10.5 and 3.0, so there's only one cure: don't stay on old versions.

HTTP/1.1: smuggling and keep-alive

A text format with two ways of specifying the body length (Content-Length and chunked) gave rise to the request smuggling class of attacks: the proxy and the backend read request boundaries differently, and part of one request becomes the start of the next. The protection is the same parsing on the whole chain and binary protocols between the proxy and the backend, where there's no ambiguity. The second classic is keep-alive: the proxy keeps the connection to the backend open, the backend closes idle ones after 5 seconds, the proxy sends a request into the closed connection and gets an error. In Traefik the idle timeout towards backends is set in serversTransport (forwardingTimeouts.idleConnTimeout), and it must be shorter than the backend's.

Observability

For HTTP/3, tcpdump shows only UDP datagrams, and Wireshark without the keys shows the same. Per-version statistics come only from the access log (RequestProtocol). If you're used to investigating problems in a traffic capture, with HTTP/3 you'll have to retrain yourself onto server logs and SSLKEYLOGFILE on the client.

The upshot

  • Three versions: one semantics and three ways of packaging it. HTTP/1.1 is text and a queue, HTTP/2 is frames and streams in one TCP connection, HTTP/3 is the same streams in QUIC, where each stream loses packets on its own.
  • HTTP/2 removed the application queue and the repeated headers; HTTP/3 removed the transport queue and the extra handshake round trip and survives network changes, but it costs CPU and needs UDP along the whole path.
  • In Traefik the first two versions are always there; the third is a flag on a TLS entrypoint, a UDP port and advertisedPort. Check Alt-Svc, curl --http3-only and the Protocol column in the browser.
  • Pitfalls: UDP on firewalls and NAT, port merging on helm upgrade, MTU in tunnels without help from clamping, kernel buffers for quic-go, replicas behind ECMP, the month-long memory of Alt-Svc. None of this breaks the site, because the client can always fall back to TCP, but it does leave HTTP/3 "enabled on paper".

Only registered users can leave a comment.

Sign in or register to leave a comment.

© 2006-2026 Vitaliy Leschenko