Networking for beginners. Part 19: IPsec and IKEv2
All parts of the series “Networking for beginners”Everything we have looked at so far traveled over the network in plain text: headers, data, PAP passwords from part four, DNS queries. We have only met protection above the transport layer — TLS, the encryption protocol inside a TCP connection that HTTPS runs on: it protects the data of one application, and only if the application uses it. IPsec provides protection at the network layer: it encrypts and signs whole IP packets, so applications don't know about it, and everything ends up protected — any protocol, any port. IPsec is the foundation of tunnels between offices, of employees connecting to the corporate network and, as we will see in the next part, of most of the "VPN" entries in a phone's settings.
IPsec has two layers. IPsec in the narrow sense is the format of protected packets and the rules for processing them: what to encrypt, with which key, how to verify. IKE — Internet Key Exchange — is the protocol by which two parties agree on algorithms, authenticate each other and derive keys for the first layer. Let's go bottom up.
What IPsec protects and how
Two protocols, both a Next Header / Protocol value in the IP header:
- ESP (Encapsulating Security Payload, protocol 50) — encryption and integrity checking. Used practically always.
- AH (Authentication Header, protocol 51) — integrity and authenticity only, no encryption, and the signature also covers the IP header. Because of this AH is incompatible with NAT, which changes the header, and today it is almost never seen: ESP with integrity checking does all the same things except protecting the IP header itself, which nobody needs.
And two modes that decide what exactly gets wrapped:
original packet: ┌─────────┬───────────────────────┐
│ IP │ TCP + data │
└─────────┴───────────────────────┘
transport mode: ┌─────────┬─────┬────────────────────────┬───────┬─────┐
│ IP │ ESP │ TCP + data │ ESP │ ICV │
│ (same) │ hdr │ ░░░░ encrypted ░░░░░░░ │ trail │ │
└─────────┴─────┴────────────────────────┴───────┴─────┘
tunnel mode: ┌─────────┬─────┬────────┬───────────────┬───────┬─────┐
│ new │ ESP │ IP │ TCP + data │ ESP │ ICV │
│ IP │ hdr │ ░░░░░░░░ encrypted ░░░░░░░░░░░ │ │
└─────────┴─────┴────────┴───────────────┴───────┴─────┘
Transport mode protects only the packet's data, keeping the original IP header: the addresses are visible, and what is protected is the communication between two hosts. Tunnel mode encrypts the original packet in full, including its addresses, and prepends a new IP header with the gateways' addresses. This is the mode for gateway-to-gateway (two offices) and for remote access: the internal 192.168.x.x addresses are hidden inside, and from the outside only the tunnel endpoints' addresses are visible.
The ESP packet
┌───────────────────┐
│ SPI (4) │ which SA: the receiver finds the keys and algorithms by it
│ Sequence (4) │ packet number, grows by 1: replay protection
├───────────────────┤
│ IV │ initialization vector for the cipher
│ data ░░░░░░░░░░░░ │ encrypted: the original packet (tunnel) or segment (transport)
│ padding ░░░░░░░░░ │ alignment to the cipher block
│ pad len │ next hdr│ next header: 4 = IPv4 inside, 41 = IPv6, 6 = TCP (transport)
├───────────────────┤
│ ICV │ integrity check value: signature of everything above with the integrity key
└───────────────────┘
The receiver uses the destination address and the SPI to find the SA (see below), checks that the Sequence has not been seen yet (a sliding window, usually 64 numbers or more), verifies the ICV with the integrity key, decrypts, reads the next header at the very end and learns whether what's inside is an IPv4 packet to be routed further or a TCP segment for itself. The order matters: integrity first, then decryption, so as not to waste time on forgeries and not to let an attacker feed garbage to the cipher.
Algorithms: today it is AES-GCM (encryption and integrity with one key, 16-byte ICV) or ChaCha20-Poly1305; the older combination is AES-CBC plus HMAC-SHA256. 3DES, MD5, SHA1 are on the "do not use" list.
The overhead of tunnel mode with AES-GCM is about 50–60 bytes per packet (20 for the new IP, 8 ESP, 8 IV, 16 ICV, padding), plus 8 if ESP is in UDP. That is why a tunnel has an MTU of 1400–1440, and the MSS clamping from part eight is present in tunnel rules almost always.
SA, SPI, SPD
A Security Association is a one-way agreement: "between A and B, for packets with this SPI, this algorithm, these keys, this mode, a lifetime of so many seconds or bytes". One-way means each direction has its own SA, and a tunnel has a pair of them. The SPI (32 bits) is a number the receiver picks itself when the SA is created, and the sender puts it into the ESP header; this is how one machine tells apart hundreds of tunnels to it.
SAs are stored in the SAD (SA database). Next to it is the SPD, the policy database: rules like "traffic with these addresses, protocol, ports — protect with this SA / pass as is / drop". Policy selectors are exactly what people usually mean by "which networks go through the tunnel". When a packet leaves the machine, the stack checks it against the SPD; if the policy says "protect" and there is no SA yet, the stack asks IKE to create one and holds the packet until it appears.
On Linux this is the xfrm subsystem, and everything listed above is visible:
$ sudo ip xfrm state # SAD
src 198.51.100.2 dst 203.0.113.5
proto esp spi 0xc1a2b3d4 reqid 1 mode tunnel
replay-window 32 flag af-unspec
aead rfc4106(gcm(aes)) 0x... 128
anti-replay context: seq 0x0, oseq 0x1a, bitmap 0x00000000
src 203.0.113.5 dst 198.51.100.2
proto esp spi 0x9e8f7a6b reqid 1 mode tunnel
...
$ sudo ip xfrm policy # SPD
src 192.168.1.0/24 dst 10.0.0.0/24
dir out priority 375423
tmpl src 198.51.100.2 dst 203.0.113.5
proto esp reqid 1 mode tunnel
src 10.0.0.0/24 dst 192.168.1.0/24
dir in priority 375423
...
Two SAs with different SPIs for the two directions, a policy "from 192.168.1.0/24 to 10.0.0.0/24 — into the tunnel between the gateways", and the oseq counter — how many packets have gone out.
The keys in the SA came from somewhere. You can enter them by hand (ip xfrm state add … key 0x…), and that is useful for understanding the mechanics, but nobody does it in real life: keys must be random, must change, and both sides must authenticate each other. That is IKE's job.
IKEv2
IKEv2 (RFC 7296) is a protocol over UDP 500 (and 4500, more on that below) through which the parties:
- agree on algorithms and derive a shared secret with Diffie-Hellman;
- authenticate each other;
- create SAs for ESP (in IKE terms, Child SAs) and later re-create them when they expire.
The first version, IKEv1 from 1998, did the same in six to nine messages in two phases with different modes, and could do neither NAT, nor EAP, nor address changes without add-ons; IKEv2 built all of that in and fit it into four messages. IKEv1 lives on in old equipment, but new tunnels are not built on it.
IKE_SA_INIT: agree and exchange keys
initiator responder
│ │
├── IKE_SA_INIT ──────────────────────────────────────────────────────►│
│ SAi1: proposals (cipher, PRF, integrity, DH group) │
│ KEi: public DH value, Ni: random number │
│ [NAT_DETECTION_*: hashes of own address and port] │
│ │
│◄── IKE_SA_INIT ──────────────────────────────────────────────────────┤
│ SAr1: chosen proposal, KEr, Nr, [CERTREQ], [NAT_DETECTION_*] │
│ │
│ both sides: SKEYSEED = prf(Ni | Nr, g^ir) │
│ from it SK_d, SK_ai, SK_ar, SK_ei, SK_er, SK_pi, SK_pr │
│ from here on everything is encrypted and signed │
Two messages in plain text. The initiator lists what it supports (for example: AES-GCM-256, PRF HMAC-SHA256, DH group 19 — the P-256 elliptic curve, or 14 — modp2048, or 31 — Curve25519), attaches its half of the Diffie-Hellman exchange and a random number. The responder picks one set and attaches its own. After that both sides have a shared secret g^ir that never went over the wire, and from it, using the PRF (a pseudorandom function based on HMAC), seven keys are derived: for deriving further keys (SK_d), for integrity and encryption of IKE messages in each direction (SK_a*, SK_e*) and for authentication (SK_p*).
Flood protection: a responder that sees too many unfinished exchanges can send a COOKIE instead of a reply and demand that the request be repeated with it — until then it creates no state, like SYN cookies in TCP.
The key point: so far nobody has verified anybody. A shared secret exists even with an impostor sitting in the middle. Verification is the next step.
IKE_AUTH: prove who you are and create the first Child SA
initiator responder
│ │
├── IKE_AUTH (encrypted with SK_ei) ──────────────────────────────────►│
│ IDi: who I am (address, FQDN, e-mail, certificate DN) │
│ [CERT]: my certificate │
│ AUTH: signature of my first message + Nr + prf(SK_pi, IDi) │
│ SAi2: proposals for ESP, TSi, TSr: traffic selectors │
│ [CP: give me an address and DNS] │
│ │
│◄── IKE_AUTH (encrypted with SK_er) ──────────────────────────────────┤
│ IDr, [CERT], AUTH: signature of its own first message + Ni + … │
│ SAr2, TSi, TSr (possibly narrowed), [CP: address, DNS] │
│ │
│ IKE SA established, first pair of Child SAs (ESP) established │
AUTH is what binds the keys to an identity. Each side signs its own first IKE_SA_INIT message (the one that went out in plain text), the other side's random number and a hash of its own identifier. An impostor in the middle would have replaced the DH values in the first message, and the signature would not match. What is used to sign:
- PSK, a pre-shared key: AUTH = prf(prf(PSK, "Key Pad for IKEv2"), data). Simple and convenient between two gateways, but it is one string for both sides that has to be delivered somehow and changed periodically.
- Certificates: a signature with an RSA or ECDSA private key, the certificate in CERT, trust through a common certificate authority. The standard for gateway-to-gateway in organizations, and mandatory for a remote access server: Windows, iOS and Android clients require the server to have a certificate with a name matching the address they connect to.
- EAP for remote access: the initiator does not send AUTH in the first IKE_AUTH, and the exchange stretches over several message pairs carrying EAP — the very request-response protocol from part three on Wi-Fi: EAP-MSCHAPv2 (a login and password, checked by the server or via RADIUS, the protocol network equipment uses to ask a central database whether to let a user in) or EAP-TLS (a user certificate). As a result of EAP both sides get the MSK key, and the last IKE_AUTH pair carries an AUTH computed with it. The server still proves itself with a certificate.
The same exchange creates the first Child SA — a pair of SAs for ESP, according to the SAi2 proposal (the ESP cipher may differ from IKE's) — and the traffic selectors TSi/TSr: the initiator says "I want to protect traffic from 192.168.1.0/24 to 10.0.0.0/24", and the responder agrees or narrows it down to what it is allowed. Narrowing is a common cause of "the tunnel is up but ping doesn't go through": the selectors at the two ends did not match, and the SA was created for the intersection, which your traffic does not fall into.
CP, the configuration payload, is for remote access: the client asks for an internal address, DNS, a domain, and the server hands them out, like DHCP but inside IKE. This is how a laptop gets 10.0.5.17 "inside" the corporate network.
CREATE_CHILD_SA and INFORMATIONAL
After that the IKE SA lives for a long time (hours) and serves as a protected management channel:
- CREATE_CHILD_SA — create one more pair of ESP SAs (for another set of selectors) or re-create an existing one when it expires — rekeying. ESP keys live for an hour or for gigabytes and are replaced invisibly to traffic: the new SA is created before the old one is deleted. Optionally a new DH exchange is included in the rekey (PFS, perfect forward secrecy): compromising the IKE key will not reveal past ESP keys.
- INFORMATIONAL — deleting SAs, notifications, and two important things. DPD (dead peer detection): an empty INFORMATIONAL that the other side must answer; no answer several times — the peer is dead, the SAs are deleted, the tunnel is re-established. MOBIKE (RFC 4555): a client whose address has changed (a phone moved from Wi-Fi to the cellular network) sends UPDATE_SA_ADDRESSES, and all SAs move to the new address without a new handshake. This is what makes IKEv2 convenient on mobile devices.
NAT-T: IPsec through NAT
NAT from part six and ESP are incompatible on both counts: NAT rewrites addresses, and ESP has no ports by which NAT could keep a translation table; two clients behind one NAT using ESP to the same server cannot be told apart at all. That is why IKE_SA_INIT includes NAT detection: each side sends a hash of its own address and port as it sees them. The receiver computes the same hash from the address and port the packet came from. If they don't match, there is a NAT along the way.
Then both sides switch to UDP 4500: IKE continues there with four zero bytes in front (the non-ESP marker), and ESP packets are wrapped in UDP on the same port — UDP-encapsulated ESP. NAT now has ports, clients behind one NAT get different external ports, and everything works. The price is 8 bytes of UDP per packet and the need to keep the NAT entry alive: the client sends a keepalive, a one-byte UDP packet 0xFF, every 20 seconds, for exactly the reasons from part nine.
$ sudo tcpdump -i eth0 -n 'udp port 500 or udp port 4500 or esp'
IP 198.51.100.2.500 > 203.0.113.5.500: isakmp: parent_sa ikev2_init[I]
IP 203.0.113.5.500 > 198.51.100.2.500: isakmp: parent_sa ikev2_init[R]
IP 198.51.100.2.4500 > 203.0.113.5.4500: NONESP-encap: isakmp: child_sa ikev2_auth[I]
IP 203.0.113.5.4500 > 198.51.100.2.4500: NONESP-encap: isakmp: child_sa ikev2_auth[R]
IP 198.51.100.2.4500 > 203.0.113.5.4500: UDP-encap: ESP(spi=0xc1a2b3d4,seq=0x1), length 132
IP 203.0.113.5.4500 > 198.51.100.2.4500: UDP-encap: ESP(spi=0x9e8f7a6b,seq=0x1), length 132
IP 198.51.100.2.4500 > 203.0.113.5.4500: isakmp-nat-keep-alive
tcpdump calls IKE by its old name, isakmp. Everything this article was about is visible: INIT on 500, NAT detected, AUTH already on 4500 with the marker, ESP in UDP with SPIs and sequence numbers, keepalives. What is inside ESP nobody can see: no addresses, no ports, no protocol — only the size and rate of packets.
What a firewall needs for IPsec: UDP 500, UDP 4500 and, if there is no NAT, the ESP protocol (50). AH (51) — if anyone still uses it.
Policies or routes
There are two ways to decide which traffic goes into the tunnel, and confusing them is the second most common cause of broken tunnels after selectors.
Policy-based is the classic: the SPD with its selectors intercepts traffic matching a policy and sends it into the SA. The tunnel has no interface, it is not visible in ip route, and traffic enters it if it matches a selector. Simple for "network A — network B", inconvenient when you need dynamic routing, several networks or a firewall "on the tunnel interface".
Route-based — the tunnel is represented by an interface (on Linux, an XFRM interface or VTI), the SA selectors are "everything" (0.0.0.0/0), and the routing table decides what to send into the tunnel. Through such an interface you can run dynamic routing protocols (OSPF within a network, BGP between networks — routers use them to tell each other about their routes, as we mentioned in part six), apply firewall rules, see counters. Modern configurations are more often like this:
$ sudo ip link add ipsec0 type xfrm dev eth0 if_id 42
$ sudo ip link set ipsec0 up
$ sudo ip route add 10.0.0.0/24 dev ipsec0
and the SAs/policies specify the same if_id 42 so the stack ties the interface to the tunnel.
Configuration and diagnostics in strongSwan
The most common IKE implementation on Linux is strongSwan. A certificate-based gateway-to-gateway tunnel in swanctl.conf:
connections {
office-b {
remote_addrs = 203.0.113.5
local { auth = pubkey; certs = gw-a.pem; id = gw-a.example.com }
remote { auth = pubkey; id = gw-b.example.com }
children {
lan {
local_ts = 192.168.1.0/24
remote_ts = 10.0.0.0/24
esp_proposals = aes256gcm16-x25519
dpd_action = restart
start_action = start
}
}
proposals = aes256gcm16-prfsha256-x25519
version = 2
dpd_delay = 30s
}
}
Every line is something from this article: proposals is the set for IKE_SA_INIT, esp_proposals for the Child SA, local_ts/remote_ts are the selectors, auth = pubkey means certificates, dpd_delay is the INFORMATIONAL interval.
State:
$ sudo swanctl --list-sas
office-b: #1, ESTABLISHED, IKEv2, 1a2b3c4d5e6f7a8b_i* 8b7a6f5e4d3c2b1a_r
local 'gw-a.example.com' @ 198.51.100.2[4500]
remote 'gw-b.example.com' @ 203.0.113.5[4500]
AES_GCM_16-256/PRF_HMAC_SHA2_256/CURVE_25519
established 1234s ago, rekeying in 12345s
lan: #1, reqid 1, INSTALLED, TUNNEL-in-UDP, ESP:AES_GCM_16-256
installed 1234s ago, rekeying in 2345s, expires in 3456s
in c1a2b3d4, 12345 bytes, 123 packets, 2s ago
out 9e8f7a6b, 23456 bytes, 234 packets, 2s ago
local 192.168.1.0/24
remote 10.0.0.0/24
ESTABLISHED — the IKE SA exists, INSTALLED — the Child SA is in the kernel, TUNNEL-in-UDP — NAT-T kicked in, then the SPIs for both directions and the counters. If in grows but out doesn't, packets from our side are not getting into the tunnel: check the policies and routes. If both are zero, the traffic does not match the selectors. If the SA is in the CONNECTING state, check the log with journalctl -u strongswan: it will show at which message things stopped (no answer to INIT — a firewall or the address; AUTH failed — keys, certificates, identifiers; NO_PROPOSAL_CHOSEN — the algorithms did not match; TS_UNACCEPTABLE — the selectors).
Key takeaways
- IPsec protects IP packets: ESP (50) encrypts and signs, AH (51) only signs and is hardly used. Transport mode is between hosts, the original header stays; tunnel mode puts the whole packet inside a new one, for gateways and remote access.
- An SA is one-way: SPI, algorithms, keys, lifetime; a pair per tunnel; the SPD says which traffic goes where. On Linux these are
ip xfrm stateandip xfrm policy. - IKEv2: IKE_SA_INIT (algorithms, DH, nonces, in the clear), IKE_AUTH (signing the first message with a PSK, a certificate or EAP; selectors; the first Child SA; an address via CP), CREATE_CHILD_SA (more SAs and rekeying), INFORMATIONAL (DPD, deletion, MOBIKE).
- NAT breaks ESP; NAT-T detects NAT in INIT and moves everything to UDP 4500 with a keepalive every 20 seconds. Open UDP 500, 4500 and ESP.
- The tunnel is up but traffic doesn't flow — selectors or policies/routes. The tunnel MTU is 50–70 bytes smaller; MSS clamping is a must.
Next — Part 20: VPN and proxies: what a VPN is in terms of layers and interfaces, how, using IKEv2 remote access as an example, a client gets an address, routes and DNS, what split tunneling and DNS leaks are, how HTTP CONNECT and SOCKS work, and why what browser extensions call a VPN is usually a proxy.
Only registered users can leave a comment.
Sign in or register to leave a comment.