15 October 2026, 10:55Виталий Лещенко

Networking for beginners. Part 18: VXLAN

All parts of the series “Networking for beginners”

The VLAN from the previous part slices one switch into many networks. But it has two limitations that turn into walls in a data center. The first is that there are only 4094 numbers, while a cloud provider wants an isolated network for every customer, and it has hundreds of thousands of customers. The second matters more: a VLAN is L2, and it ends where a router begins. A virtual machine moving from a server in one rack to a server in another wants to keep its IP address and stay in "its" network, and between the racks there are routers. Stretching L2 across the whole data center with VLANs and STP means one huge broadcast domain with a single STP tree, and that stops working at a few hundred switches.

VXLAN, Virtual eXtensible LAN, RFC 7348, solves both problems with a trick from part one — a protocol inside a protocol: an Ethernet frame is wrapped in a UDP datagram and travels across any IP network like an ordinary packet. For the hosts inside it is an L2 segment; for the network outside it is UDP traffic between servers.

Overlay and underlay

Two words you can't talk about VXLAN without:

Underlay — the physical IP network: switches, routers, routes between servers. It doesn't care what it carries.
Overlay — the virtual network on top of it: frames of virtual machines or containers that think they are connected to a shared switch.

The point where one turns into the other is the VTEP (VXLAN Tunnel Endpoint): a device or program with an address in the underlay that takes a frame from the overlay, wraps it and sends it to another VTEP, and unwraps what it receives and hands it to the overlay. A VTEP can live in a switch (hardware) or in the Linux kernel, a hypervisor or Open vSwitch (software). In a cloud, every virtualization server is a VTEP.

   VM-A 10.100.0.1                                                     VM-B 10.100.0.2
   MAC aa:…:01                                                         MAC bb:…:02
        │  overlay (VNI 100)                                      overlay  │
   ┌────┴──────────┐                                            ┌──────────┴────┐
   │ server 1      │                                            │ server 2      │
   │ VTEP          │                                            │ VTEP          │
   │ 198.51.100.2  │                                            │ 203.0.113.5   │
   └────┬──────────┘                                            └──────────┬────┘
        │  underlay: ordinary IP packets, UDP 4789                         │
        └──────────── routers, any number of hops ─────────────────────────┘

The packet

A frame from VM-A to VM-B looks like this in the underlay:

 ┌───────────┬───────────┬──────────┬──────────┬───────────────────────────────┐
 │ Ethernet  │ IP        │ UDP      │ VXLAN    │ original Ethernet frame       │
 │ outer     │ outer     │ dst 4789 │ 8 bytes  │ aa:…:01 → bb:…:02, IP, TCP, … │
 │ 14        │ 20        │ 8        │          │ (without FCS)                 │
 └───────────┴───────────┴──────────┴──────────┴───────────────────────────────┘
   MAC and IP of the VTEPs, src port = hash of the inner frame

Outer IP — the VTEP addresses: 198.51.100.2 → 203.0.113.5. The underlay routes by them and knows nothing about the inner frame.
Outer UDP: destination port 4789 (registered for VXLAN; Linux historically used 8472, and that is the first thing to check when "it doesn't work between two implementations"). The source port is a hash of the addresses and ports of the inner frame. This is deliberate: underlay routers balance traffic across equal-cost paths by hashing addresses and ports (we came across this in part eight, when traceroute "jumped" between paths), and if the port were constant, the whole tunnel between two VTEPs would take a single path. With a varying port, different flows inside the tunnel are spread across different links.
The VXLAN header, 8 bytes:

 ┌────────┬────────────────────┬────────────────────────┬────────┐
 │ flags  │ reserved           │ VNI                    │ rsvd   │
 │ 8 bits │ 24 bits            │ 24 bits                │ 8 bits │
 │ I = 1  │                    │ network number, 16M    │        │
 └────────┴────────────────────┴────────────────────────┴────────┘

The I flag says the VNI is valid; the rest are zeros. The VNI (VXLAN Network Identifier) is the counterpart of the VLAN VID, but 24 bits: 16 million networks instead of 4094.

The inner frame — as it was, with the virtual machines' MAC addresses; only the FCS is dropped, since the outer Ethernet will compute its own.

The arithmetic: 14 + 20 + 8 + 8 = 50 bytes of overhead over IPv4 (70 over IPv6). An inner frame with 1500 bytes of data becomes an outer packet of 1550, and it will not fit into an underlay with MTU 1500. There are two ways out: raise the underlay MTU to 1550 or jumbo 9000 (the right one if the underlay is yours), or lower the MTU of the overlay interfaces to 1450 (the only one if the underlay belongs to someone else). Forgetting about this is a classic: ping works, large transfers hang — the very same black hole from part eight, only inside a tunnel.

In tcpdump on the underlay interface you can see both nesting dolls at once:

$ sudo tcpdump -i eth0 -n udp port 4789
IP 198.51.100.2.49152 > 203.0.113.5.4789: VXLAN, flags [I] (0x08), vni 100
ARP, Request who-has 10.100.0.2 tell 10.100.0.1, length 28
IP 203.0.113.5.52201 > 198.51.100.2.4789: VXLAN, flags [I] (0x08), vni 100
ARP, Reply 10.100.0.2 is-at bb:bb:bb:bb:bb:02, length 28
IP 198.51.100.2.49152 > 203.0.113.5.4789: VXLAN, flags [I] (0x08), vni 100
IP 10.100.0.1 > 10.100.0.2: ICMP echo request, id 1, seq 1, length 64

ARP from part seven, whole, inside UDP: for the virtual machines this is an ordinary Ethernet segment.

How a VTEP knows where to send

The switch in part two learned its "MAC → port" table from source addresses. A VTEP does the same, except that its "port" is the address of a remote VTEP: when it receives a frame from the tunnel, it remembers "inner MAC bb:…:02 lives behind VTEP 203.0.113.5". For known unicast everything is simple: wrap it and send it to that VTEP.

The problem is BUM traffic: broadcast, unknown unicast, multicast. A switch flooded it to all ports. What are a VTEP's "ports"? All the other VTEPs with the same VNI, and there may be hundreds of them, not on one segment. Three ways:

Multicast in the underlay. Each VNI is mapped to a group address, for example 239.1.1.100; all VTEPs with that VNI join the group, and BUM frames are wrapped in a packet to the group address. The underlay delivers it to all subscribers, provided it can route multicast: for that it has to run a protocol like PIM, with which routers build a delivery tree from the sender to the subscribers. In your own data center this is possible, in someone else's cloud it is not. This is how VXLAN was designed in the RFC.

Head-end replication. A VTEP keeps a list of the other VTEPs in its VNI and sends a BUM frame to each one separately as unicast. The underlay does not need multicast, but with a hundred VTEPs every ARP request turns into a hundred packets. The list is set by hand or by a controller.

Both methods together with learning from frames are called flood-and-learn: a VTEP learns where MAC addresses are by watching traffic, like a switch. It is fine for a dozen VTEPs, not for a thousand.

Control plane: EVPN. Instead of learning from traffic, VTEPs tell each other their tables through a routing protocol. The protocol is BGP — the one providers use to exchange routes (covered in part 22 and, in practice, in a separate series) — with the EVPN extension: each VTEP announces "in VNI 100 I have MAC aa:…:01 with IP 10.100.0.1" and "I participate in VNI 100". The others receive this before a frame even appears. The consequences: unknown unicast disappears (all MACs are known in advance), and a VTEP can answer ARP requests itself from the EVPN table without letting them into the tunnels (ARP suppression). This is how modern data center fabrics are built, and in them the underlay is pure L3 between switches (spine-leaf: every leaf switch is connected to every spine, with only routing between them), while L2 exists only inside VXLAN at the edge.

Static entries. A fourth option, with no flooding and no BGP: an external program that already knows where each machine is (an orchestrator) writes the tables into the VTEPs directly. This is how Kubernetes works.

Where it's used

Clouds. Each customer gets its own VNIs; its machines on any servers see each other in "their" L2 network, but not their neighbors on the same server. Millions of networks, customer addresses overlap (everyone has 10.0.0.0/24), and that is not a problem: they are in different VNIs.
Data centers with an EVPN fabric: the L2 network stretches across all racks without STP between switches, and a VM moves anywhere with the same address.
Kubernetes. Pods on different nodes must see each other by IP, and the nodes may sit in networks where you cannot simply add routes. Network plugins (Flannel in vxlan mode, Calico in VXLAN mode) set up a VTEP on every node, assign each node a pod subnet and write static entries "node B's subnet → VTEP B" from the API server's data. No flooding, no BGP: the orchestrator knows everything. Pod-to-pod traffic across nodes is VXLAN between nodes on port 4789 (or 8472 with Flannel), and you can see it in tcpdump on a node. Docker Swarm does the same for overlay networks.
Stretching L2 between sites for VM migration. Technically possible, practically dangerous: a broadcast domain with 20 ms of latency and a single gateway for two sites means both broadcast storms over the WAN and traffic that travels to the other site only to come back to its own. It is done, but with caveats.

A couple of colleagues in the trade, to avoid confusion: Geneve is the successor of VXLAN with an extensible header (port 6081); OVN and NSX-T run on it, and the idea is the same. GRE is an older encapsulation directly in IP (protocol 47) without UDP, and therefore without entropy for load balancing and with problems behind NAT. VLAN is a tag in a frame within one L2; VXLAN is a frame in UDP across L3.

What VXLAN lacks

Encryption and authentication. The inner frame travels in the clear, and the VNI is just a number in the header: anyone with access to the underlay can read other networks and inject frames into them with any VNI. Isolation rests on the underlay being a closed data center network. Over the internet, VXLAN is wrapped in IPsec (the next part) or another encrypted tunnel.
Protection from the fact that it is still L2. A large broadcast domain stays large: ARP storms, loops inside the overlay, MAC flapping — everything from part two, only now across routers.
Automatic MTU. The overlay does not know it is an overlay.

And a practical note: the outer UDP checksum may be zero (the RFC allows it for IPv4), many network cards compute and verify VXLAN in hardware (offload), and underlay firewalls must allow UDP 4789 between VTEPs.

VXLAN between two Linux hosts

Two servers with underlay addresses 198.51.100.2 and 203.0.113.5, with ordinary routing between them. Point-to-point, no flooding — the remote VTEP is set explicitly:

# server 1
$ sudo ip link add vxlan100 type vxlan id 100 local 198.51.100.2 remote 203.0.113.5 dstport 4789
$ sudo ip link set vxlan100 mtu 1450 up
$ sudo ip addr add 10.100.0.1/24 dev vxlan100

# server 2
$ sudo ip link add vxlan100 type vxlan id 100 local 203.0.113.5 remote 198.51.100.2 dstport 4789
$ sudo ip link set vxlan100 mtu 1450 up
$ sudo ip addr add 10.100.0.2/24 dev vxlan100

$ ping 10.100.0.2

vxlan100 is an ordinary Ethernet interface with a MAC address, ARP and everything else; its "cable" just happens to be a tunnel. If instead of addresses you put a bridge on it and attach virtual machines, you get a shared L2 segment for VMs on two servers.

Several VTEPs with head-end replication: no remote is set, and the list of neighbors goes into the bridge FDB table as entries with an all-zero MAC "for flooding":

$ sudo ip link add vxlan100 type vxlan id 100 local 198.51.100.2 dstport 4789
$ sudo bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst 203.0.113.5
$ sudo bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst 203.0.113.6

Now BUM traffic goes to both, and unicast goes to wherever the MAC was learned. Learned entries show up in the same table with a VTEP address instead of a port:

$ bridge fdb show dev vxlan100
00:00:00:00:00:00 dst 203.0.113.5 self permanent
00:00:00:00:00:00 dst 203.0.113.6 self permanent
bb:bb:bb:bb:bb:02 dst 203.0.113.5 self

The multicast variant is group 239.1.1.100 dev eth0 instead of remote; the static one, as in Kubernetes, is nolearning at creation and bridge fdb add <mac> dev vxlan100 dst <vtep> for each machine.

All of the interface's parameters: ip -d link show vxlan100 shows the id, local, remote or group, port, learning/nolearning, udpcsum.

Key takeaways

VXLAN wraps an Ethernet frame in UDP (port 4789) with an 8-byte header and a 24-bit VNI: 16 million L2 networks on top of any IP network. Overlay is the virtual networks, underlay is the physical one, the VTEP is the boundary.
50 bytes of overhead: overlay MTU 1450 or underlay 1550+. The source port is a hash for load balancing in the underlay.
A VTEP learns "MAC → VTEP address" from traffic; BUM traffic goes via multicast in the underlay or head-end replication (flood-and-learn), or the tables come from a control plane: EVPN over BGP in data centers, static entries from the orchestrator in Kubernetes and Docker.
There is no encryption: isolation rests on a closed underlay, and over the internet it goes inside IPsec. The overlay remains L2 with all its ailments, only bigger.
Linux: ip link add … type vxlan id 100 … dstport 4789, the table is bridge fdb show dev vxlan100, and a capture with tcpdump udp port 4789 shows both nesting dolls.

Next — Part 19: IPsec and IKEv2: how to encrypt packets at the network layer so that applications don't even know. ESP and its modes, Security Associations and SPI, policies and selectors, and the four IKEv2 exchanges that agree on keys, authenticate the parties and survive NAT.