Networking for beginners. Part 2: Ethernet
All parts of the series “Networking for beginners”In the first part we saw that the network stack is a nesting doll of headers, and that its outermost layer, the link layer, delivers data to a neighbour on the cable. Today is about the most widespread link-layer protocol: Ethernet. It runs the wall socket at home, the switch port in the rack, 400-gigabit links between data centres, and even Wi-Fi pretends to be Ethernet to the operating system.
Ethernet was invented by Robert Metcalfe at Xerox PARC in 1973 for a network made of one thick coaxial cable that all the office machines were attached to. Since then everything has changed: the cable, the speed, the topology, the electronics. Only the frame format has stayed the same, and that is the most instructive part of its history.
The frame
An Ethernet frame, as tcpdump sees it, looks like this:
bytes: 6 6 2 46…1500 4
┌────────────┬──────────┬────────┬───────────────┬─────┐
│ MAC │ MAC │ Ether │ data │ FCS │
│ destination│ source │ Type │ (IP packet, │ │
│ │ │ │ ARP, …) │ │
└────────────┴──────────┴────────┴───────────────┴─────┘
└────── header, 14 bytes ────────┘
Before the frame, another 8 bytes go into the cable that programs never see: the preamble — seven bytes of 10101010 that the receiver uses to lock onto the clock — and the start frame delimiter (SFD) 10101011, whose final one says "the header comes next". After the frame comes the interframe gap (IFG) of 96 bit times, that is, a pause as long as 12 bytes, so the receiver has time to process what it got. At gigabit speed that is 96 nanoseconds.
Field by field:
- Destination MAC — first, so that the network card and the switch can drop a frame not meant for them without reading it to the end.
- Source MAC — the switch uses it for learning, more on that below.
- EtherType — the very "what's inside" field from the first part:
0x0800is an IPv4 packet and0x86DDan IPv6 packet, two generations of the network layer (parts 5–6 and 13);0x0806is ARP, the request "what MAC has this IP address" (part 7);0x8100is a VLAN tag with the number of a virtual network (part 17);0x8863/0x8864is PPPoE, a subscriber session at an ISP (part 4). There is a subtlety: if the value is less than or equal to0x05DC(1500), it is not a type but the length of the data, and we are looking at an old-format IEEE 802.3 frame with an LLC header inside. Today practically only STP control frames use that format. Everything else is Ethernet II with an EtherType starting at0x0600. - Data — from 46 to 1500 bytes. If an IP packet is shorter than 46 bytes (and a SYN without options is 40), the network card pads it with zeros up to 46. That is why
tcpdumpshows 60-byte frames for packets that would fit in 54: that is padding. The upper limit of 1500 is Ethernet's MTU, which we will keep tripping over throughout the series. - FCS — a CRC-32 checksum over the whole frame, from the header through the data. The network card computes and checks it. If it does not match, the frame is silently dropped: Ethernet does not report errors to the sender and does not retransmit anything. Retransmission is the job of TCP, eight parts later. The only trace is the
rx_crc_errorscounter in the interface statistics; it is worth checking when "the network works, but slowly".
So the minimum frame is 64 bytes (14 + 46 + 4), the maximum 1518 (14 + 1500 + 4). With a VLAN tag, inserted between the source MAC and the EtherType, it is four bytes more. Some networks allow jumbo frames with up to 9000 bytes of data, but that is an agreement rather than a standard: every device along the path must be configured the same way, otherwise the frame disappears without a trace.
Remember the number 64: further down it will become clear where it came from.
The MAC address
Six bytes, written in hex separated by colons or hyphens: 3c:52:82:1a:2b:3c. The address is burned into the network card by the manufacturer and is supposed to be globally unique, although in fact the operating system can substitute any address it likes.
The first three bytes are the OUI (Organizationally Unique Identifier), the manufacturer code it buys from IEEE: 3c:52:82 is Hewlett Packard, 00:50:56 VMware, b8:27:eb Raspberry Pi. Wireshark uses the OUI to print the manufacturer's name next to the address, and you can use it to tell what kind of device has appeared on the network. The manufacturer assigns the remaining three bytes itself.
The two low-order bits of the first byte have a special meaning:
first address byte: x x x x x x U I
│ │
│ └── I/G: 0 — individual, 1 — group
└──── U/L: 0 — global (from the manufacturer), 1 — local
- I/G. If the lowest bit is 1, this is a group address, and the frame is meant not for one card but for everyone subscribed. A special case is the broadcast address
ff:ff:ff:ff:ff:ff: everyone receives it. IPv4 multicast group addresses start with01:00:5e, IPv6 ones with33:33; we will meet them in the parts on ARP and Neighbor Discovery. - U/L. If the second bit is 1, the address is locally administered rather than assigned by the manufacturer. In practice this means the second hex digit is
2,6,aore:02:42:ac:11:00:02is a Docker container,52:54:00:…a QEMU virtual machine, and phones and laptops using a random MAC address on Wi-Fi set this bit too.
Each interface of a machine has its own MAC address. To see it: ip link on Linux, ipconfig /all on Windows. On Linux you can change it with one command, ip link set eth0 address 02:00:00:00:00:01, and that is a legitimate trick, for example when the ISP has tied the service to the MAC address of your old router.
Where 64 bytes came from: shared medium and collisions
The original Ethernet was one coaxial cable for everyone. One station transmits, everyone hears, and two stations cannot transmit at the same time: the signals overlap and you get garbage. A way to share the cable is needed. It is called CSMA/CD — Carrier Sense Multiple Access with Collision Detection — and it works like this:
- Want to transmit — listen to the cable. Busy — wait until it is free, plus the interframe gap.
- Free — transmit and keep listening. If what you receive is not what you send, someone started at the same time as you: a collision.
- On detecting a collision, send another 32 bits of garbage (jam) so that everyone is sure to notice it, and stop.
- Wait a random time and repeat from step 1. The random time is
rslot times, whereris chosen from 0 to 2ⁿ − 1 andnis the attempt number (capped at 10). After 16 attempts the frame is dropped. This is called exponential backoff: the more collisions, the wider the spread and the lower the chance of colliding again.
Now the key point. A collision can only be noticed while you are still transmitting. If station A at one end of the cable finished sending a frame before the signal from station B at the other end reached it, A will decide everything went fine, when in fact the frame was corrupted. So a transmission must last at least as long as it takes the signal to reach the farthest end of the cable and come back.
For 10 Mbit/s Ethernet the maximum network diameter with repeaters was set at about 2500 metres, and the round trip of the signal including electronics delays was fitted into 512 bit times — that is the slot time, 51.2 microseconds. 512 bits is 64 bytes. A frame shorter than 64 bytes is padded with zeros not out of love for round numbers, but so that the sender is guaranteed to notice a collision.
At 100 Mbit/s a bit time is ten times shorter, and the network diameter had to be cut to ~200 metres. In half-duplex gigabit the frame would have to be stretched to 512 bytes, so half-duplex gigabit never caught on at all. Today collisions exist nowhere except on old equipment, and yet the number 64 lives on in every Ethernet standard, because changing the frame format costs more than carrying history around.
Hubs, which you can still find in old cabinets, are the same shared cable in a box: the signal from one port is repeated to all the others. All devices behind a hub form one collision domain.
The switch
The switch replaced the shared cable. Each port is a separate cable to a single device, and the switch forwards a frame only where it needs to go. To know where, it keeps a MAC address table: a mapping from an address to the port behind which that address was seen.
The algorithm fits into four rules:
- Learning. A frame arrived on a port — remember: the source MAC address lives behind this port. By default an entry lives 300 seconds after the last frame from that address, then it is removed (aging).
- Forwarding. Look up the destination MAC in the table. Found — send the frame to that port, and only to it.
- Flooding. Not found (unknown unicast), or the address is broadcast or multicast — send it to all ports except the one it came from.
- Filtering. If the destination address lives behind the same port the frame came from, do not forward it anywhere.
Let's follow an example. Three machines are connected to ports 1, 2 and 3, the table is empty, the switch has just been powered on.
┌─────────────────┐
A aa:…:01 ── 1 ────┤ ├──── 3 ── C cc:…:03
│ switch │
B bb:…:02 ── 2 ────┤ │
└─────────────────┘
| Step | Event | Table afterwards | Where the frame went |
|---|---|---|---|
| 1 | A sends a frame to C | aa:…:01 → port 1 |
C's address is unknown: flooded to ports 2 and 3 |
| 2 | C replies to A | aa:…:01 → 1, cc:…:03 → 3 |
A's address is known: only to port 1 |
| 3 | A sends to C again | unchanged | only to port 3 |
| 4 | B sends a broadcast frame | bb:…:02 → 2 added |
to all ports except 2 |
Machine B received someone else's frame at step 1, but no longer at step 3. This gives the switch its main properties: each port is its own collision domain, two conversations between different pairs of ports go on simultaneously, and other machines' unicast traffic does not reach a port. Broadcast, however, still reaches everyone: a switch does not split the broadcast domain — that is the job of routers and VLANs (part 17).
A couple of technical details that will come in handy:
- A switch that receives the whole frame, checks the FCS and only then forwards it is called store-and-forward; that is the norm. Cut-through starts forwarding after reading only the destination MAC: faster by fractions of a microsecond, but it passes corrupted frames on.
- The MAC address table is finite: from a few thousand entries in home devices to hundreds of thousands in rack switches. If it overflows (the old MAC flooding attack — bombarding the switch with frames from random source addresses), the switch stops learning and floods everything, that is, it turns into a hub.
- A switch changes nothing in the frame. Not the MAC addresses, not the EtherType, not the data. That is the main difference from a router, which strips the Ethernet header entirely.
On Linux there is a switch in the kernel — the bridge — and you can look at its table:
$ bridge fdb show br br0
3c:52:82:1a:2b:3c dev eth0 master br0
02:42:ac:11:00:02 dev veth1 master br0
33:33:00:00:00:01 dev eth0 self permanent
The permanent entries are the addresses of the bridge itself and the group addresses it must accept. On Cisco the same thing is show mac address-table, on MikroTik /interface bridge host print.
Duplex and auto-negotiation
Twisted pair, which replaced coax, solved the collision problem radically: the cable has separate wire pairs for transmitting and receiving. 10BASE-T and 100BASE-TX use two of them (pins 1–2 transmit, 3–6 receive), and if the other end is not a hub but a switch or another machine, both sides can transmit at the same time. This is full duplex: CSMA/CD is off, collisions are impossible by construction, and the full speed is available in each direction.
But 100-megabit cards also have half duplex, for compatibility with hubs, and both sides of a link have to agree. That is what auto-negotiation is for: when the link comes up, each side sends bursts of pulses (FLP, fast link pulses) encoding what it supports: 10 or 100 Mbit/s, half or full duplex. The best common option is chosen.
A classic mistake worth knowing even today is the duplex mismatch. An admin hard-set a switch port to 100/full, while the server was left on auto. The server receives no FLP, detects the speed of 100 Mbit/s from the shape of the signal (parallel detection), but has no way to learn the duplex, and in that case the standard says to pick half duplex. The link comes up, ping works, but under load the server sees collisions where there can be none, aborts transmissions, and the switch receives truncated frames and counts them as CRC errors. Symptoms: speed tens of times lower than expected, late collisions growing on one side and CRC errors/runts on the other. There is only one cure: either auto on both sides, or hard-set on both.
Gigabit put an end to this story: in 1000BASE-T auto-negotiation is mandatory, half duplex is practically never implemented, and ethtool shows a picture like this:
$ ethtool eth0
Settings for eth0:
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Speed: 1000Mb/s
Duplex: Full
Auto-negotiation: on
Link detected: yes
If you unexpectedly see Speed: 100Mb/s here on a gigabit card, it is most often the cable: gigabit needs all four pairs, and one bad contact in a connector is enough for auto-negotiation to fall back to 100 Mbit/s over two pairs.
Full duplex has its own throttling mechanism — flow control: a receiver that cannot keep up with its buffer sends a PAUSE frame to the group address 01:80:c2:00:00:01 with EtherType 0x8808 and the time for which the sender must go quiet. Home switches usually enable it by default; data centres more often turn it off: a pause stops the whole port rather than one connection, and one slow receiver slows everyone down.
Physics: what is in the cable
To find your way among the names. The name format is <speed>BASE-<medium>: BASE means baseband transmission without modulation onto a carrier, and the letters after the hyphen denote the medium.
| Standard | Medium | Pairs | Reach | How it is encoded |
|---|---|---|---|---|
| 10BASE-T | Cat3 twisted pair | 2 | 100 m | Manchester code, 1 bit = 1 transition |
| 100BASE-TX | Cat5 | 2 | 100 m | 4B/5B + MLT-3, three signal levels |
| 1000BASE-T | Cat5e | 4, both directions on each | 100 m | PAM-5, five levels, 250 Mbit/s per pair |
| 2.5G/5GBASE-T | Cat5e/Cat6 | 4 | 100 m | like 10GBASE-T at a lower frequency |
| 10GBASE-T | Cat6a | 4 | 100 m | PAM-16, LDPC coding |
| 1000BASE-SX / 10GBASE-SR | multimode fibre | — | 300–550 m | 850 nm laser |
| 1000BASE-LX / 10GBASE-LR | single-mode fibre | — | 10 km | 1310 nm laser |
Gigabit has an interesting trick: all four pairs transmit in both directions at once, and each side subtracts its own signal from the received one (echo cancellation). That is why gigabit needs the whole cable, while 100 Mbit/s was fine with half of it — which is exactly what the old "splitters for two computers on one cable" relied on.
Gigabit's second trick is auto MDI-X: a port detects by itself which of the neighbour's pairs are transmitting and swaps them if needed. Crossover cables for connecting two computers directly went away together with 100 Mbit/s.
Fibre in switches lives in pluggable modules: SFP (1 Gbit/s), SFP+ (10), SFP28 (25), QSFP28 (100). You can plug in a module for the right fibre and distance, and Ethernet does not change because of it. For home and office it is enough to know: twisted pair up to 100 metres; beyond that, either fibre or another switch.
Loops and STP
Connect two ports of the same switch with a cable. Or two switches with two cables "for redundancy". The very first broadcast frame goes out on both cables, comes back through the other one, goes out on both again, and so on forever: Ethernet has no TTL field, and a frame caught in a loop will circulate until someone pulls the cable. Within a second thousands of copies are spinning in the loop, the port LEDs stay solidly lit, and the MAC address table goes crazy: the same source address is seen first on one port, then on another (MAC flapping). The whole network is down. This is called a broadcast storm, and it is the simplest way to take down an office with a single cable.
Loop protection is the STP protocol (Spanning Tree Protocol, 802.1D) and its fast version RSTP (802.1w). Switches exchange BPDU control frames every two seconds (those very 802.3 frames with LLC), elect a root switch with the lowest identifier, compute the shortest path from each switch to the root and block all other ports between switches. The result is a tree without cycles; if a working link fails, a blocked port opens — STP takes 30–50 seconds for that, RSTP seconds. Redundant cables between switches are fine, but only when STP is enabled and configured; unmanaged home switches usually do not have it, and there a loop is just a loop.
Seeing it for yourself
Interface statistics, including error counters:
$ ip -s link show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/ether 3c:52:82:1a:2b:3c brd ff:ff:ff:ff:ff:ff
RX: bytes packets errors dropped missed mcast
9876543210 8765432 0 0 0 123456
TX: bytes packets errors dropped carrier collsns
1234567890 2345678 0 0 0 0
Non-zero receive errors, transmit collsns on a full-duplex link, or growing rx_crc_errors in ethtool -S eth0 are a reason to look at the cable, the connector and the duplex settings.
Frames with the Ethernet header:
$ sudo tcpdump -i eth0 -n -e -c 3
12:00:00.000001 3c:52:82:1a:2b:3c > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.1.1 tell 192.168.1.10, length 28
12:00:00.000250 d8:47:32:0a:0b:0c > 3c:52:82:1a:2b:3c, ethertype ARP (0x0806), length 60: Reply 192.168.1.1 is-at d8:47:32:0a:0b:0c, length 46
12:00:00.000300 3c:52:82:1a:2b:3c > d8:47:32:0a:0b:0c, ethertype IPv4 (0x0800), length 74: 192.168.1.10.51234 > 203.0.113.10.443: Flags [S], ...
Everything this article covered is here: the broadcast destination address in the first frame, the EtherType saying "ARP inside", and the length of 60 in the second frame — that is padding up to the minimum 64 bytes (tcpdump does not count the FCS, so it shows 60). The first frame is shown as 42 bytes long because it was captured on the sender before the network card appended the zeros.
In Wireshark, expand the Ethernet II line of any frame: there you will find the addresses with manufacturer names from the OUI, the I/G and U/L bits right in the address breakdown, and the EtherType decoded.
Key takeaways
- The frame: 14 bytes of header (two MAC addresses and the EtherType), 46–1500 bytes of data, 4 bytes of CRC. Minimum 64 bytes, maximum 1518. Data larger than 1500 — the MTU — does not fit in a frame.
- The MAC address: 3 bytes of manufacturer and 3 bytes of serial number; the lowest bit of the first byte marks a group address, the next one a locally administered address. Everyone receives
ff:ff:ff:ff:ff:ff. - 64 bytes is a legacy of the shared cable and CSMA/CD: a frame must last longer than the signal's round trip so that a collision can be noticed.
- A switch learns its table from source addresses, forwards by destination addresses, floods unknown and broadcast frames, and does not change a single byte of the frame.
- Full duplex eliminated collisions; a duplex mismatch is when one side is on auto and the other is hard-set.
- A loop without STP takes the network down: a frame has no TTL.
Next — Part 3: Wi-Fi: why radio cannot work like Ethernet, CSMA/CA and acknowledgements, channels and interference, how a client connects to an access point and what happens during the WPA2 four-way handshake.
Only registered users can leave a comment.
Sign in or register to leave a comment.