Networking for beginners. Part 10: TCP
All parts of the series “Networking for beginners”In the ninth part UDP added nothing to IP but ports: a packet either arrives or it does not. For most applications that is not enough. A file must arrive whole, the bytes in the order they were sent, and the sender must not flood the receiver and the network faster than they can cope. All of this is done by TCP, the Transmission Control Protocol, RFC 793 (now RFC 9293), and it does so on top of the same unreliable IP that loses, duplicates and reorders packets.
This will be a long conversation, because TCP is not a format but an algorithm: a set of rules by which two parties agree, number bytes, acknowledge, retransmit and slow down. We will follow the life of a connection: the header, setup, transfer, reliability, speed, teardown — and at the end look at all of it in a single capture.
What TCP promises
An application opens a connection and gets a byte stream in both directions: whatever is written on one side is read on the other, whole, without losses, in the same order. How this is achieved, the application does not need to know. The price:
- before data transfer there is a handshake, an extra round trip to the server and back;
- every segment gets an acknowledgement in the opposite direction;
- message boundaries are not preserved: write two messages and you may read one, one and a half, or three — a stream is a stream;
- losing one packet delays everything that arrived after it: the receiver will not hand byte 3000 to the application until it has received byte 2000;
- exactly two parties, no broadcast.
The header
Twenty mandatory bytes and up to forty bytes of options:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
┌───────────────────────────────┬───────────────────────────────┐
│ Source Port │ Destination Port │
├───────────────────────────────┴───────────────────────────────┤
│ Sequence Number │
├───────────────────────────────────────────────────────────────┤
│ Acknowledgment Number │
├───────┬───────┬─┬─┬─┬─┬─┬─┬─┬─┼───────────────────────────────┤
│ Data │ Rsrvd │C│E│U│A│P│R│S│F│ Window │
│ Offset│ │W│C│R│C│S│S│Y│I│ │
│ │ │R│E│G│K│H│T│N│N│ │
├───────┴───────┴─┴─┴─┴─┴─┴─┴─┴─┼───────────────────────────────┤
│ Checksum │ Urgent Pointer │
├───────────────────────────────┴───────────────────────────────┤
│ Options (0–40 bytes) │
└───────────────────────────────────────────────────────────────┘
- Ports — as in UDP.
- Sequence Number — the number of the first data byte of this segment in the stream. In TCP every byte is numbered, not every segment. This is the heart of the protocol.
- Acknowledgment Number — "the next byte I am expecting": everything before it has been received. Valid when the ACK flag is set.
- Data Offset — the header length in 32-bit words, like IHL in IP; 5 without options.
- Flags: SYN — setup, synchronising the numbers; ACK — the acknowledgement field is valid; FIN — the sender has finished sending; RST — reset, "there is no such connection"; PSH — hand the data to the application without waiting for the buffer to fill; URG — urgent data, practically unused; ECE, CWR — the ECN congestion signals from the sixth part.
- Window — how many bytes the sender of this segment is ready to accept. This is flow control; see the section below.
- Checksum — over the header, data and a pseudo-header with the IP addresses, as in UDP, but mandatory.
- Urgent Pointer — for URG.
- Options — negotiated at setup and in effect for the whole connection: MSS (maximum segment size), Window Scale (the window multiplier), SACK Permitted and SACK (selective acknowledgements), Timestamps. We will meet each of them below.
It helps to keep the size in mind: 20 bytes of IP, 20 bytes of TCP, usually another 12 bytes of the Timestamps option in every segment — 52 bytes per segment before the data.
Setup: three steps
A connection begins with a handshake in which the parties exchange initial sequence numbers and options:
client server (LISTEN)
│ │
├── SYN, seq = x ──────────────────────────────────────────►│ SYN-SENT → …
│ MSS 1460, SACK ok, Timestamps, WScale 7 │ … SYN-RECEIVED
│ │
│◄── SYN + ACK, seq = y, ack = x + 1 ───────────────────────┤
│ MSS 1460, SACK ok, Timestamps, WScale 7 │
│ │
├── ACK, seq = x + 1, ack = y + 1 ─────────────────────────►│ ESTABLISHED
│ ESTABLISHED │
- The client sends a segment with the SYN flag and its initial number x. The number is random: so that stale segments from a previous connection between the same ports are not mistaken for new ones, and so that an outsider cannot guess it. The options say what the client supports.
- The server replies with SYN + ACK: its own initial number y and the acknowledgement
ack = x + 1. The SYN flag takes up one sequence number even though it carries no data, hence the plus one. - The client acknowledges:
ack = y + 1. The connection is established on both sides, and this third segment can already carry data.
Why three and not two: each side must make sure its SYN arrived (get an ACK for it), and both sides send. Two segments would only be enough for one direction.
What happens if the port is closed: the server replies with RST, and the client instantly gets "Connection refused". If the port is silently blocked by a firewall, there is no reply, and the client retransmits the SYN: after 1, 2, 4, 8, 16 and 32 seconds — six attempts in Linux, about two minutes until "Connection timed out". The difference between "refused immediately" and "timeout after two minutes" is the first diagnostic clue: the former means the host is alive and the port is closed, the latter that packets are not getting through or replies are being filtered.
On the server, a listening socket keeps a queue of half-open connections (SYN received, waiting for the third segment). It can be overflowed by a stream of SYNs with spoofed addresses — a SYN flood, the oldest attack on TCP. The defence is SYN cookies: the server keeps no state but encodes it into its initial number y, and reconstructs it from ack = y + 1 when and if the third segment arrives.
Data transfer
After the handshake each side knows the number of the next byte it expects from the other. The sender cuts the stream into segments of up to the MSS announced in the SYN: usually the MTU minus 40 bytes of headers, 1460 for Ethernet, 1452 for PPPoE. Each segment's Sequence Number holds the number of its first byte; the receiver replies with a segment carrying an ACK with the number of the next expected byte.
A B
├── seq 1, 1448 bytes ─────────────────────────────►│ received 1…1448
├── seq 1449, 1448 bytes ──────────────────────────►│ received 1449…2896
│◄── ack 2897 ──────────────────────────────────────┤ "expecting 2897", both acknowledged
├── seq 2897, 1448 bytes ──────────────────────────►│
Acknowledgements are cumulative: ack 2897 means "everything up to and including 2896 has been received", and one ACK can acknowledge several segments. The receiver does not have to answer every one: the delayed ACK rule is to acknowledge every second full segment, or on a timer after 40–200 ms if the second does not arrive. This halves the number of packets, but it has a side effect, described below.
A reply can piggyback: if B has data of its own for A, it puts the ACK flag and the acknowledgement number into the same segment that carries its data. That is why in real traffic almost every segment has the ACK flag, and "pure" acknowledgements without data are rare.
The PSH flag is a hint to the receiver to hand what it has accumulated to the application now, without waiting for the buffer to fill; it is set on the last segment of each application write. In captures, P. on a segment means "a write() ended here".
A stream, not messages
An application wrote "HELLO" into the socket and then "WORLD". TCP may send them as one segment "HELLOWORLD", as two, or as "HEL" and "LOWORLD" if the first segment was cut short by the window. The receiver will read whatever has arrived by the time of read(). A program that assumes "one read() — one message" works on the local machine and breaks on the internet when segments arrive differently. All protocols on top of TCP mark message boundaries themselves: with a length at the start (as in TLS, the encryption protocol, where each record begins with a type and a length), with a delimiter (the blank line between headers and body in HTTP, the protocol a browser uses to request pages), or both.
Reliability: retransmissions
IP loses segments. TCP finds out about it in two ways.
By timer. Having sent a segment, the sender starts an RTO (Retransmission Timeout) timer. If no acknowledgement arrives, it sends the segment again. The RTO is computed from the measured round-trip time RTT: a smoothed average plus four smoothed deviations. For the first segment, when there are no measurements yet, the RTO is one second; in Linux it cannot be less than 200 ms and doubles after each retransmission (exponential backoff, as in Ethernet). The Timestamps option puts the send time into every segment, the reply echoes it back, and RTT is measured for every segment rather than once per window.
By duplicate acknowledgements. A receiver that gets a segment out of order (2897…4344 arrived, but it was expecting 1449) still replies, but with the previous ack 1449 — it is still waiting for 1449. A sender that sees three identical ACKs in a row does not wait for the timer: segment 1449 is lost and must be retransmitted. This is fast retransmit, and it kicks in after one RTT instead of the RTO.
Without extra information, after losing one segment the sender does not know whether the following ones arrived, and may retransmit the whole window. The SACK (Selective Acknowledgment) option solves this: in a duplicate ACK the receiver lists the blocks it has received beyond what is acknowledged — "expecting 1449, but I have 2897–5792". The sender retransmits exactly the hole. SACK is enabled everywhere TCP is younger than the year 2000.
Flow control: the receive window
The receiver's buffer is finite: the application may read from the socket more slowly than data arrives. So that the sender does not send too much, the receiver announces a window in every segment — how many bytes are free in the buffer. The sender keeps no more than the window "in flight" (sent but not acknowledged). As the application reads from the buffer, the window grows, and the receiver reports this in the next ACK.
A window of 0 means "stop". The sender stops and every few seconds sends a window probe, a one-byte segment, so that the receiver replies with its current window. In Wireshark such pauses are labelled TCP ZeroWindow and TCP Window Full, and they mean that the receiving application is what is slow, not the network.
The Window field is 16 bits, a maximum of 65,535 bytes. In 1981 that was enough, but consider: at 100 Mbit/s and an RTT of 50 ms, keeping the line continuously busy requires 100,000,000 / 8 × 0.05 = 625,000 bytes in flight. With a 64 KB window the sender would sit idle 90% of the time waiting for ACKs. This is the product of bandwidth and delay, the bandwidth-delay product, and it is why the Window Scale option appeared: a multiplier of 2ⁿ, announced in the SYN, by which the Window field is multiplied for the whole connection. wscale 7 is 128, a window of up to 8 MB; the maximum is n = 14, a window of up to a gigabyte. After the handshake, tcpdump shows the field as is: win 502 with wscale 7 means 64,256 bytes. Modern stacks size the buffer on the fly to match the measured BDP.
Congestion control: the sender's window
The receive window protects the receiver. But who protects the network — a router in the middle with a queue that overflows if all senders transmit at full speed? The router announces nothing; it simply drops the excess. The sender must guess on its own how much the network will carry, and the only signal it has is loss.
For this the sender has a second window, cwnd (the congestion window), and the real limit on data in flight is the smaller of the two: min(rwnd, cwnd). cwnd is managed by an algorithm, and its classic version (RFC 5681) goes like this:
cwnd
▲
│ loss (3 dup ACKs)
│ ╱│ ╱│
│ ╱ │ ╱ │
│ ╱ │ ssthresh ╱ │
│ ╱ ├──────────────── ╱ │ ───── …
│ ╱ │ ╱ │
│ ╱ │ ╱ │
│ ╱ │ ╱ │
│ ╱ slow │ ╱ │
│ ╱ start │ ╱ avoidance │
│ ╱ (×2 per RTT) │ ╱ (+1 per RTT) │
├────────────────────┴─────┴──────────────────┴───────────► time
- Slow start. cwnd starts at 10 segments (about 14 KB) and doubles every RTT: every ACK adds one segment. The name is historical: it is not slow, it is exponential, but compared with the "send everything at once" of the 1980s it is slow.
- Congestion avoidance. On reaching the ssthresh threshold, growth becomes linear: one segment more per RTT. We cautiously probe whether there is still room.
- Loss detected by three duplicate ACKs — the network is overloaded but working. ssthresh = cwnd / 2, cwnd is halved too, and we continue linearly from the half. This is fast recovery.
- Loss detected by timer — the network is silent, things are bad. cwnd = 1 segment, slow start again up to ssthresh.
The result is a sawtooth: the speed grows until the first loss, drops by half, grows again. Three consequences follow that you can see in real life:
- A short connection never reaches full speed. A 100 KB page is done while cwnd is still in slow start. That is why browsers keep connections open and reuse them.
- One loss per thousand packets costs not 0.1% of the speed, but a multiple. Each loss halves cwnd, and on long links with a large RTT recovery takes seconds.
- TCP cannot tell losses due to interference from losses due to congestion. Wi-Fi from the third part retransmits frames itself so that TCP does not see losses and does not cut its speed; on satellite and mobile links intermediate devices do the same.
The classic algorithm was replaced long ago. Linux defaults to CUBIC: after a loss the window grows along a cubic curve — quickly back towards the previous value, cautiously around it, and then quickly again if there are no losses; on fast links it recovers much faster than linear growth. BBR from Google goes further and does not wait for losses at all: it measures the available bandwidth and the minimum RTT and keeps exactly their product in flight, without filling router queues. With BBR losses stop being the signal, and it copes better with lossy links. The algorithm is chosen by the sender, sysctl net.ipv4.tcp_congestion_control, and the two sides do not have to use the same one.
A separate note on bufferbloat: routers with large buffers do not drop packets but accumulate them, and TCP, seeing no losses, speeds up until the queue grows to seconds of delay. A file download "kills" a video call not because of bandwidth but because of the queue. The fix is on the router: active queue management (fq_codel), which drops packets or marks them with ECN before the queue grows too long.
Teardown: four steps
A connection is closed in each direction separately. The side that has nothing more to say sends FIN; the other acknowledges it and may keep sending until it is done itself:
client server
ESTABLISHED ESTABLISHED
├── FIN, seq = m ───────────────────────────────────────►│
│ FIN-WAIT-1 │ CLOSE-WAIT
│◄── ACK, ack = m + 1 ───────────────────────────────────┤ (the server's application
│ FIN-WAIT-2 │ learns: read() returned 0)
│ │
│ … the server may still send … │
│ │
│◄── FIN, seq = n ───────────────────────────────────────┤ LAST-ACK
├── ACK, ack = n + 1 ───────────────────────────────────►│ CLOSED
│ TIME-WAIT, 60 seconds, then CLOSED │
FIN, like SYN, takes up one number. Often the second and third segments are combined: the server replies with FIN + ACK at once, and you get three segments. The CLOSE-WAIT state on the server means "the other side has closed, I am waiting for my application to call close()"; many connections in CLOSE-WAIT indicate a leak in an application that does not close its sockets.
TIME-WAIT — the side that closed first keeps the connection in memory for another minute after the last ACK (exactly 60 seconds in Linux; by the standard, twice the maximum segment lifetime). There are two reasons: if the last ACK was lost, the server will retransmit its FIN, and someone needs to answer; and so that late segments of the old connection do not end up in a new one with the same ports. A side effect: a client opening thousands of short connections per second to one server exhausts its ephemeral ports, because each one is occupied for a minute. The cure is reusing connections, not shortening the timer.
RST is an emergency close: "there is no connection, forget it". The stack sends it in response to a segment for a non-existent connection or a closed port, and an application sends it on close() with unread data. RST is not acknowledged and does not create TIME-WAIT. Connection reset by peer in the logs is this.
Small segments: Nagle and delayed ACK
An application writes to the socket one byte at a time — for example, ssh, a remote terminal, sends each keystroke separately. Each byte in a segment with 52 bytes of headers is 2% payload. Nagle's algorithm says: while there is an unacknowledged segment, accumulate small writes and send them as one segment when the ACK arrives. For ssh this is tolerable, but for a "request — short response" protocol it clashes with delayed ACK: the client sent a request in two write() calls, the second piece waits for an ACK on the first because of Nagle, and the server does not send the ACK because it is waiting for the second segment because of delayed ACK. Both wait until the delayed ACK timer expires — 40 ms in Linux. A request that should take a millisecond takes forty. It is a well-known symptom, and the cure is TCP_NODELAY on the socket, which almost all modern libraries set by default, or one write instead of two.
Keepalive: by default TCP is silent while the applications are silent, and a connection without traffic can live for days, even if the other side was switched off long ago. The keepalive option makes the stack send empty probes after two hours of silence (in Linux tcp_keepalive_time = 7200), nine of them 75 seconds apart, and close the connection if there is no reply. Applications that need to learn about a broken connection sooner lower the timer or send their own pings.
One whole connection
A capture of an HTTPS request from start to finish, tcpdump -i eth0 -n 'tcp port 443'. In SYN and SYN+ACK tcpdump shows the real initial numbers, and after the handshake numbers relative to the initial ones, since otherwise it would be impossible to read.
The full exchange (addresses shortened: C = 192.168.1.10.51234, S = 203.0.113.10.443):
C > S: Flags [S], seq 1000000000, win 64240, options [mss 1460,sackOK,TS val 1 ecr 0,nop,wscale 7], length 0
S > C: Flags [S.], seq 2000000000, ack 1000000001, win 65160, options [mss 1460,sackOK,TS val 9 ecr 1,nop,wscale 7], length 0
C > S: Flags [.], ack 1, win 502, length 0
C > S: Flags [P.], seq 1:518, ack 1, win 502, length 517
S > C: Flags [.], ack 518, win 507, length 0
S > C: Flags [.], seq 1:1449, ack 518, win 507, length 1448
S > C: Flags [P.], seq 1449:2897, ack 518, win 507, length 1448
C > S: Flags [.], ack 2897, win 501, length 0
C > S: Flags [P.], seq 518:644, ack 2897, win 501, length 126
S > C: Flags [P.], seq 2897:3151, ack 644, win 507, length 254
C > S: Flags [.], ack 3151, win 501, length 0
… data …
C > S: Flags [F.], seq 644, ack 3151, win 501, length 0
S > C: Flags [F.], seq 3151, ack 645, win 507, length 0
C > S: Flags [.], ack 3152, win 501, length 0
Reading it:
1–3. The handshake. In the SYN: MSS 1460, SACK permitted, timestamps, a window with a multiplier of 128. win 64240 in the SYN is without the multiplier, which is not yet in effect. The third segment: ack 1 is a relative number, "expecting the first byte".
The client sends 517 bytes (
seq 1:518— bytes 1 through 517) with PSH: this is the ClientHello, the start of TLS.win 502× 128 = a 64,256-byte window.The server acknowledges:
ack 518.
6–7. The server sends two segments of 1448 bytes. Not 1460, because 12 bytes in each segment are taken by the Timestamps option: the MSS is counted including them. The first has no PSH flag, the second does: this is a single 2896-byte server write cut into two segments.
- The client acknowledges both with one ACK:
ack 2897— delayed ACK in action.
9–11. Another exchange: the client sends 126 bytes, the server 254, an acknowledgement.
12–14. The client is done: FIN with seq 644 (the one after the last byte sent). The server replies with FIN + ACK in one segment, ack 645 — the FIN took number 644. The client acknowledges with ack 3152 and goes into TIME-WAIT.
In Wireshark the same thing is even clearer: Follow TCP Stream assembles the stream into text, Statistics → TCP Stream Graphs → Time/Sequence plots the growth of sequence numbers over time, where steps are waits for ACKs and plateaus are losses, and the filters tcp.analysis.retransmission, tcp.analysis.duplicate_ack, tcp.analysis.zero_window immediately show what went wrong.
Seeing it for yourself
All connections with their states:
$ ss -tan
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
ESTAB 0 0 192.168.1.10:51234 203.0.113.10:443
TIME-WAIT 0 0 192.168.1.10:51230 203.0.113.10:443
Recv-Q on an established connection is bytes that have arrived but the application has not read yet; Send-Q is sent but not acknowledged. A growing Recv-Q means your application is slow; a growing Send-Q means the network or the remote side is slow.
The internals of one connection — everything this article covered — in one line:
$ ss -tin dst 203.0.113.10
cubic wscale:7,7 rto:216 rtt:12.3/1.5 mss:1448 cwnd:10 ssthresh:7 bytes_sent:644 bytes_acked:644
segs_out:8 segs_in:9 send 9.4Mbps lastsnd:120 lastrcv:118 retrans:0/0 rcv_space:64256 minrtt:11.9
The CUBIC algorithm, the window multipliers, an RTO of 216 ms, an RTT of 12.3 ms, cwnd of 10 segments, 0 retransmissions. If retrans is growing, the network is losing packets; if cwnd is small with a large RTT, that is why the speed is low.
System-wide retransmission counters: nstat -az | grep -i retrans. And checking a port without an application: nc -vz 203.0.113.10 443 — "succeeded" means the handshake went through, "refused" means RST, a long pause means the SYN is going nowhere.
Key takeaways
- TCP numbers every byte; Sequence is the number of the segment's first byte, ACK is the next expected one, acknowledgements are cumulative. SYN and FIN each take up a number.
- Setup is SYN, SYN+ACK, ACK with an exchange of initial numbers and options; RST means "port closed" immediately, silence means two minutes of SYN retransmissions.
- Reliability: retransmission on an RTO timer derived from the measured RTT and fast retransmit on three duplicate ACKs; SACK says which blocks arrived.
- The receive window protects the receiver (Window Scale makes it large enough for the bandwidth-delay product); the congestion window protects the network: slow start, linear growth, halving on loss. A loss costs not a percentage but a multiple.
- Teardown is a FIN in each direction; TIME-WAIT lasts a minute on the side that closed first; CLOSE-WAIT in large numbers is a socket leak.
- A stream keeps no message boundaries; Nagle plus delayed ACK gives a 40 ms delay; keepalive by default kicks in after two hours.
This concludes the transport layer. Next come the service protocols, and the first of them is Part 11: DHCPv4: how a machine without an address gets an address, mask, gateway and DNS in four broadcast packets, what a lease is and when it is renewed, how a request passes through a router into another network, and what happens when there are two DHCP servers on the network.
Only registered users can leave a comment.
Sign in or register to leave a comment.