On 26 May 2025, at 4:41 PM, Christof Meerwald <cmeerw%cmeerw.org@localhost> wrote:
Noticed a lot of retransmitted data packets showing up in "netstat
-ss", and after some experimentation it appears to be some issue with
path MTU discovery on IPv6 (or the lack of it).
Test VPS is running current: NetBSD 10.99.14 (GENERIC) #0: Sun May 25 16:59:07 UTC 2025 mkrepro%mkrepro.NetBSD.org@localhost:/usr/src/sys/arch/amd64/compile/GENERIC amd64
but I have also seen it on NetBSD 10.1.
The NetBSD VPS is using vioif0 with MTU 1500, but I am actually
connecting to the IPv6 address via an HE tunnel (so the MTU on the
path is lower).
On the NetBSD VPS I am running:
dd if=/dev/zero bs=16k count=1 | nc -6 -l 12345
And on a Linux machine (also with MTU 1500) I am then using
nc -6 green.cmeerw.net 12345 >/dev/null
I would expect that after the first few packets with a too big MTU, it
will then use the lower MTU from an ICMP6 packet too big. However, it
appears that NetBSD tries to retransmit the first few packets at full
speed for a few seconds until finally falling back to a lower MTU.
tcpdump shows this:
15:48:31.482763 IP6 2001:470:597a:bbb:419:1719:4348:1725.43226 > 2a03:d9c2:100:49::b5d.12345: Flags [S], seq 2776544030, win 64800, options [mss 1440,sackOK,TS val 1231408504 ecr 0,nop,wscale 7], length 0
15:48:31.489363 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [S.], seq 3901012170, ack 2776544031, win 32768, options [mss 1440,nop,wscale 3,sackOK,TS val 1 ecr 1231408504], length 0
15:48:31.535870 IP6 2001:470:597a:bbb:419:1719:4348:1725.43226 > 2a03:d9c2:100:49::b5d.12345: Flags [.], ack 1, win 507, options [nop,nop,TS val 1231408569 ecr 1], length 0
15:48:31.536080 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1429, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.536116 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1429:2857, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.536137 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 2857:4285, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.544930 IP6 2001:470:0:221::2 > 2a03:d9c2:100:49::b5d: ICMP6, packet too big, mtu 1460, length 1240
15:48:31.545013 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1429, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.545040 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1429:2857, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.545058 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 2857:4285, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.545430 IP6 2001:470:0:221::2 > 2a03:d9c2:100:49::b5d: ICMP6, packet too big, mtu 1460, length 1240
15:48:31.545465 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1429, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.545484 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1429:2857, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.545502 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 2857:4285, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.545981 IP6 2001:470:0:221::2 > 2a03:d9c2:100:49::b5d: ICMP6, packet too big, mtu 1460, length 1240
15:48:31.546016 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1429, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.546035 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1429:2857, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.546052 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 2857:4285, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.553659 IP6 2001:470:0:221::2 > 2a03:d9c2:100:49::b5d: ICMP6, packet too big, mtu 1460, length 1240
15:48:31.553729 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1429, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.553750 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1429:2857, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.553768 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 2857:4285, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.554185 IP6 2001:470:0:221::2 > 2a03:d9c2:100:49::b5d: ICMP6, packet too big, mtu 1460, length 1240
15:48:31.554219 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1429, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.554238 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1429:2857, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.554255 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 2857:4285, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.554747 IP6 2001:470:0:221::2 > 2a03:d9c2:100:49::b5d: ICMP6, packet too big, mtu 1460, length 1240
15:48:31.554780 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1429, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.554799 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1429:2857, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.554816 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 2857:4285, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.555303 IP6 2001:470:0:221::2 > 2a03:d9c2:100:49::b5d: ICMP6, packet too big, mtu 1460, length 1240
15:48:31.555337 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1429, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.555355 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1429:2857, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.555373 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 2857:4285, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.555823 IP6 2001:470:0:221::2 > 2a03:d9c2:100:49::b5d: ICMP6, packet too big, mtu 1460, length 1240
15:48:31.555856 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1429, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.555885 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1429:2857, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.555903 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 2857:4285, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.556376 IP6 2001:470:0:221::2 > 2a03:d9c2:100:49::b5d: ICMP6, packet too big, mtu 1460, length 1240
15:48:31.556410 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1429, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.556428 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1429:2857, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.556446 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 2857:4285, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.556905 IP6 2001:470:0:221::2 > 2a03:d9c2:100:49::b5d: ICMP6, packet too big, mtu 1460, length 1240
15:48:31.556939 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1429, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.556958 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1429:2857, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.556975 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 2857:4285, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.557516 IP6 2001:470:0:221::2 > 2a03:d9c2:100:49::b5d: ICMP6, packet too big, mtu 1460, length 1240
15:48:31.557549 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1429, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.557567 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1429:2857, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
15:48:31.557584 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 2857:4285, ack 1, win 4140, options [nop,nop,TS val 1 ecr 1231408569], length 1428
this continues for a few seconds until:
15:48:38.629056 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1429, ack 1, win 4140, options [nop,nop,TS val 15 ecr 1231408569], length 1428
15:48:38.629130 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1209, ack 1, win 4140, options [nop,nop,TS val 15 ecr 1231408569], length 1208
15:48:38.629572 IP6 2001:470:0:221::2 > 2a03:d9c2:100:49::b5d: ICMP6, packet too big, mtu 1460, length 1240
15:48:38.629607 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1209, ack 1, win 4140, options [nop,nop,TS val 15 ecr 1231408569], length 1208
15:48:38.630118 IP6 2001:470:0:221::2 > 2a03:d9c2:100:49::b5d: ICMP6, packet too big, mtu 1460, length 1240
15:48:38.630152 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1209, ack 1, win 4140, options [nop,nop,TS val 15 ecr 1231408569], length 1208
15:48:38.630739 IP6 2001:470:0:221::2 > 2a03:d9c2:100:49::b5d: ICMP6, packet too big, mtu 1460, length 1240
15:48:38.630772 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1209, ack 1, win 4140, options [nop,nop,TS val 15 ecr 1231408569], length 1208
and even that continues for half a second:
15:48:39.133856 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1209, ack 1, win 4140, options [nop,nop,TS val 16 ecr 1231408569], length 1208
15:48:39.134355 IP6 2001:470:0:221::2 > 2a03:d9c2:100:49::b5d: ICMP6, packet too big, mtu 1460, length 1240
15:48:39.134390 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1:1209, ack 1, win 4140, options [nop,nop,TS val 16 ecr 1231408569], length 1208
15:48:39.511721 IP6 2001:470:597a:bbb:419:1719:4348:1725.43226 > 2a03:d9c2:100:49::b5d.12345: Flags [.], ack 1209, win 498, options [nop,nop,TS val 1231416205 ecr 15], length 0
15:48:39.511803 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 1209:2417, ack 1, win 4140, options [nop,nop,TS val 16 ecr 1231416205], length 1208
15:48:39.511830 IP6 2a03:d9c2:100:49::b5d.12345 > 2001:470:597a:bbb:419:1719:4348:1725.43226: Flags [.], seq 2417:3625, ack 1, win 4140, options [nop,nop,TS val 16 ecr 1231416205], length 1208
So we end up with something like:
17943 data packets (25439300 bytes) retransmitted
just from those 16 kB.
Christof
--
https://cmeerw.org sip:cmeerw at cmeerw.org
mailto:cmeerw at cmeerw.org xmpp:cmeerw at cmeerw.org