Re: kern/57049: large TCP transfers NetBSD-Xen-Guest -> NetBSD-Xen-DOM0 abort with EHOSTDOWN

[Frank Kardel <kardel%netbsd.org@localhost>]

More data points:

The arp state loops through following state sequence:

host (ip-address) at MAC on xennet0 expired U
host (ip-address) at MAC on xennet0 expired U
host (ip-address) at MAC on xennet0 30s R
host (ip-address) at MAC on xennet0 28s R
host (ip-address) at MAC on xennet0 27s R
host (ip-address) at MAC on xennet0 26s R
host (ip-address) at MAC on xennet0 25s R
host (ip-address) at MAC on xennet0 23s R
host (ip-address) at MAC on xennet0 22s R
host (ip-address) at MAC on xennet0 21s R
host (ip-address) at MAC on xennet0 20s R
host (ip-address) at MAC on xennet0 18s R
host (ip-address) at MAC on xennet0 17s R
host (ip-address) at MAC on xennet0 16s R
host (ip-address) at MAC on xennet0 15s R
host (ip-address) at MAC on xennet0 14s R
host (ip-address) at MAC on xennet0 13s R
host (ip-address) at MAC on xennet0 11s R
host (ip-address) at MAC on xennet0 10s R
host (ip-address) at MAC on xennet0 9s R
host (ip-address) at MAC on xennet0 7s R
host (ip-address) at MAC on xennet0 5s R
host (ip-address) at MAC on xennet0 4s R
host (ip-address) at MAC on xennet0 3s R
host (ip-address) at MAC on xennet0 2s R
host (ip-address) at MAC on xennet0 1s R
host (ip-address) at MAC on xennet0 5s D
host (ip-address) at MAC on xennet0 4s D
host (ip-address) at MAC on xennet0 3s D
host (ip-address) at MAC on xennet0 1s D
host (ip-address) at MAC on xennet0 1s P
host (ip-address) at MAC on xennet0 1s P
host (ip-address) at MAC on xennet0 1s P
host (ip-address) at MAC on xennet0 expired U
host (ip-address) at MAC on xennet0 expired U
host (ip-address) at MAC on xennet0 expired U
host (ip-address) at MAC on xennet0 30s R
host (ip-address) at MAC on xennet0 29s R
host (ip-address) at MAC on xennet0 28s R
host (ip-address) at MAC on xennet0 27s R

tcpdump on xennet0:
16:13:26.259366 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 
host (MAC) tell guest-host, length 28
16:13:27.259380 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 
host (MAC) tell guest-host, length 28
16:13:28.259414 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 
host (MAC) tell guest-host, length 28
16:13:29.259411 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 
host (MAC) tell guest-host, length 28
16:13:30.259434 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 
host (MAC) tell guest-host, length 28
16:13:31.259465 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 
host (MAC) tell guest-host, length 28
16:13:32.259516 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 
host tell guest-host, length 28
16:13:32.259606 ARP, Ethernet (len 6), IPv4 (len 4), Reply host is-at 
MAC, length 28
16:13:32.259608 ARP, Ethernet (len 6), IPv4 (len 4), Reply host is-at 
MAC, length 28
16:13:32.259609 ARP, Ethernet (len 6), IPv4 (len 4), Reply host is-at 
MAC, length 28
16:13:32.259609 ARP, Ethernet (len 6), IPv4 (len 4), Reply host is-at 
MAC, length 28
16:13:32.259610 ARP, Ethernet (len 6), IPv4 (len 4), Reply host is-at 
MAC, length 28
16:13:32.259610 ARP, Ethernet (len 6), IPv4 (len 4), Reply host is-at 
MAC, length 28

tcpdump on guest interfac in DOM0 connected to bridge0:
16:26:03.745507 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 
host (MAC) tell guest-host, length 28
16:26:04.745537 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 
host (MAC) tell guest-host, length 28
16:26:05.745559 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 
host (MAC) tell guest-host, length 28
16:26:06.745599 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 
host (MAC) tell guest-host, length 28
16:26:07.745636 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 
host (MAC) tell guest-host, length 28
16:26:08.745725 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 
host tell guest-host, length 28
16:26:08.745793 ARP, Ethernet (len 6), IPv4 (len 4), Reply host is-at 
MAC, length 28
16:26:08.745797 ARP, Ethernet (len 6), IPv4 (len 4), Reply host is-at 
MAC, length 28
16:26:08.745800 ARP, Ethernet (len 6), IPv4 (len 4), Reply host is-at 
MAC, length 28
16:26:08.745803 ARP, Ethernet (len 6), IPv4 (len 4), Reply host is-at 
MAC, length 28
16:26:08.745812 ARP, Ethernet (len 6), IPv4 (len 4), Reply host is-at 
MAC, length 28
16:26:08.745815 ARP, Ethernet (len 6), IPv4 (len 4), Reply host is-at 
MAC, length 28

So we see that Replys are delayed (possibly queued) for 6 seconds. of 
the 7 requests 6 are being replied to.
Why do we see this delay/queueing on a busy bridge interface?
arp.c has bridge conditional code.