NetBSD 9.0 vs 8.0 network slow

[Dima Veselov <kab00m%lich.phys.spbu.ru@localhost>]

Greetings,

having two similar servers I tried to update one of them
to be first working on NetBSD 9. It seem to work fine, but
sometimes it freeze, that usually happens on network
services. This short freezes are actually mess up, so
I had to remove the server from production use.

Hardware is identical (maybe slightly different firmware)
and they both were proved to work very fast on NetBSD
8-STABLE.

Would be nice if someone can point me out what and
where I may debug to find the cause.

N9 ethernet:
[     1,008595] bge1 at pci2 dev 0 function 1: Broadcom BCM5720 Gigabit 
Ethernet
[     1,008595] bge1: APE firmware NCSI 1.5.12.0
[     1,008595] bge1: interrupting at msix2 vec 0
[     1,008595] bge1: HW config 002b1194, 00006014, 4000aa08, 00000000 
00000000
[     1,008595] bge1: ASIC BCM5720 A0 (0x5720000), Ethernet address 
20:47:47:87:3f:22
[     1,008595] bge1: setting short Tx thresholds
[     1,008595] brgphy1 at bge1 phy 2: BCM5720C 1000BASE-T media 
interface, rev. 0

N8 ethernet:
bge1 at pci2 dev 0 function 1: Broadcom BCM5720 Gigabit Ethernet
bge1: APE firmware NCSI 1.3.7.0
bge1: interrupting at msi2 vec 0
bge1: HW config 002b1194, 00006014, 4000aa08, 00000000 00000000
bge1: ASIC BCM5720 A0 (0x5720000), Ethernet address 20:47:47:8f:00:02
bge1: setting short Tx thresholds
brgphy1 at bge1 phy 2: BCM5720C 1000BASE-T media interface, rev. 0

both sysctl and kernel parameters are default.

Both interfaces have vlans, one on each and both interfaces
on N9 are affected.

Both servers hit port-amd64/47016 and port-amd64/53687, but
on NetBSD 8 this cause no noticeable performance issues.

Sympthoms that I ever noticed:
1. named was awfully slow and it could take seconds to answer
on local zone.

2. host request (without named) usually take 0.2 sec, but one of
ten will take up to 10 sec. tcpdump show that request is
not going out on interface for all that time.

3. lighttpd on N8 does the same work, but take 0% always.
same work on N9 can make it to take up to 30% CPU and always
will be in system part and almost always in kqueue. Any other
process using network will show the same, for example
find over NFS directory took 40% system CPU time in kqueue.

4. testing network speed with /dev/zero->netcat->netcat->/dev/null
is showing 7-8 Mb/s between N8 and N9 and more than 70 Mb/s
between N8 and anything else.

5. constant ping have much more varying times around 1ms
from N9 to N8, often raised to 10 and once even to 300 ms,
but ping from N8 to N9 is much more stable like pinging
any other host.

6. Other NetBSD-9 system show nice performance in same
network.

And for now I have OS reverted back and ensured that
NetBSD 8 do not cause any of the above.

Thanks in advance.

--
Dima Veselov
Physics R&D Establishment of Saint-Petersburg University