Subject: Etherexpress works but Very Very Slow ?
To: None <port-arm32@netbsd.org>
From: Dave McConnell <davem@eastcoast.co.za>
List: port-arm32
Date: 10/29/1999 17:26:51
Hi

I'm busy debugging the slow etherexpress performance on EBSA285. I have a
few "clues" that might suggest something to somebodey. I'd appreciate
it.....

On ebsa285:

Using intel etherexpress (Intel 82558 chip=fxp driver) the ethercard
actually seems to work, BUT

An NFS boot takes about an hour, but finally does boot. (NFS boot on same
system using ne2000 or DEC tulip ethernet card works fast, around 30 secs or
so, never timed it....)

If I boot using another card (e.g. ne0) and with fxp0 in the system, pinging
thru fxp0 to a number on the same (uncongested 100mb/s) lan segment:

Packet size 2048:

2056 bytes from 196.7.114.64: icmp_seq=9 ttl=128 time=2.972 ms

----196.7.114.64 PING Statistics----
10 packets transmitted, 9 packets received, 10.0% packet loss
round-trip min/avg/max/stddev = 2.295/2.674/2.995/0.347 ms

Packet size 4096:
4104 bytes from 196.7.114.64: icmp_seq=9 ttl=128 time=3294.221 ms

----196.7.114.64 PING Statistics----
10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 24.351/1121.960/3294.221/822.059 ms

Note the varying RTT AND the fact that on average it takes around 1000ms as
opposed to 2.6 ms for 2048 byte packets. The "lost packet" in to 2048 test
also happens quite frequently.

Similar tests thru the ne0 interface (on a 10mb/s LAN) show around 15ms rtt
for 4096 byte pings.

In light of the above (naively?) tried to "fix" this problem by rebuilding
NFS with all max sizes set to 1024 (figuring it seemed to handle smaller
packets OK). Under the circumstances, the ne0 still NFS boots fine, but the
fxp0 is the same (waaaay to slow by about 2 orders of magnitude).

The fxp0 also seems to take about 2 to 3 DHCP request/ack pairs to get an IP
(the ne0 gets it first time every time).

Some sample tcpdump output on the NFS server side while the very slow fxp0
boot is in progress may also provide clues?

09:34:34.216105 192.168.1.1.nfs > ebsa285.nanoteq.co.za.2630410246: reply ok
112
getattr [|nfs]
09:34:38.354911 ebsa285.nanoteq.co.za.2630410246 > 192.168.1.1.nfs: 96
getattr [
|nfs]
09:34:38.355201 192.168.1.1.nfs > ebsa285.nanoteq.co.za.2630410246: reply ok
112
getattr [|nfs]
09:34:38.365628 ebsa285.nanoteq.co.za.2630410247 > 192.168.1.1.nfs: 108 read
[|n
fs]
09:34:38.383887 192.168.1.1.nfs > ebsa285.nanoteq.co.za.2630410247: reply ok
147
2 read [|nfs] (frag 4406:1480@0+)
09:34:38.383914 192.168.1.1 > ebsa285.nanoteq.co.za: (frag 4406:1480@1480+)
09:34:38.384003 192.168.1.1 > ebsa285.nanoteq.co.za: (frag 4406:1480@2960+)
09:34:38.384023 192.168.1.1 > ebsa285.nanoteq.co.za: (frag 4406:1480@4440+)
09:34:38.384043 192.168.1.1 > ebsa285.nanoteq.co.za: (frag 4406:1480@5920+)
09:34:38.384080 192.168.1.1 > ebsa285.nanoteq.co.za: (frag 4406:928@7400)
09:34:41.384853 ebsa285.nanoteq.co.za.2630410247 > 192.168.1.1.nfs: 108 read
[|n
fs]
09:34:41.385566 192.168.1.1.nfs > ebsa285.nanoteq.co.za.2630410247: reply ok
147
2 read [|nfs] (frag 4409:1480@0+)
09:34:41.385604 192.168.1.1 > ebsa285.nanoteq.co.za: (frag 4409:1480@1480+)
09:34:41.385684 192.168.1.1 > ebsa285.nanoteq.co.za: (frag 4409:1480@2960+)
09:34:41.385705 192.168.1.1 > ebsa285.nanoteq.co.za: (frag 4409:1480@4440+)
09:34:41.385737 192.168.1.1 > ebsa285.nanoteq.co.za: (frag 4409:1480@5920+)
09:34:41.385761 192.168.1.1 > ebsa285.nanoteq.co.za: (frag 4409:928@7400)
09:34:41.389063 ebsa285.nanoteq.co.za.2630410248 > 192.168.1.1.nfs: 108 read
[|n
fs]
09:34:41.403355 192.168.1.1.nfs > ebsa285.nanoteq.co.za.2630410248: reply ok
147
2 read [|nfs] (frag 4410:1480@0+)
09:34:41.403396 192.168.1.1 > ebsa285.nanoteq.co.za: (frag 4410:1480@1480+)
09:34:41.403474 192.168.1.1 > ebsa285.nanoteq.co.za: (frag 4410:1272@2960)
09:34:44.404835 ebsa285.nanoteq.co.za.2630410248 > 192.168.1.1.nfs: 108 read
[|n
fs]
09:34:44.405332 192.168.1.1.nfs > ebsa285.nanoteq.co.za.2630410248: reply ok
147
2 read [|nfs] (frag 4413:1480@0+)
09:34:44.405370 192.168.1.1 > ebsa285.nanoteq.co.za: (frag 4413:1480@1480+)
09:34:44.405448 192.168.1.1 > ebsa285.nanoteq.co.za: (frag 4413:1272@2960)
09:34:44.416583 ebsa285.nanoteq.co.za.2630410249 > 192.168.1.1.nfs: 108 read
[|n
fs]
09:34:44.418298 192.168.1.1.nfs > ebsa285.nanoteq.co.za.2630410249: reply ok
147
2 read [|nfs] (frag 4414:1480@0+)
09:34:44.418330 192.168.1.1 > ebsa285.nanoteq.co.za: (frag 4414:1480@1480+)
09:34:44.418418 192.168.1.1 > ebsa285.nanoteq.co.za: (frag 4414:1480@2960+)
09:34:44.418438 192.168.1.1 > ebsa285.nanoteq.co.za: (frag 4414:1480@4440+)
09:34:44.418475 192.168.1.1 > ebsa285.nanoteq.co.za: (frag 4414:1480@5920+)
09:34:44.418496 192.168.1.1 > ebsa285.nanoteq.co.za: (frag 4414:928@7400)
09:34:47.434803 ebsa285.nanoteq.co.za.2630410249 > 192.168.1.1.nfs: 108 read
[|n
fs]
09:34:47.435826 192.168.1.1.nfs > ebsa285.nanoteq.co.za.2630410249: reply ok
147
2 read [|nfs] (frag 4417:1480@0+)
09:34:47.435862 192.168.1.1 > ebsa285.nanoteq.co.za: (frag 4417:1480@1480+)
09:34:47.435942 192.168.1.1 > ebsa285.nanoteq.co.za: (frag 4417:1480@2960+)
09:34:47.435960 192.168.1.1 > ebsa285.nanoteq.co.za: (frag 4417:1480@4440+)
09:34:47.435981 192.168.1.1 > ebsa285.nanoteq.co.za: (frag 4417:1480@5920+)
09:34:47.436018 192.168.1.1 > ebsa285.nanoteq.co.za: (frag 4417:928@7400)
09:34:47.438240 ebsa285.nanoteq.co.za.2630410250 > 192.168.1.1.nfs: 108 read
[|n
fs]

Thanks for any suggestions.
David