Subject: Re: wm(4) with i82544EI on the AlphaServer ES40...
To: None <tls@rek.tjls.com>
From: Greg A. Woods <woods@weird.com>
List: tech-net
Date: 11/30/2004 20:04:40
[ On Monday, November 29, 2004 at 20:57:01 (-0500), Thor Lancelot Simon wrote: ]
> Subject: Re: wm(4) with i82544EI on the AlphaServer ES40...
>
> I've repeatedly seen strange, unexpectedly low throughput numbers from
> wm cards hooked up back-to-back (including with the Intel-supplied driver
> on Linux).  I was able to reproduce this with Marvell cards that supposedly
> use the same PHY block, as well, which makes me think that there may be
> something about that combination of that PHY and the use of another such
> (instead of a switch) as the link partner that causes poor performance.

Weird.  It does seem to come up with the right link status on both ends
and I never see any errors on the interfaces at either end, but I guess
that doesn't mean much...  :-)

> Can you throw a cheap (but modern) switch in the middle and see if you
> get the same or different results?

I think I have a decent switch in there now (at least that's what my
on-site eyes and hands tell me they've done :-), but it made no
difference.

I have also managed to convince them to "borrow" a 2-port fibre card
without any PHY (i82546GB) from another not-quite-production box and
sadly it's no better either.  It's hooked back-to-back with an identical
card in the dual-Opteron FreeBSD box I've been testing against.  I can
just barely push 36 MB/s through it with ideal(?) TTCP parameters and
with all the hardware checksum features turned on, and about 10% less
without them on.

If I had not seen the bge0 card do well over 50 MB/s under True64 I'd be
worried that there was some more fundamental problem with the bus or
such.  Hopefully Jason is able to wrangle if_bge.c into better shape for
the alpha soon.  :-)


[console]<@> # dmesg | fgrep -e wm0 -e wm1 
wm0 at pci1 dev 1 function 0: Intel i82546GB 1000BASE-X Ethernet, rev. 3
wm0: interrupting at dec 6600 irq 24
wm0: 64-bit 33MHz PCI bus
wm0: 256 word (8 address bits) MicroWire EEPROM
wm0: Ethernet address 00:04:23:a8:7c:02
wm0: 1000baseSX, 1000baseSX-FDX, auto
wm1 at pci1 dev 1 function 1: Intel i82546GB 1000BASE-X Ethernet, rev. 3
wm1: interrupting at dec 6600 irq 25
wm1: 64-bit 33MHz PCI bus
wm1: 256 word (8 address bits) MicroWire EEPROM
wm1: Ethernet address 00:04:23:a8:7c:03
wm1: 1000baseSX, 1000baseSX-FDX, auto

[console]<@> # ttcp -l 32768 -b 131072 -v -t -s -n 10000 10.20.20.1
ttcp-t: buflen=32768, nbuf=10000, align=16384/0, port=5001, sockbufsize=131072  tcp  -> 10.20.20.1
ttcp-t: socket
ttcp-t: sndbuf
ttcp-t: connect
ttcp-t: 327680000 bytes in 10.67 real seconds = 29992.58 KB/sec +++
ttcp-t: 327680000 bytes in 10.67 CPU seconds = 29998.35 KB/cpu sec
ttcp-t: 10000 I/O calls, msec/call = 1.09, calls/sec = 937.27
ttcp-t: 0.0user 10.6sys 0:10real 100% 0i+0d 0maxrss 0+40006pf 3+119csw
ttcp-t: buffer address 0x20050000

[console]<@> # ifconfig wm1 ip4csum tcp4csum udp4csum
[console]<@> # ifconfig wm1
wm1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        capabilities=7<IP4CSUM,TCP4CSUM,UDP4CSUM>
        enabled=7<IP4CSUM,TCP4CSUM,UDP4CSUM>
        address: 00:04:23:a8:7c:03
        media: Ethernet autoselect (1000baseSX full-duplex)
        status: active
        inet 10.20.20.2 netmask 0xffffff00 broadcast 10.20.20.255

[console]<@> # ttcp -l 32768 -b 131072 -v -t -s -n 10000 10.20.20.1
ttcp-t: buflen=32768, nbuf=10000, align=16384/0, port=5001, sockbufsize=131072  tcp  -> 10.20.20.1
ttcp-t: socket
ttcp-t: sndbuf
ttcp-t: connect
ttcp-t: 327680000 bytes in 9.86 real seconds = 32441.06 KB/sec +++
ttcp-t: 327680000 bytes in 9.86 CPU seconds = 32449.39 KB/cpu sec
ttcp-t: 10000 I/O calls, msec/call = 1.01, calls/sec = 1013.78
ttcp-t: 0.0user 9.8sys 0:09real 99% 0i+0d 0maxrss 0+40006pf 4+110csw
ttcp-t: buffer address 0x20050000

[console]<@> # ttcp -l 65536 -b 131072 -v -t -s -n 10000 10.20.20.1
ttcp-t: buflen=65536, nbuf=10000, align=16384/0, port=5001, sockbufsize=131072  tcp  -> 10.20.20.1
ttcp-t: socket
ttcp-t: sndbuf
ttcp-t: connect
ttcp-t: 655360000 bytes in 17.76 real seconds = 36030.71 KB/sec +++
ttcp-t: 655360000 bytes in 11.28 CPU seconds = 56740.17 KB/cpu sec
ttcp-t: 10000 I/O calls, msec/call = 1.82, calls/sec = 562.98
ttcp-t: 0.0user 11.2sys 0:17real 63% 0i+0d 0maxrss 0+80020pf 19+197csw
ttcp-t: buffer address 0x20050000



For reference, the Tru64 numbers on the Broadcom/DEGXA card.
Unfortunately Tru64 doesn't have a driver for the i8254x cards...

bcm0: Link down
bcm0 at pci1 slot 6
bcm0: DEGXA (1000BaseT) Gigabit Ethernet Interface, hardware address: 00-08-02-91-89-AE

# uname
OSF1
# ifconfig -v bcm0
bcm0: flags=c63<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST,SIMPLEX>
     HWaddr 0:8:2:91:89:ae
     inet 10.30.30.2 netmask ff000000 broadcast 10.255.255.255 ipmtu 1500 

# ./ttcp -v -t -s -n 100000 10.30.30.1 
ttcp-t: buflen=8192, nbuf=100000, align=16384/0, port=5001  tcp  -> 10.30.30.1
ttcp-t: socket
ttcp-t: connect
ttcp-t: 819200000 bytes in 15.50 real seconds = 51612.90 KB/sec +++
ttcp-t: 819200000 bytes in 10.20 CPU seconds = 78407.51 KB/cpu sec
ttcp-t: 100000 I/O calls, msec/call = 0.16, calls/sec = 6451.61
ttcp-t: 0.0user 10.1sys 0:15real 65% 0i+2d 108maxrss 0+1pf 335169+15897csw
ttcp-t: buffer address 0x40008000

# ./ttcp -l 65536 -b 131072 -v -t -s -n 10000 10.30.30.1
ttcp-t: buflen=65536, nbuf=10000, align=16384/0, port=5001, sockbufsize=131072  tcp  -> 10.30.30.1
ttcp-t: socket
ttcp-t: sndbuf
ttcp-t: connect
ttcp-t: 655360000 bytes in 15.12 real seconds = 42322.25 KB/sec +++
ttcp-t: 655360000 bytes in 3.96 CPU seconds = 161432.23 KB/cpu sec
ttcp-t: 10000 I/O calls, msec/call = 1.55, calls/sec = 661.29
ttcp-t: 0.0user 3.9sys 0:15real 26% 0i+2d 136maxrss 0+7pf 255202+248csw
ttcp-t: buffer address 0x40008000
# 

-- 
						Greg A. Woods

+1 416 218-0098                  VE3TCP            RoboHack <woods@robohack.ca>
Planix, Inc. <woods@planix.com>          Secrets of the Weird <woods@weird.com>