Subject: Re: atrocious tx performance of gigabit cardbus re(4)
To: None <jonathan@dsg.stanford.edu>
From: Steven M. Bellovin <smb@cs.columbia.edu>
List: tech-net
Date: 09/25/2007 20:29:30
On Tue, 25 Sep 2007 13:00:23 -0700
jonathan@dsg.stanford.edu wrote:

> 
> In message <20070925195809.6b4fb60a@berkshire.machshav.com>,
> "Steven M. Bellovin" writes:
> 
> >On Tue, 25 Sep 2007 14:32:32 -0500
> >David Young <dyoung@pobox.com> wrote:
> >
> >...
> >> 
> >> It may be possible to get even higher performance by tweaking
> >> parameters both on the bridge and on the NIC.  For example, the
> >> latency timer on the primary (PCI) bus may be rather small.  On the
> >> NIC, both the PCI latency timer and the DMA burst parameters
> >> (highly NIC-specific) may be too small.
> >> 
> >I'm trying to find my notes, but if memory serves correctly I had
> >serious performance issues with a PCI re(4) on a gigE it.
> 
> [The driver and cardbus frontend are quite likely poorly
> tuned or untuned.
> 
> If dim memory serves, I split the re driver and added the cardbus
> frontend onne weekend, while the dongle on my old 3com cardbus card
> was dying. The replacement card I had picked up at Fry's was one of
> the first re(4) cardbus cards to hit the market. I did no more than
> get the darn thing to work at 100Mbit and get the link LEDs working
> and transmit of *some* packets working at gigabit speed.

I found an old private email I sent on my tests.  While I was certainly
seeing better performance than the OP reported here, the re(4) card was
much worse than wm on a gigE network talking to another NetBSD machine
that had a wm card.


-----
Here are four ttcp tests.  The first two were with the re card; the
second two, in the same slot on the same machine, same cable, etc.,
were with a wm card.

b139$ ttcp -s -fk -r
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001  tcp
ttcp-r: socket
ttcp-r: accept from 192.168.2.199
ttcp-r: 167772160 bytes in 3.66 real seconds = 357701.53 Kbit/sec +++
ttcp-r: 25991 I/O calls, msec/call = 0.14, calls/sec = 7093.06
ttcp-r: 0.0user 0.4sys 0:03real 11% 0i+0d 0maxrss 0+2pf 24920+39csw
b140$ ttcp -n 20480 -t -fk -s fubar
ttcp-t: buflen=8192, nbuf=20480, align=16384/0, port=5001  tcp  -> fubar
ttcp-t: socket
ttcp-t: connect
ttcp-t: 167772160 bytes in 3.41 real seconds = 383868.23 Kbit/sec +++
ttcp-t: 20480 I/O calls, msec/call = 0.17, calls/sec = 5997.94
ttcp-t: 0.0user 0.4sys 0:03real 12% 0i+0d 0maxrss 0+40961pf 18890+35csw




b142$ ttcp -n 20480 -t -fk -s fubar
ttcp-t: buflen=8192, nbuf=20480, align=16384/0, port=5001  tcp  -> fubar
ttcp-t: socket
ttcp-t: connect
ttcp-t: 167772160 bytes in 2.48 real seconds = 529463.40 Kbit/sec +++
ttcp-t: 20480 I/O calls, msec/call = 0.12, calls/sec = 8272.87
ttcp-t: 0.0user 0.5sys 0:02real 22% 0i+0d 0maxrss 0+40961pf 17057+46csw
b143$ ttcp -s -fk -r               
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001  tcp
ttcp-r: socket
ttcp-r: accept from 192.168.2.199
ttcp-r: 167772160 bytes in 2.50 real seconds = 524481.43 Kbit/sec +++
ttcp-r: 33499 I/O calls, msec/call = 0.08, calls/sec = 13404.54
ttcp-r: 0.0user 0.3sys 0:02real 15% 0i+0d 0maxrss 0+2pf 16841+46csw




		--Steve Bellovin, http://www.cs.columbia.edu/~smb