Re: NetBSD 5.1 TCP performance issue (lots of ACK)

[Thor Lancelot Simon <tls%panix.com@localhost>]

On Thu, Oct 27, 2011 at 05:31:47PM +0200, Manuel Bouyer wrote:
> On Thu, Oct 27, 2011 at 04:02:12PM +0200, Manuel Bouyer wrote:
> > > Do you have actual problems if gluster doesn't force the buffer to be
> > > large?
> > 
> > that's interesting: I now have 78MB/s with tso4, and 48MB/s without
> > tso4. Just as if the setsockopt would turn tso4 off.
> 
> Even more interesting: without changes on the linux side, a linux client
> gets only 25MB out of the NetBSD server without setsockopt (it gets 95MB/s
> when the netbsd server sets the snd/rcv buf size),
> and the NetBSD client without setsockopt gets only 73MB/s out of the linux
> server (it gets 95MB/s)

It's possible this has to do with the interrupt moderation tuning.  I
believe we've been pending the checkin of better values than the ones
I worked out from the documentation for quite some time -- there were
highly unobvious performance effects with small buffers.  Simon did
a bunch of testing and concluded, as I recall, that the values used
by Intel in the Linux driver were "magic" and that we should use
those, not mine.

If this hasn't been adjusted to match the Linux driver, you might
want to take a quick look at the values it uses and see whether
they yield better small-buffer performance in your case.

Also it may be worth checking we are not doing TSO twice, once in
hardware and once in software.  There was a bug a long time ago
that could cause that but last time I looked I determined by setting
breakpoints in the relevant segmentation functions that it was,
as far as I could tell, fixed.

There should never be a performance loss from TSO, given a sane
implementation.  In fact there shouldn't be much gain either on
modern systems _unless_ they are under very heavy load, since the
real savings is in transmitting host CPU time.

Thor