Re: NetBSD 5.1 TCP performance issue (lots of ACK)

[Manuel Bouyer <bouyer%antioche.eu.org@localhost>]

On Wed, Oct 26, 2011 at 08:15:44PM -0400, Greg Troxel wrote:
> 
> The speed
> 
>   with glusterfs: seems to be a combination of 74 MB/s and pauses
> 
>   with ttcp: seems to be 112 MB/s burst (for 0.2s) and some smaller
>   pauses, which gut checks with 95 MB/s as reported by ttcp.
> 
> So are you seeing high 40s MB/s out of glusterfs?

Yes, between 40 and 50MB/s

> 
> 
> What is between these two devices?  Is this just a gigabit switch, or
> anything more complicated?

they're all (the 2 NetBSD and the linux host)  connected to a cisco 3750
gigabit switch. I also tested with a single crossover cable, this doens't
change anything .

> We are seeing reordering which I would not
> expect on an ethernet.  I wonder if the tso4 option is causing that.
> What if you turn off the offload options?  (I realize it may slow down,
> but if both are then equal, that's a clue.)

that's easy. And yes, I get better performances: 77MB/s instead of < 50.
So it looks like we have something wrong with TSO.
The traces are still at ftp://asim.lip6.fr/outgoing/bouyer/
(netbsd-{client,server}-notso.pcap.gz).

Did you see the reordering in the ttcp trace too ?

But, that still doesn't explain why I get good performances when one
of the host is linux. NetBSD used tso as well, and it didn't seem to cause
problems for linux ...

BTW, how is TSO working ? does the adapter get a single data block of
a full window size ? if so, maybe the transmit ring just isn't big
enough ...

> 
> 
> I wonder if the very large buffers get full and that causes cache
> thrashing.  What happens if you change gluster to have smaller buffers
> (I don't understand why it's ok to have the FS change the tcp socket
> buffer options from system default)?

Because it knows the size of its packets, or its internal receive buffers ?

-- 
Manuel Bouyer <bouyer%antioche.eu.org@localhost>
     NetBSD: 26 ans d'experience feront toujours la difference
--