Re: Network performance issues

[Steven Bellovin <smb%cs.columbia.edu@localhost>]

On Aug 26, 2010, at 8:59 33AM, Sad Clouds wrote:

> Hi, I'm testing some networking code I've written that uses I/O
> multiplexing with kqueue and poll. The test consists of the following:
> 
> Client opens 100 simultaneous connections and sends N bytes of data on
> each connection.
> 
> Server accepts connections and replies N bytes of data back to the
> client.
> 
> When the client's send and receive data counters reach N bytes for a
> connection, it closes this connection.
> 
> I'm seeing some performance issues with larger data segments:
> 
> Time for client to open 100 connections, send and receive 500 bytes on
> each connection ranges from 0.07 to 1.54 seconds.
> 
> Time for client to open 100 connections, send and receive 1000 bytes on
> each connection gives 31.05 seconds.
> 
> The hardware is two fast x86 machines, both using hme0 network
> interfaces. The machines are connected to 100Mbps Ethernet switch.
> Client is running NetBSD 5.0.2, server is running NetBSD 5.1_RC3.
> 
> I've gone through my code and can't see any problems. The hardware has
> plenty of bandwidth, so that shouldn't slow things down. What I find
> strange is that going from 500 byte data segments to 1000 byte,
> increases total time wait by so much. If I had bugs in my code, surely
> that would result in identical issues with 500 and 1000 byte data
> segments. I tested this with kqueue and poll and got similar results.
> 
> Can this be a kernel issue? What sysctl tunable parameters could have
> influence on this?

I suspect that the underlying problem is packet loss.  If a segment gets 
dropped and retransmitted, the sending TCP will slow down.  But that's 
connection-specific; other connections won't be affected.  You can verify that 
that's happening by running tcpdump on both ends, though I confess I don't know 
of a good analysis tool.  The question is why packets are getting lost.

Several possibilities occur to me.  First, and most likely: you're exceeding 
the buffer capacity of the switch.  I don't think the problem is on the sending 
side, since as I recall the queue length limit on NetBSD is in terms of 
packets, not bytes, but the switch may be different.  Alternatively, it may be 
the receiving NIC -- it may have buffer management problems, too.

Second: perhaps there's a cable problem -- I've often seen long packets fail 
under such circumstances.  Try using ping at various packet sizes and see if 
there's greater packet loss.


                --Steve Bellovin, http://www.cs.columbia.edu/~smb