On Thu, Jun 10, 2004 at 09:34:51PM +0300, Arto Huusko wrote:
> Hi all,
> 
> I have a very weird problem with network transfer speed, and I have had
> this for quite some time.
> 
> In short, the problem is this: reading data from NIC and writing to disk
> is *VERY* slow. Easiest way to see this is just to ftp to another
> machine (on local 100mbit network) and get some big file: the transfer
> speed is ~50kb/sec.
> 
> Today I finally decided to investigate further. I wrote two very
> simple programs:
> 
> The first program waits for connections on a TCP socket. When a connection
> is opened, this program writes a file to the socket and then closes the
> socket. The file is written to the socket by reading a buffer and then
> writing the buffer to the socket. The buffer size is configurable, I
> used 4096 bytes.
> 
> The second program connects to a TCP port, reads all the data from the
> socket and writes the data to a file. This is done by reading a buffer
> from the socket and then writing the buffer to the file.
> 

I assume it also has buffer size configurable.

> I ran the "server" program on NetBSD/alpha, and served netbsd.generic
> file (~7mb). I ran the client program on NetBSD/i386.
> 
> Depending on the buffer size used on the client, the transfer speed
> varies from unbearably slow to extremely fast.
> 
> When 4096 byte buffer was used in the client, I didn't even wait for the
> transfer to finish: I aborted after about a minute. When using
> 409600 byte buffer, the transfer lasted about 30 seconds. When using
> 10 megabyte buffer, the transfer took about 1 second.
> 
> When I did this the other way around -- run server on i386, client on
> alpha -- there was no such disparity. The buffer on server was 4096.
> Different buffer sizes for the client resulted in more or less the
> same transfer speed every time: about two seconds.
> 
> I wonder if there is some kind of interaction with the same buffer
> being used for reading from NIC and writing to disk. IIRC, there is
> some kind of buffer sharing added to NetBSD some time ago, to
> reduce number of copies. So could it be that, the buffer is first
> filled from NIC, then it is queued to disk, but before being able
> to read new data to the buffer, the system stops to wait until
> the buffer is actually flushed to disk?
> 
> In that case, the 10mb buffer the transfer is fast, because it first fills
> the memory, then dumps the stuff to disk, and then there's nothing more to 
> do.
> With small buffer there is a problem, because the read from network must
> always wait for the buffer to be flushed to disk?
> 

I suspect that disk traffic worsen nic interrupt latencies and causes
packet loss (say not large enought memory buffer on nic) which causes
timeouts and retransmissions. You should be able to see it in tcpdump.

	Andrey