Hi all,

I have a very weird problem with network transfer speed, and I have had
this for quite some time.

In short, the problem is this: reading data from NIC and writing to disk
is *VERY* slow. Easiest way to see this is just to ftp to another
machine (on local 100mbit network) and get some big file: the transfer
speed is ~50kb/sec.

Today I finally decided to investigate further. I wrote two very
simple programs:

The first program waits for connections on a TCP socket. When a connection
is opened, this program writes a file to the socket and then closes the
socket. The file is written to the socket by reading a buffer and then
writing the buffer to the socket. The buffer size is configurable, I
used 4096 bytes.

The second program connects to a TCP port, reads all the data from the
socket and writes the data to a file. This is done by reading a buffer
from the socket and then writing the buffer to the file.

I ran the "server" program on NetBSD/alpha, and served netbsd.generic
file (~7mb). I ran the client program on NetBSD/i386.

Depending on the buffer size used on the client, the transfer speed
varies from unbearably slow to extremely fast.

When 4096 byte buffer was used in the client, I didn't even wait for the
transfer to finish: I aborted after about a minute. When using
409600 byte buffer, the transfer lasted about 30 seconds. When using
10 megabyte buffer, the transfer took about 1 second.

When I did this the other way around -- run server on i386, client on
alpha -- there was no such disparity. The buffer on server was 4096.
Different buffer sizes for the client resulted in more or less the
same transfer speed every time: about two seconds.

I wonder if there is some kind of interaction with the same buffer
being used for reading from NIC and writing to disk. IIRC, there is
some kind of buffer sharing added to NetBSD some time ago, to
reduce number of copies. So could it be that, the buffer is first
filled from NIC, then it is queued to disk, but before being able
to read new data to the buffer, the system stops to wait until
the buffer is actually flushed to disk?

In that case, the 10mb buffer the transfer is fast, because it first fills
the memory, then dumps the stuff to disk, and then there's nothing more to do.
With small buffer there is a problem, because the read from network must
always wait for the buffer to be flushed to disk?

My setup is this, if it makes any difference: -current 2.0E on
both machines. Alpha is AlphaStation 500/266, NIC is realtek. i386
is Athlon XP 2400+, MSI KT4V mobo, 3c905C NIC.

-- 
Arto Huusko