tech-kern archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: posix message queues and multiple receivers

On 2013-12-07 01:02, David Laight wrote:
On Fri, Dec 06, 2013 at 06:08:23PM +0100, Johnny Billquist wrote:

i see slowness from multiple clients when i run nfsd with just
one thread.  i've never seen the problem dsl has seen with a
netbsd nfs server (only other problems! :-)

I agree. I think it's nonsense. The idea that dsl have is that having
several I/O requests running in parallel against one server would cause
starvation on I/O in one thread because the others keeps the disk busy.
However, that assumes that I/O gets reordered, and that this reordering
can actually cause some I/O to get starved for long enough that NFS
times out and retries.

I've seen in on two different systems with different OS.
NFS/TCP probably doesn't suffer - assuming it doesn't run timeouts
at either the NFS or RPC layers.

The LAN traces for the first one were absolutely horrid.
Someone had NFS mounted an IDE disk on an i386 (ok - might have been
a i486) that would have been using 8 bit PIO from one of our much faster
sparc systems.
They then tried to write a large file. This generated a lot of concurrent
8k NFS writes as the buffer cache got cleared.
I believe that the disk driver on the server selected the disk transfers
using the 'elevator' algorithm. Since the writes were for more or less
sequential sectors, as soon as they got out of sequence one of the write
requests would have to wait until all the others completed.
The other 3 nfsd processes keep finding requests and the disk heads
keep movng in the same direction.
Once the RPC transfer times out all hell breaks loose.

Run one nfsd and it all runs smoothly.
Two might also be ok.

The second timne I saw similar nfs issues we just reduced the number of
servers and the problems went away.

You know, you might also hit a different problem, which I have had on many occasions. NFS using 8k transfers saturating the ethernet on the server, making the server drop IP fragemnts. That in turn force a resend of the whole 8k after a nfs timeout. That will totally kill your nfs performance. (Obviously, even larger nfs buffers make the problem even worse.)

Even with an elevator scan algorithm and four concurrent nfs clients, you're disk operation will complete within a few hundred ms at most.


Johnny Billquist                  || "I'm on a bus
                                  ||  on a psychedelic trip
email:             ||  Reading murder books
pdp is alive!                     ||  tryin' to stay hip" - B. Idol

Home | Main Index | Thread Index | Old Index