Re: nfsd "serializing" patch

[Manuel Bouyer <bouyer%antioche.eu.org@localhost>]

On Fri, Dec 07, 2012 at 09:05:55AM -0800, Chuck Silvers wrote:
> On Wed, Dec 05, 2012 at 05:11:37PM +0100, Manuel Bouyer wrote:
> > On Wed, Dec 05, 2012 at 08:01:13AM -0800, Chuck Silvers wrote:
> > > this seems to be trying to work around our current limitation that all 
> > > writes
> > > to a given vnode are serialized, is that right?
> > 
> > It is partially. Another aspect is CPU cache trashing, by processing
> > the same socket by manu CPUs we load and eventually write the
> > same variables in different CPUs cache, and this could well make processing
> > slower than a pure sequencial processing.
> > 
> > > eventually we'd like to
> > > improve things such that I/O to non-overlapping ranges of the file can be
> > > processed in parallel. 
> > 
> > nice goal, but it doesn't look easy, and I guess you'll still need a lock
> > at last to gain exclusive ownerchip of a range. So there will still
> > be some lock contention. This, with the CPU cache issue mentionned
> > above, may still not make a multithread nfsd servicing a single socket
> > faster than a single nfsd thread.
> 
> I'm thinking more of the case where the thread processing a request has to
> wait for disk I/O.  in that case, having multiple threads can be a huge 
> benefit.
> also, if the nfsd threads operate on separate files, multiple threads will 
> also
> help even today.  you mentioned that requests that are pending simultaneously
> being for the same file is "the" common case... I'll agree that it's
> "a" common case, but parallel operations on multiple files is hardly uncommon.
> I don't think we should optimize one at the expense of the other.

Well, on my nfs server, which I think is quite standard (it has
home directories and work space for about 100 users), I see mostly writes,
and mostly burst of writes to a single file from a given host.
This is probably because once read, the data are cached on the client side.

The main problem with the current concurent approach is that a single
host can keep busy all nfsd threads with writes, and other, less expensive
operations from other hosts have to wait.

And, from what I've seen, the client won't send much concurent request
for differents files, because its queue is already full with
write requests.


> 
> > > this change appears to help in the context of the
> > > current limitation, but it sounds like the opposite direction for what 
> > > we'd
> > > like to do in the long run.  it would be better if we could find some way
> > > improve things now that won't make life harder for ourselves later.
> > 
> > We still want to process UDP sockets in parallel, so the code to do things
> > in multiple threads is still there, and will stay. So going back to a
> > multithreaded processing for connected sockets shouldn't be hard in
> > the future, if we want to.
> 
> one way to accomodate these various workloads would be to have a sysctl
> (or set of them) to control how many threads will work on each socket.
> then we can debate what the default setting should be.  :-)

That should be doable.

-- 
Manuel Bouyer <bouyer%antioche.eu.org@localhost>
     NetBSD: 26 ans d'experience feront toujours la difference
--