Subject: curproc and nfsd
To: None <tech-kern@netbsd.org>
From: None <rick@snowhite.cis.uoguelph.ca>
List: tech-kern
Date: 05/26/2004 11:17:00
Well, I'm enjoying the curproc debate and will be happy to stay on the
sidelines:-) However, I will throw in my $0.00 worth w.r.t. nfsd context.

If a "kcont" (I have no idea what that is) can't sleep, it will be very hard
to make use of it for the nfs server and avoid a context switch to an nfsd.
(I once played around, trying to get the socket upcall to do Getattr and
 Lookup RPCs for the case where the reply info was cached. It was messy
 and I ended up tossing it. There's just too many places where you might
 need to sleep waiting for a buffer or need to do a VOP that might sleep
 or... For NFS over TCP (this is formalized in the NFSv4 RFC), you can't
 just drop a request, so you have to hand these over to a context that
 can sleep for continuation.)

What I think might be a good idea and would avoid a fair number of
the context switches is to have a kernel context (aka kernel thread)
that can sleep, which is dedicated to handling network
input. It would pick up the received packets straight from the network
device driver and either:
- drop them off in the receive queue of a socket
OR
- continue on, if there is a socket upcall
If the thread doing the socket upcall can sleep, it can do an NFS RPC and
there would be no need for separate nfsd threads and context switches to them.
(Related to your discussion, these threads would have to have enough
 "proc like" information that they would satisfy the needs of the VFS/VOP
 calls. It would be really nice to know what uses are made of uio_procp,
 cn_procp (or thread * for FreeBSD5) and minimize that. Maybe a small
 structure that just has the fields of "struct proc" or "struct thread"
 that the VFS/VOP calls need from these?)

Since they're kernel only and network receive only, they could be trimmed
down and get special treatment from the scheduler.

w.r.t. smp and locks:
There aren't too many shared data structures in the nfs server, but they
do need to be protected for SMP, sometimes in cases where a sleep is
possible. (For FreeBSD5, I've used a mix of the mutex locks and some of
my own locking flags with the usual "sleep if you want it and the lock
flag is set" code.) Although my current nfsd isn't really fine grained
locked, I think it's in fairly good shape until you hit the VFS/VOP calls.
(Since FreeBSD5 still requires Giant on the VFS/VOP calls, I haven't yet
 bothered to put non-sleep locks around the v4 state data structures, but
 that can be easily done.)

rick
ps: I just cc'd a couple of you non-NetBSDers since, after typing this in,
    I thought it might interest you. (Apologies in advance for spamming you:-)