tech-kern archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: Attempt to fix nfs client

In article <>,
Tobias Ulmer  <> wrote:
>TLDR: nfs_reqq crrptin, see diff below.
>I've been trying to track down the source of NFS client hangs and
>crashes over the last weeks, in order to get back to the things I really
>wanted to do...
>No special setup, the NFS Server is an OpenBSD machine serving files
>(pkgsrc, src, xsrc) from SSD (= fast replies). gbit net, udp intr
>There are any number of bug reports in gnats about this.
>1) Can't interrupt hanging process on 'intr' mount.
>2) Client goes dead silent and hangs despite tcpdump showing correct
>   reply from server.
>3) Client crashes in all places dealing with nfsreq/nfsmount pointers.
>   Mostly nfs_timer.
>The first was easy to fix. The main loop of nfs_receive() doesn't check
>for signals.
>The second issue is looping in nfs_receive, waiting for any reply to
>arrive. This is supposed to be fulfilled by nfs_timer re-sending the
>request, but investigation shows nfs_timer isn't scheduled because
>nfs_reqq is empty(!)
>I'm not 100% certain multiple lwps are racing to add and remove their
>requests to the global list. I could never spot contention between them
>(using mutex_trylock), but I assume they're doing so.  Once insert and
>nfsreq removal is locked, crashes were confined to nfs_timer.
>Before one could observe funny situations where nfs_receive() would
>return the desired reply but nfs_reply() could not find the outstanding
>request in the nfs_reqq, then discarding the valid reply as unexptected
>Finally, the crashes in nfs_timer. Allocating zero'ed nfsreq and
>overwriting with junk before freeing will quickly crash the machine.
>Sprinkling a few printfs around, it's clear that nfs_timer (callout,
>kernel_lock'ed, blocked from starting by splsoftnet) is giving up its
>lock at some point (presumably when calling into pr_send). Lwps
>will then modify and free their nfsreq in between nfs_timer
>operating on the nfs_reqq.
>PR 40491 (
>basically spells it out in one sentence, it just doesn't go far
>enough. Both nfs_reqq and nfsreq can evaporate from under you.
>Below is my attempt at patching this up. It seems a bit heavy handed but
>so far it's been 100% reliable (even serving nfs mounts) and perfomance is
>unchanged.  Contention is low because nfs_timer is mostly blocked from
>running by splsoftnet, and it's a recovery thing anyway.

That is fine as it is! Thanks for working on it, committed.


Home | Main Index | Thread Index | Old Index