[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
Re: Attempt to fix nfs client
In article <20180120144510.GA61048%atom.tmux.org@localhost>,
Tobias Ulmer <tobiasu%tmux.org@localhost> wrote:
>TLDR: nfs_reqq crrptin, see diff below.
>I've been trying to track down the source of NFS client hangs and
>crashes over the last weeks, in order to get back to the things I really
>wanted to do...
>No special setup, the NFS Server is an OpenBSD machine serving files
>(pkgsrc, src, xsrc) from SSD (= fast replies). gbit net, udp intr
>There are any number of bug reports in gnats about this.
>1) Can't interrupt hanging process on 'intr' mount.
>2) Client goes dead silent and hangs despite tcpdump showing correct
> reply from server.
>3) Client crashes in all places dealing with nfsreq/nfsmount pointers.
> Mostly nfs_timer.
>The first was easy to fix. The main loop of nfs_receive() doesn't check
>The second issue is looping in nfs_receive, waiting for any reply to
>arrive. This is supposed to be fulfilled by nfs_timer re-sending the
>request, but investigation shows nfs_timer isn't scheduled because
>nfs_reqq is empty(!)
>I'm not 100% certain multiple lwps are racing to add and remove their
>requests to the global list. I could never spot contention between them
>(using mutex_trylock), but I assume they're doing so. Once insert and
>nfsreq removal is locked, crashes were confined to nfs_timer.
>Before one could observe funny situations where nfs_receive() would
>return the desired reply but nfs_reply() could not find the outstanding
>request in the nfs_reqq, then discarding the valid reply as unexptected
>Finally, the crashes in nfs_timer. Allocating zero'ed nfsreq and
>overwriting with junk before freeing will quickly crash the machine.
>Sprinkling a few printfs around, it's clear that nfs_timer (callout,
>kernel_lock'ed, blocked from starting by splsoftnet) is giving up its
>lock at some point (presumably when calling into pr_send). Lwps
>will then modify and free their nfsreq in between nfs_timer
>operating on the nfs_reqq.
>PR 40491 (http://gnats.netbsd.org/cgi-bin/query-pr-single.pl?number=40491)
>basically spells it out in one sentence, it just doesn't go far
>enough. Both nfs_reqq and nfsreq can evaporate from under you.
>Below is my attempt at patching this up. It seems a bit heavy handed but
>so far it's been 100% reliable (even serving nfs mounts) and perfomance is
>unchanged. Contention is low because nfs_timer is mostly blocked from
>running by splsoftnet, and it's a recovery thing anyway.
That is fine as it is! Thanks for working on it, committed.
Main Index |
Thread Index |