Hi,

Jim Reid wrote:
[...]
>     Todd> When I start running, say, -j16 rm's, very shortly the iMac
>     Todd> complains that the server is not responding, exactly once. I
>     Todd> believe NFS is hung from that point on, because further
>     Todd> attempts to use the iMac result in more and more processes
>     Todd> getting blocked on something over NFS, until the whole iMac
>     Todd> is effectively wedged.
> 
> Yeah. It looks like sending 16 simultaneous NFS requests to the HPUX
> server - plus whatever others are doing NFS to that box - gives it a
> very hard time. As a general rule of thumb, NFS clients and servers

My experience is, that the NFS implementation in HP-UX is really
bad.

[...]
>     Todd> nfsiod is at the default of 4, but I don't understand why
>     Todd> overloading them would have such a nasty effect.
> 
> I doubt your nfsiod processes are the problem. They're probably not
> doing anything while your highly parallel remove is in progress. It's
> the nfs code on the server that'll be hurting. And you're stressing
> that very heavily.

Wouldn't more nfsiod processes keep at least a few free for other
NFS access (to other servers)?

> Each NFS unlink request forces the server to write the directory file
> back to disk so there may well be nasty synchronisation issues for the
> HPUX kernel in such extreme circumstances. Directory operations are
> one of the most expensive operations in NFS. And if the server is too
> slow at replying, your client rm processes repeat the previous unlink
> request because the previous one timed out, producing a vicious
> infinite loop. Experimenting with the NFS mount parameters - timeouts,

Well, the loop shouldn't be infinite, should it? It should just take
a long time to complete. But I agree that tuning the parameters
might help.

> etc - and the number of *server* side NFS daemons might alleviate the
> problem. So might NFS-over-TCP because this'll introduce some
> flow-control and backoff when the other end is busy.

And so might a different operating system on the server :-)

> The default number of NFS processes is usually sub-optimal too. A good
> heuristic there is to have 1 process for each disk spindle and network
> interface plus a couple of "spares". ie Each process can be in a busy
> I/O wait and there are still some processes available to be scheduled
> for further NFS requests. [The number 4 seems to be a legacy from the
> earliest days of NFS on a Sun3.] Other considerations for the number
> of NFS processes are the thundering herd problem - waking every
> nfs(io)d for 1 incoming query - and CPU/MMU context thrashing. IIRC
> this was why some early SPARCs didn't run more than 8 nfsd's. Sun
> wanted to be sure that some of the MMU's contexts could be kept free
> so that they'd be available for applications.

Agreed with that.

  Guenther