Re: NFS client renders system unusable

["Brad Pitney" <pitney.brad%googlemail.com@localhost>]

On Sat, Mar 15, 2008 at 5:39 AM, Sarton O'Brien 
<bsd-xen%roguewrt.org@localhost> wrote:
>
> Sarton O'Brien wrote:
>  > Sarton O'Brien wrote:
>  >> I have an NFS server running with samba on lfs which occaisionally
>  >> panics for reasons I haven't bothered to delve into yet, I'm almost
>  >> positive it is either samba or lfs related ... but most likely the
>  >> combination.
>  >>
>  >> When the NFS server dies, the client console fills up with:
>  >>
>  >> yp_order: clnt_call: RPC: Unable to send; errno = No buffer space
>  >> available
>  >> yp_order: clnt_call: RPC: Unable to send; errno = No buffer space
>  >> available
>  >> yp_order: clnt_call: RPC: Unable to send; errno = No buffer space
>  >> available
>  >> yp_order: clnt_call: RPC: Unable to send; errno = No buffer space
>  >> available
>  >> yp_order: clnt_call: RPC: Unable to send; errno = No buffer space
>  >> available
>  >> yp_order: clnt_call: RPC: Unable to send; errno = No buffer space
>  >> available
>  >>
>  >> And the client system becomes completely unusable.
>  >>
>  >> In this instance it occurred within minutes of the server dying.
>  >>
>  >> Googling the error doesn't reveal much so I'm not sure if there is
>  >> something I can do to mitigate the effects, if it's a known issue or
>  >> if it's a bug. As the NFS server used to be almost completely stable,
>  >> this is a problem I've never really encountered.
>  >
>
> > OK, now it just appears to be happening when under load, the NFS
>  > server hasn't died and the client is producing those messages endless ...
>  >
>  > kernel and userland are only a few days old for both server and client.
>  >
>  > Any ideas?
>
>  Sorry, I should have mentioned that the load is client connections, not
>  data. I've a p2p app on the client using minimal BW but supporting a
>  large amount of connections ... all those p2p connections would equate
>  to nfs client conections. The problem only occurs when the app is
>  running. All other clients, including ones with high data usage are fine.
>
>  I can't provide any details from the client due to it being unusable but
>  nfsstat on the server reports:
>
>  spike# nfsstat -s
>  Server Info:
>  RPC Counts: (75842 calls)
>       null         getattr         setattr          lookup          access
>          0  0%       25052 33%         760  1%       28762 37%
>  6518  8%
>   readlink            read           write          create           mkdir
>          0  0%        6926  9%        3797  5%         764  1%
>  0  0%
>    symlink           mknod          remove           rmdir          rename
>          0  0%           0  0%        1102  1%           0  0%
>  359  0%
>       link         readdir     readdirplus          fsstat          fsinfo
>        493  0%         671  0%           0  0%          49  0%
>  5  0%
>   pathconf          commit
>          0  0%         584  0%
>  Server Errors:
>  RPC errors          faults
>       2321               0
>  Server Cache Stats:
>  inprogress            idem        non-idem          misses
>          0               0               0           75842
>  Server Write Gathering:
>     writes       write RPC       OPs saved
>       3797            3797               0  0%
>
>  Though I'm guessing the problem is with the client ... buffer? I have no
>  idea ...
>

Strange, when my NFS server goes away (maybe I've updated it and
rebooted), my NFS clients just stall and wait for the server to come
back, once it does, just resume as if nothing happened with exception
of a few messages like this:
Mar 13 00:13:44 nfs-client /netbsd: nfs server
nfs-server:/media/data/netbsd/current/obj: not responding
Mar 13 00:13:44 nfs-client /netbsd: nfs server
nfs-server:/media/data/netbsd/current/obj: is alive again
Mar 13 13:37:48 nfs-client /netbsd: nfs server
nfs-server:/media/data/netbsd/pkgsrc: not responding
Mar 13 13:37:48 nfs-client /netbsd: nfs server
nfs-server:/media/data/netbsd/pkgsrc: is alive again
Mar 14 21:13:33 nfs-client /netbsd: nfs server
nfs-server:/media/data/netbsd/current/obj: not responding
Mar 14 21:14:49 nfs-client /netbsd: nfs server
nfs-server:/media/data/netbsd/current/obj: is alive again

is this with DomUs? if so, is it just the one of them?

what does netstat -m show? are you using NIS/YP?

is it possible for you to try FFS as opposed to LFS?

>  Sarton
>



-- 
Best regards,
 Brad