Re: Dom0 xvif mbuf issues

[Manuel Bouyer <bouyer%antioche.eu.org@localhost>]

On Wed, Sep 26, 2018 at 01:14:40PM -0700, Harry Waddell wrote:
> 
> I have a server where Dom0 started becoming unusable as of a few months ago
> where previously it ran for years with few issues. 
> 
> netbsd-7 branch, never more than a month behind. 
> BRIDGE_IPF is enabled and these options set in sysctl.conf: 
> 
> kern.sbmax=1048576
> net.inet.tcp.recvbuf_max=1048576
> net.inet.tcp.sendbuf_max=1048576
> kern.mbuf.nmbclusters=300000
> kern.maxfiles=3000
> 
> Xen 4.8.3 similarly updated.
> 
> One of the xvif devices "could not allocate a new mbuf". I enabled MBUF debugging
> and netstat didn't seem to point to a leak on any of the devices.

Looks like temporary memory shortage in the dom0 (this is a MGETHDR failing,
not MCLGET, so the nmbclusters limit is not relevant).
How many mbufs were allocated ?


> It hung again, but with a new 
> error scrolling on the console. "xennetback: got only 63 new mcl pages"

This would point to a memory shortage in the hypervisor itself.
Do you have enough free memory (xl info) ?

> 
> My suspicion is that either one of the guests started doing a lot more nfs activity OR
> that a VM I created which uses a fuse filesystem to move large dumpfile to azure blob
> storage may be what pushed this previously working system off the edge. 
> 
> I'm moving the azure fuse system to another server and plan to disable ipf on the bridge. 

ipf shouldn't be a problem, I'm using it extensively on bridges here.

> 
> Beyond that, and any suggestions? Should I just upgrade to netbsd 8 and/or xen 4.11?
> ( even if it's just to make debugging easier since this is where current work is taking place? )

I'm not sure it would change something

> 
> This is a production system with about 30 guests. I just want it to work like it used to. 

how many vifs is there in the dom0 ?

-- 
Manuel Bouyer <bouyer%antioche.eu.org@localhost>
     NetBSD: 26 ans d'experience feront toujours la difference
--