domU's report "Out of memory allocating ksiginfo for pid ###"

[Mike Bowie <mbowie%buzmo.com@localhost>]

Good day folks... long time listener, first time caller.

First up, I'd like to praise all of those involved in the NetBSD xen 
project... I've not been a NetBSD user in the past, but the experience 
I've had with the project thus far has definitely ensured I will be in 
the future.  The cohesion between the OS and the xen kernels etc is 
simply superb and makes Good Logical Sense(tm)... which really is a 
welcome change.  All in all, *very* impressive.

I have a Dell PowerEdge 1950-III which I've been working at getting 
online with xen.  The ultimate goal is to bring four matching machines 
into production in this same configuration.  I had hoped to do so with 
4.0, however the mfi driver for the PERC 6/i (which features in this 
box) wasn't ported from OpenBSD until February.  I'd also looked at 
installing various incarnations of HEAD, however the last one I managed 
to install had trouble with the bnx driver core-dumping and the ISOs for 
the past week or two all core-dump when loading sysinst.

So, I've cvsup'd a source tree from late February, when the mfi driver 
gained support for the PERC device and rolled my own ISOs and kernels as 
4.99.54.

Having tooled around with this for a while now, I have a fairly happy 
dom0 up and running with xen 3.1.4 build from relatively current pkgsrc. 
 I did have some trouble with dom0's RAM allocation, and admittedly 
have given it a whole gigabyte at this point... not ideal, but it is 
working. http://mail-index.netbsd.org/port-xen/2008/04/21/msg003604.html 
references problems similar to those I was seeing, however other issues 
such as the bnx driver stalling and xend failing to start were resolved 
by adding more (and more) RAM to dom0.  (I started @ 64MB and doubled 
incrementally from there.)

So to go along with my nice dom0, I've installed a couple of NetBSD 
domU's, also running 4.99.54... and this is where my "real trouble" 
starts.  If I perform any "strenuous" activity on either domU it 
consumes 100% of it's RAM and 100% of it's available CPU time.  If I'm 
on a xen console and I send it a couple of cr's (and I'm lucky) it will 
output "Out of memory allocating ksiginfo for pid ###".  Ultimately, all 
I can do is "xm destroy" the domU and start it again.

If they're idle or doing little, there's no problem... but if I build a 
large package, install a large package or cvsup, they'll top out and 
choke.  One domU has just 128MB of RAM, while the other has 1GB... the 
former dies much sooner than the latter.

I did ask this question in #netbsd%irc.freebode.net@localhost, and YazzY was kind 
enough to suggest looking into the "elevator" scheduling option, however 
this has not yielded any success or change as of yet.

YazzY also suggested that I should prove my 4.99.54 build on a different 
system, which I've not yet done.  It's worth noting that I do not have 
physical access to these machines... they're in a data center some 1500 
miles away.

I'll stop there, before this post becomes a novel... seemingly relevant 
files and output:
http://www.buzmo.com/projects/netbsd-xen/pe1950-build/menu.lst
http://www.buzmo.com/projects/netbsd-xen/pe1950-build/boot.log
http://www.buzmo.com/projects/netbsd-xen/pe1950-build/buzmo0.dfw
http://www.buzmo.com/projects/netbsd-xen/pe1950-build/netbsd4-99-54-amd64.dfw

Any suggestions are sincerely appreciated.

Cheers,

Mike.