netbsd dom0 + Xen 4.1 + netbsd pv domU + xl => works! (with caveat)

[Toby Karyadi <toby.karyadi%gmail.com@localhost>]

Hi,

I just found out you can run a netbsd pv domU using the 'xl' command (vs 
'xm'). hvm domu + xl has never been a problem. The caveat is this: the 
disk(s) has to be of 'phy' type, as opposed to 'file' type. That is, in 
the domU config file, instead of defining disk as such:
    disk = [ 'file:/mnt/v01/vhosts/dadomu01/dadomu01.disk0.img,hda,w ]
the disk can be defined as such:
    disk = [ 'phy:/dev/vnd0d,hda,w' ]
where vnd0 has been vnconfiged to point to the dadomu01.disk0.img file 
above. I've also tested the config using an lvm logical volume and it 
worked fine, and I would assume that any other physical device, like a 
dk wedge would work as well.

I'm probably not the first one to find this out, however, I think it is 
important to bring this up, especially since the last information that 
I'm aware of on Xen 4, domU, and xl was this: 
http://mail-index.netbsd.org/port-xen/2011/04/01/msg006575.html.

My understanding on the problem with netbsd dom0 + file based disk image 
+ xl is that we don't have support for blktap and in that case xl will 
start qemu-dm. In my case qemu-dm wasn't able to setup the disk image 
file as a device (don't know why). The domU was shown to be in the 'b' 
state, but I was unable to connect via console and the network was not 
up either. This thread at the Xen ml gave me the idea of using phy disk 
type: 
http://lists.xen.org/archives/html/xen-devel/2010-12/msg01217.html. But 
this is from Dec of 2010, I wonder what has changed since then.

Also, AFAIK, with xm and netbsd, the xenbackendd will use the 
/usr/pkg/etc/xen/scripts/block sh script to setup the vnd device. I 
wonder why the block script doesn't get executed when using xl. I guess 
this is a part of xm + xend, since xl bypasses xend altogether. I don't 
think we should go through the route of using qemu-dm since that's 
another user process and I'd imagine the context switching would 
dramatically affect disk io performance and plus we already have an 
excellent kernel space vnd device. I think it's a matter of fixing some 
glue logic, but what do I know. Should I file a PR to track this?

Btw, thanks to all of who have been working on improvements to netbsd + 
xen, such as lvm and smp support in netbsd pv domu.

Cheers,
Toby

PS: details:

netbsd version: cvs co -D 2012.03.16.23.20.00 src
pkgsrc version: HEAD, about a day prior to pkgsrc-2012Q1
xenkernel41
xentools41

pv domu config:

name = "dadomu01"
kernel = "/mnt/v01/vhosts/dadomu01/netbsd-XEN3_DOMU.gz"
memory = 128
vcpus = 2
#disk = [ 'file:/mnt/v01/vhosts/dadomu01/dadomu01.disk0.img,hda,w',]
#disk = [ 'phy:/dev/dk1,hda,w' ]
#disk = [ 'phy:/dev/mapper/vg00-lv01,hda,w']
disk = [ 'phy:/dev/vnd0d,hda,w']
vif = [ 'mac=00:16:3e:55:cd:aa,bridge=bridge0' ]
on_reboot = 'restart'
on_crash = 'destroy'