FreeBSD HVM+PV domU hangs on boot -- but not if using file-backed storage

[Hugo Silva <hugo%barafranca.com@localhost>]

Hello everyone,

Got another Zen mystery to share with the bunch :-)

Some time ago I noticed that my FreeBSD HVM domUs were hanging on boot 
if using the XENHVM (+PV drivers) kernel. I posted on the freebsd-xen 
list and no one else seemed to be suffering the same fate; this prompted 
some more investigation.

I dd'd a file and reinstalled a pristine FreeBSD 8/amd64 there, then 
compiled the XENHVM kernel, then rebooted; The result is below:

----
Using 'file:/usr/ftest,hda,w':

(...)
GEOM: new disk ad0
ad0: Intel check1 failed
ad0: Adaptec check1 failed
ad0: LSI (v3) check1 failed
ad0: LSI (v2) check1 failed
ad0: FreeBSD check1 failed
ata1: Identifying devices: 00010000
ata1: New devices: 00010000
ata1-master: pio=PIO4 wdma=WDMA2 udma=UDMA100 cable=40 wire
acd0: setting PIO4 on PIIX3 chip
acd0: setting WDMA2 on PIIX3 chip
acd0: <QEMU CD-ROM/0.9.0> CDROM drive at ata1 as master
acd0: read 689KB/s (689KB/s), 512KB buffer, WDMA2
acd0: Reads:
acd0: Writes:
acd0: Mechanism: ejectable tray, unlocked
acd0: Medium: no/blank disc
xbd0: 2000MB <Virtual Block Device> at device/vbd/768 on xenbus0
xbd0: attaching as ad0
GEOM: new disk ad0
xbd1: 658MB <Virtual Block Device> at device/vbd/5632 on xenbus0
xbd1: attaching as ad2
xn0: <Virtual Network Interface> at device/vif/0 on xenbus0
xn0: bpf attached
xn0: Ethernet address: 00:16:3e:fa:42:03
ATA PseudoRAID loaded
WARNING: WITNESS option enabled, expect reduced performance.
GEOM: new disk ad2
GEOM: ad0s1: geometry does not match label (16h,63s != 64h,63s).
Trying to mount root from ufs:/dev/ad0s1a
WARNING: / was not properly dismounted
ct_to_ts([2010-05-27 08:52:53]) = 1274950373.000000000
start_init: trying /sbin/init
WARNING: /usr was not properly dismounted
WARNING: /var was not properly dismounted
re0: link state changed to UP

FreeBSD/amd64 (Amnesiac) (cuau0)

login: root
Password:
Last login: Wed May 26 19:18:17 on cuau0
Copyright (c) 1992-2009 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
       The Regents of the University of California. All rights reserved.

FreeBSD 8.0-RELEASE (XENHVM) #0: Wed May 26 19:16:42 UTC 2010
----


Odd. Since I use LVM by default these days, I had never noticed that the 
domU would boot if using file-backed storage.
Now I changed the config to use a logical volume *:

* after creating a lvm and dd'ing the contents of the file on the 
previous step to it*
** same thing happens if you install directly to a lv.

----
Using 'phy:/dev/mapper/rvg1-ftest,hda,w':

(...)
ad0: setting PIO4 on PIIX3 chip
ad0: setting WDMA2 on PIIX3 chip
ad0: 4096MB <QEMU HARDDISK 0.9.0> at ata0-master WDMA2
ad0: 8388608 sectors [8322C/16H/63S] 16 sectors/interrupt 1 depth queue
GEOM: new disk ad0
ad0: Intel check1 failed
ad0: Adaptec check1 failed
ad0: LSI (v3) check1 failed
ad0: LSI (v2) check1 failed
ad0: FreeBSD check1 failed
ata1: Identifying devices: 00010000
ata1: New devices: 00010000
ata1-master: pio=PIO4 wdma=WDMA2 udma=UDMA100 cable=40 wire
acd0: setting PIO4 on PIIX3 chip
acd0: setting WDMA2 on PIIX3 chip
acd0: <QEMU CD-ROM/0.9.0> CDROM drive at ata1 as master
acd0: read 689KB/s (689KB/s), 512KB buffer, WDMA2
acd0: Reads:
acd0: Writes:
acd0: Mechanism: ejectable tray, unlocked
acd0: Medium: no/blank disc
( nothing happens anymore )
----


Something seems to be going on, but top isn't helping:
CPU states:  0.0% user,  0.0% nice, 20.5% system,  2.5% interrupt, 77.0% 
idle
Memory: 87M Act, 24K Wired, 9456K Exec, 40M File, 330M Free
Swap: 2048M Total, 2048M Free
 PID USERNAME PRI NICE   SIZE   RES STATE      TIME   WCPU    CPU COMMAND
   0 root      96    0     0K   36M mcpair     2:01  0.00%  0.00% [system]
1555 root      85    0    61M   15M aiowork    0:03  0.00%  0.00% qemu-dm
1678 root      43    0    15M 1544K CPU        0:00  0.00%  0.00% top
 306 root      85    0   128M   18M select     0:00  0.00%  0.00% python2.6
 298 root      85    0   108M 4756K wait       0:00  0.00%  0.00% python2.6


While it says 20.5% CPU is being used by the kernel, all processes show 
0.00%. hmm.
xm top says the dom0 is using ~29% CPU and the FreeBSD domU is always 
around 10%

I've let it run like this for as much as 40 minutes in several 
occasions, never getting to the login prompt. Also attached a vnc, just 
in case.. it's also stuck there.


---- begin domU.conf ----
import os, re
arch = os.uname()[4]
if re.search('64', arch):
   arch_libdir = 'lib64'
else:
   arch_libdir = 'lib'

kernel = "/usr/pkg/lib/xen/boot/hvmloader"

builder='hvm'
memory=256
shadow_memory=8
name="ftest"

vif = [ 'mac=00:16:3e:fa:42:03, bridge=bridge0' ]

disk =  [
   # 'file:/usr/ftest,hda,w',
   'phy:/dev/mapper/rvg1-ftest,hda,w',
   # 'file:/usr/pkg/etc/xen/iso/8.0-RELEASE-amd64-disc1.iso,hdc:cdrom,r'
   # 'file:/usr/pkg/etc/xen/iso/8.0-RELEASE-amd64-livefs.iso,hdc:cdrom,r'
   ]

boot="c"

vnc = 1
sdl = 0

vncconsole=0
vncdisplay=0

cpus="1"
vcpus=1
keymap="pt"

acpi=1
apic=1
device_model = '/usr/pkg/libexec/qemu-dm'
stdvga=0
serial='pty'
--- end domU.conf ----


I am not entirely sure this is a LVM problem - maybe physical 
disks/partitions also fail to work with FreeBSD+HVM+PV. All my disks are 
taken for raids, so I was unable to test this possibility. If you're 
willing to do so, you can get the ISO here: 
http://darkstar.ist.utl.pt/freebsd/8.0/amd64/8.0-RELEASE-amd64-disc1.iso. 
The steps are:

- Install FreeBSD, making sure you install the sources
- cd /usr/src && make kernel KERNCONF=XENHVM && shutdown -r now

It doesn't mean much but I was able to run HVM+PV FreeBSD many months 
ago on a Debian dom0, also with lvm-backed storage. I can't test that 
now either, as the machine happily runs NetBSD -current these days.


The only remotely suspect message I was able to find while booting the 
domUis:
xbdback backend/vbd/3/768: unknown device 0xc207 (major=194)

I checked, and:
- I don't get this message on a NetBSD PV domU
- I don't get it either if using file-backed storage on the FreeBSD domU

Suggestions?