"Hotplug scripts not working" again

[Eric Haszlakiewicz <erh%nimenees.com@localhost>]

Previously, I had attempted to get Xen working on one of my machines, and ran 
into problems mostly due to not starting the appropriate daemons.  I'm now 
attempting to setup a domU on another machine, with a newly installed Xen, 
and I can't get it to work using what I thought was the same steps as I
used before.
All of "xend start", xenstored, xenconsoled, and xenbackendd are running, but
I'm still getting errors about not being able to create the vif interfaces.
Any ideas for what I should try?

When I run:
xm create foo
It hangs for a while, then prints:
Error: Device 0 (vif) could not be connected. Hotplug scripts not working.

In the xend.log, the last few lines when it hangs are:
[2009-06-06 21:26:54 12883] DEBUG (DevController:151) Waiting for devices vif.
[2009-06-06 21:26:54 12883] DEBUG (DevController:151) Waiting for devices usb.
[2009-06-06 21:26:54 12883] DEBUG (DevController:151) Waiting for devices vbd.
[2009-06-06 21:26:54 12883] DEBUG (DevController:156) Waiting for 1.
[2009-06-06 21:26:54 12883] DEBUG (DevController:590) hotplugStatusCallback 
/local/domain/0/backend/vbd/9/1/hotplug-status.

but eventually it figures out things aren't working and destroys the domain.

In xend-debug.log I see these lines:
Xend started at Sat Jun  6 21:22:56 2009.
brctl: not found
brctl: not found

The machine has a wm0 network interface.  I created bridge0 with:
ifconfig bridge0 create
brconfig bridge0 add wm0 up
(This step causes all network access to temporarily hang.  Is that normal?)

/usr/pkg/etc/xen/foo:
kernel = "/netbsd-INSTALL_XEN3_DOMU"
memory = 128
name = "foo"
vcpus = 1
vif = [ 'mac=00:16:3e:50:02:f0, bridge=bridge0' ]
#disk = [ 'phy:/dev/wd1d,0x1,w', 'phy:/dev/wd1j,0x2,w' ]
disk = [ 'file:/tmp/xendisk,0x01,w' ]
root = "xbd0"
extra = "-as"

I noticed that when it's hanging, if I run xenstore-ls, that also gets stuck,
but that happens just before /local/domain/0/backend/vbd/13/2/online
"13" isn't the current domain id.  It looks like the entries from the
 previous attempts are still left behind.  Also, depending on where I
tell xenstore-ls to start, the behaviour is different.
e.g. if I do "xenstore-ls /local" it prints:
...other domain/backend/vbd/* entries omitted...
    17 = ""
     1 = ""
      ð »in:

xenstore-ls: xs_directory (/local/domain/0/backend/vbd/17/1/ð »in): Invalid 
argument

but with "xenstore-ls /local/domain" I get this and it hangs:
...
   17 = ""
    1 = ""
     domain = "foo"

Other times when I've run xenstore-ls it's gotten past the vbd section and
failed with this error instead:
    vif = ""
     11 = ""
      0 = ""
       bridge = "bridge0"
xenstore-ls: xs_directory (/local/domain/0/backend/vif/11/0/bridge): Cannot 
allocate memory

Killing and restarting all the xen daemons seems to clean up all the extra
stuff I see in xenstore-ls, but to restart xend I had to use kill -9.

This machines has:
xenkernel3-3.1.4nb3 Xen3 Kernel
xentools3-3.1.4nb4  Userland Tools for Xen
and a 5.0_RC2 XEN3_DOM0 kernel running.

Oh, and I just managed to get xenstore-ls to hang in "D" wait on channel "rplq"
:(  (it un-hung when I restarted xenstored though)