PCI passthru with Xen 3.0, NetBSD 5.1 dom0, Linux domU

[Julien Oster <julien%julien-oster.de@localhost>]

Hello,

I recently installed Xen on my NetBSD 5.1 server (now acting as dom0)
and then Debian (wheezy) as domU. The server contains an Intel Atom
CPU without HVM, so I used paravirtualization. I simply used the
xenkernel33 and xentools33 packages and apart from some small problems
at first (which had nothing to do with NetBSD, though), everything
turned out fine.

Because my Atheros AR5008 WiFi PCI card doesn't really work under
NetBSD, at least not in host mode (frequent timeouts, some devices
don't associate at all, general flakyness), I decided to give it a try
under Debian, using Xen PCI passthru.

This isn't supported with a NetBSD dom0 under Xen 3.3 so I had to
downgrade to Xen 3.0. (After taking a quick look at the patch that
enables PCI passthru in NetBSD with Xen 3.0, but it isn't really
straightforward to adapt to 3.3. I *did* manage to horribly fake the
PciDevice class, but got the exact same result as with an unfaked but
patched class using 3.0 in the end.)

Downgrading also worked well. I didn't have to do much except
replacing xenkernel33 and xentools33 with xenkernel3 and xentools3, as
well as uncompressing the Linux kernel image (from vmlinuz to vmlinux,
otherwise it wouldn't boot).

So, after setting everything up for PCI passthru
(pciback.hide=(04:00.0) in the dom0 bootloader, pci = [...] in the
domain config) I booted up the Debian domU and at first it seemed like
a success.

Linux saw the Atheros network card and seemingly initialized it. This
was evident by some kernel messages, automatic loading of the correct
modules and the presence of the wlan0 network interface. I could also
get the interface up with "ifconfig wlan0 up". However, when trying to
scan for APs with "iwlist wlan0 scan", the kernel generated a lot of
these messages:

[  341.696272] ath: Could not kill baseband RX
[  341.875586] ath: Could not kill baseband RX
[  341.940445] ath: Failed to stop TX DMA in 100 msec after killing last frame
[  341.940535] ath: Failed to stop TX DMA!
[  342.120306] ath: Could not kill baseband RX

And no result came back. Putting the device into monitor mode and
listening with tcpdump gave no errors, but no packets either. There
should have been a metric ton of beacons, at least.

So I think it's pretty clear that PCI passthru didn't really work in
this case. The device is seen by domU kernel, but isn't operable.
Without knowing anything about it, I suspect that either the device's
registers (memory mapped only, it seems) aren't really accessed or
that the device's interrupts aren't really seen by the domU kernel.

Any idea what could be wrong?

I attached the output of "pcictl pci4 dump -d 0" below, which shows
the PCI configuration registers for that device. I have no experience
with PCI from anything else than a user's viewpoint, but at first
glance it seems that the device only uses one memory region and one
IRQ line, so nothing special I guess?

Thanks,
Julien


PCI configuration registers:
  Common header:
    0x00: 0x0023168c 0x02b00216 0x02800001 0x00002010

    Vendor Name: Atheros Communications (0x168c)
    Device ID: 0x0023
    Command register: 0x0216
      I/O space accesses: off
      Memory space accesses: on
      Bus mastering: on
      Special cycles: off
      MWI transactions: on
      Palette snooping: off
      Parity error checking: off
      Address/data stepping: off
      System error (SERR): off
      Fast back-to-back transactions: on
      Interrupt disable: off
    Status register: 0x02b0
      Capability List support: on
      66 MHz capable: on
      User Definable Features (UDF) support: off
      Fast back-to-back capable: on
      Data parity error detected: off
      DEVSEL timing: medium (0x1)
      Slave signaled Target Abort: off
      Master received Target Abort: off
      Master received Master Abort: off
      Asserted System Error (SERR): off
      Parity error detected: off
    Class Name: network (0x02)
    Subclass Name: miscellaneous (0x80)
    Interface: 0x00
    Revision ID: 0x01
    BIST: 0x00
    Header Type: 0x00 (0x00)
    Latency Timer: 0x20
    Cache Line Size: 0x10

  Type 0 ("normal" device) header:
    0x10: 0x48100000 0x00000000 0x00000000 0x00000000
    0x20: 0x00000000 0x00000000 0x00000000 0x00611737
    0x30: 0x00000000 0x00000040 0x00000000 0x000001c2

    Base address register at 0x10
      type: 32-bit nonprefetchable memory
      base: 0x48100000, not sized
    Base address register at 0x14
      not implemented(?)
    Base address register at 0x18
      not implemented(?)
    Base address register at 0x1c
      not implemented(?)
    Base address register at 0x20
      not implemented(?)
    Base address register at 0x24
      not implemented(?)
    Cardbus CIS Pointer: 0x00000000
    Subsystem vendor ID: 0x1737
    Subsystem ID: 0x0061
    Expansion ROM Base Address: 0x00000000
    Capability list pointer: 0x40
    Reserved @ 0x38: 0x00000000
    Maximum Latency: 0x00
    Minimum Grant: 0x00
    Interrupt pin: 0x01 (pin A)
    Interrupt line: 0xc2

  Capability register at 0x40
    type: 0x80 (unknown)
  Capability register at 0x80
    type: 0x00 (reserved)

  Device-dependent header:
    0x40: 0x00008080 0x00000000 0x00000000 0x00000000
    0x50: 0x00000000 0x00000000 0x00000000 0x00000000
    0x60: 0x00000000 0x00000000 0x00000000 0x00000000
    0x70: 0x00000000 0x00000000 0x00000000 0x00000000
    0x80: 0x00000000 0x00000000 0x00000000 0x00000000
    0x90: 0x00000000 0x00000000 0x00000000 0x00000000
    0xa0: 0x00000000 0x00000000 0x00000000 0x00000000
    0xb0: 0x00000000 0x00000000 0x00000000 0x00000000
    0xc0: 0x00000000 0x00000000 0x00000000 0x00000000
    0xd0: 0x00000000 0x00000000 0x00000000 0x00000000
    0xe0: 0x00000000 0x00000000 0x00000000 0x00000000
    0xf0: 0x00000000 0x00000000 0x00000000 0x00000000