Re: PCI passthru with Xen 3.0, NetBSD 5.1 dom0, Linux domU

[Konrad Rzeszutek Wilk <konrad.wilk%oracle.com@localhost>]

On Fri, Jun 03, 2011 at 02:33:09PM +0200, Julien Oster wrote:
> Hello,
> 
> I recently installed Xen on my NetBSD 5.1 server (now acting as dom0)
> and then Debian (wheezy) as domU. The server contains an Intel Atom
> CPU without HVM, so I used paravirtualization. I simply used the
> xenkernel33 and xentools33 packages and apart from some small problems
> at first (which had nothing to do with NetBSD, though), everything
> turned out fine.
> 
> Because my Atheros AR5008 WiFi PCI card doesn't really work under
> NetBSD, at least not in host mode (frequent timeouts, some devices
> don't associate at all, general flakyness), I decided to give it a try
> under Debian, using Xen PCI passthru.
> 
> This isn't supported with a NetBSD dom0 under Xen 3.3 so I had to
> downgrade to Xen 3.0. (After taking a quick look at the patch that
> enables PCI passthru in NetBSD with Xen 3.0, but it isn't really
> straightforward to adapt to 3.3. I *did* manage to horribly fake the
> PciDevice class, but got the exact same result as with an unfaked but
> patched class using 3.0 in the end.)
> 
> Downgrading also worked well. I didn't have to do much except
> replacing xenkernel33 and xentools33 with xenkernel3 and xentools3, as
> well as uncompressing the Linux kernel image (from vmlinuz to vmlinux,
> otherwise it wouldn't boot).
> 
> So, after setting everything up for PCI passthru
> (pciback.hide=(04:00.0) in the dom0 bootloader, pci = [...] in the
> domain config) I booted up the Debian domU and at first it seemed like
> a success.
> 
> Linux saw the Atheros network card and seemingly initialized it. This
> was evident by some kernel messages, automatic loading of the correct
> modules and the presence of the wlan0 network interface. I could also
> get the interface up with "ifconfig wlan0 up". However, when trying to
> scan for APs with "iwlist wlan0 scan", the kernel generated a lot of
> these messages:
> 
> [  341.696272] ath: Could not kill baseband RX
> [  341.875586] ath: Could not kill baseband RX
> [  341.940445] ath: Failed to stop TX DMA in 100 msec after killing last frame
> [  341.940535] ath: Failed to stop TX DMA!
> [  342.120306] ath: Could not kill baseband RX
> 
> And no result came back. Putting the device into monitor mode and
> listening with tcpdump gave no errors, but no packets either. There
> should have been a metric ton of beacons, at least.
> 
> So I think it's pretty clear that PCI passthru didn't really work in
> this case. The device is seen by domU kernel, but isn't operable.
> Without knowing anything about it, I suspect that either the device's
> registers (memory mapped only, it seems) aren't really accessed or
> that the device's interrupts aren't really seen by the domU kernel.
> 
> Any idea what could be wrong?

Did you set 'iommu=soft' in the Linux kernel? And are you using a kernel
older than 2.6.37 for this?
> 
> I attached the output of "pcictl pci4 dump -d 0" below, which shows
> the PCI configuration registers for that device. I have no experience
> with PCI from anything else than a user's viewpoint, but at first
> glance it seems that the device only uses one memory region and one
> IRQ line, so nothing special I guess?
> 
> Thanks,
> Julien
> 
> 
> PCI configuration registers:
>   Common header:
>     0x00: 0x0023168c 0x02b00216 0x02800001 0x00002010
> 
>     Vendor Name: Atheros Communications (0x168c)
>     Device ID: 0x0023
>     Command register: 0x0216
>       I/O space accesses: off
>       Memory space accesses: on
>       Bus mastering: on
>       Special cycles: off
>       MWI transactions: on
>       Palette snooping: off
>       Parity error checking: off
>       Address/data stepping: off
>       System error (SERR): off
>       Fast back-to-back transactions: on
>       Interrupt disable: off
>     Status register: 0x02b0
>       Capability List support: on
>       66 MHz capable: on
>       User Definable Features (UDF) support: off
>       Fast back-to-back capable: on
>       Data parity error detected: off
>       DEVSEL timing: medium (0x1)
>       Slave signaled Target Abort: off
>       Master received Target Abort: off
>       Master received Master Abort: off
>       Asserted System Error (SERR): off
>       Parity error detected: off
>     Class Name: network (0x02)
>     Subclass Name: miscellaneous (0x80)
>     Interface: 0x00
>     Revision ID: 0x01
>     BIST: 0x00
>     Header Type: 0x00 (0x00)
>     Latency Timer: 0x20
>     Cache Line Size: 0x10
> 
>   Type 0 ("normal" device) header:
>     0x10: 0x48100000 0x00000000 0x00000000 0x00000000
>     0x20: 0x00000000 0x00000000 0x00000000 0x00611737
>     0x30: 0x00000000 0x00000040 0x00000000 0x000001c2
> 
>     Base address register at 0x10
>       type: 32-bit nonprefetchable memory
>       base: 0x48100000, not sized
>     Base address register at 0x14
>       not implemented(?)
>     Base address register at 0x18
>       not implemented(?)
>     Base address register at 0x1c
>       not implemented(?)
>     Base address register at 0x20
>       not implemented(?)
>     Base address register at 0x24
>       not implemented(?)
>     Cardbus CIS Pointer: 0x00000000
>     Subsystem vendor ID: 0x1737
>     Subsystem ID: 0x0061
>     Expansion ROM Base Address: 0x00000000
>     Capability list pointer: 0x40
>     Reserved @ 0x38: 0x00000000
>     Maximum Latency: 0x00
>     Minimum Grant: 0x00
>     Interrupt pin: 0x01 (pin A)
>     Interrupt line: 0xc2
> 
>   Capability register at 0x40
>     type: 0x80 (unknown)
>   Capability register at 0x80
>     type: 0x00 (reserved)
> 
>   Device-dependent header:
>     0x40: 0x00008080 0x00000000 0x00000000 0x00000000
>     0x50: 0x00000000 0x00000000 0x00000000 0x00000000
>     0x60: 0x00000000 0x00000000 0x00000000 0x00000000
>     0x70: 0x00000000 0x00000000 0x00000000 0x00000000
>     0x80: 0x00000000 0x00000000 0x00000000 0x00000000
>     0x90: 0x00000000 0x00000000 0x00000000 0x00000000
>     0xa0: 0x00000000 0x00000000 0x00000000 0x00000000
>     0xb0: 0x00000000 0x00000000 0x00000000 0x00000000
>     0xc0: 0x00000000 0x00000000 0x00000000 0x00000000
>     0xd0: 0x00000000 0x00000000 0x00000000 0x00000000
>     0xe0: 0x00000000 0x00000000 0x00000000 0x00000000
>     0xf0: 0x00000000 0x00000000 0x00000000 0x00000000