PCI passthrough, SR-IOV, MSI-X

[Brian Marcotte <marcotte%panix.com@localhost>]

I have some network cards which support SR-IOV (ixv, mcx). I've been
attempting to pass through one of the virtual interfaces to a NetBSD
guest without success. The NetBSD guest can send, but not receive.

Some debugging information is below.

I remembered that NetBSD/xen has issues with MSI-X and a workaround
can be to disable MSI-X in the driver, but the drivers for ixv and mcx
suggest MSI-X is required.

I've tried various workarounds including:

  PV 	                    - wouldn't compile after I added PCI devices
  Pure HVM                  - no change
  -current                  - no change
  pciback "permissive mode" - no change

Is there any way to make this work right now? This would be a
significant performance win if it could be made to work.

Thanks.

- Brian

---------------------------------------------------------------------
Xen 4.19, dom0 is Linux. Linux guest works fine.

QEMU log:

[00:04.0] Write-back to unknown field 0x9c (partially) inhibited (0xffff0000)
[00:04.0] If the device doesn't work, try enabling permissive mode
[00:04.0] (unsafe) and if it helps report the problem to xen-devel
[00:04.0] xen_pt_msix_update_remap: Error: unbind MSI-X entry 103 failed (err: 22)
[00:04.0] [last line repeats many times]

NetBSD guest messages:

[   1.0264498] ixv0 at pci0 dev 4 function 0: Intel(R) PRO/10GbE Virtual Function Network Driver, Version - 2.0.1-k
[   1.0264498] ixv0: clearing prefetchable bit
[   1.0264498] ixv0: device 82599 VF
[   1.0264498] ixv0: Mailbox API 1.3
[   1.0264498] ixv0: Using MSI-X interrupts with 3 vectors
[   1.0264498] ixv0: for TX/RX, interrupting at msix0 vec 0, bound queue 0 to cpu 0
[   1.0264498] ixv0: for TX/RX, interrupting at msix0 vec 1, bound queue 1 to cpu 1
[   1.0264498] ixv0: for link, interrupting at msix0 vec 2, affinity to cpu 2
[   1.0264498] ixv0: Ethernet address a6:cf:c3:f0:8d:19