Re: Installer Boot Failure with 10.0_STABLE

[Greg Troxel <gdt%lexort.com@localhost>]

Regarding

  http://gnats.netbsd.org/58395

I am having trouble understanding a few things about this.

As I understand it, grant frames are created when asked for.  So while
NetBSD seeing max_nr_frames=64 makes sense, I don't follow why
nr_frames=32 already.

I have a dom0 running netbsd-10, and domUs running netbsd9 i386 and
netbsd9 amd64 as 1 and 2.  Each domU has a two xbd, one for data and one
for swap, and a single xennet, bridged to the dom0's single physical
interface; they are very straightforward.

  $ for domid in 0 1 2; do xen-diag gnttab_query_size $domid; done
  domid=0: nr_frames=64, max_nr_frames=64
  domid=1: nr_frames=1, max_nr_frames=64
  domid=2: nr_frames=2, max_nr_frames=64

I guess the dom0 must allocate all the frames it can, just because.  And
the domU is only allocating a few.

So it seems, as Manuel has suggested, that the problem is that on
tornadovps's dom0, for reasons that are so far unclear, there is a
configured limit of 64, at least as reported to the domU, but when the
33rd entry is requested, the response is -1 (an invalid address) but an
ok status.  So there is really a limit of 32 before things get buggy.

It also seems that NetBSD until 9, and Linux, only ask for grant table
entries as they need them, rather than getting a lot up front.

I have the impression that Linux guests will blow up at some point from
the same bug, but when they try to get the 33rd entry, not at boot.  And
likely NetBSD 9 guests as well.

So it sounds like:

  The hypervisor should be fixed.

  Until then, the hypervisor should advertise max_nr_frames=32.

  NetBSD 10 should treat a grant value of -1 as invalid, stop iterating,
  skip it, and just set up the previous ones, logging this once.

  NetBSD 10 probably gets all available and then will never try to get
  more, but if it would try to get more, it should be careful in light
  of the previous bullet.


Did I get that right?



Links that might be helpful to people that started out with the level of
clue about grant tables I had before reading this PR and searching:

  https://cristian.livadaru.net/xen-max-grant-frames/
  https://xcp-ng.org/blog/2022/07/27/grant-table-in-xen/