Re: Xen MP panics in cpu_switchto()

[Andrew Doran <ad%netbsd.org@localhost>]

On Mon, Jan 13, 2020 at 07:36:41PM +0100, Manuel Bouyer wrote:

> On Mon, Jan 13, 2020 at 06:33:08PM +0000, Andrew Doran wrote:
> > On Mon, Jan 13, 2020 at 05:43:51PM +0100, Manuel Bouyer wrote:
> > 
> > > On Mon, Jan 13, 2020 at 04:59:50PM +0100, Manuel Bouyer wrote:
> > > > It also sets rsp and rbp. I think rbp is not set by anything else, at last
> > > > in the Xen case.
> > > > The different rbp value would explain why in one case we hit a KASSERT()
> > > > in lwp_startup later.
> > > > But I don't know what pcb_rbp contains; I couldn't find where the pcb for
> > > > idlelwp is initialized.
> > > 
> > > I tried the attached patch, which should set rsp/rbp as cpu_switchto()
> > > does. It doens't cause the lwp_startup() KASSERT as calling cpu_switchto()
> > > does; it also doesn't change the scheduler behavior.
> > 
> > Wait - do you mean that everything works now?  Or that everything still runs
> > on CPU0?
> 
> No, everything still runs on CPU0

Hmm, I don't understand why.  I'll set up Xen and try it out.  It might take
me a day or two.

> > The very first thing that idle_loop() does on amd64/i386 is set up the frame
> > pointer - ebp/rbp.
> > 
> > 0000000000000000 <idle_loop>:
> >    0:   55                      push   %rbp
> >    1:   48 89 e5                mov    %rsp,%rbp
> >    4:   41 56                   push   %r14
> >    6:   41 55                   push   %r13
> 
> OK, so it's OK that my patch doesn't changes anything.
> And so I still don't understand the KASSERT when cpu_switchto() is called
> before idle_loop().

The assertion in lwp_startup() is because I made MI changes so that prevlwp
is never NULL when calling cpu_switchto(), when fixing some bugs problems MP
support on !x86 and make things more correct.  lwp_startup()/mi_switch() now
need to unlock prevlwp after it is finished in cpu_switchto().  I never
expected anybody but mi_switch() to call cpu_switchto().

Andrew