Re: [RESEND] User-level window trap when booting NetBSD kernel under QEMU SPARC64

[Eduardo Horvath <eeh%NetBSD.org@localhost>]

On Mon, 19 May 2014, Mark Cave-Ayland wrote:

> On 19/05/14 17:14, Eduardo Horvath wrote:

> > It's been a while since I last looked at the SPARC V9 manual, but ISTR
> > %wstate register controls which of the window fill/spill traps is taken
> > for regular and "other" states.
> > 
> > You need to dump the contents of %wstate.
> 
> I know that when the fill_0_normal trap is taken on the restore at the end of
> openfirmware(), the QEMU internal variables look like this:
> 
> (gdb) p/x env->cwp
> $7 = 0x3
> (gdb) p/x env->canrestore
> $8 = 0x0
> (gdb) p/x env->cansave
> $9 = 0x6
> (gdb) p/x env->cleanwin
> $10 = 0x7
> (gdb) p/x env->otherwin
> $11 = 0x0
> (gdb) p/x env->wstate
> $12 = 0x0
> (gdb)

A %wstate of 0 is why you're taking the fill_0_normal trap.

*snip*

> > I'm not entirely sure what's going on here since you didn't have symbols
> > in the disassembly and there's no stack trace, but I assume the routine
> > generating the fault is openfirmware() in the kernel.
> 
> Yes, that's correct. Unfortunately I don't have a NetBSD build environment so
> I've been doing most of the work by disassembling the kernel via QEMU's
> gdbstub and comparing against the source in a web browser. So based upon what
> you're saying it appears we have a stack like this (indented to show window
> saves/restores):
> 
>   cpu_initialize() {
>     prom_set_trap_table() {
>       openfirmware() {
>           /* OpenBIOS C code */
>           of_client_interface() {
>              enter_forth() {
>                set_trap_table() {
>                   SUNW,set-trap-table
>                }
>              }
>           }
>       }
>       /* fill_0_normal trap occurs here */
>     }
> 
>     /* Switch to kernel mode */
>     wrpr %g0, WSTATE_KERN, %wstate
>   }
> 
> The assumption has to be that in order for this to work without errors then no
> window/fault traps can occur between calling SUNW,set-trap-table in OpenBIOS
> and getting back to cpu_initialize() to set the correct value for %wstate
> which is quite a few window levels. AIUI data faults can't happen because the
> ASI is set to 0x82 (no fault) which is why it is the fill_0_normal window
> fault which is triggering this.
> 
> I'm starting to wonder if setting %wstate to use trap vector 6 should happen
> *before* calling prom_set_trap_table()? At the point SUNW,set-trap-table is
> called then the kernel is effectively saying "I am taking responsibility for
> handling all traps from now on", and so if the kernel cannot handle traps
> after this point for any reason, then it is not honouring its contract to
> manage the trap table.

The problem comes down to setting %wstate and callng SUNW,set-trap-table 
really needs to be an atomic operation, which is not really possible.

> Regardless of this, now I understand this further I need to look into the
> OpenBIOS CIF interface in order to see if I can preserve the entire window
> state across CIF calls which I suspect might be what Sun's OBP does. Otherwise
> it would not be possible to run many versions of NetBSD (and OpenBSD which
> suffers from the same problem) under emulation :/

Yes, OBP preserves the window state on entry and restores it on exit.  It 
also written in Forth, so doesn't use any register windows internally.  
It issues one `save' instruction on entry to get a set of registers to use 
for the Forth engine and one `restore' when the CIF returns.

If OpenBIOS uses register windows, things will get really complicated.  
You enter the CIF handler with the PROM's register window settings, but 
return with the kernel's settings.  

Looking at locore.s:

1:

        /* set trap table */
#ifdef SUN4V
        cmp     %l6, CPU_SUN4V
        bne,pt  %icc, 6f
         nop
        /* sun4v */
        set     _C_LABEL(trapbase_sun4v), %o0
        GET_MMFSA %o1
        call    _C_LABEL(prom_set_trap_table_sun4v)     ! Now we should be 
running 100% from our handlers
         nop

        ba      7f
         nop
6:
#endif
        /* sun4u */
        set     _C_LABEL(trapbase), %l1
        call    _C_LABEL(prom_set_trap_table_sun4u)     ! Now we should be 
running 100% from our handlers
         mov    %l1, %o0
7:
        wrpr    %l1, 0, %tba                    ! Make sure the PROM 
didn't foul up.

        /*
         * Switch to the kernel mode and run away.
         */
        wrpr    %g0, WSTATE_KERN, %wstate


So the kernel calls prom_set_trap_table_sun4u() and then immediately sets 
the %tba address itself.  You could try moving that call to after the 
register window state has been set up so you call 
prom_set_trap_table_sun4u() with the kernel trap table and the kernel 
window state.  I don't know what either OpenBIOS or OBP would do in that 
situation.

Otherwise, you can just skip the prom_set_trap_table_sun4u() call.  I 
added the SUNW,set-trap-table call to make sure OBP was aware the trap 
table changed in case it needed to do something funny.

Option 2 is to rewrite the OpenBIOS to have SUNW,set-trap-table just 
execute the load of the %trapbase register and not do any other fancy stuff.

Option 3 is to save and restore the entire window state on CIF entry and 
exit.  This will be a pain in the neck because you not only need to save 
all the regiser window control registers, you also need to cycle through 
and save or restore any dirty register windows contents.  Or I suppose you 
could always save all the contents of all the register windows.  This will 
slow down CIF entry and exit.


Eduardo