Re: sun4v trap table

[Eduardo Horvath <eeh%NetBSD.org@localhost>]

On Fri, 10 Oct 2025, Sad Clouds wrote:

> On Thu, 9 Oct 2025 18:53:48 +0000 (UTC)
> Eduardo Horvath <eeh%NetBSD.org@localhost> wrote:
> 
> > 
> > Don't look at the Solaris, Illuminos, or Linux kernels.  They use a 
> > completely different trap and address space model from NetBSD.
> > 

> Maybe ddb (which itself relies on traps) cannot stop at breakpoints in
> trap handlers?

You can't use DDB on trap code.  That's what TRAPTRACE is for.

> 
> > Remember, sun4u can use all 4 trap levels.  sun4v can only use 2 because 
> > the other 2 are reserved for the hypervisor.  On sun4v the hypervisor and 
> > LDOMs are always active, whether configured or not.
> > 
> 
> I'm confused about how the transition between Trap Levels (TLs)
> happens. Hardware documentation gives this table for sun4v TLs usage:
> 
> TL Exec Mode        Usage
> ---------------------------------------------------------------------------
> 0  Nonprivileged    Normal execution
> 1  Privileged       System calls; interrupt handlers; instruction emulation
> 2  Privileged       Window spill/fill handler
> 3  Hyperprivileged  Real address TLB miss handler
> 4  Hyperprivileged  Reserved for error handling
> 5  Hyperprivileged  RED_state handler

Ignore that.  Those are just guidelines.

Trap levels are determined by the %tl register,  Whenever a trap is taken 
it is inremented.  Trap returns decrement it.

I beleive the hypervisor trap levels are entered through a hypercall.  But 
once again, ignore all that.  Unlike x86 or ARM, consider  all of the 
hypervisor operation as just part of the H/W.

> So let's assume a user process makes a function call which results in a
> window spill trap. The user process is running at TL=0, but the
> documentation states that TLs are incremented sequentially, so how does
> it get to TL=2 as stated in the table above? It would be running at TL=1.

That is misleading.  Window trap handlers run at TL=1 unless they occur 
during trap processing when the CPU is already at TL=1.

> 
> When I look at NetBSD sun4u spill trap handlers, these are virtually
> the same for TL=0 and TL>0. I assume the "supervisor mode" is a synonym
> for the "nucleus mode".

Nucleus mode always uses TLB context 0.

> 
> /* Traps from TL=0 -- traps from user mode */
> TABLE(uspill):
>   SPILL64(uspill8,ASI_AIUS)   ! 0x080 spill_0_normal -- used to save user windows in user mode
>   SPILL32(uspill4,ASI_AIUS)   ! 0x084 spill_1_normal
>   SPILLBOTH(uspill8,uspill4,ASI_AIUS)  ! 0x088 spill_2_normal
>   UTRAP(0x08c); TA32          ! 0x08c spill_3_normal
> TABLE(kspill):
>   SPILL64(kspill8,ASI_N)      ! 0x090 spill_4_normal -- used to save supervisor windows
>   SPILL32(kspill4,ASI_N)      ! 0x094 spill_5_normal
>   SPILLBOTH(kspill8,kspill4,ASI_N) ! 0x098 spill_6_normal
>   UTRAP(0x09c); TA32          ! 0x09c spill_7_normal
> TABLE(uspillk):
>   SPILL64(uspillk8,ASI_AIUS)  ! 0x0a0 spill_0_other -- used to save user windows in supervisor mode
>   SPILL32(uspillk4,ASI_AIUS)  ! 0x0a4 spill_1_other
>   SPILLBOTH(uspillk8,uspillk4,ASI_AIUS) ! 0x0a8 spill_2_other
> ...
> /* Traps from TL>0 -- traps from supervisor mode */
> TABLE(uspill):
>   SPILL64(1,ASI_AIUS)         ! 0x080 spill_0_normal -- used to save user windows
>   SPILL32(2,ASI_AIUS)         ! 0x084 spill_1_normal
>   SPILLBOTH(1b,2b,ASI_AIUS)   ! 0x088 spill_2_normal
>   UTRAP(0x08c); TA32          ! 0x08c spill_3_normal
> TABLE(kspill):
>   SPILL64(1,ASI_N)            ! 0x090 spill_4_normal -- used to save supervisor windows
>   SPILL32(2,ASI_N)            ! 0x094 spill_5_normal
>   SPILLBOTH(1b,2b,ASI_N)      ! 0x098 spill_6_normal
>   UTRAP(0x09c); TA32          ! 0x09c spill_7_normal
> TABLE(uspillk):
>   SPILL64(1,ASI_AIUS)         ! 0x0a0 spill_0_other -- used to save user windows in nucleus mode
>   SPILL32(2,ASI_AIUS)         ! 0x0a4 spill_1_other
>   SPILLBOTH(1b,2b,ASI_AIUS)   ! 0x0a8 spill_2_other
> ...
> 
> So how do they differ between various TLs?
> 
> When TL=0:
>   uspill:  spill windows when in user address space?
> 
>   kspill:  spill windows when in kernel address space?
> 
>   uspillk: spill windows for user space process when in kernel space?
>            When would this happen? User process making a syscall which
>            then results in that syscall running in kernel address space
>            but needing to spill windows for user process?
> 
> When TL>1:
>   These are the same, so when would these traps occur? Are these nested
>   traps, e.g. a user process causes a trap (syscall, pagefault, etc)
>   which then causes another trap related to window spill?

The comments there originally came from the v9 manual and are probably a 
bit misleading.

What this comes down to are two things:

Is the stack frame 32-bit or 64-bit or do you need to figure that out in 
the trap handler from the stack pointer?

Does the stack frame need to be saved to the user address space or kernel 
address space?

The fist is determined by the macro, the second by the ASI.

In SPARC v7 there was only one trap level and you are not allowed to take 
a trap in a trap handler or you get a watchdog reset.  So window trap 
handlers had to make sure the destination was mapped in the MMU before 
accessing stack frame memory.

In v8 you can take a trap while handling a trap so you don't need to check 
MMU mappings before spilling a window,  You just take a TL>0 MMU trap.

TL=0 traps happen during normal execution.  TL>0 traps happenwhile already 
taking a trap, say a window spill causes a MMU miss,  They are usually 
triggerd by a savewin instruction in a trap hander that needs to enter the 
kernel and do something.

Solaris will typically be executing user code, take a window spill trap, 
then take a MMU miss trap, return to the MMU miss trap, then return to 
userland.

NetBSD will take a window trap, then a MMU miss trap, clean up the whole 
mess and directly return to userland.

Eduardo