Sun3 MMU design memo

To: port-sun3%netbsd.org@localhost
Subject: Sun3 MMU design memo
From: Izumi Tsutsui <tsutsui%ceres.dti.ne.jp@localhost>
Date: Sat, 30 Dec 2023 04:51:04 +0900
I'm not sure if there is any official/unofficial documents
about Sun proprietary MMU used on sun3 (not sun3x) machines,
but I would like to post my old "sun3 MMU design memo"
I surveyed back in 2020 during investigating PR/55990
so that this will be archived under NetBSD.org hosts.

All descriptions are quite dumb and still incomplete,
but I hope this would be worth for future archaeologists.


--- Cut here ---

Sun3 (not sun3x) MMU design memo
================================

Written by Izumi Tsutsui (20200812)

Quick reversed from NetBSD/sun3 src/sys/arch/sun3/sun3/pmap.c implementation.

VA space
--------

0x00000000 - 0x0FFFFFFF (256MB)
+-------------------------------------------------------------------------------------------------------------------------------+
| 31| 30| 29| 28| 27| 26| 25| 24| 23| 22| 21| 20| 19| 18| 17| 16| 15| 14| 13| 12| 11| 10|  9|  8|  7|  6|  5|  4|  3|  2|  1|  0|
+---------------+-----------------------------------------------------------+---------------------------------------------------|
| (should be 0) | <--------- segment map number ----------> | <- PTE num -> | <------------  page offset (PGOFSET) -----------> |
+-------------------------------------------------------------------------------------------------------------------------------+


PA space
--------

0x00000000 - 0xFFFFFFFF (whole 32bit 4GB?)
+-------------------------------------------------------------------------------------------------------------------------------+
| 31| 30| 29| 28| 27| 26| 25| 24| 23| 22| 21| 20| 19| 18| 17| 16| 15| 14| 13| 12| 11| 10|  9|  8|  7|  6|  5|  4|  3|  2|  1|  0|
+---------------+-----------------------------------------------------------+---------------------------------------------------|
| <--------------       page frame number (PG_FRAME)        --------------> | <------------  page offset (PGOFSET) -----------> |
+-------------------------------------------------------------------------------------------------------------------------------+


context (CONTEXT)
------------------

Maybe introduced to avoid extra flush ops against pmap and cache on switching userland processes

- 8 hardware "context"s
- switched by writing context number at CONTEXT_REG (0x30000000) with SFC/DFC=0x3
  (see sun3/control.c and sun68k/ctrlsp.S)
  - no other valid address?

- pmap.c comment says:
 * In this pmap design, the kernel is mapped into all contexts
 * Processes take up a known portion of the context
 * Processes compete for the available contexts on a LRU basis
 * each "context"'s address space is defined by the 2048 one-byte entries in the segment map.
 * kernel ptes are in all contexts, and are always in the mmu

- context 0 (EMPTY_CONTEXT) is used for kernel_pmap (in current pmap.c implementation)
- context 1 - 7 are used for userland pmap and managed by context_free_queue TAILQ 
 - see also pmap.c comments for future "projects"

- All PTEs are corresponding to contexts
 - kernel PTEs are mapped to ALL context via set_segmap_allctx() in locore.s

- per implementation of set_segmap_allctx():
 * write context number to CONTEXT_REG
 * write SME to (SEGMAP_BASE | (va & CONTROL_ADDR_MASK))
 * loop above through all context (i.e. 7 to 0)
 -> This also implies the current context number infomation is also stored
    (or used to choose hardware SEGMAP entry) on writing SME to SEGMAP space, 

- cache.c also has `cache_flush_context()` function
 - (probably) VA cache also records and refers context number of the VA and hits only if the VA is in the same context

- allocated by context_allocate() for each userland mappings in pmap_enter_user() and static pmap_fault_reload()
- freed by context_free() in pmap_destroy() via static pmap_release()


segment map (SEGMAP)
--------------------

PV mapping management mechanism per segement basis.

- 2048 one-byte entries per each context
 - corresponding to VA bit 27-17

- Accessed via address space 0x20000000 - 0x2FFFFFFC with SFC/DFC=0x3
  (see sun3/control.c and sun68k/ctrlsp.S)
 - set_segmap() and get_segmap() functions are used to access
 - Maybe lower address bits (16-0) are ignored
 - Probably "current" context number is implicitly decoded to choose SEGMAP per context

- Each segment map entry has one byte "SME" (segment map entry?) corresponding to "PMEG" number
 - actually "SME" index (segment number) is calculated from va using VA_SEGNUM()

```
/* pmap3.h */
#define SEGSHIFT        17              /* LOG2(NBSG) */

/* pte3.h */
#define VA_SEGNUM(x)    ((u_int)(x) >> SEGSHIFT)
```


Page Map Entry Group (PMEG)
---------------------------

- pmap.c comment says:
 * sun3s also have this evil "PMEG" crapola
 * PMEG contains the mappings for that virtual segment
 * Each PMEG maps a segment of 128Kb length, with 16 pages of 8Kb each.

- Up to 255 entries
 - PMEG number 255 is used for `SEGINV` (so actually 254 entries are available?)

This means:
- 2048 * 8 segment maps are necessary to handle whole 256MB VA space for all contexts
- Only 256 (or 255) sets of hardware (= PMEGs) are available in MMU to handle actual PV mappings per each segment map
as another pmap.c comment says:
 * As you might guess, these PMEGs are in short supply and heavy demand.
 * PMEGs allocated to the kernel are "static" in the sense that they can't be stolen from it.
 * PMEGs allocated to a particular segment of a pmap's virtual space will be fought over by the other pmaps.


page map (PGMAP)
----------------

Contains page table entries (PTEs).

- Accessed via address space 0x10000000 - 0x1FFFFFFC with SFC/DFC=0x3
  (see sun3/control.c and sun68k/ctrlsp.S)
 - set_pte() and get_pte() functions are used to access

- VA (i.e. segment map number) is used to specify PTE address
 - Probably PMEG hardware looks up "SME" number set by set_segmap() from segment map number in VA
   and update PTEs in the selected "PMEG" implemented as MMU hardware
 - Maybe "PTE num" bits (16-13) in VA are also used to specify PTE number (0-15) in each PMEG hardware

```
#define VA_PTE_NUM_SHIFT  13
#define VA_PTE_NUM_MASK (0xF << VA_PTE_NUM_SHIFT)
#define VA_PTE_NUM(va) (((va) & VA_PTE_NUM_MASK) >> VA_PTE_NUM_SHIFT)
```

page table entry (PTE)
----------------------

32 bit PTE
- per definitions in sun3/include/pte3.h:

+-------------------------------------------------------------------------------------------------------------------------------+-
|       31      |       30      |       29      |       28      |       27      |       26      |       25      |       24      |
+---------------------------------------------------------------+-------------------------------+-------------------------------+-
|                            PG_PERM                            |           PG_TYPE             |           PG_MODREF           |
+---------------------------------------------------------------+-------------------------------+-------------------------------+
|    PG_VALID   |   PG_WRITE    |   PG_SYSTEM   |    PG_NC      |  (OBMEM/OBIO/VME_D16/VMED32)  |     PG_REF    |     PG_MOD    |
+-------------------------------------------------------------------------------------------------------------------------------+-

-+-----------------------------------------------------------------------------------------------+
 | 23| 22| 21| 20| 19| 18| 17| 16| 15| 14| 13| 12| 11| 10|  9|  8|  7|  6|  5|  4|  3|  2|  1|  0|
-+-------------------+---------------------------------------------------------------------------|
 |<--- unused ? ---> | <--------------       page frame number (PG_FRAME)        --------------> |
-+-----------------------------------------------------------------------------------------------+

(BTW, all pte variables in pmap.c should be declared as unsigned, i.e. uint32_t rather than int?)


PMEG management in pmap implementation
--------------------------------------

```
#define NPMEG   256
```
```
struct pmeg_state {
        TAILQ_ENTRY(pmeg_state) pmeg_link;	/* opaque to handle PMEGs in TAILQs (free / inactive / active / kernel (wired)) */ 
        int            pmeg_index;		/* index # of NPMEG (0-255) */
        pmap_t         pmeg_owner;		/* which pmap owns this pmeg (kernel_pmap or user pmaps) */
        int            pmeg_version;		/* copy of pmap->pm_version; incremented on each pmap_create(9) call */
        vaddr_t        pmeg_va;			/* which VA where this PMEG belongs */
        int            pmeg_wired;		/* bitmap info of wired mapping for each PTE (0-15) XXX: should be unsigned? */
        int            pmeg_reserved;		/* indicates reserved PMEG (for PROM mappings etc.) */
        int            pmeg_vpages;		/* a number of valid PV mappings */
        int            pmeg_qstate;		/* which TAILQ this PMEG belongs (PMEGQ_FREE / PMEGQ_INACTIVE / PMEGQ_ACTIVE / PMEGQ_KERNEL / PMEGQ_NONE) */
};
```

Consideration
-------------

I'm afraid it is a bit hard to implement wired map handling due to limited number of context..
 - Should we keep all context and pmeg which contain PTE mapped as wired??
 - Maybe I should learn and clarify definition and behavior of "wired" in MD pmap and MI uvm implementation..
   (keep P-V mappings in any case??)

- Note current `pmeg_wired` in struct pmeg_state seems only used for accounting in pmap_wired_pages(9)
  (as exported pmap_wired_count(9) macro)

Random memo
-----------

- context_allocate() / context_free() pair
- pmeg_allocate() / pmeg_free() pair
- pmeg_release() is used to move pmeg from active TAILQ to inactive TAILQ as cached entries
  when context is stolen by other process (see pmap.c comments)
- pv_link (PV mapping) is only managed for RAM (to handle cache alias?)
- get_pte_pmeg() refers "pmeg_num" as SME number and writes segmap at corresponding VA offset with pmeg_num by set_segmap()
- set_pte_pmeg() vice versa

--- Cut here ---

---
Izumi Tsutsui
Follow-Ups:
- Re: Sun3 MMU design memo
  - From: Romain Dolbeau
Prev by Date: Re: port-sun3/57741: sshd fails to open a PTY on NetBSD/sun3
Next by Date: Re: Sun3 MMU design memo
Previous by Thread: Re: port-sun3/57741: sshd fails to open a PTY on NetBSD/sun3
Next by Thread: Re: Sun3 MMU design memo
Indexes:
Home | Main Index | Thread Index | Old Index