Subject: Alchemy wired memory regions
To: None <port-evbmips@netbsd.org, port-mips@netbsd.org>
From: Garrett D'Amore <garrett_damore@tadpole.com>
List: port-mips
Date: 10/20/2005 11:12:34
I've been thinking about the memory map for Alchemy parts, and I want to 
post my proposal out here for review.  Please let me know your thoughts 
on it.

Alchemy parts have 32 TLB entries.  Each entry can arrange for mappings 
of up to 32MB.

PCI and PCMCIA addresses on Au15XX boards at least are well above the 
32-bit address space.

Here's what I'm thinking:

For PCI configuration space accesses, I can use a single wired entry of 
32MB, and move that wired entry on pci_conf_XXX accesses when I need 
to.  I've already written this and works great.  It consumes 1 wired 
entry if PCI is configured into the kernel.

For PCI memory accesses via the bus_space framework (excluding mmap) I  
am planning on using 32 MB wired entries.  For most purposes, this means 
if you have any devices configured in your kernel that have registers in 
PCI memory space, you are going to consume one more wired entry.  (For 
most purposes, one wired entry will cover pretty much all of the devices 
you have on your system.)

For PCI I/O accesses, I can use the same kind of logic as the PCI 
configuration space (one shared entry that moves), but in reality a 
single 32MB entry should cover all the needs without having to move it.  
This also consumes a wired entry, but only if you have I/O space devices 
configured in the kernel and on your bus.

For PCI framebuffers that do mmap (and any other similiar devices), they 
get a non-wired TLB entry as per normal.

PCMCIA also has three address spaces, unfortunately.   My treatment of 
these would be the same as for PCI, I think, where memory space gets 
wired entries on demand, and I/O and ATTR space each have a single 
shared wired entry.

It is also true that for access that do not have BUS_SPACE_LINEAR 
mappings, I could have a single system wide shared mapping, that is 
moved as needed under protection of a lock.  While this will work, I 
wonder if it will hurt performance more to move this around, as I 
suspect that we may find ourselves flip-flopping back and forth across 
two or more devices in different parts of the address space.

In a fully configured system, with stuff in all address spaces, the 
worst case looks like it will typically be about 6 wired entries (plus 
the one that NetBSD uses for context switching).  If I try to mitigate 
this by sharing, I can save 3 wired entires, at the cost of more 
frequent changes to the TLB.

I guess in theory it is possible to avoid the wired entries altogether 
and use ordinary TLB entries, but this looks a lot harder, and I'm not 
sure what the implications of trying to support register access from 
interrupt context are likely to be.  Frankly, that's an avenue I don't 
want to explore yet, unless someone feels a compelling need for it.  
(And I think not having wired spaces will impact performance too, as 
most devices will be interrupting frequently I think, thereby 
prematurely forcing out TLB entries at random that might otherwise get 
to stay.)

Thanks in advance for your thoughts on this.

    -- Garrett