Simon Burge wrote:
> On Mon, Jan 30, 2006 at 02:49:26PM -0800, Garrett D'Amore wrote:
>
>   
>> Good.  What I will do is go ahead and commit *that* change seperately,
>> and then commit the PCI code which depends on it after I have changed
>> it. :-)
>>
>> Any objections (Simon?)  Speak up now, please!
>>     
>
> My first broad comment on the PCI-specific parts was the name of the
> device (aupb) - I would have preferred aupci, or at a stretch aupcib
> to keep in line with the current naming conventions.  There was a stub
> entry already in the aubus attachment code for aupci.
>
> I'm still not conviced about wiring down gobs of memory for PCI vs
> normal allocation and faulting in the pages to the TLB when needed, but
> I'm still open to being convinced this is a win for large areas of PCI
> memory space like frame buffers.
>
> Just to check my memory, we wire down a TLB per 32MB of wired space we
> reserve?  So your PCI patch will wire down 288MB (256MB of PCI mem, 16MB
> of PCI IO and 16MB of PCI config) or 9 TLB entries regardless of how
> much space is needed?  This is close to a third of our total TLB count
> and I'm worried about the performance hit that might cause.  Perhaps
> we could do the wiring after calling pci_configure_bus() (or otherwise
> in the non-PCI_NETBSD_CONFIGURE case) when we know how much space we
> actually need, and then wire only what we do need.
>   
This isn't quite right.  We wire down 16MB per entry, because they are
split into halves (and we are clever about this specifically to avoid
wasting entries).

We only allocate entries when we need to map the actual devices.  The
vast majority of PCI devices only need a few K (under a meg at least)
for registers and such, and so will all fit within a single wired
entry.  (The major exception to this is framebuffers.)

A typical configuration will use a page for IO space (if any IO space is
required) and a page for memory space.  This will result in 16MB for IO,
and 16 MB for memory, and the total will consume (typically) only 1
actual TLB entry (hi and lo), plus one more for config space.

I'm hoping that framebuffer memory can be set up without wired accesses,
using ordinary user-level mappings.  But I think the need to support a
wscons framework in the kernel might require otherwise.  In such a case
your wired entries grows to be as large as your framebuffer.  I'm open
to other, better, ways to handle this.  I hate the idea of sucking up a
couple of entries just to support a framebuffer.  (Alchemy has 16
entries, as I recall, which actually can give 32 separate mappings.)
> PCI config space access pretty much only occur during system startup,
> and otherwise if you run "pcictl dump" or similar.  It seems a waste to
> burn a wired TLB entry for the life of the system for this use.  I think
> we can get away with just using uvm_km_alloc/pmap_kenter_pa when needed,
> or perhaps just reserve a single 4kB page with uvm_km_alloc once and
> enter a different PA each time we need to do a config cycle.
>   
I disagree with this statement.  PCI config accesses can occur to read
fault addresses, access CardBus CIS, etc.  I do use a single wired
entry, but I'm thinking of changing this so that it could (in theory)
share a wired entry with other non-linear mappings -- e.g. IO accesses
and PCMCIA attribute space, etc.  This is a future enhancement I want to
do.  It won't get rid of the wired entry, but it will use it for more
stuff, thereby minimizing the waste.
> I'm really trying to get away from using as many wired TLB entries as we
> can - the more often we need to recycle non-wired TLB entries the more
> performance in general is going to get hurt.
>   
Agreed.  I've even gone so far as to consider using a single entry for
*all* non-linear mapped stuff (basically if the driver didn't say
BUS_SPACE_MAP_LINEAR), but I think ultimately this risks hurting IO
performance more.
>
> With the existing wired entries you have now, I wonder if something like:
>
> #define        AU_PCI_MEM_VA           0xf0000000
> #define        AU_PCI_MEM_SZ           0x0e000000    /* 224 MB */
>
> #define        AU_PCI_IO_VA            0xfe000000
> #define        AU_PCI_IO_SZ            0x01000000    /* 16 MB */
>
> #define        AU_PCI_CFG_VA           0xff000000
> #define        AU_PCI_CFG_SZ           0x01000000    /* 16 MB, overkill */
>
> would work better, with all the entries being contiguous?  This
> gives KSEG2 more room for growth if the kernel needs VA for other
> purposes.  Or if we don't need to wire down an entry for PCI config,
> then 240MB/16MB for PCI mem/IO.
>   
I'm open to tuning this somewhat.  I was motivated to try to get up to
256 MB in PCI space, but that was a fairly arbitrary limit.  I want to
support at least 128MB for a large framebuffer, and then maybe ~32 or 64
MB for all other stuff to fit in.  256MB was just the next power of 2. :-)

Now the minimum size is certainly a 16MB boundary (due to alignment
req'ts with the large entries I'm using).

Future stuff that has to be considered:

PCMCIA -- needs separate attribute, memory, and IO space mappings.  I
think its needs are a lot more modest  though, so that helps.

LCD panels on Au1100/1200 parts.  I've not looked at this other than to
realize it maps up above 4GB.

My first inclination is to avoid tuning too much of this now, but commit
what I have to get basic functionality, and then tweak as we either need
or have time to perf. tune it.

By the way, none of these TLB entries are *ever* mapped on a system that
doesn't need them (i.e. unless you have an Alchemy part with PCI support
configured in).

    -- Garrett
> Simon.
> --
> Simon Burge                                   <simonb@wasabisystems.com>
> NetBSD Development, Support and Service:   http://www.wasabisystems.com/
>   


-- 
Garrett D'Amore, Principal Software Engineer
Tadpole Computer / Computing Technologies Division,
General Dynamics C4 Systems
http://www.tadpolecomputer.com/
Phone: 951 325-2134  Fax: 951 325-2191