Re: evbppc reserved-tlb cleanup

[Simon Burge <simonb%NetBSD.org@localhost>]

Jachym Holecek wrote:

> the patch below converts ibm4xx-based evbppc from static reserved-TLB
> entry allocation to recently introduced ppc4xx_tlb_reserve() API (and
> fixes a stupid bug therein, too ;-). Some side effects:
> 
>   o ibm405gp UART0 used to be linear mapped. The VA happens to be
>     inside kernel segment, giving us the possibility of multiple
>     VA matches in the TLB. This is considered "programming error"
>     by 405 core and results in "undefined behaviour". We now avoid
>     mapping peripherals in kernel segment.
> 
>   o Some boards used to map hardwired RAM size. We now use the real
>     size as passed in by boot firmware.
> 
>   o TLB_NRESERVED is (finally) gone.
> 
> I don't have any of the affected boards to play with, so it would be
> great to know how it works on Walnut, Explora451 and OpenBlocks.

With this change, or at least the version of it that got committed on
16th October, my Walnut with a 256MB DIMM now has 17 wired TLB entries
(1 for each 16MB of ram, and an extra for the OPB devices).  I have run
this in the past with 512MB, which would have made half the available
TLBs wired.

Previously, pmap_tlbmiss() added 16MB page table entries on the fly
if there was a kernel page miss.  I haven't measured the impact of
this, but I suspect that wiring that percentage of TLBs will have some
negative effect on overall performance, even though we'd be taking
kernel page misses.

I've changed pmap_tlbmiss() to log kernel and non-kernel TLB misses
separately, and booting to multiuser results in on 3.99.17:

        evcnt type 2: cpu tlbmiss  =  433543
        evcnt type 2: cpu ktlbmiss =   17236

and with 4.99.3:

        evcnt type 2: cpu tlbmiss  =  596968

No kernel TLB misses with 4.99.3 obviously, but around 150k or 33% more
overall TLB misses were needed with needed with -current.  To show the
effect of wiring too many TLBs, I rigged initppc() to wired enough TLBs
for 512MB of RAM (32 wired TLBs), and see:

        evcnt type 2: cpu tlbmiss  = 1275548

which results in nearly double the TLB misses as used by 16 wired TLBs.
And here's with initppc() only wiring one TLB for RAM:

        evcnt type 2: cpu tlbmiss  =  406119
        evcnt type 2: cpu ktlbmiss =   16270

This last test was done with this fragment in walnut/machdep.c:

        /* Map first large TLB page of kernel */
        ppc4xx_tlb_reserve(0, 0, TLB_PG_SIZE, TLB_EX);

        /* Map console just after RAM */
        /* XXX magic number here!!! */
        ppc4xx_tlb_reserve(0xef000000, board_data.mem_size, TLB_PG_SIZE,
            TLB_I | TLB_G);

I think we should look at using code like this in all the machdep.c's
instead of wiring down all RAM.  Any thoughts from anyone to the
contrary on this?

I'm also not sure if we should map to the end of kernel text and data
(for say when a large memory disk is in the kernel) or not.  It's
probable that if an extra 16MB (or two or ...) is needed to cover a
memory disk, it'll get hit often.  Something to test one day...

Cheers,
Simon.