Re: Anyone working on Octeon SMP?

[Nick Hudson <nick.hudson%gmx.co.uk@localhost>]

On 10/04/2026 11:18, Kevin Bowling wrote:
> On Mon, Apr 6, 2026 at 6:42 AM Nick Hudson <nick.hudson%gmx.co.uk@localhost> wrote:
>> 
>> On 06/04/2026 10:49, Kevin Bowling wrote:
>>> On Sat, Mar 28, 2026 at 8:28 AM Andrew Parker <andrew%pmk1.net@localhost> wrote:
>>>> 
>>>> On Thursday, 19 March 2026 23:31:36 EDT Kevin Bowling wrote:
>>>>> On Sun, Feb 2, 2025 at 6:52 AM Andrew Parker <andrew%pmk1.net@localhost> wrote:
>>>>>> On 1/29/25 06:23, Nick Hudson wrote:
>>>>>>> On 20/12/2024 22:01, Andrew Parker wrote:
>>>>>>>> Hi, I've been working towards getting a MULTIPROCESSOR build more
>>>>>>>> stable
>>>>>>>> on my ER4 and would like to know if anyone else may be working on the
>>>>>>>> same?
>>>>>>>> 
>>>>>>>> I've found a few areas that were causing instability on my machine and
>>>>>>>> have some patches that help (but mostly just for debugging at this
>>>>>>>> point). If there's any interest in teaming up and exchanging ideas or
>>>>>>>> patches for SMP on Octeon please let me know.
>>>>>>> 
>>>>>>> Sure.
>>>>>>> 
>>>>>>> I've dropped the ball on this and said I had a couple of fixes in mind,
>>>>>>> but done nothing to share them. Hopefully we can make it stable.
>>>>>>> 
>>>>>>> Nick
>>>>>> 
>>>>>> Great!  I'm curious about what you have in mind for improvement.  I've
>>>>>> mostly been looking around TLB invalidation and perhaps a missing memory
>>>>>> barrier.
>>>>>> 
>>>>>> Anyway, I'll work on getting some stuff cleaned up and contact directly
>>>>>> if that works for you.
>>>>> 
>>>>> I'm interested in this as well, is there anything to share around
>>>>> current status or issues as well as if anything is pending out of
>>>>> tree?
>>>> 
>>>> I hoped to spend more time on this over the winter but ended up moving and
>>>> some of my networking gear is still packed up.
>>>> 
>>>> It's been a slow process getting my test environment setup again but it would
>>>> be great to pick this back up...especially if there's continued interest in
>>>> it.
>>>> 
>>>> Give me a week or two to see what I can dig up and I'll be in touch.
>>> I've made some progress but it turned into a much deeper hole than I
>>> was anticipating.  I can increase SMP stability with some changes in
>>> pmap_tlb.c to add some icache syncs
>> 
>> I have a change I'll commit soon to handle EXECness (more) correctly.

This is committed

https://mail-index.netbsd.org/source-changes/2026/04/10/msg161522.html


>> 
>> Am I right in thinking at least some octeon processors icaches are VIVT?
>> I've forgotten most of the mips stuff I knew... If so there will be more flushing
>> required for VIVT.
> VIPT.  But it has an assortment of fun issues.

fun issues?



>>> and guarding around an assert that
>>> is easy to trigger
>>> @@ -735,7 +767,9 @@ pmap_tlb_shootdown_bystanders(pmap_t pm)
>>>                         * And best of all, we avoid an IPI.
>>>                         */
>>>                        KASSERT(!kernel_p);
>>> -                       pmap_tlb_pai_reset(ti, pai, pm);
>>> +                       if (pai->pai_asid > KERNEL_PID) {
>>> +                               pmap_tlb_pai_reset(ti, pai, pm);
>>> +                       }
>> 
>> 
>> you mean this KASSERT in pmap_tlb_pai_reset?
> Yeah.  But let me think harder about this.
>>    252  /*
>>    253   * We must have an ASID but it must not be onproc (on a processor).
>>    254   */
>>    255  KASSERT(pai->pai_asid > KERNEL_PID);
>> 
>>> I think there are a variety of pmap changes needed.  But I wonder if
>>> any MIPS SMP or other PMAP_TLB_NEED_SHOOTDOWN has been heavily
>>> exercised?
>> 
>> Almost certainly not.
> Good to know, I was a bit shy about looking at the MI pmap at first.
>>> IPI and TLB stuff is probably "simpler" on ARM.  On
>>> FreeBSD (13), OpenBSD, Linux the MIPS TLB shootdowns are synchronous.
>> 
>> No other NetBSD architecture (not sure of powerpc booke status actually) uses
>> the PMAP_TLB_NEED_SHOOTDOWN stuff.
>> 
>> I'm about to switch aarch64 to sys/uvm/pmap which uses architecture defined
>> broadcast TLB operations. RISC-V uses SBI remote fence operations.
> Arch defined broadcast sounds very suitable for what we'll need to do
> here.  Is that out of tree?

Arm architecture defined the broadcast TLB operations long ago. Others have caught up...

MI PMAP uses tlb_* functions
https://nxr.netbsd.org/xref/src/sys/arch/aarch64/aarch64/aarch64_tlb.c#84
https://nxr.netbsd.org/xref/src/sys/arch/aarch64/aarch64/cpufunc_asm_armv8.S#221