Re: Anyone working on Octeon SMP?

[Kevin Bowling <kevin.bowling%kev009.com@localhost>]

On Fri, Apr 10, 2026 at 11:45 PM Nick Hudson <nick.hudson%gmx.co.uk@localhost> wrote:
>
> On 10/04/2026 11:18, Kevin Bowling wrote:
> > On Mon, Apr 6, 2026 at 6:42 AM Nick Hudson <nick.hudson%gmx.co.uk@localhost> wrote:
> >>
> >> On 06/04/2026 10:49, Kevin Bowling wrote:
> >>> On Sat, Mar 28, 2026 at 8:28 AM Andrew Parker <andrew%pmk1.net@localhost> wrote:
> >>>>
> >>>> On Thursday, 19 March 2026 23:31:36 EDT Kevin Bowling wrote:
> >>>>> On Sun, Feb 2, 2025 at 6:52 AM Andrew Parker <andrew%pmk1.net@localhost> wrote:
> >>>>>> On 1/29/25 06:23, Nick Hudson wrote:
> >>>>>>> On 20/12/2024 22:01, Andrew Parker wrote:
> >>>>>>>> Hi, I've been working towards getting a MULTIPROCESSOR build more
> >>>>>>>> stable
> >>>>>>>> on my ER4 and would like to know if anyone else may be working on the
> >>>>>>>> same?
> >>>>>>>>
> >>>>>>>> I've found a few areas that were causing instability on my machine and
> >>>>>>>> have some patches that help (but mostly just for debugging at this
> >>>>>>>> point). If there's any interest in teaming up and exchanging ideas or
> >>>>>>>> patches for SMP on Octeon please let me know.
> >>>>>>>
> >>>>>>> Sure.
> >>>>>>>
> >>>>>>> I've dropped the ball on this and said I had a couple of fixes in mind,
> >>>>>>> but done nothing to share them. Hopefully we can make it stable.
> >>>>>>>
> >>>>>>> Nick
> >>>>>>
> >>>>>> Great!  I'm curious about what you have in mind for improvement.  I've
> >>>>>> mostly been looking around TLB invalidation and perhaps a missing memory
> >>>>>> barrier.
> >>>>>>
> >>>>>> Anyway, I'll work on getting some stuff cleaned up and contact directly
> >>>>>> if that works for you.
> >>>>>
> >>>>> I'm interested in this as well, is there anything to share around
> >>>>> current status or issues as well as if anything is pending out of
> >>>>> tree?
> >>>>
> >>>> I hoped to spend more time on this over the winter but ended up moving and
> >>>> some of my networking gear is still packed up.
> >>>>
> >>>> It's been a slow process getting my test environment setup again but it would
> >>>> be great to pick this back up...especially if there's continued interest in
> >>>> it.
> >>>>
> >>>> Give me a week or two to see what I can dig up and I'll be in touch.
> >>> I've made some progress but it turned into a much deeper hole than I
> >>> was anticipating.  I can increase SMP stability with some changes in
> >>> pmap_tlb.c to add some icache syncs
> >>
> >> I have a change I'll commit soon to handle EXECness (more) correctly.
>
> This is committed
>
> https://mail-index.netbsd.org/source-changes/2026/04/10/msg161522.html
>m
>
> >>
> >> Am I right in thinking at least some octeon processors icaches are VIVT?
> >> I've forgotten most of the mips stuff I knew... If so there will be more flushing
> >> required for VIVT.
> > VIPT.  But it has an assortment of fun issues.
>
> fun issues?

It seems to require pretty deliberate management that is different
than other MIPS and archs.

I was able to get SMP stabilized to my own satisfaction last night, in
so far as it can hold up cnmac driver modifications which is what I
was originally started with.

There are a few categories of mandatory fixes:  membar_release is
missing a SYNC_PLUNGER (second syncw), INT_MASKs critically missing in
octeon_intr.c, and a variety of locore fixups.  Beyond that I ended up
redoing octeon_intr.c to distribute interrupts and fix my previous
octeon III patch.  I have changes to the PMAP_TLB_NEED_SHOOTDOWN path
primarily that I am least confident about.  I'll try and organize
everything into a more deliberate patch series, right now it is pretty
messy with attempts and debugging.

>
>
> >>> and guarding around an assert that
> >>> is easy to trigger
> >>> @@ -735,7 +767,9 @@ pmap_tlb_shootdown_bystanders(pmap_t pm)
> >>>                         * And best of all, we avoid an IPI.
> >>>                         */
> >>>                        KASSERT(!kernel_p);
> >>> -                       pmap_tlb_pai_reset(ti, pai, pm);
> >>> +                       if (pai->pai_asid > KERNEL_PID) {
> >>> +                               pmap_tlb_pai_reset(ti, pai, pm);
> >>> +                       }
> >>
> >>
> >> you mean this KASSERT in pmap_tlb_pai_reset?
> > Yeah.  But let me think harder about this.
> >>    252  /*
> >>    253   * We must have an ASID but it must not be onproc (on a processor).
> >>    254   */
> >>    255  KASSERT(pai->pai_asid > KERNEL_PID);
> >>
> >>> I think there are a variety of pmap changes needed.  But I wonder if
> >>> any MIPS SMP or other PMAP_TLB_NEED_SHOOTDOWN has been heavily
> >>> exercised?
> >>
> >> Almost certainly not.
> > Good to know, I was a bit shy about looking at the MI pmap at first.
> >>> IPI and TLB stuff is probably "simpler" on ARM.  On
> >>> FreeBSD (13), OpenBSD, Linux the MIPS TLB shootdowns are synchronous.
> >>
> >> No other NetBSD architecture (not sure of powerpc booke status actually) uses
> >> the PMAP_TLB_NEED_SHOOTDOWN stuff.
> >>
> >> I'm about to switch aarch64 to sys/uvm/pmap which uses architecture defined
> >> broadcast TLB operations. RISC-V uses SBI remote fence operations.
> > Arch defined broadcast sounds very suitable for what we'll need to do
> > here.  Is that out of tree?
>
> Arm architecture defined the broadcast TLB operations long ago. Others have caught up...
>
> MI PMAP uses tlb_* functions
> https://nxr.netbsd.org/xref/src/sys/arch/aarch64/aarch64/aarch64_tlb.c#84
> https://nxr.netbsd.org/xref/src/sys/arch/aarch64/aarch64/cpufunc_asm_armv8.S#221
>
>
>