Re: Anyone working on Octeon SMP?

[Kevin Bowling <kevin.bowling%kev009.com@localhost>]

On Sat, Mar 28, 2026 at 8:28 AM Andrew Parker <andrew%pmk1.net@localhost> wrote:
>
> On Thursday, 19 March 2026 23:31:36 EDT Kevin Bowling wrote:
> > On Sun, Feb 2, 2025 at 6:52 AM Andrew Parker <andrew%pmk1.net@localhost> wrote:
> > > On 1/29/25 06:23, Nick Hudson wrote:
> > > > On 20/12/2024 22:01, Andrew Parker wrote:
> > > >> Hi, I've been working towards getting a MULTIPROCESSOR build more
> > > >> stable
> > > >> on my ER4 and would like to know if anyone else may be working on the
> > > >> same?
> > > >>
> > > >> I've found a few areas that were causing instability on my machine and
> > > >> have some patches that help (but mostly just for debugging at this
> > > >> point). If there's any interest in teaming up and exchanging ideas or
> > > >> patches for SMP on Octeon please let me know.
> > > >
> > > > Sure.
> > > >
> > > > I've dropped the ball on this and said I had a couple of fixes in mind,
> > > > but done nothing to share them. Hopefully we can make it stable.
> > > >
> > > > Nick
> > >
> > > Great!  I'm curious about what you have in mind for improvement.  I've
> > > mostly been looking around TLB invalidation and perhaps a missing memory
> > > barrier.
> > >
> > > Anyway, I'll work on getting some stuff cleaned up and contact directly
> > > if that works for you.
> >
> > I'm interested in this as well, is there anything to share around
> > current status or issues as well as if anything is pending out of
> > tree?
>
> I hoped to spend more time on this over the winter but ended up moving and
> some of my networking gear is still packed up.
>
> It's been a slow process getting my test environment setup again but it would
> be great to pick this back up...especially if there's continued interest in
> it.
>
> Give me a week or two to see what I can dig up and I'll be in touch.

I've made some progress but it turned into a much deeper hole than I
was anticipating.  I can increase SMP stability with some changes in
pmap_tlb.c to add some icache syncs and guarding around an assert that
is easy to trigger

@@ -735,7 +767,9 @@ pmap_tlb_shootdown_bystanders(pmap_t pm)
                        * And best of all, we avoid an IPI.
                        */
                       KASSERT(!kernel_p);
-                       pmap_tlb_pai_reset(ti, pai, pm);
+                       if (pai->pai_asid > KERNEL_PID) {
+                               pmap_tlb_pai_reset(ti, pai, pm);
+                       }

I think there are a variety of pmap changes needed.  But I wonder if
any MIPS SMP or other PMAP_TLB_NEED_SHOOTDOWN has been heavily
exercised?  IPI and TLB stuff is probably "simpler" on ARM.  On
FreeBSD (13), OpenBSD, Linux the MIPS TLB shootdowns are synchronous.

My test is iperf3 -P4 --bidir against the ER4 (4 cores).  With my
current changes I can get it to run about 900 seconds, where before it
often would die in a few or a dozen seconds.  I'm having trouble
catching the ultimate source of the instability, whether it is TLB or
something else.

Regards,
Kevin