Re: Question about a comment

[Juli Mallett <jmallett%freebsd.org@localhost>]

On Mon, Jul 4, 2016 at 5:40 PM,  <coypu%sdf.org@localhost> wrote:
> I'm curious about what is the reason for the following comment
> in sys/arch/mips/mips/locore.S:
>
> #ifdef MIPS64_OCTEON
>         //
>         // U-boot on the erlite starts all cpus at the kernel entry point.
>         // Use EBASE to find our CPU number and if it's not 0, call
>         // octeon_cpu_spinup if MP or loop using the wait instruction since
>         // non-primary CPUs can't do anything useful.
>         //
>         mfc0    a0, MIPS_COP_0_EBASE            # EBASE
>         COP0_SYNC
>         ext     a1, a0, 0, 10                   # get CPU number
>         beqz    a1, 2f                          # normal startup if 0
>          nop
> #ifdef MULTIPROCESSOR
>         j       _C_LABEL(octeon_cpu_spinup)
>          nop
> #else
> 1:      wait
>         b       1b
>          nop
> #endif /* MIPS64_OCTEON */
>
> What makes octeon special, why are we unable to use the other CPU?

I'm not sure exactly what you're asking, and I can't answer for NetBSD
as well as I can for FreeBSD, but actually this code is not optimal
(and nor is FreeBSD's.)  You need to allow one core to do the initial
startup and then launch the additional cores as what FreeBSD calls
application processors; basically, start shoving work at them.
Limiting it to core 0 is wrong; other cores could be your startup
core, and it doesn't matter which, really; you just need a consistent
way to pick one.  The trouble is, it takes a fair bit of work to get
the coremask and determine what the lowest running core is; I guess I
and whoever did the NetBSD code opted to be lazy rather than do all
that heavy lifting in the assembly.

Now, you might also be asking about how the Octeon launches its cores.
To start with, it's worth remembering that the Octeon is primarily
used as a networking (or related) application processor of one sort or
another, not a general-purpose computer.  As such, a lot of
carrier-grade equipment that they're designed for actually runs
different applications on different cores, or runs packet processing
on most of the cores with a few set aside for management/control stuff
with a general-purpose (or more general-purpose) OS.

You could write your operating system or application to behave like a
Unix-like OS does on a general-purpose computer, and assume that all
of the cores are its for the taking.  You can send NMIs and put other
cores into special reset states and whatever and force the cores to
boot up later.  There are middle-ground positions, but for ease-of-use
I think, Cavium hasn't gone with them.  So instead, this task is
relegated to the firmware (or, rather, the U-Boot bootloader, which in
Cavium's case is loaded with a *lot* of additional board-specific
knowledge.)

The bootloader lets you tell it what cores to launch a given
application on, and the application or OS receives a structures which
tells it what its "core mask" is.  These are the cores which have been
assigned to it.  All of them are started (pretend simultaneously) by
U-Boot at the same entry point address.  As a result, you need for one
of them to do the single-core startup stuff, and then release the
others to do work.

I had patches for FreeBSD, but I guess I abandoned them for some
reason, looking at what's checked in; you ought to be able to run
NetBSD or FreeBSD with a coremask that does not include the 0 bit set,
i.e. not having management/control/whatever happening on core 0, but
both of them simply assume core 0 is the one you're going to have
handle the single-threaded work and then launch the others.  There are
ways you can make it a lot more elegant, but so far it doesn't seem a
pressing issue for anyone using NetBSD or FreeBSD unmodified.

It may also be that the comment is just mildly misleading if you don't
parse it right; it's not that the other cores _can't_ do anything
useful, it's that NetBSD chooses to not let them do anything useful
until the 0 core wakes them up, because you can't actually, say, have
all of your cores initialize the VM system in parallel.  That all just
needs done once, on one core, in a specified order.  If you're not
running a MULTIPROCESSOR kernel, then they'd never get woken up, so
they just run in a loop.  I think this is wrong, though.  I think the
kernel ought to be happy being run single-core on a non-0 core.  It's
just a limitation of how the startup code was written here.  Getting
rid of the #else case and putting the whole thing under
defined(MIPS64_OCTEON) && defined(MULTIPROCESSOR) seems more correct
to me.  If you're running a UP kernel and the user passes
coremask=0xffff, then I guess they must want 16 separate instances of
it to try to run!?  But if I pass coremask=0x0700, it's probably the
case that I really do have a single core I want to run on.

Fixing the MULTIPROCESSOR case to better determine what core should do
the single-core/single-threaded stuff (what FreeBSD calls the BSP or
bootstrap processor) more thoroughly is probably reasonable, but it's
a bunch of crummy assym macros or hard-coded offsets to probe into
structures whose layout and size can change.  If someone gets it right
and actually cares to see it through, I'd love to see it be harmonized
between NetBSD and FreeBSD as much as possible.  A single assembly
routine in a single self-contained .S file could do the work and
provide a routine for locore.S to use to determine whether we're
running on the BSP, and that would be very easy to share.  It's only
tedious.  Hell, you could even do it in *very* careful C, but best not
to, especially because you have the challenge of per-core stacks at
this stage of the game.

Hope that helps,
Juli.