Re: What if the console device is only accessible from one CPU in a multiprocessor system?

[Jason Thorpe <thorpej%me.com@localhost>]

> On Mar 6, 2025, at 1:57 AM, Christoph Badura <bad%bsd.de@localhost> wrote:
> 
> On Tue, Mar 04, 2025 at 01:26:42PM -0800, Jason Thorpe wrote:
>> The firmware selects which CPU module gets to boot the system (the
>> criteria used is unimportant here), and the UART on that module is
>> connected to the console port and the RTC on that module is the
>> authoritative RTC for the system.
>> 
>> Obviously, this is a lot more problematic for the console device.  We
>> currently have no restrictions in the kernel overall around which CPU
>> gets to access the console device for printing messages, and don’t
>> even get me started on how weird this all gets with DDB.
> 
> My first thought over morning coffe was to add a thin layer on the
> cnputc/cngetc/cnpollc interface level.  Basically have a stack of
> cn_tabs (only the necessary parts perhaps).  On the Laser/TurboLaser
> machines have "virtual" cnputc/cngetc/cnpollc that, on the boot CPU,
> calls the "real" cnputc/cngetc that frobs the hardware and, on the other
> CPUs, xcalls to the boot CPU.  I presume you can easily identify the
> boot CPU.  I don't see a way around the xcall.

Yes, it is trivial to identify the boot CPU.

That’s more or less what I did in the “mcclock” driver.  In this particular case, I wasn’t thinking of doing it at the cn*c layer, but more in the driver itself; even though it’s a Z8530 and could use the zsc / zstty code in theory, in practice I don’t think it’s worth it and would probably just do a stripped down “gbuscn” driver (DEC calls this CPU module local bus the “GBus”) that does the bare minimum.  That driver would have all of the necessary xcall goop.

What I really don’t like, though, is e.g. a kernel printf that happens to run on a non-primary CPU could incur possibly a LOT of xcalls.  Maybe I shouldn’t really care too much because how often do I really expect that to happen?  OTOH, it’s also the case that just regular user logging in on the console could easily trigger this as well.

> The "thin layer" may come in handy when we want to deal with multiple
> devices acting as console like e.g. Linux has it.  There you can have
> console on e.g. frame buffer and serial simultaneously.

I think wscons already has a way to do this, actually?

> I'm surprised you expect weirdness with DDB.  I thought DDB runs on the
> boot CPU and pauses all other CPUs.  Doing xcalls to the other CPUs for
> stack traces etc.  I.e. it's a non-issue for DDB.

Well, on Alpha, currently, the way it works is the BUGCHK handler (which is invoked on whatever CPU calls Debugger()) calls alpha_debug(), which sends an IPI to all other CPUs telling them to pause (this is a tight loop in the IPI handler until that CPU sees its bit in the “paused CPUs” bitmask get cleared), switches to the debug stack, enters DDB, switches back to the previous stack, and then un-pauses all other CPUs.  There is not currently anything in DDB that enforces “must be running on the primary CPU” to my knowledge, and it’s definitely not implemented that way on Alpha, currently.

It is perhaps worth changing this, but it could make entering DDB problematic if the primary CPU was stuck at a sufficiently high spl to block IPIs.  Of course, that wouldn’t make much difference on a Laser/TurboLaser, but it would unnecessarily cripple other systems, potentially.

> I did a bit of nxr'ing on cnputc and on subr_prf.c but that quickly
> turned into a maze of twisty little passages to large for my short term
> memory.  The main thing I remember is that kprintf ends up cnputc'ing
> every single character.  Sending a bunch of characters in an xcall might
> be less overhead.

Yah, I am definitely thinking an “cnputstr()” or similar type of routine could be desirable.

> You are looking for an excuse to pull out the rototiller, aren't you?  :-)

In this case, no :-)

-- thorpej