Arm MP startup issues [was Re: 2021-10-18-netbsd-raspi-earmv6hf.img (Re: Raspberry Pi update please.)]

[Rory Bolt <rory.bolt%gmail.com@localhost>]

Hello all,

As noted in PR 56459, I too analyzed this issue and found a workaround
documented in the problem report.

I just wanted to let you all know the other things I tried along the way...

Basically I tried all the suggestions in the mailing list:

Tara-san's suggestion "Increasing PERCPU_IMPORT_SIZE may solve this
issue." had no effect.
Jason Thorpe's suggestion to mark the idle LWP LSONPROC causes the
boot to hang forever...
Nick Hudson's suggestion: https://www.netbsd.org/~skrll/mp_ready.diff
had no effect.

As stated in my PR, anything that causes the idle thread to block
during initialization will cause bad things to happen... and there are
many calls with the potential to block on the initialization path. I
am concerned that my work around (delaying cold=0 until after the
secondary processors initialize) is not in the ARM specific code, and
thus may have unintended side effects on other platforms.

I see two paths forward:

1) introduction of a new ARM specific "I am still initializing flag"
that can be used to prevent the calls to broadcast and wait during
secondary processor initialization without affecting other platforms.
This fixes the problem for now until something else causes the
initialization code to block...

or

2) Rework of the entire arm secondary processor initialization for ARM
to eliminate the possibility of blocking on the idle thread. This
seems like the right thing to do as it would be ARM specific and
should not affect other platforms as well as eliminating
"whack-a-mole" fixes.

I would be concerned with altering code that affects other platforms
so the other approach that I think is interesting (allowing the idle
thread to block during initialization) is probably not reasonable.

-Rory