Re: Generic startup code maps memory non-cacheable non-shareable

[Nick Hudson <nick.hudson%gmx.co.uk@localhost>]

On 02/11/2018 16:56, Aymeric Vincent wrote:
> Nick Hudson <nick.hudson%gmx.co.uk@localhost> writes:
>
> > Because ap_mpstart is called on BP where L2 cache initialisation is
> > done before the APs are started and the arm_cpu_hatched dance is
> > needed
> For it to work, arm_cpu_hatched and arm_cpu_mbox need to be mapped with
> the cacheable and shareable bits on. When is this supposed to be done?
> (I guess this incurs switching level 1 page tables.) I don't see in the
> code why it would be done in time on the iMX6 and not on the Cyclone V
> given arm_cpu_hatched is accessed very early during bootup of the
> secondary processors.

OK, Here goes...

To quote the commit message...

"Provide generic start code that assumes the MMU is off and caches are
disabled as per the linux booting protocol for ARMv6 and ARMv7 boards.
u-boot image type should be changed to 'linux' for correct behaviour.

The new start code builds a minimal "bootstrap" L1PT with cached access
disabled and uses the same table for all processors. "

generic_start builds the translation tables (with cached access 
disabled), turns the MMU
on,  and starts running the kernel at KERNEL_BASE_VIRT by calling into 
'start'. Not much is
done by 'start' before it calls the board specific 'initarm', e.g.

http://src.illumos.org/source/xref/netbsd-src/sys/arch/evbarm/fdt/fdt_machdep.c#420
http://src.illumos.org/source/xref/netbsd-src/sys/arch/evbarm/nitrogen6/nitrogen6_machdep.c#284

both of these use arm32_bootmem_init/arm32_kernel_vm_init to build full 
blown
cached, RO, RW and XN mappings for the kernel and a bunch of 
data/stacks, etc. The BP is
then switched to these translation tables and they're used for the life 
time of the kernel.

http://src.illumos.org/source/xref/netbsd-src/sys/arch/arm/arm32/arm32_kvminit.c#953

'initarm_common' then runs and once this has completed the APs are 
kicked into life.

http://src.illumos.org/source/xref/netbsd-src/sys/arch/evbarm/fdt/fdt_machdep.c#548
http://src.illumos.org/source/xref/netbsd-src/sys/arch/evbarm/nitrogen6/nitrogen6_machdep.c#398

The APs will also use the bootstrap tables initially in cpu_mpstart

http://src.illumos.org/source/xref/netbsd-src/sys/arch/arm/arm/armv6_start.S#720

but jump to full blown tables in cpu_init_secondary_processors

http://src.illumos.org/source/xref/netbsd-src/sys/arch/arm/arm/armv6_start.S#804
http://src.illumos.org/source/xref/netbsd-src/sys/arch/arm/arm32/arm32_machdep.c#726

The first atomic is used after the jump to full blown tables

http://src.illumos.org/source/xref/netbsd-src/sys/arch/arm/arm32/arm32_machdep.c#768

I hope that explains it all.

It's made a bit more complicated by all the non-FDT boards, and older 
cpu versions, etc.

The idea behind starting with caches disabled is that there are no cache 
artifacts to worry about
along the way and we have control of the startup timing/sequence.

Nick