Anybody running NetBSD/cats on bare metal?

[Harold Gutch <logix%foobar.franken.de@localhost>]

Hi,

I recently played around with NetBSD/cats in gxemul.  Releases up to
NetBSD 7.0 work, NetBSD 8.0 and 9.0 boot but OpenSSL seems to be
broken on them in a similar manner to PR 53670, but 10.0 and current
don't boot at all.  I bisected this to the switch from gcc 9 to gcc 10
in https://mail-index.netbsd.org/source-changes/2021/06/19/msg130281.html

Booting a kernel built after this commit results in no console output
in gxemul and it then exiting with the error
"arm_load_register_bank: unimplemented mode 0".

This here is the output of gxemul -t on a kernel from just before the
commit above (i.e., built with gcc 9):

<start("STAC",0,0,0,..)>
  <initarm("STAC",&__end__,&__end__,0x4401a103,..)>
    <set_cpufuncs("STAC",&__end__,&__end__,0x4401a103,..)>
      <cpufunc_id(0,&__end__,&__end__,0x4401a103,..)>
      <__aeabi_memcpy8(&cpufuncs,&sa110_cpufuncs,132,0x4401a100,..)>
    <__udivsi3(32,32,0,"$",..)>
    <__udivsi3(32,32,32,"$",..)>
    <pmap_pte_init_sa1(1,32,32,31,..)>
      <pmap_pte_init_generic(1,32,32,31,..)>
  <pmap_devmap_bootstrap(0x6f8000,&cats_devmap,0,"$",..)>
    <pmap_map_chunk(0x6f8000,0xfd000000,0x42000000,0x100000,..)>
  <pmap_map_chunk(0x6f8000,0xfd100000,0x50000000,0x100000,..)>
<pmap_map_chunk(0x6f8000,0xfd200000,0x7c000000,0x100000,..)>
<pmap_map_chunk(0x6f8000,0xfd300000,0x79000000,0x100000,..)>
<pmap_map_chunk(0x6f8000,0xfe000000,0x7a000000,0x1000000,..)>
<pmap_map_chunk(0x6f8000,0xff000000,0x7b000000,0x1000000,..)>
<pmap_map_chunk(0x6f8000,0xfd400000,0x80000000,0x100000,..)>
<printf("NetBSD/cats booting ...\n",0xfffffff4,0,0,..)>
  <vprintf_flags(5,"NetBSD/cats booting ...\n",0xf03efeb4,0,..)>
    <kprintf("NetBSD/cats booting ...\n",5,0,0,..)>
      <putchar(78,5,0,1,..)>
        <getnanouptime(0xf03efd68,5,0,1,..)>
        <snprintf(0xf03efd78,64,"[% 4jd.%.*ld] ",0,..)>
          <vsnprintf(0xf03efd78,64,"[% 4jd.%.*ld] ",0xf03efd54,..)>
            <kprintf("[% 4jd.%.*ld] ",8,0xf03efd14,0xf03efd78,..)>
              <0xf025f37c(0,0xffffffff,"4jd.%.*ld] ",32,..)>
                <0xf025efc0(20,0xffffffff,"jd.%.*ld] ",52,..)>
                  <0xf025f088(74,0xffffffff,"d.%.*ld] ",106,..)>
                    <0xf025efe4(68,0xffffffff,".%.*ld] ",100,..)>
                      <0xf025f2e4(14,0xffffffff,"*ld] ",46,..)>
[...]

and this here on a kernel from after the commit above:

<start("STAC",0,0,0,..)>
  <initarm("STAC",&__end__,&__end__,0x4401a103,..)>
    <set_cpufuncs("STAC",&__end__,&__end__,0x4401a103,..)>
      <cpufunc_id(0,&__end__,&__end__,0x4401a103,..)>
      <memcpy(&cpufuncs,"P",132)>
    <__udivsi3(32,32,"X",0,..)>
    <__udivsi3(32,32,32,0,..)>
    <pmap_pte_init_sa1("X",32,32,31,..)>
      <pmap_pte_init_generic("X",32,32,31,..)>
  <pmap_devmap_bootstrap(0x6f8000,&cats_devmap,0,"X",..)>
    <pmap_map_chunk(0x6f8000,0xfd000000,0x42000000,"\t",..)>
  <pmap_map_chunk(0x6f8000,0xfd100000,0x50000000,"\t",..)>
<pmap_map_chunk(0x6f8000,0xfd200000,0x7c000000,"\t",..)>
<pmap_map_chunk(0x6f8000,0xfd300000,0x79000000,"\t",..)>
<pmap_map_chunk(0x6f8000,0xfe000000,0x7a000000,0x1000000,..)>
<pmap_map_chunk(0x6f8000,0xff000000,0x7b000000,0x1000000,..)>
<pmap_map_chunk(0x6f8000,0xfd400000,0x80000000,"\t",..)>
<printf("NetBSD/cats booting ...\n",0xfffffff4,0,0,..)>
  <kprintf("NetBSD/cats booting ...\n",5,0,0,..)>
    <putchar(78,5,0,1,..)>
      <addtstamp(5,0,0,1,..)>
        <getnanouptime(0xf03fed70,0,0,&v_putc,..)>
        <snprintf(0xf03fed80,64,"[% 4jd.%.*ld] ",1,..)>
          <kprintf("[% 4jd.%.*ld] ",8,0xf03fed34,0xf03fed80,..)>
            <0xf0268df8(0,0xffffffff,"4jd.%.*ld] ",0,..)>
              <0xf0268ab8(0,0xffffffff,"jd.%.*ld] ",20,..)>
                <0xf0268b78(4,0xffffffff,"d.%.*ld] ",74,..)>
                  <0xf0268adc(4,0xffffffff,".%.*ld] ",68,..)>
                    <__aeabi_uldivmod(1,0,10,0,..)>
                      <__udivmoddi4(1,0,10,0,..)>
                        <__aeabi_uidivmod(1,10,10,0,..)>
                          <__udivmodsi4(1,10,0xf03fec58,0,..)>
                          <0xf0349f7c(0,1,0xf03fec58,0,..)>
                            <__udivsi3(1,10,0xf03fec58,0,..)>
                        <__aeabi_uldivmod(1,0,10,0,..)>
                          <__udivmoddi4(1,0,10,0,..)>
[...]


The differences are:
1) a call to memcpy() (working) vs __aeabi_memcpy8() (nonworking)
2) printf() -> vprintf_flags() -> kprintf() (working) vs
   printf() -> kprintf() (nonworking) -- i.e., without going through
   vprintf_flags() inbetween
3) putchar() calling getnanouptime() and snprintf() (working) vs
   putchar() calling addtimestamp() which then calls getnanouptime()
   and snprintf() (nonworking)
4) snprintf() -> vsnprintf() -> kprintf() (working) vs
   snprintf() -> kprintf() (nonworking)

and finally kprintf() going into a loop when dividing by 10 when
handling decimals (nonworking).

I assume at least some of these are due to differences in gcc 9 vs gcc
10 (or perhaps our local patches), but before I dig further:  has
anybody on the list run NetBSD 10.0 or a current after June 19, 2021 on
their cats, i.e., on actual hardware?  I'd just first like to rule out
an emulation bug or some lack of support in gxemul.


thanks,
  Harold