> > Try adding a `nop', e.g.:
> > 
> > 	call _C_LABEL(proc_trampoline_mp)
> > 	 nop

Hmm, so with all my printfs turned on, this seemed to help a big deal.
If I turned all my printfs off, looks like I putmyself to sleep at a
bad time - never get woken up...

BTW I did check the spinwait version again - locks up solid,
powercycle needed.  Its probably the case that I need to protect some
of this with spl*(), but I'm not yet sure - clue welcome ;-)

So, I turned on the semaphore printfs just for smp_cache_flush which
is what was being done when I went to sleep, so I could see if the
right things were happening.  Now I get a watchdog reset again.

NetBSD 1.5S (TOO.MP) #27: Fri Mar  2 09:46:21 PST 2001
    root@too:/l0/NetBSD/obj/sys/compile/TOO.MP
total memory = 159 MB
avail memory = 144 MB
using 896 buffers containing 8264 KB of memory
bootpath: /iommu@f,e0000000/sbus@f,e0001000/espdma@f,400000/esp@f,800000/sd@3,0
mainbus0 (root): SUNW,SPARCstation-20
cpu0 at mainbus0: mid 8: RT620/625 @ 100 MHz, on-chip FPU
cpu0: 256K byte write-back, 64 bytes/line, sw flush: cache enabled
cpu1 at mainbus0: mid 9: RT620/625 @ 100 MHz, on-chip FPU
cpu1: 256K byte write-back, 64 bytes/line, sw flush: cache enabled
obio0 at mainbus0
clock0 at obio0 slot 0 offset 0x200000: mk48t08: hostid 72725820
timer0 at obio0 slot 0 offset 0x300000 delay constant 48
zs0 at obio0 slot 0 offset 0x100000 level 12 softpri 6
zstty0 at zs0 channel 0 (console i/o)
zstty1 at zs0 channel 1
zs1 at obio0 slot 0 offset 0x0 level 12 softpri 6
kbd0 at zs1 channel 0
ms0 at zs1 channel 1
fdc0 at obio0 slot 0 offset 0x700000 level 11: no drives attached
auxreg0 at obio0 slot 0 offset 0x800000
power0 at obio0 slot 0 offset 0xa01000 level 2
iommu0 at mainbus0 ioaddr 0xe0000000: version 0x3/0x1, page-size 4096, range 64MB
sbus0 at iommu0: clock = 25 MHz
dma0 at sbus0 slot 15 offset 0x400000: rev 2
esp0 at dma0 slot 15 offset 0x800000 level 4: ESP100A, 40MHz, SCSI ID 7
scsibus0 at esp0: 8 targets, 8 luns per target
ledma0 at sbus0 slot 15 offset 0x400010: rev 2
le0 at ledma0 slot 15 offset 0xc00000 level 6
{0}sema_init(0xf02b14dc, 0, cache_flush)

{0}sema_signal(0xf02b14dc,1) == 1

{0}sema_wait(0xf02b14dc) == 0

Watchdog Reset
Type  help  for more information
<#1> ok .registers
          %g0      %g1      %g2      %g3      %g4      %g5      %g6      %g7
            0        0        5        0        0 f62431a0 f6243000 28313327
           PC      nPC        Y      PSR      WIM      TBR
     f0006124 f00060fc        0 1e001fe2        2 f0004050
<#1> ok .locals
            0        1        2        3        4        5        6        7
IN:         1        5        0        0 f6248e60        0 f6248db0 f00633e0
LOC: 1e401dc2       83 f1f00000 f1800000 1e001dc0 e0002000        1 f0205791
OUT:        a        5        0        9 f7fffff9    f8081 f6248c50 f0063abc
<#1> ok 1 .window
            0        1        2        3        4        5        6        7
IN:  f0205790        0 f0257800        1        1    40000 f6248e18 f0054278
LOC:        0        0       fc 20000000 f1000004        0 f1000006      1ff
OUT:        1        5        0        0 f6248e60        0 f6248db0 f00633e0
<#1> ok 0 .window
            0        1        2        3        4        5        6        7
IN:         1        5        0        0 f6248e60        0 f6248db0 f00633e0
LOC: 1e401dc2       83 f1f00000 f1800000 1e001dc0 e0002000        1 f0205791
OUT:        a        5        0        9 f7fffff9    f8081 f6248c50 f0063abc
<#1> ok 

Which is the same as we were getting originally on cpu1.

This GDB was configured as "sparc--netbsd"...
(gdb) x/i 0xf0006124
0xf0006124 <ctw_invalid+56>:    save  %g5, 0x40, %g5
(gdb) x/i 0xf00060fc
0xf00060fc <ctw_invalid+16>:    std  %l0, [ %g5 ]
(gdb)