Re: QEMU SMP idle

[Julian Coleman <jdc%coris.org.uk@localhost>]

Hi,

> seems reasonable to me.  and yeah, qemu sparcs all seem to have an apc(4)
> when they don't exist on most platforms.  but it exists on qemu platform
> now and so netbsd should support it.  and it's a great benefiti...
> 
> please commit!

Committed (a little while back)!

I've played with this a little and I'm trying to get SMP running with serial
console.  The current state is that qemu continues to assert IPL 12 (serial)
but neither serial chip has any interrupt bits set when we query them.  We
would normally panic with crazy interrupts here, but we ignore this specific
case (stray serial interrupt with MULTIPROCESSOR) so we loop here forever.

So far I have discovered, mostly by using extra printf's in our interrupt
routines plus combinations of the QEMU trace flags:

  -trace sun4m_cpu_interrupt
  -trace sun4m_cpu_reset_interrupt
  -trace sun4m_cpu_set_irq_raise
  -trace sun4m_cpu_set_irq_lower
  -trace escc_update_irq
  -trace escc_mem_writeb_ctrl
  -trace escc_mem_writeb_data
  -trace escc_mem_readb_data
  -trace escc_mem_readb_ctrl

that we (almost) always query the status of the chip twice, even with
single-CPU configurations (seen with escc_mem_readb_ctrl).  Our code path
here is z8530sc.c:zsc_intr_hard() then z8530tty.c:zstty_txint() where I can
see the read of rr3 then the writes of wr0 with ZSWR0_CLR_INTR and then
ZSWR0_RESET_TXINT.  The latter causes QEMU to deassert the interrupt and
remove CPU IQR 12.  We then read rr3 again and it is 0.  We sometimes
enter the hard interrupt handle on character output on the console because
we have transmit interrupts enabled.  We don't always - I assume this is
a race between the interrupt being deasserted and us invoking the interrupt
handler.

The annotated QEMU trace output looks like (for example):

zs.c:zs_putc()
  escc_mem_readb_ctrl Read channel a, reg[0] = 0x44
  escc_mem_writeb_data Write channel a, ch 93
  escc_update_irq IRQ = 0
  sun4m_cpu_set_irq_lower Lower CPU IRQ 12
  sun4m_cpu_reset_interrupt Reset CPU IRQ 12
]  escc_update_irq IRQ = 1
  sun4m_cpu_set_irq_raise Raise CPU IRQ 12
  sun4m_cpu_interrupt Set CPU IRQ 12
??
  escc_mem_writeb_ctrl Write channel a, reg[0] = 0x03
z8530sc.c:zsc_intr_hard()
  escc_mem_readb_ctrl Read channel a, reg[3] = 0x10
  escc_mem_writeb_ctrl Write channel a, reg[0] = 0x38
  escc_update_irq IRQ = 1
z8530tty.c:zstty_txint()
  escc_mem_writeb_ctrl Write channel a, reg[0] = 0x28
  escc_update_irq IRQ = 0
  sun4m_cpu_set_irq_lower Lower CPU IRQ 12
  sun4m_cpu_reset_interrupt Reset CPU IRQ 12
??
  escc_mem_writeb_ctrl Write channel a, reg[0] = 0x03
z8530sc.c:zsc_intr_hard()
  escc_mem_readb_ctrl Read channel a, reg[3] = 0x00

I'm not sure where we write 0x03 to reg[0] (rr0) nor why we read reg[3] (rr3)
twice.  Looking at the interrupt handler in locore.s, we should stop if a
handler returns 1 and the QEMU output doesn't distinguish chips, so it's not
clear to me if the second read is the other chip (zs1) or not.

Finally, running with the all the trace flags on alters the timing, so that
I can usually get to the "root device" prompt, and trying to output anything
in the interrupt handler when we successfully handle a transmit interrupt is
prone to generating infinite loops, so I'm resorting to updating counters
and querying them later.

With extra printf's, I do see that we often write the output on CPU 1, 2,
or 3 after we have booted the secondary processors.  However, at the point
we spin with IRQ 12 set and both ESCC's returning 0, the previous write is
often done by CPU 0.  The hard interrupt handle always seems to run on CPU
0 though.

I suspect that there is a timing bug in QEMU's interrupt handling that means
it will sometimes fail to deassert the CPU IRQ when the the ESCC is not
interrupting.  However, there is also something odd with our handling of
interrupts with SMP, so presumably the real chip behaves oddly as well.  I
hope to have my SS20 unpacked soon, so I'll get a chance to compare with
real hardware.

Regards,

Julian

--