Re: Per-CPU Unit (PCU) interface

[Andrew Doran <ad%netbsd.org@localhost>]

On Mon, Jan 24, 2011 at 03:27:58PM +0000, Mindaugas Rasiukevicius wrote:
> Hello,
> 
> While looking at the bugs on still work-in-progress mips64 FPU code on
> Matt's branch, it occurred to me that we could abstract SMP handling
> complexity into MI interface.  Basically, all architectures are using
> similar logic for FPU handling on SMP, but each have own variations,
> confusions, and therefore each fall into the bugs.  Hence, PCU:
> 
> http://www.netbsd.org/~rmind/subr_pcu.c
> http://www.netbsd.org/~rmind/pcu.h
> http://www.netbsd.org/~rmind/pcu.diff

I have three problems with this:

- It uses our xcall interface which provides a mailbox to pass messages 
  between CPUs.  The intent seems to be to change the above code to use
  IPIs in the future. An MI IPI interface would likely not have mailbox 
  capability, as it would probably render the interface+implementation
  too specific and too inefficient for general use.  I can envison some
  other framework built on top of a raw IPI interface.

- When re-implementing the x86 lazy FPU switching for 5.0 I dropped
  the notion of discarding state.  There's very little benefit, but
  many drawbacks with potential race conditions.  It's easier just to
  save whatever state is on the CPU.  You lose an optimization for a 
  corner condition and gain code that's easier to maintain and prove.

- I would like to see immediate users of this interface, ideally at
  the time of commit.  In its current form it seems not an acceptable
  replacement for the x86
  lazy FPU code because it uses xcalls and those are somewhat 
  heavyweight.  Efficiency is important in this case because it's
  the raison d'etre for lazy FPU context switch.

> Few notes:
> 
> - MD code provides pcu_state_save() and pcu_state_load() primitives
> via pcu_ops_t.
> 
> - PCU interface provides pcu_load(), pcu_save_lwp(), pcu_discard()
> and other routines, which carry the synchronisation logic.  See the
> "Concurrency notes" description in the top.
> 
> - There can be multiple PCUs, so this can be re-used not only for FPU,
> but any similar MD functionality, e.g. PowerPC AltiVec.
> 
> - Instead of IPIs, PCU is currently using XC_HIGHPRI cross-calls and
> therefore is running at IPL_SOFTCLOCK.
> 
> - Once there is MI IPI support, it is ~trivial to convert the code to
> use them by: 1) splsoftclock() -> splhigh() 2) replacing xc_unicast()
> calls with cpu_send_ipi() and moving them *before* splx().
> 
> API is not yet set in stone, but I think the essential logic is there.
> Matt is trying this code on mips64.  I will try to adapt x86 to it.
> 
> Please review.
> 
> -- 
> Mindaugas