Re: Ubiquitous ucas(9)

[Mindaugas Rasiukevicius <rmind%netbsd.org@localhost>]

Jason Thorpe <thorpej%me.com@localhost> wrote:
> <...>
> 
> a) acquire the ucas_critical_mutex.
> b) disable preemption.
> c) go to splhigh().
> d) set ucas_critical_owning_cpu to curcpu().
> e) membar_enter()
> f) Trigger an IPI on all-but-owning-cpu to enter the ucas_critical_gate.
> g) wait for all-but-owning cpu to enter the gate.
> 
> Then we perform the ufetch-compare-maybe-ustore.  Then:
> 
> h) membar_exit()
> i) set ucas_critical_owning_cpu to NULL to release the other CPUs from
> the gate. j) splx(s)
> k) re-enable preemption
> l) release ucas_critical_mutex.
> 
> <...>

A few comments:

- Kernel preemption is implicitly disabled at IPL_HIGH, so you do not need
to do it explicitly.

- Consider re-using the cpu_lock instead of adding ucas_critical_mutex.

- Also, consider using kcpuset_copy(&ucas_kcpuset, kcpuset_attached),
plus clearing the current CPU; ucas_kcpuset can be global and it would
be already protected by cpu_lock.  Alternatively, you can add a third
parameter to ipi_trigger_multi(), e.g. "bool exclude_curcpu", as it is
arguably a common case.  Then you would just use kcpuset_attached.

- You can get away with one counter (eliminating ucas_critical_owning_cpu)
if you set it to ncpu (well, kcpuset_countset(kcpuset_attached) - 1) and
decrement.  Note: once it reaches zero, you can use -1 in order to release
the waiters on other CPU.

- The counter must be volatile or accessed through a volatile cast.

-- 
Mindaugas