Re: evtchn_upcall_mask/pending synchronization

[Taylor R Campbell <riastradh%NetBSD.org@localhost>]

> Date: Wed, 13 Jul 2022 12:00:54 +0000
> From: "Mathew\, Cherry G." <c%bow.st@localhost>
> 
> Taylor R Campbell <riastradh%NetBSD.org@localhost> writes:
> 
> > In any context where curcpu() is stable, under this contract, it
> 
> Can you elaborate on what you mean by "stable" ? If you mean the
> VCPU executing the current critical section code will not be rescheduled
> onto a different physical CPU, I don't think there is any such explicit
> guarantee.

In NetBSD/xen, curcpu() corresponds to a Xen VCPU.  This changes only
if NetBSD itself chooses, in a timer interrupt, to preempt the
interrupted lwp and migrate it to another CPU (i.e., another Xen
VCPU).

What Xen does behind the scenes with PCPUs is not relevant -- if Xen
preempts a VCPU running on one PCPU and chooses to migrate it to
another PCPU, then it will do so transparently to the guest by issuing
the necessary memory barriers (and, for the clock, updating the tsc
generation number, &c.); I would expect the guest has no control over
when Xen does this, since otherwise a recalcitrant guest could hold up
Xen suspend/migrate/resume indefinitely or similar.

NetBSD curcpu() is stable in the following contexts:

- interrupt context (hard or soft)
- while the IPL is raised with splvm or higher
- between kpreempt_disable and kpreempt_enable
- while curlwp->l_pflag has LP_BOUND set
- while cold

In any other context, a timer interrupt (presumably delivered, in
NetBSD/xen, by an upcall) can trigger preemption and migration, at
which point whatever value you read out of curcpu() is stale and you
are no longer allowed to touch it.

> > If it is, maybe it should assert the fact; if not, then the definition
> > in sys/arch/xen/x86/xen_intr.c is currently broken, and it needs to
> > use kpreempt_disable/enable.
> 
> I'm not sure how kpreeempt_disable/enable will help in the case where
> curcpu() could randomly be switched by the hypervisor to reflect a VCPU
> rescheduled onto a different physical CPU ?

kpreempt_disable ensures that NetBSD doesn't migrate the lwp to
another CPU (i.e., Xen VCPU), so between kpreempt_disable/enable, it
is safe to call curcpu() and use the result.


(curlwp_bind/bindx also makes it safe to use curcpu().  The difference
is that with curlwp_bind/bindx, even if the current lwp is not
_migrated_ to another (V)CPU, other lwps may preempt it and run on the
same (V)CPU before it is allowed to continue on that (V)CPU.  With
kpreempt_disable, other lwps can't run on the (V)CPU until
kpreempt_enable -- only curlwp and interrupt handlers can run on it.
Either one might work, and they cost approximately the same to
execute, but kpreempt_disable/enable makes reasoning a little easier
because you don't have to think about whether other lwps can run in
this very small window.)