Re: evtchn_upcall_mask/pending synchronization

["Mathew\, Cherry G." <c%bow.st@localhost>]

Taylor R Campbell <riastradh%NetBSD.org@localhost> writes:

>> Date: Wed, 13 Jul 2022 12:00:54 +0000
>> From: "Mathew\, Cherry G." <c%bow.st@localhost>
>> 
>> Taylor R Campbell <riastradh%NetBSD.org@localhost> writes:
>> 
>> > In any context where curcpu() is stable, under this contract, it
>> 
>> Can you elaborate on what you mean by "stable" ? If you mean the
>> VCPU executing the current critical section code will not be rescheduled
>> onto a different physical CPU, I don't think there is any such explicit
>> guarantee.
>
> In NetBSD/xen, curcpu() corresponds to a Xen VCPU.  This changes only
> if NetBSD itself chooses, in a timer interrupt, to preempt the
> interrupted lwp and migrate it to another CPU (i.e., another Xen
> VCPU).
>
> What Xen does behind the scenes with PCPUs is not relevant -- if Xen
> preempts a VCPU running on one PCPU and chooses to migrate it to
> another PCPU, then it will do so transparently to the guest by issuing
> the necessary memory barriers (and, for the clock, updating the tsc
> generation number, &c.); I would expect the guest has no control over
> when Xen does this, since otherwise a recalcitrant guest could hold up
> Xen suspend/migrate/resume indefinitely or similar.
>

Sorry, my mistake - I fully confused myself wrt that ( was misremembering a
swapgs dance related to TLS from reading code 10years ago).

In summary, all I have to add (hopefully of value) is that the xen docs
don't explicitly guarantee any VCPU/pCPU pinning unless explicitly
requested (after suitable checks).


> NetBSD curcpu() is stable in the following contexts:
>
> - interrupt context (hard or soft)
> - while the IPL is raised with splvm or higher
> - between kpreempt_disable and kpreempt_enable
> - while curlwp->l_pflag has LP_BOUND set
> - while cold
>
> In any other context, a timer interrupt (presumably delivered, in
> NetBSD/xen, by an upcall) can trigger preemption and migration, at
> which point whatever value you read out of curcpu() is stale and you
> are no longer allowed to touch it.
>
>> > If it is, maybe it should assert the fact; if not, then the definition
>> > in sys/arch/xen/x86/xen_intr.c is currently broken, and it needs to
>> > use kpreempt_disable/enable.
>> 
>> I'm not sure how kpreeempt_disable/enable will help in the case where
>> curcpu() could randomly be switched by the hypervisor to reflect a VCPU
>> rescheduled onto a different physical CPU ?
>
> kpreempt_disable ensures that NetBSD doesn't migrate the lwp to
> another CPU (i.e., Xen VCPU), so between kpreempt_disable/enable, it
> is safe to call curcpu() and use the result.
>
>
> (curlwp_bind/bindx also makes it safe to use curcpu().  The difference
> is that with curlwp_bind/bindx, even if the current lwp is not
> _migrated_ to another (V)CPU, other lwps may preempt it and run on the
> same (V)CPU before it is allowed to continue on that (V)CPU.  With
> kpreempt_disable, other lwps can't run on the (V)CPU until
> kpreempt_enable -- only curlwp and interrupt handlers can run on it.
> Either one might work, and they cost approximately the same to
> execute, but kpreempt_disable/enable makes reasoning a little easier
> because you don't have to think about whether other lwps can run in
> this very small window.)

I don't have anything useful to add here - it looks to me like the code
could behave similar to what is on native - and that would be an added
opportunity to unify xen and native x86 NetBSD code.

Thanks,
-- 
~cherry