Re: percpu_foreach() does not execute remotely

[Jason Thorpe <thorpej%me.com@localhost>]

> On Jan 28, 2020, at 7:11 PM, Taylor R Campbell <campbell+netbsd-tech-kern%mumble.net@localhost> wrote:
> 
>> Date: Tue, 28 Jan 2020 18:54:59 -0800
>> From: Jason Thorpe <thorpej%me.com@localhost>
>> 
>> Something like this.  I haven't tested this yet, but I will shortly.
>> Only high-pri xcalls are supported in this -- to make low-pri xcalls
>> work, you'd need to change how percpu_cpu_swap() interlocks with
>> percpu_foreach().  High-pri doesn't need to take the
>> percpu_swap_lock because percpu_cpu_swap() protects its critical
>> section with splhigh().
> 
> Why is there a restriction to priority and why is percpu_swap_lock
> relevant?  I'm probably missing something else that should be obvious,

Hm, actually, I think you are right, but because of a property of xcalls themselves...

-- percpu_cpu_swap() is a low-pri xcall, and thus won't run concurrently with a low-pri percpu_foreach_xcall(), so no need to grab percpu_swap_lock, the way that the basic percpu_foreach() does.

-- And as stated before, percpu_cpu_swap()'s critical section is protected by splhigh(), which provides synchronization on the remote cpu with a high-pri percpu_foreach_xcall().

It warrants a comment, so I'll add one.


> but why not just...
> 
> struct percpu_foreach_xcall {
> 	percpu_callback_t	cb;
> 	void			*cookie;
> };
> 
> static void
> percpu_foreach_xc(void *vpercpu, void *vpfx)
> {
> 	struct percpu *percpu = vpercpu;
> 	struct percpu_foreach_xcall *pfx = vpfx;
> 	void *ptr;
> 
> 	ptr = percpu_getref(percpu);
> 	(*pfx->cb)(ptr, pfx->cookie, curcpu());
> 	percpu_putref(percpu);
> }
> 
> void
> percpu_foreach_xcall(struct percpu *percpu, unsigned xcflags,
>    percpu_callback_t cb, void *cookie)
> {
> 	struct percpu_foreach_xcall pfx = { .cb = cb, .cookie = cookie };
> 	CPU_INFO_ITERATOR cii;
> 	struct cpu_info *ci;
> 
> 	for (CPU_INFO_FOREACH(cii, ci))
> 		xc_wait(xc_unicast(xcflags, &percpu_foreach_xc, percpu, &pfx));
> }

-- thorpej