Re: Atomic migration chains, balancing fixes.

[Mindaugas Rasiukevicius <rmind%NetBSD.org@localhost>]

Andrew Doran <ad%netbsd.org@localhost> wrote:
> - I think the KASSERT(l->l_stat == LSRUN) in cpu_xc_offline() is broken
> because something could, for example, make an LWP stopped or suspended.

Correct. I think we can double-lock run-queues immediately, and migrate
everything in a single loop.

> - For cpu_need_resched(l->l_cpu, RESCHED_KPREEMPT), spc_mutex also needs to
> be held, not just spc_lwplock.

OK.

> - l->l_stat = LSIDL: I think it will break lwp_suspend().

As noted in first email - it would return EINTR. We may handle that in same
way as LSONPROC (just checking for l_target_cpu != NULL), or use LSONPROC as
migration state (but then additional holding of spc_lwplock would be needed).

> - In mi_switch: if (__predict_false(l->l_stat == LSONPROC ...) Why check
> something that is known to be true?

True. Though in else case only.

> I still prefer the scheme described in PR/38169. In fact, I think it could
> be even simpler - I will update the PR. ...

OK, I have committed balancing fixes separately.
Here are the clean patches for both approaches:

http://www.netbsd.org/~rmind/mchains_3.diff
vs
http://www.netbsd.org/~rmind/midle_1.diff

> ... The cost of switching to the idle LWP to do a forced migration is
> proably small when compared to the penalty that the migrating LWP will
> suffer because of cache traffic and TLB refill.

Can you explain (at least shortly, please) your point about cache traffic and
TLB refill, or how it suffer more than migration via idle LWP?

Thanks.

-- 
Best regards,
Mindaugas
www.NetBSD.org