kern/38169: Simplifcation for running LWP migration

[ad%NetBSD.org@localhost]

>Number:         38169
>Category:       kern
>Synopsis:       Simplifcation for running LWP migration
>Confidential:   no
>Severity:       non-critical
>Priority:       low
>Responsible:    kern-bug-people
>State:          open
>Class:          change-request
>Submitter-Id:   net
>Arrival-Date:   Wed Mar 05 08:35:00 +0000 2008
>Originator:     ad%netbsd.org@localhost
>Release:        4.99.55
>Organization:
The NetBSD Project
>Environment:
n/a
>Description:
To handle migration of LWPs in the ONPROC state, mi_switch() double
locks run queues. It locks the run queues of the current CPU and also
those of the target CPU.

Technically that's fine, but human factors come into play. It makes the 
locking in mi_switch() more complicated and as a result it's more
difficult to understand what's going on. That might not be a problem,
but even without the double runqueue locking, the locking dances are
already quite complicated.

>How-To-Repeat:
Code inspection.
>Fix:
One possible way to make it less complicated:

- Maintain a per-CPU linked list which contains threads to be migrated. 
  An LWP would be inserted onto this list by sched_enqueue() if (l == 
  curlwp) and it is due to be migrated.

- Add to the meaning of the LSRUN state somehow so that it also means
  a thread could be on a CPU-local migration queue. This could be a bit
  in lwp_t::l_flag.

- In sched_nextlwp(), return NULL if a migration is pending on the local
  CPU. This will force a switch to the idle loop. Thread priority comes
  into question here: should we always migrate or only if the next 
  thread on the runqueue is of lower priority?

- In the idle loop, before other processing, examine the local migration
  queue. While it's not empty, do the double locking dance to migrate
  threads to their target CPUs. Then look for more jobs to run (as
  usual).