rwlock changes

[Andrew Doran <ad%netbsd.org@localhost>]

Hi,

http://www.netbsd.org/~ad/rwlock.diff

This diff does the following:

- Use atomic ops directly, since rwlocks work the same way on all platforms.
- Try to make it a bit more cache efficient, and use branch hints.
- Fix a bug in rw_downgrade() where the turnstile lock was not released.
- Remove a couple of redundant assertions.
- Use atomic_swap instead of atomic_cas in one place where it's safe.
- After acquiring turnstile lock in rw_vector_enter, check if owner is
  running again and spin if so.

It also changes the handoff/release algorithm to work like this:

         * If we are releasing a write lock, then prefer to wake all
         * outstanding readers.  Otherwise, wake one writer if there
         * are outstanding readers, or all writers if there are no
         * pending readers.  If waking one specific writer, the writer
         * is handed the lock here.  If waking multiple writers, we
         * set WRITE_WANTED to block out new readers, and let them
         * do the work of acquring the lock in rw_vector_enter().

So, direct handoff is only done when there are readers involved (wakeing
readers, or readers pending on the lock). Otherwise threads being awoken
must acquire the lock themselves, making it work like an adaptive mutex in
that case. It weakens the defence against starvation slightly but I'm
fairly confident that it shouldn't be a problem.

The sum of these changes is a reduction of 4% in build time on an 8-core
machine. There is less idle time during the build because the windows where
a rwlock can be held but the owner is not running is reduced.

Comments?

Thanks,
Andrew