Subject: Concern about IPL levels and spinlocking
To: None <tech-kern@netbsd.org>
From: Bill Studenmund <wrstuden@netbsd.org>
List: tech-kern
Date: 12/17/2005 23:32:50
--XF85m9dhOBO43t/C
Content-Type: text/plain; charset=unknown-8bit
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

It's been looking into what it would take for us to start spinlocking our=
=20
interrupt handlers. It's been pointed out that we really need to spinlock=
=20
things in interrupt-handler order, and I agree with that. So I started=20
looking at what our interrupt handler levels were, assuming we would have=
=20
to fix things in order from the highest to lowest.

I'd expected that I'd find one common hierarchy, with a few additions and
a few deletions (like no audio if the system didn't have audio). =20
Unfortunately that's not what I found.

I looked at the IPL levels and spl=A0code for the architectures which have a
GENERIC.MP. The architectures (machines) I found were alpha, amd64, i386
[same as amd64], macppc, sparc, sparc64, and vax. I didn't look at non-SMP
architectures as spinlocks won't matter for them.

I found some common threads. spllock =3D=3D splhigh on all architectures, a=
nd=20
splnet > splbio > splsoftnet. Also, splstatclock >=3D splclock.

But I found some spl levels swap relative ordering. For instance, on amd64
& i386, splipi > splhigh. On alpha and vax, though, splhigh < splipi. That
worries me, as if we ever write MI code that ends up depending on being
able to pass IPIs around, it'll fail on alpha & vax. And since splipl is=20
an MD definition (macppc and sparc don't seem to have it), MI code would=20
not directly see the problem. In fact, on vax, splipi < most other spl=20
levels.

Another issue I found was that I expected spltty > splnet > splbio >=20
splsoftserial. I guess that's from my mac68k days. Alpha is close, with=20
spltty =3D=3D splnet =3D=3D splbio > splsoftserial. But on amd64/i386,=20
spltty > splsoftserial > splnet > splbio, and on vax splnet > spltty =3D=3D=
=20
splbio > splsofterial. I'm really not sure how to really make something=20
like ppp work reliably like this. :-|

I guess one final irritation is that on amd64/i386, macppc, and vax,=20
splaudio < splsched, but on sparc and sparc64, splsched > splaudio!

I'm really not sure what to do about this, but it'll be real hard to move=
=20
to a fine-grained spin locking on all MP architectures if MI code doesn't=
=20
have one hierarchy to code to.

Thoughts? I'm not trying to say we have to fix this _now_, but thoughts on=
=20
the right thing to do will be good to have. :-)

Take care,

Bill

--XF85m9dhOBO43t/C
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (NetBSD)

iD8DBQFDpRCiWz+3JHUci9cRAj8GAJ4mHiLc0Nxwa2ug+yVZ1tJA/bu84QCgiPhX
LIDsAXpon4o4QYN7PvGHw7g=
=nUq8
-----END PGP SIGNATURE-----

--XF85m9dhOBO43t/C--