Subject: Re: Best practice to stack LKMs (export a service hook)?
To: None <tech-kern@NetBSD.org>
From: Bill Studenmund <wrstuden@netbsd.org>
List: tech-kern
Date: 12/19/2005 16:22:09
--UHN/qo2QbUvPLonB
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Mon, Dec 19, 2005 at 10:35:39PM +0100, Gerhard Sittig wrote:
> [ in case you remember, this is for my NetBSD port of www.dazuko.org,
>   current work is done under NetBSD 2.0.2 ]
>=20
> The situation is this:  There are two LKMs, one providing a service for
> another -- or vice versa, one hooking into the other by means of a
> callback.  What is the correct way to export the callback hook from the
> first module for the second module?
>=20
> The alternatives I can see:  A function pointer variable which simply
> gets set and unset from the second module.  Some register and unregister
> functions.  If so, how to export them?  Introduce new syscalls?  Or just
> export the routines?  Although there is only one "user" module for the
> exporting module.  No need for chains or something along these lines.

Just export the routine.

Actually, export the routine and also an addref/decrec routine.

> Here is what I did, which works but I'm just not sure if it's an
> acceptable way.  I took the first, simple approach.
>=20
> The base LKM named dazukofs(4) (a VFS module, hooking all appropriate
> vnode ops and "asking" the upper module for permission for the access)
> has a declaration in its src/miscfs/dazukofs/dazuko.h header file:
>=20
>     enum dazukofs_opcode {
> 	DAZ_OP_OPEN, DAZ_OP_CREATE, DAZ_OP_CLOSE, ...
>     };
>=20
>     union dazukofs_vop_extra { ... };
>=20
>     extern int (*dazukofs_vop_hook) __P((
> 	enum dazukofs_opcode code,
> 	struct proc *procp,
> 	struct ucred *credp,
> 	struct vnode *vp,
> 	const char *fn,
> 	union dazukofs_vop_extra *extra));
>=20
> and an appropriate variable in its implementation .c file:
>=20
>   int (*dazukofs_vop_hook) __P(... params ...) =3D NULL;
>=20
> The unload() routine refuses to unload the base LKM when the hook is
> set:
>=20
>     if (dazukofs_vop_hook !=3D NULL) {
> 	return(EBUSY);
>=20
>=20
> The client LKM named dazuko(4) (a DEV module implementing access control
> and providing a character device to the user space for Dazuko enabled
> daemons) implements the callback (or better: upcall) routine:
>=20
>   static int netbsd_dazuko_vn_hook(... params ...);
>=20
> and simply registers and unregisters this routine by means of a simple
> assignment to the base LKM's variable:
>=20
>   dazukofs_vop_hook =3D netbsd_dazuko_vn_hook;
>   ...
>   dazukofs_vop_hook =3D NULL;
>=20
> As stated above there is only this one dazuko(4) module hooking into
> dazukofs(4).  Multiple registered hooks are not supported.  Neither in
> parallel (in a table) nor in sequence (in a chain).
>=20
>=20
> Using the variable from within the base dazukofs(4) module is rather
> straight forward:
>=20
>   if (dazukofs_vop_hook !=3D NULL) {
>     ...
>     rc =3D dazukofs_vop_hook(...);
>     ...
>   }
>=20
> Although now that I cite the code I noticed that the pointer might get
> unset between the check and the call, so I may have to insert some kind
> of locking here.  And export the lock as well as the pointer variable so
> the hooking module will take part in the synchronization.

As above, swap them around and just have a register/deregister use=20
routine.

> Which again suggests that an "accessor" function is more appropriate
> than I thought above.  So I would have to export a set, a query, a lock
> and an unlock routine instead of the variable itself and the lock.
>=20
> The most important question here is:  Can it happen that the dazuko(4)
> module gets unloaded while in parallel a vnode op takes place?  That's
> when the pointer might get dereferenced while it's NULL.

Yes. Unlikely, since the administrator has to undo it, but it can happen.=
=20
Thus the need for a "don't unload me" routine.

> PS:  Yes, I noticed that the CVS repo suggests that the NetBSD project
> now prefers ANSI C style over K&R declarations.  I will gladly switch to
> ANSI style later (actually have been there first but switched to K&R
> after seeing the 2.0.2 source tree :) ), but I want to get the function
> right before doing style improvements.
>=20
> Another PS:  I did try to put both parts into a single LKM.  This worked
> in principle, but the modstat(8) output with only one LKM type spec does
> not look half as nice as the current setup with one VFS and one DEV
> module separately listed.  Plus I wanted to keep the VFS hooks and the
> access control parts split if possible.

Sounds reasonable. I had a custom vfs and device driver when I did the=20
nastore work.

> Yet another PS:  The above approach needs the -s switch for modload(8)
> when loading dazukofs(4).  Not exporting the module's symbols will
> prevent the dazuko(4) module from loading.  This is not too nice for
> users, but maybe could be solved by means of /etc/lkm.conf (haven't
> checked yet).

We're still improving this part of things. :-)

Take care,

Bill

--UHN/qo2QbUvPLonB
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (NetBSD)

iD8DBQFDp06xWz+3JHUci9cRAl0+AJ9b7wPs0gEdg+VtOKDB6mI/Pd6PYQCfTKi/
Zfzu9KRWywYykZ6q2OmaSAc=
=LcQI
-----END PGP SIGNATURE-----

--UHN/qo2QbUvPLonB--