Re: sysctl_teardown() with dynamic nodes?

[Paul Goyette <paul%vps1.whooppee.com@localhost>]

On Thu, 22 Oct 2015, Paul Goyette wrote:

> A little bit of progress in the on-going saga of sysctl_teardown() not
> working as expected...
>
> I still can't determine what the problem is, but I can reproduce it at
> will.  I put in some extra debugging stuff (a few calls to
> sysctl_log_print()) and here's what I see (manual line-breaks inserted
> for readability)
>
> * The sysv_ipc module gets autoloaded and sets up its sysctl stuff.
>
> 	sysv_ipc_modcmd: clog = 0xfffffe81f04dc318
> 	root 0xffffffff806c0420 left 62 size 128 content \
> 		version 715 type 2 len 3: 1 82 1369 \
> 		version 714 type 2 len 3: 1 82 1368 \
> 		version 713 type 2 len 3: 1 82 1367 \
> 		version 712 type 2 len 3: 1 82 9 \
> 		version 711 type 2 len 3: 1 82 8 \
> 		version 710 type 2 len 3: 1 82 7 \
> 		version 709 type 2 len 3: 1 82 6 \
> 		version 708 type 4 len 3: 1 82 5 \
> 		version 707 type 2 len 3: 1 82 1366 \
> 		version 706 type 2 len 3: 1 82 1365 \
> 		version 705 type 5 len 3: 1 82 1 end
>
> * The module gets auto-unloaded, and syscall_teardown() returns
>  successfully.
>
> 	sysv_ipc_modcmd: clog = 0xfffffe81f04dc318
> 	root 0xffffffff806c0420 left 62 size 128 content \
> 		<remainder of sysctl_log_print() elided>
>
> * The module gets reloaded, and once again the sysctllog output
>  looks correct.  Everything is the same except for the version and
>  node numbers:
>
> 	sysv_ipc_modcmd: clog = 0xfffffe810f3b97d0
> 	root 0xffffffff806c0420 left 62 size 128 content \
> 		version 752 type 2 len 3: 1 83 1374 \
> 		version 751 type 2 len 3: 1 83 1373 \
> 		version 750 type 2 len 3: 1 83 1372 \
> 		version 749 type 2 len 3: 1 82 9 \
> 		version 748 type 2 len 3: 1 82 8 \
> 		version 747 type 2 len 3: 1 82 7 \
> 		version 746 type 2 len 3: 1 82 6 \
> 		version 745 type 4 len 3: 1 82 5 \
> 		version 744 type 2 len 3: 1 83 1371 \
> 		version 743 type 2 len 3: 1 83 1370 \
> 		version 742 type 5 len 3: 1 82 1 end

YIKES!!!

Notice that the newly-created entries with dynamically-allocated
numbers have the wrong value in the penultimate part of the mib-name!

These should all be 1.82.xxx (for kern.ipc.xxx) but the dynamic
entries are all 1.83.xxx.

This is not good :(

> * Now, when auto-unload comes around, we get a failure for each of
>  the dynamically-assigned nodes (1370 thru 1374).  Each of them
>  fails with error == 20 (ENOTDIR).  Here's the debug output from
>  the first failure (note that the first entry in the log has been
>  consumed before the call to sysctl_log_print()).
>
> 	sysctl_teardown: log 0xfffffe810f3b97d0 error 20 \
> 		type 2 ver 752 name 1 83 1374
> 	root 0xffffffff806c0420 left 68 size 128 content \
> 		version 751 type 2 len 3: 1 83 1373 \
> 		version 750 type 2 len 3: 1 83 1372 \
> 		version 749 type 2 len 3: 1 82 9 \
> 		version 748 type 2 len 3: 1 82 8 \
> 		version 747 type 2 len 3: 1 82 7 \
> 		version 746 type 2 len 3: 1 82 6 \
> 		version 745 type 4 len 3: 1 82 5 \
> 		version 744 type 2 len 3: 1 83 1371 \
> 		version 743 type 2 len 3: 1 83 1370 \
> 		version 742 type 5 len 3: 1 82 1 end

The failure is certainly expected, given the corruption in the clog
(noted above).  There certainly aren't any children of 1.83, which
is kern.boottime (a "leaf" in the tree).

> * And according to sysctl, these entries still exist in the tree:
>
> 	# sysctl -M kern.ipc | sed -e "s/ flag/\\
> 		flag/"
> 	kern.ipc (1.82): CTLTYPE_NODE, children 8/16, size 96,
> 		flags 0x200<READONLY,PERMANENT>, ver=758
> 	kern.ipc.sysvmsg (1.82.2): CTLTYPE_INT, size 4,
> 		flags 0x200<READONLY,PERMANENT>, ver=104
> 	kern.ipc.sysvsem (1.82.3): CTLTYPE_INT, size 4,
> 		flags 0x200<READONLY,PERMANENT>, ver=105
> 	kern.ipc.sysvshm (1.82.4): CTLTYPE_INT, size 4,
> 		flags 0x200<READONLY,PERMANENT>, ver=106
> 	kern.ipc.msgmni (1.82.1370): CTLTYPE_INT, size 4,
> 		flags 0x70<READWRITE>, func=0xffffffff808fc1db, ver=743
> 	kern.ipc.msgseg (1.82.1371): CTLTYPE_INT, size 4,
> 		flags 0x70<READWRITE>, func=0xffffffff808fc0eb, ver=744
> 	kern.ipc.semmni (1.82.1372): CTLTYPE_INT, size 4,
> 		flags 0x70<READWRITE>, func=0xffffffff808fdd93, ver=750
> 	kern.ipc.semmns (1.82.1373): CTLTYPE_INT, size 4,
> 		flags 0x70<READWRITE>, func=0xffffffff808fdccc, ver=751
> 	kern.ipc.semmnu (1.82.1374): CTLTYPE_INT, size 4,
> 		flags 0x70<READWRITE>, func=0xffffffff808fdc05, ver=752

Again, since the clog was corrupt, these entries weren't deleted, so
perfectly understandable that they're still in the tree.

This might seem to prove that the entries were correctly created, and
that only the clog is corrupt.  But the way the clog is constructed,
it is possible that the newly-created nodes were attached to the wrong
parent!  (The mib-name in the clog entry is created by walking back
_UP_ the tree, towards the root.)

> I haven't tried this time, but on previous iterations, any attempt to
> actually read these lingering entries results in a crash, since the
> func pointers are no longer valid (the module has been unloaded).
>
> It doesn't make a lot of sense to me how this could happen.  The
> failing entries are on either side of five entries that are torn-
> down successfully.  The only code I can find that would return a
> ENOTDIR for sysctl_teardown() is in routine sysctl_locate()
>
> int
> sysctl_locate(struct lwp *l, const int *name, u_int namelen,
> 	      const struct sysctlnode **rnode, int *nip)
> {
> 	...
> 	for (ni = 0; ni < namelen; ni++) {
> 		/*
> 		 * walked off bottom of tree
> 		 */
> 		if (node == NULL) {
> 			if (SYSCTL_TYPE(pnode->sysctl_flags) == CTLTYPE_NODE)
> 				error = ENOENT;
> 			else
> 				error = ENOTDIR;
> 			break;
> 		}
> 	...
>        }
>
> 	*rnode = pnode;
> 	if (nip)
> 		*nip = ni;
>
> 	return (error);
> }
>
> I just don't see how this could return ENOTDIR for some entries and
> succeed for others.
>
>
> +------------------+--------------------------+-------------------------+
> | Paul Goyette     | PGP Key fingerprint:     | E-mail addresses:       |
> | (Retired)        | FA29 0E3B 35AF E8AE 6651 | paul at whooppee.com    |
> | Kernel Developer | 0786 F758 55DE 53BA 7731 | pgoyette at netbsd.org  |
> +------------------+--------------------------+-------------------------+

+------------------+--------------------------+-------------------------+
| Paul Goyette     | PGP Key fingerprint:     | E-mail addresses:       |
| (Retired)        | FA29 0E3B 35AF E8AE 6651 | paul at whooppee.com    |
| Kernel Developer | 0786 F758 55DE 53BA 7731 | pgoyette at netbsd.org  |
+------------------+--------------------------+-------------------------+