Subject: Re: MAKEDEV/MAKEDEV.subr issues in today's i386/amd64 builds (1/2)
To: Alan Barrett <apb@cequrux.com>
From: Greg A. Woods <woods@planix.com>
List: current-users
Date: 03/01/2007 16:20:22
From: "Greg A. Woods" <woods@planix.com>
To: Alan Barrett <apb@cequrux.com>,
	NetBSD-current Users's Discussion List <current-users@netbsd.org>
Subject: Re: MAKEDEV/MAKEDEV.subr issues in today's i386/amd64 builds
In-Reply-To: <20070228203251.GE14658@snowdrop.l8s.co.uk>
References: <6e9741c60702281105o17726a06gb6298ea76540630b@mail.gmail.com>
	<20070228192423.GF8801@apb-laptoy.apb.alt.za>
	<bc8ff1fd0702281140p33625816yfbe0592eb61362a4@mail.gmail.com>
	<20070228194957.GH8801@apb-laptoy.apb.alt.za>
	<20070228195651.GC14658@snowdrop.l8s.co.uk>
	<20070228200142.GI8801@apb-laptoy.apb.alt.za>
	<20070228203251.GE14658@snowdrop.l8s.co.uk>
User-Agent: Wanderlust/2.14.0 (Africa) SEMI/1.14.6 (Maruoka) FLIM/1.14.6 (Marutamachi) APEL/10.6 Emacs/21.3 (alpha--netbsd) MULE/5.0 (SAKAKI)
Reply-To: NetBSD-current Users's Discussion List <current-users@netbsd.org>
 lz;@-iwMv_u\6uIEKR0KY"=MzoQH#CrqBN`nG_5B@rrM8,f~Gr&h5a\=<t0loVf0$}bP=]i3OMh"n_
 _@m4/,~2`V=(-9LyW.)'`@E_fE^<4y7)BIe`A''/j-Y#gDNZERh%CCij'q-NA4F<|yjznEhd7=l^xH
 2.qD3o0IanGHERTW+z$G
MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka")
Content-Type: text/plain; charset=US-ASCII

At Wed, 28 Feb 2007 20:32:51 +0000,
David Laight wrote:
> 
> IIRC init handles MAKEDEV being somewhere else (either /etc or /sbin).

Perhaps you're thinking of my ongoing harping to move MAKEDEV into
/sbin.  I've been running my systems with such changes ever since early
in 2004 or so.


> (Or maybe /dev/MAKEDEV being a symlink)

I do that too, just in case someone is too hung up running "./MAKEDEV"
instead of using one more pathname component.


> MAKEDEV.local is a config file (of sorts) to could be deemed to belong
> in /etc.

Exactly!  I do that too.  :-)


>  Something about the way the scripts are run needs them to be
> in the same directory as each other.

With my changes below that's not necessary.

-- 
						Greg A. Woods

H:+1 416 218-0098 W:+1 416 489-5852 x122 VE3TCP RoboHack <woods@robohack.ca>
Planix, Inc. <woods@planix.com>       Secrets of the Weird <woods@weird.com>


These diffs are actually against the head of the netbsd-4 branch where
they've been well tested (except for init.8, which is against netbsd-1-6
as I have not yet had a chance to merge them forward), and the
distrib/sets/lists diffs are left as an exercise for the reader...

cvs diff: Diffing sbin/init
Index: sbin/init/Makefile
===================================================================
RCS file: /cvs/master/m-NetBSD/main/src/sbin/init/Makefile,v
retrieving revision 1.36
diff -u -r1.36 Makefile
--- sbin/init/Makefile	18 Apr 2006 11:40:26 -0000	1.36
+++ sbin/init/Makefile	14 Feb 2007 19:50:21 -0000
@@ -3,6 +3,7 @@
 
 PROG=	init
 MAN=	init.8
+#INSTALLFLAGS=-fschg
 DPADD=	${LIBUTIL}
 LDADD=	-lutil
 CPPFLAGS+=	-DMFS_DEV_IF_NO_CONSOLE -DSUPPORT_UTMP -DSUPPORT_UTMPX
@@ -10,9 +11,19 @@
 .ifdef	SMALLPROG
 CPPFLAGS+=	-DLETS_GET_SMALL
 .else
+
+# Enable "Enter pathname of shell or RETURN for sh: " prompt in single
+# user mode
+#CPPFLAGS+=	-DALTSHELL
+
+# Require the root password before entering single-user mode if
+# console is "insecure"
 CPPFLAGS+=	-DALTSHELL -DSECURE -DCHROOT
 DPADD+=		${LIBCRYPT}
 LDADD+=		-lcrypt
+
 .endif
 
 .include <bsd.prog.mk>
+
+LDSTATIC=-static
Index: sbin/init/NOTES
===================================================================
RCS file: /cvs/master/m-NetBSD/main/src/sbin/init/NOTES,v
retrieving revision 1.3
diff -u -r1.3 NOTES
--- sbin/init/NOTES	18 Apr 2006 11:40:26 -0000	1.3
+++ sbin/init/NOTES	14 Feb 2007 19:21:35 -0000
@@ -9,8 +9,10 @@
 
 	This is part of the extensive 'job control' glossary entry.
 	This specific reference says that 'init' must by default provide
-	protection from job control signals to jobs it starts --
-	it sets SIGTSTP, SIGTTIN and SIGTTOU to SIG_IGN.
+	protection from job control signals to jobs it starts -- it sets
+	SIGTSTP, SIGTTIN and SIGTTOU to SIG_IGN.  We use SIGTSTP as the
+	logical choice (by connotation) to initiate "boring" mode so we
+	don't ignore it.
 
 B.2.2.2, p206 line 889:
 
@@ -82,38 +84,98 @@
 Design notes:
 ------------
 
-your generic finite state machine
-we are fascist about which signals we elect to receive,
-	even signals purportedly generated by hardware
-handle fatal errors gracefully if possible (we reboot if we goof!!)
-	if we get a segmentation fault etc., print a message on the console
-	and spin for a while before rebooting
-	(this at least decreases the amount of paper consumed :-)
+in an ideal world init would be started with stdin/stdout/stderr
+attached to whatever the kernel found as the active console device, and
+it would be our controlling terminal so that we could receive SIGINT
+from it.  However in a the current world we must explicitly
+open("/dev/console").  We make sure to first call revoke("/dev/console")
+first though.  (There'd need to be an fdrevoke() to eliminate the hard
+pathname.)
+
+we are fascist about which signals we elect to receive, even signals
+purportedly generated by hardware.  We either catch or block signals
+rather than ignore them, so that they get reset on exec.
+
+handle fatal errors gracefully if possible (the kernel will reboot if we
+goof!!)  if we get a segmentation fault etc., print a message on stderr
+and spin for a while before rebooting (this at least decreases the
+amount of paper consumed on a printing console by constant reboots :-)
+
 apply hysteresis to rapidly exiting gettys
+
 check wait status of children we reap
-	don't wait for stopped children
-don't use SIGCHILD, it's too expensive
-	but it may close windows and avoid races, sigh
+
+don't wait for stopped children
+
+don't use SIGCHILD, it's too expensive (but it may close windows and
+avoid races, sigh)
+
 look for EINTR in case we need to change state
-init is responsible for utmp and wtmp maintenance (ick)
-	maybe now we can consider replacements?  maintain them in parallel
-	init only removes utmp and closes out wtmp entries...
 
-necessary states and state transitions (gleaned from the man page):
-	1: single user shell (with password checking?); on exit, go to 2
+init is responsible for utmp and wtmp maintenance (ick) maybe now we
+can consider replacements?  maintain them in parallel init only
+removes utmp and closes out wtmp entries...
+
+Note that 'init' cannot call reboot(2) directly because there's no way
+for 'reboot' to pass the rebootstr to it, so most process killing, and
+calling of reboot(2) is all done by one of shutdown, poweroff, reboot,
+halt, or poweroff.
+
+shutdown/poweroff/reboot/halt first tell 'init' to go catatonic (send it
+a SIGTSTP) just in case init doesn't get its SIGTERM before some tty
+session dies for whatever reason.
+
+shutdown runs /etc/rc.shutdown and then goes on a killing spree with
+kill(-1, SIGTERM), pauses, and then kills again with kill(-1, SIGKILL);
+and finally if it was given -p|-r|-h it will call reboot(2) as
+appropriate; and if not it will send 'init' a SIGUSR1 to trigger
+creation of a single-user shell, and then exit itself.
+
+powerdown|reboot|halt just go on a killing spree like shutdown does and
+then call reboot(2) as appropriate.
+
+In the course of the killing done by shutdown/powerdown/reboot/halt,
+'init' will also get a SIGTERM at which point it'll kill it's TTY
+sessions with SIGHUP just in case any of them ignore SIGTERM.
+
+shutdown/poweroff/reboot/halt MUST NOT use SIGHUP at all during their
+killing spree since that signal is often used to signify "config reload"
+to various long-running daemon processes -- SIGHUP is only valid to TTY
+session groups, just as it is in init.
+
+init is your generic finite state machine, here are the necessary states
+and state transitions:
+
+	0: start:  Given the -s flag, we start at state 1; otherwise
+	   state 2
+
+	1: single user shell: (optionally check the root password first)
+	   on exit, go to 2
+
 	2: run rc script, on exit 0 check if init.root sysctl != "/", if it
            differs then fork + chroot into the value of init.root and run
            /etc/rc inside the chroot: on exit 0, go to 3; on exit N (error),
            go to 1 (applies also to /etc/rc when init.root == "/")
+
 	3: read ttys file: on completion, go to 4.  If we did chroot in
 	   state 2, we chroot after forking each getty to the same dir
 	   (init.root is not re-read)
-	4: multi-user operation: on SIGTERM, go to 7; on SIGHUP, go to 5;
-		on SIGTSTP, go to 6
-	5: clean up mode (re-read ttys file, killing off controlling processes
-		on lines that are now 'off', starting them on lines newly 'on')
-		on completion, go to 4
-	6: boring mode (no new sessions); signals as in 4
-	7: death: send SIGHUP to all controlling processes, reap for 30 seconds,
-		then go to 1 (warn if not all processes died, i.e. wait blocks)
-Given the -s flag, we start at state 1; otherwise state 2
+
+	4: multi-user operation: wait for processes to die and collect
+	   them, respawn any that are on active ttys.  If respawn fails
+	   goto 3.  On SIGTERM go to 7; on SIGHUP go to 5; on SIGTSTP go
+	   to 6; on SIGUSR1 go to 8; on SIGINT go to 1.
+
+	5: clean up mode (re-read ttys file, killing off controlling
+	   processes on lines that are now 'off', starting them on lines
+	   newly 'on') on completion, go to 4
+
+	6: catatonic mode (no new sessions); all ttys entries are
+	   internally marked as 'off', go to 4
+
+	7: kill all ttys: logwtmp(shutdown), send SIGHUP to all spawned
+	   processes, reap for a few seconds, repeat with SIGKILL, then
+	   go to 8
+
+	8: prepare single user mode: mark all ttys internally as 'off',
+	   logwtmp(singluser), go to 1
Index: sbin/init/init.c
===================================================================
RCS file: /cvs/master/m-NetBSD/main/src/sbin/init/init.c,v
retrieving revision 1.81.2.1
diff -u -r1.81.2.1 init.c
--- sbin/init/init.c	16 Feb 2007 20:31:20 -0000	1.81.2.1
+++ sbin/init/init.c	18 Feb 2007 19:38:09 -0000
@@ -78,17 +78,14 @@
 
 #include "pathnames.h"
 
-#define XSTR(x) #x
-#define STR(x) XSTR(x)
-
 /*
  * Sleep times; used to prevent thrashing.
  */
 #define	GETTY_SPACING		 5	/* N secs minimum getty spacing */
 #define	GETTY_SLEEP		30	/* sleep N secs after spacing problem */
 #define	WINDOW_WAIT		 3	/* wait N secs after starting window */
-#define	STALL_TIMEOUT		30	/* wait N secs after warning */
-#define	DEATH_WATCH		10	/* wait N secs for procs to die */
+#define	STALL_TIMEOUT		20	/* wait N secs after warning */
+#define	DEATH_WATCH		30	/* wait N secs for procs to die */
 
 const struct timespec dtrtime = {.tv_sec = 0, .tv_nsec = 250000};
 
@@ -113,6 +110,10 @@
     __attribute__((__format__(__printf__,1,2)));
 void emergency(const char *, ...)
     __attribute__((__format__(__printf__,1,2)));
+void notice(const char *, ...)
+    __attribute__((__format__(__printf__,1,2)));
+void info(const char *, ...)
+    __attribute__((__format__(__printf__,1,2)));
 void disaster(int);
 void badsys(int);
 
@@ -137,8 +138,9 @@
 state_func_t read_ttys(void);
 state_func_t multi_user(void);
 state_func_t clean_ttys(void);
-state_func_t catatonia(void);
-state_func_t death(void);
+state_func_t go_catatonic(void);
+state_func_t go_single_user(void);
+state_func_t kill_ttys(void);
 
 enum { AUTOBOOT, FASTBOOT } runcom_mode = AUTOBOOT;
 
@@ -191,7 +193,7 @@
 state_func_t runetcrc(int);
 #ifdef SUPPORT_UTMPX
 static struct timeval boot_time;
-state_t current_state = death;
+state_t current_state = kill_ttys;
 static void session_utmpx(const session_t *, int);
 static void make_utmpx(const char *, const char *, int, pid_t,
     const struct timeval *, int);
@@ -212,26 +214,11 @@
 
 #ifdef MFS_DEV_IF_NO_CONSOLE
 
-#define NINODE 1280
-#define FSSIZE ((8192		/* boot area */				\
-	+ 2 * 8192		/* two copies of superblock */		\
-	+ 4096			/* cylinder group info */		\
-	+ NINODE * (128 + 18)	/* inode and directory entry */		\
-	+ mfile[0].len		/* size of MAKEDEV file */		\
-	+ 2 * 4096) / 512)	/* some slack */
-
-struct mappedfile {
-	const char *path;
-	char	*buf;
-	int	len;
-} mfile[] = {
-	{ "/dev/MAKEDEV",	NULL,	0 },
-	{ "/dev/MAKEDEV.local",	NULL,	0 }
-};
+#ifndef FSSIZE
+# define FSSIZE	768		/* ... in sectors */
+#endif
 
 static int mfs_dev(void);
-static void mapfile(struct mappedfile *);
-static void writefile(struct mappedfile *);
 
 #endif
 
@@ -282,10 +269,9 @@
 
 #ifndef LETS_GET_SMALL
 	/*
-	 * Note that this does NOT open a file...
 	 * Does 'init' deserve its own facility number?
 	 */
-	openlog("init", LOG_CONS, LOG_AUTH);
+	openlog("init", LOG_PERROR, LOG_AUTH);
 #endif /* LETS_GET_SMALL */
 
 
@@ -308,7 +294,7 @@
 		}
 
 	if (optind != argc)
-		warning("ignoring excess arguments");
+		warning("ignoring %d excess arguments", argc - optind);
 #else /* LETS_GET_SMALL */
 	requested_transition = single_user;
 #endif /* LETS_GET_SMALL */
@@ -320,17 +306,17 @@
 	handle(badsys, SIGSYS, 0);
 	handle(disaster, SIGABRT, SIGFPE, SIGILL, SIGSEGV,
 	       SIGBUS, SIGXCPU, SIGXFSZ, 0);
-	handle(transition_handler, SIGHUP, SIGTERM, SIGTSTP, 0);
+	handle(transition_handler, SIGHUP, SIGINT, SIGTERM, SIGTSTP, SIGUSR1, 0);
 	handle(alrm_handler, SIGALRM, 0);
 	(void)sigfillset(&mask);
 	delset(&mask, SIGABRT, SIGFPE, SIGILL, SIGSEGV, SIGBUS, SIGSYS,
-	    SIGXCPU, SIGXFSZ, SIGHUP, SIGTERM, SIGTSTP, SIGALRM, 0);
-	(void)sigprocmask(SIG_SETMASK, &mask, NULL);
+	    SIGXCPU, SIGXFSZ, SIGHUP, SIGINT, SIGTERM, SIGTSTP, SIGUSR1, SIGALRM, 0);
+	(void)sigprocmask(SIG_SETMASK, &mask, (sigset_t *) NULL);
 	(void)sigemptyset(&sa.sa_mask);
 	sa.sa_flags = 0;
 	sa.sa_handler = SIG_IGN;
-	(void)sigaction(SIGTTIN, &sa, NULL);
-	(void)sigaction(SIGTTOU, &sa, NULL);
+	(void)sigaction(SIGTTIN, &sa, (struct sigaction *) NULL);
+	(void)sigaction(SIGTTOU, &sa, (struct sigaction *) NULL);
 
 	/*
 	 * Paranoia.
@@ -352,7 +338,8 @@
 	/*
 	 * Should never reach here.
 	 */
-	return 1;
+	_exit(1);		/* avoid atexit() and other sticky goo */
+	/* NOTREACHED */
 }
 
 /*
@@ -375,7 +362,7 @@
 		sa.sa_mask = mask_everything;
 		/* XXX SA_RESTART? */
 		sa.sa_flags = sig == SIGCHLD ? SA_NOCLDSTOP : 0;
-		(void)sigaction(sig, &sa, NULL);
+		(void) sigaction(sig, &sa, (struct sigaction *) NULL);
 	}
 	va_end(ap);
 }
@@ -399,7 +386,6 @@
 /*
  * Log a message and sleep for a while (to give someone an opportunity
  * to read it and to save log or hardcopy output if the problem is chronic).
- * NB: should send a message to the session logger to avoid blocking.
  */
 void
 stall(const char *message, ...)
@@ -410,13 +396,12 @@
 	vsyslog(LOG_ALERT, message, ap);
 	va_end(ap);
 	closelog();
+	notice("pausing now for %d seconds to prevent thrashing", STALL_TIMEOUT);
 	(void)sleep(STALL_TIMEOUT);
 }
 
 /*
- * Like stall(), but doesn't sleep.
- * If cpp had variadic macros, the two functions could be #defines for another.
- * NB: should send a message to the session logger to avoid blocking.
+ * Log a warning message (like stall(), but doesn't sleep).
  */
 void
 warning(const char *message, ...)
@@ -424,7 +409,7 @@
 	va_list ap;
 
 	va_start(ap, message);
-	vsyslog(LOG_ALERT, message, ap);
+	vsyslog(LOG_EMERG, message, ap);
 	va_end(ap);
 	closelog();
 
@@ -454,7 +439,6 @@
 
 /*
  * Log an emergency message.
- * NB: should send a message to the session logger to avoid blocking.
  */
 void
 emergency(const char *message, ...)
@@ -468,6 +452,52 @@
 }
 
 /*
+ * Log a notice message.
+ */
+void
+notice(const char *message, ...)
+{
+	va_list ap;
+
+	va_start(ap, message);
+	vsyslog(LOG_NOTICE, message, ap);
+	va_end(ap);
+	closelog();
+}
+
+/*
+ * Log an informational message.
+ */
+void
+info(const char *message, ...)
+{
+	va_list ap;
+
+	va_start(ap, message);
+	vsyslog(LOG_INFO, message, ap);
+	va_end(ap);
+	closelog();
+}
+
+/*
+ * return name of signal signum.
+ */
+static char *
+strsigname(int signum)
+{
+	static char signame[32];		/* SIG2STR_MAX */
+
+	if (signum >= sys_nsig || signum < 0)
+		snprintf(signame, sizeof(signame), "#%d", signum);
+	else {
+		strncpy(signame, sys_signame[signum], sizeof(signame));
+		signame[sizeof(signame) - 1] = '\0';
+	}
+
+	return signame;
+}
+
+/*
  * Catch a SIGSYS signal.
  *
  * These may arise if a system does not support sysctl.
@@ -478,8 +508,10 @@
 {
 	static int badcount = 0;
 
-	if (badcount++ < 25)
+	if (badcount++ < 25) {
+ 		warning("signal SIG%s: %s", strsigname(sig), strsignal(sig));
 		return;
+	}
 	disaster(sig);
 }
 
@@ -490,8 +522,7 @@
 disaster(int sig)
 {
 
-	emergency("fatal signal: %s", strsignal(sig));
-	(void)sleep(STALL_TIMEOUT);
+	stall("fatal signal SIG%s: %s", strsigname(sig), strsignal(sig));
 	_exit(sig);		/* reboot */
 }
 
@@ -569,7 +600,6 @@
 #ifndef LETS_GET_SMALL
 /*
  * Close out the accounting files for a login session.
- * NB: should send a message to the session logger to avoid blocking.
  */
 void
 clear_session_logs(session_t *sp, int status)
@@ -588,8 +618,10 @@
 #endif
 
 /*
- * Start a session and allocate a controlling terminal.
- * Only called by children of init after forking.
+ * Start a session and allocate a controlling terminal via the specified name
+ *
+ * Only called by children of init (i.e. for /etc/rc and the single-user shell)
+ * after forking the process that will run them.
  */
 void
 setctty(const char *name)
@@ -637,6 +669,8 @@
 	did_multiuser_chroot = 0;
 #endif /* !LETS_GET_SMALL && CHROOT */
 
+	warning("starting single user mode...");
+
 	/*
 	 * If the kernel is in secure mode, downgrade it to insecure mode.
 	 */
@@ -725,8 +759,7 @@
 		argv[0] = "-sh";
 #endif /* ALTSHELL */
 		(void)execv(INIT_BSHELL, __UNCONST(argv));
-		emergency("can't exec `%s' for single user: %m", INIT_BSHELL);
-		(void)sleep(STALL_TIMEOUT);
+		stall("can't exec %s for single user: %m", INIT_BSHELL);
 		_exit(1);
 	}
 
@@ -749,12 +782,11 @@
 		if (wpid == -1) {
 			if (errno == EINTR)
 				continue;
-			warning("wait for single-user shell failed: %m; "
-			    "restarting");
+			warning("waitpid() for single-user shell failed: %m; restarting single-user mode");
 			return (state_func_t)single_user;
 		}
 		if (wpid == pid && WIFSTOPPED(status)) {
-			warning("shell stopped, restarting");
+			warning("init: shell[%d] stopped -- sending SIGCONT to it\n", pid);
 			kill(pid, SIGCONT);
 			wpid = -1;
 		}
@@ -768,14 +800,16 @@
 
 	if (WIFSIGNALED(status)) {
 		if (WTERMSIG(status) == SIGKILL) {
-			/* executed /sbin/reboot; wait for the end quietly */
+			/* executed /sbin/reboot; wait for the end */
 			sigset_t s;
 	
+			warning("single user shell killed with SIGKILL, waiting for kernel halt or reboot");
 			(void)sigfillset(&s);
 			for (;;)
 				(void)sigsuspend(&s);
 		} else {	
-			warning("single user shell terminated, restarting");
+			warning("single user shell terminated with SIG%s, restarting single user mode",
+				strsigname(WTERMSIG(status)));
 			(void)sigaction(SIGTSTP, &satstp, NULL);
 			(void)sigaction(SIGHUP, &sahup, NULL);
 			return (state_func_t) single_user;
@@ -811,7 +845,7 @@
 		(void)sigaction(SIGTSTP, &sa, NULL);
 		(void)sigaction(SIGHUP, &sa, NULL);
 
-		setctty(_PATH_CONSOLE);
+		setctty(_PATH_CONSOLE);	/* XXX or _PATH_CONSDEV ??? */
 
 		argv[0] = "sh";
 		argv[1] = _PATH_RUNCOM;
@@ -825,7 +859,7 @@
 			if (chroot(rootdir) != 0) {
 				warning("failed to chroot to `%s': %m",
 				    rootdir);
-				_exit(1); 	/* force single user mode */
+				_exit(1);	/* force single user mode */
 			}
 #endif /* CHROOT */
 
@@ -838,7 +872,6 @@
 		    _PATH_RUNCOM);
 		while (waitpid(-1, NULL, WNOHANG) > 0)
 			continue;
-		(void)sleep(STALL_TIMEOUT);
 		return (state_func_t)single_user;
 	default:
 		break;
@@ -866,23 +899,34 @@
 	} while (wpid != pid);
 
 	if (WIFSIGNALED(status) && WTERMSIG(status) == SIGTERM &&
-	    requested_transition == catatonia) {
+	    requested_transition == go_catatonic) {
 		/* /etc/rc executed /sbin/reboot; wait for the end quietly */
 		sigset_t s;
 
+		warning("%s on %s killed with SIGTERM, waiting for kernel halt or reboot",
+			_PATH_BSHELL, _PATH_RUNCOM);
 		(void)sigfillset(&s);
 		for (;;)
 			(void)sigsuspend(&s);
 	}
 
 	if (!WIFEXITED(status)) {
-		warning("`%s' on `%s' terminated abnormally, going to "
-		    "single user mode", INIT_BSHELL, _PATH_RUNCOM);
+		if (WIFSIGNALED(status)) {
+			warning("%s on %s terminated with SIG%s, going to single user mode",
+				_PATH_BSHELL, _PATH_RUNCOM, strsigname(WTERMSIG(status)));
+		} else
+			warning("%s on %s terminated abnormally, going to single user mode",
+				INIT_BSHELL, _PATH_RUNCOM);
 		return (state_func_t)single_user;
 	}
 
-	if (WEXITSTATUS(status))
+	if (WEXITSTATUS(status)) {
+		warning("%s on %s terminated with non-zero exit status (%d), going to single user mode",
+			INIT_BSHELL, _PATH_RUNCOM, WEXITSTATUS(status));
 		return (state_func_t)single_user;
+	}
+
+	logwtmp("~", "reboot", "");	/* this really means "boot"! */
 
 	return (state_func_t) read_ttys;
 }
@@ -1181,7 +1225,7 @@
 #ifdef CHROOT
 		/* If /etc/rc ran in chroot, we want to kill any survivors. */
 		if (did_multiuser_chroot)
-			return (state_func_t)death;
+			return (state_func_t)kill_ttys;
 		else
 #endif /* CHROOT */
 			return (state_func_t)single_user;
@@ -1337,7 +1381,7 @@
 		return MULTI_USER;
 	if (s == (state_t)clean_ttys)
 		return CLEAN_TTYS;
-	if (s == (state_t)catatonia)
+	if (s == (state_t)go_catatonic)
 		return CATATONIA;
 	return DEATH;
 }
@@ -1409,6 +1453,8 @@
 	(void)gettimeofday(&sp->se_started, NULL);
 	add_session(sp);
 #endif /* LETS_GET_SMALL */
+
+	return;
 }
 
 /*
@@ -1424,13 +1470,20 @@
 		requested_transition = clean_ttys;
 		break;
 	case SIGTERM:
-		requested_transition = death;
+		requested_transition = kill_ttys;
 		break;
 	case SIGTSTP:
-		requested_transition = catatonia;
+		requested_transition = go_catatonic;
+		break;
+	case SIGUSR1:
+		requested_transition = go_single_user;
+		break;
+	case SIGINT:
+		requested_transition = single_user;
 		break;
 #endif /* LETS_GET_SMALL */
 	default:
+		warning("ignoring unknown state transition [%d]", sig);
 		requested_transition = 0;
 		break;
 	}
@@ -1447,6 +1500,8 @@
 	int status;
 	session_t *sp;
 
+	info("(re)entering multi-user state");
+
 	requested_transition = 0;
 
 	/*
@@ -1471,6 +1526,13 @@
 		add_session(sp);
 	}
 
+	/*
+	 * the value of requested_transition could be changed either by
+	 * collect_child() if start_getty() encounters a failure from fork()
+	 * fails (it'll be clean_ttys), or by transition_handler() if a signal
+	 * caught by it is triggered (in which case waitpid() will also be
+	 * interrupted)
+	 */
 	while (!requested_transition)
 		if ((pid = waitpid(-1, &status, 0)) != -1)
 			collect_child(pid, status);
@@ -1489,6 +1551,8 @@
 	int session_index = 0;
 	int devlen;
 
+	notice("(re)loading %s", _PATH_TTYS);
+
 	for (sp = sessions; sp; sp = sp->se_next)
 		sp->se_flags &= ~SE_PRESENT;
 
@@ -1513,8 +1577,10 @@
 			if ((typ->ty_status & TTY_ON) == 0 ||
 			    typ->ty_getty == 0) {
 				sp->se_flags |= SE_SHUTDOWN;
-				if (sp->se_process != 0)
+				if (sp->se_process != 0) {
+					info("kill -HUP %d", sp->se_process);
 					(void)kill(sp->se_process, SIGHUP);
+				}
 				continue;
 			}
 			sp->se_flags &= ~SE_SHUTDOWN;
@@ -1522,8 +1588,10 @@
 				warning("can't parse getty for port `%s'",
 				    sp->se_device);
 				sp->se_flags |= SE_SHUTDOWN;
-				if (sp->se_process != 0)
+				if (sp->se_process != 0) {
+					info("kill -HUP %d", sp->se_process);
 					(void)kill(sp->se_process, SIGHUP);
+				}
 			}
 			continue;
 		}
@@ -1536,8 +1604,10 @@
 	for (sp = sessions; sp; sp = sp->se_next)
 		if ((sp->se_flags & SE_PRESENT) == 0) {
 			sp->se_flags |= SE_SHUTDOWN;
-			if (sp->se_process != 0)
+			if (sp->se_process != 0) {
+				info("kill -HUP %d", sp->se_process);
 				(void)kill(sp->se_process, SIGHUP);
+			}
 		}
 
 	return (state_func_t)multi_user;
@@ -1545,17 +1615,45 @@
 
 /*
  * Block further logins.
+ *
+ * transiting back to clean_ttys, i.e. SIGHUP, will re-enable things....
  */
 state_func_t
-catatonia(void)
+go_catatonic(void)
 {
 	session_t *sp;
 
+	warning("stopping tty respawning...");
+
 	for (sp = sessions; sp; sp = sp->se_next)
 		sp->se_flags |= SE_SHUTDOWN;
 
 	return (state_func_t)multi_user;
 }
+
+
+/*
+ * prepare to spawn a single user shell....
+ */
+state_func_t
+go_single_user(void)
+{
+	session_t *sp;
+
+	warning("single user mode transition initiated...");
+
+	/*
+	 * First we make double sure we don't respawn anything!  (we should
+	 * already have transited through go_catatonic(), but not if someone
+	 * happens to trigger this state directly.
+	 */
+	for (sp = sessions; sp; sp = sp->se_next)
+		sp->se_flags |= SE_SHUTDOWN;
+
+	logwtmp("~", "singleuser", "");
+
+	return (state_func_t) single_user;
+}
 #endif /* LETS_GET_SMALL */
 
 /*
@@ -1571,18 +1669,16 @@
 
 #ifndef LETS_GET_SMALL
 /*
- * Bring the system down to single user.
+ * kill all our own directly spawned sessions
+ *
+ * Note that reboot/halt/poweroff will kill(-1, SIGTERM) and thus init will be
+ * put in the kill_ttys() state by default.  This code may help shut down TTY
+ * sessions that have for some reason ignored SIGTERM.
  */
 state_func_t
-death(void)
+kill_ttys(void)
 {
 	session_t *sp;
-	int i, status;
-	pid_t pid;
-	static const int death_sigs[3] = { SIGHUP, SIGTERM, SIGKILL };
-
-	for (sp = sessions; sp; sp = sp->se_next)
-		sp->se_flags |= SE_SHUTDOWN;
 
 	/* NB: should send a message to the session logger to avoid blocking. */
 #ifdef SUPPORT_UTMPX
@@ -1592,22 +1688,59 @@
 	logwtmp("~", "shutdown", "");
 #endif
 
-	for (i = 0; i < 3; ++i) {
-		if (kill(-1, death_sigs[i]) == -1 && errno == ESRCH)
-			return (state_func_t)single_user;
-
-		clang = 0;
-		alarm(DEATH_WATCH);
-		do
-			if ((pid = waitpid(-1, &status, 0)) != -1)
-				collect_child(pid, status);
-		while (clang == 0 && errno != ECHILD);
+# ifndef LET_SHUTDOWN_DO_IT
+	warning("shutting down all sessions...");
+	/*
+	 * First we make double sure we don't respawn anything!  Note that if
+	 * we don't do this first then collect_child() will respawn any entry
+	 * not turned 'off'
+	 */
+	for (sp = sessions; sp; sp = sp->se_next)
+		sp->se_flags |= SE_SHUTDOWN;
 
-		if (errno == ECHILD)
-			return (state_func_t)single_user;
+	/*
+	 * put them down easy....
+	 *
+	 * NOTE:  These are TTY sessions, not daemons, so SIGHUP is indeed the
+	 * right way to tell them to quit.
+	 */
+	for (sp = sessions; sp; sp = sp->se_next) {
+		if (sp->se_process) {
+			info("kill -HUP %d", sp->se_process);
+			kill(sp->se_process, SIGHUP);
+		}
 	}
+	clang = 0;
+	alarm(DEATH_WATCH);
+	do {
+		int pid;
+		int status;
 
-	warning("some processes would not die; ps axl advised");
+		errno = 0;
+		if ((pid = waitpid(-1, &status, 0)) != -1)
+			collect_child(pid, status);
+		else if (errno == ECHILD) {
+			/* 
+			 * nobody left but us!!!
+			 */
+			alarm(0);
+			return (state_func_t) go_single_user;
+		} else if (errno != EINTR)
+			warning("waitpid(-1) failed: %m");
+	} while (clang == 0);
+
+	/* put any remaining down hard.... */
+	for (sp = sessions; sp; sp = sp->se_next) {
+		if (sp->se_process) {
+			info("kill -KILL %d", sp->se_process);
+			kill(sp->se_process, SIGKILL);
+			sp->se_process = 0;
+		}
+		/* don't bother to reap them */
+ 	}
+
+	/* ... and keep them down. */
+# endif /* !LET_SHUTDOWN_DO_IT */
 
 	return (state_func_t)single_user;
 }
@@ -1615,56 +1748,6 @@
 
 #ifdef MFS_DEV_IF_NO_CONSOLE
 
-static void
-mapfile(struct mappedfile *mf)
-{
-	int fd;
-	struct stat st;
-
-	if (lstat(mf->path, &st) == -1)
-		return;
-
-	if ((st.st_mode & S_IFMT) == S_IFLNK) {
-		mf->buf = malloc(st.st_size + 1);
-		if (mf->buf == NULL)
-			return;
-		mf->buf[st.st_size] = 0;
-		if (readlink(mf->path, mf->buf, st.st_size) != st.st_size)
-			return;
-		mf->len = -1;
-		return;
-	}
-
-	if ((fd = open(mf->path, O_RDONLY)) == -1)
-		return;
-	mf->buf = mmap(0, (size_t)st.st_size, PROT_READ,
-			MAP_FILE|MAP_SHARED, fd, (off_t)0);
-	(void)close(fd);
-	if (mf->buf == MAP_FAILED)
-		return;
-	mf->len = st.st_size;
-}
-
-static void
-writefile(struct mappedfile *mf)
-{
-	int fd;
-
-	if (mf->len == -1) {
-		symlink(mf->buf, mf->path);
-		free(mf->buf);
-		return;
-	}
-
-	if (mf->len == 0)
-		return;
-	fd = open(mf->path, O_WRONLY | O_CREAT | O_TRUNC, 0755);
-	if (fd == -1)
-		return;
-	(void)write(fd, mf->buf, mf->len);
-	(void)munmap(mf->buf, mf->len);
-	(void)close(fd);
-}
 
 static int
 mfs_dev(void)
@@ -1681,25 +1764,33 @@
 	size_t olen;
 #endif
 
+	/*
+	 * XXX !!! The kernel should pass us the right console FDs as
+	 * stdin/stdout/stderr, just like the shell does. !!!
+	 */
+
 	/* If we have /dev/console, assume all is OK  */
 	if (access(_PATH_CONSOLE, F_OK) != -1)
 		return(0);
 
-	/* Grab the contents of MAKEDEV */
-	mapfile(&mfile[0]);
-
-	/* Grab the contents of MAKEDEV.local */
-	mapfile(&mfile[1]);
+	warnx("Could not find /dev/console, mounted a MFS on /dev, now running %s...", _PATH_MAKEDEV);
 
 	/* Mount an mfs over /dev so we can create devices */
 	switch ((pid = fork())) {
 	case 0:
+#if 0
+		(void) dup2(STDERR_FILENO, STDOUT_FILENO); /* Give it stdout */
+		/* XXX or even login_tty() */
+#endif
 		asprintf(&fs_size, "%d", FSSIZE);
-		if (fs_size == NULL)
+		if (fs_size == NULL) {
+			warn("Error formatting string for mount_mfs params!");
 			return(-1);
+		}