Re: Hanging at shutdown with mystery "file system full" error

[Robert Elz <kre%munnari.OZ.AU@localhost>]

    Date:        Wed, 21 Oct 2015 23:07:53 +0200
    From:        "Ian D. Leroux" <idleroux%fastmail.fm@localhost>
    Message-ID:  <20151021230753.d8eab7389a3df445751e1987%fastmail.fm@localhost>

  | umount -f /dev
  | 
  | Causes an immediate hang with the same kernel error message
  | (..., on /var: file system full), attributed this time to the login
  | command.

Not sure exactly why the full message, but in a way that makes sense, forcibly
unmounting will invalidate any vnodes from the filesystem, which would include
those for the mount devices for all of  the (normal) filesystems - there'd
be no way to access their vops for performing any i/o.

  | The options as I currently see them are:
  | Maybe the hang has to do with unmounting a *busy* tmpfs filesystem?

You'd be able to test/provoke that by just opening a file on /tmp and
then umount -f'ing it.   I doubt there'd be a problem.

  | - /dev should never be unmounted, and /dev is allowed to be on tmpfs.

Those are probably both true.

  | Then swap1_stop() should not forcibly unmount all tmpfs filesystems.

Well, perhaps not that one.

  | Would it make sense to replace "umount -aft tmpfs" with
  | "umount -at tmpfs", so as to remove only filesystems that aren't busy?

No, other busy tmpfs's (such as a command hung waiting on some device that
isn't responding with its cwd there) wouldn't get unmounted.   Not
unmounting tnpfs at all wouldn't be so bad (their state after shutdown is
hardly important!) but that could prevent other filesystems being unmounted
(a mounted /var/shm would block a normal unmount of /var for example).

  | - /dev is not allowed to be on tmpfs.  Then I'm not seeing a bug, I'm
  | just using a bogus configuration.  Note, however, that the automatic
  | creation of a tmpfs-mounted /dev is documented in init(8),

Yes, at least until we have a devfs having init build /dev on a tmpfs
(or an mfs would work I guess, and maybe a better choice?) is a good
way to handle /dev directories trhat are obviously not populated correctly.

  | and it's the only way I know of running with a read-only root partition.

But that's not true, which you can easily see by just booting single
user, and mounting /tmp (and if needed) /usr, /var, etc.   Most things
work fine.

A bunch of things don't work (like changing passwords, and chowning
/dev/console (etc) to the user who logs in) but none of that is actually
essential to operation.

For what it's worth I saw a shutdown hang on a fairly recent current
recently - but it was much later than you're describing, after the
"syncing disks 3 2 1 ..." message (never got to "Done" after 1).   I
waited a while (several minutes at least) and it just sat there - till
it needed to be powered off [that's why it was being shut down.] (Have not
yet restarted that system to see if there were any uncleaned filesystems
left over.)

That system has the "normal" tmpfs's (/tmp and /var/shm) neither of
which would have been busy - what's more, /etc/rc.shutdown had completed
I believe, so they should have been unmounted well before.

kre