Re: Hanging at shutdown with mystery "file system full" error

[Robert Elz <kre%munnari.OZ.AU@localhost>]

    Date:        Thu, 22 Oct 2015 07:52:06 +0200
    From:        "Ian D. Leroux" <idleroux%fastmail.fm@localhost>
    Message-ID:  <20151022075206.995f2caa63b153174b7f7ead%fastmail.fm@localhost>

  | That last part I don't understand, or rather I don't understand
  | why /etc/rc.d/swap1 should worry about that case.

I'll explain what I think the issue is, and then others can correct it...

  | I would have
  | thought that any subsidiary mount point, tmpfs or not, would prevent an
  | unmount of (say) /var.

It would.

  | If the rc system is meant to unmount all
  | filesystems at shutdown time, then it already needs some logic to do so
  | in the right order.

It does (reverse order of mounts).

  | How are tmpfs filesystems special in this regard?

Well....

The problem is that it is permitted to swap onto files.   If we did away
with that, there would be no problems to solve.

But since we allow swapping to files (as distinct from disc partitions)
that swapping has to be disabled, before the filesystems on which the
files live can be unmounted.

When that swapping is disabled, anything that was swapped out there needs
to be swapped in again (at this stage, during rc.shutdown processing, the
kernel is not aware that the system is about to terminate, this is all just
using the same facility as any other removal of a swap file/device).

One of the (potentially) big users of space on swap, is files on a tmpfs.
In fact, during shutdown, it is probably the only significant user.

On systems with limited ram, reading all those files into mem can cause
memory exhaustion, and hangs...   That's kind of unfortunate, especially when
you're in a situation where all you are really wanting to do is trash
everything.

So, to avoid that, tmpfs filesystems are forcibly unmounted first, before
swap is disabled, then that user of swap space is gone - if we are really
in the process of shutting down, most user processes should have already been
terminated, so there should be very limited (if any) process consumption of
swap space, so disabling swap should then function normally.

Once swap is disabled (at least swap on files) the filesystems can all be
unmounted (the question of swap on a file on a tmpfs and what would happen to
that I have no idea about, but it is such an absurd concept that I doubt
anyone cares ... it should just be prohibited, and perhaps is.)

At least that's the idea, but /dev on tmpfs kind of screws with this model...

  | My understanding of the not-so-recent change to /etc/rc.d/swap1 is that
  | it is meant to ensure that there is swap space left for swapctl to
  | function.  That seems to me to be orthogonal to the question of
  | ensuring that all filesystems can be unmounted.

It it perhaps a little clearer now (assuming my perceptions about the
problem are correct.)

  | That sounds to me like there are several possibly-unrelated shutdown
  | hangs,

Could be ... or it is also possible that the /dev problem and no space
message you're seeing are co-incidental. and while annoying, not actually
related to the hang you're seeing.

And, please treat my report of a hang as no more than a rumour .. I know
a hang happened to me (once) but I have not investigated at all, and almost
anything might have been the cause.

  | To the extent that read-only-/ with tmpfs-/dev is a reasonable
  | configuration (and at least Joerg and I seem to think so),

Yes, so do I (though devfs would be better) - I didn't mean to suggest 
otherwise, just that tmpfs /dev is not mandatory for a read only root.
(and that since /dev consists of just the directory, and a whole bunch
of inodes, and needs nothing that actually consumes space - or more
than the directory itself, mfs would actually work for it, perhaps
petter than tmpfs ... but that is a whole separate discussion.)

  | then /etc/rc.d/swap1 needs a more precise notion of which filesystems
  | it can safely unmount to make space on swap.

One notion I had was to perhaps add an "permanent" flag to mount.
If set, no umount would ever be able to remove it (with or without -f).
(Of course, kernel shutdown still happens, syncing and marking the
filesystem clean - just as happens with root, the one current filesystem
that cannot be unmounted.)   That would avoid embedding magic into weird
places about what can, and cannot, safely be unmounted.

  | On systems like mine, which have way more swap than they need, it seems
  | practical just to omit the 'umount -aft tmpfs' from swap1_stop(), but
  | that doesn't address the problem that the change was initially meant to
  | solve.

You could certainly do that.   Perhaps replace it with a "rm -rf /tmp/*"
(repeated for each tmpfs that you have - other than /dev, though even
it could probably be cleaned safely that way at shutdown time.)

The rm should be enough to remove anything consuming space (probably needs
to also deal with . files, since rm -fr /tmp itself is probably not safe,
just in case /tmp was not mounted) - there should be no processes left running
holding large unlinked files open.   That would be slower than the forced
unmount, but probably not by a lot, and would still achieve the purpose of
avoiding the need to swap in lots of meaningless garbage when swap is
disabled.    Unfortunately it is harder to script in general than the
simple "umount -aft tmpfs" ...

kre