Choosing ordering of pseudo-interface config

[Robert Elz <kre%munnari.OZ.AU@localhost>]

As things are setup currently, /etc/rc.d/network configures all
real hardware interfaces first (which makes sense - in more or
less lexical order, which shouldn't matter to anything, but is
what most people would expect).

[Aside: I'm assuming that most people here use auto interface
discovery, and don't specify a list of interfaces to configure,
if you're in the 2nd group, nothing here applies to you, and
you can stop reading... -- unless you're interested in rcorder(8)]

Any pseudo-interfaces (cloning interfaces) are configured after the
hardware interfaces (which is fine, bridges, vlans, tunnels, etc
aren't useful without actually having a real interface somewhere
to use.)   They're configured in whatever order ifconfig -C produces
the interface names (the source of which is a total mystery to me,
but it doesn't really matter).

The issue is that there is no one right order to configure these
things - there are some orders that makes more sense than others.
but nothing is going to be right for everyone.

On my system (well, the one of interest), ifconfig -C produces
bridge vlan stf ... which for many people would be a fine order to
configure things, but it happens for me to be inappropriate, as I
need to bridge a vlan (the other connection on the bridge will be
a xen interface, to a DomU).

It turns out there's a simple mod hat can me made to rc.d/network
that allows people to configure the order, which is ...

--- network.orig        2013-12-10 08:33:37.000000000 +0700
+++ network     2013-12-10 08:36:49.000000000 +0700
@@ -205,12 +205,16 @@
                ifaces="$(/sbin/ifconfig -l)"
                if checkyesno auto_ifconfig; then
                        tmp="$ifaces"
+                       cftmp=
                        for cloner in $(/sbin/ifconfig -C); do
                                for int in /etc/ifconfig.${cloner}[0-9]*; do
                                        [ ! -f $int ] && break
-                                       tmp="$tmp ${int##*.}"
+                                       cftmp="$cftmp ${int}"
                                done
                        done
+                       for int in $(/sbin/rcorder $cftmp); do
+                               tmp="$tmp ${int#*.}"
+                       done
                else
                        tmp="$net_interfaces"
                fi

That just runs rcorder on the /etc/ifconfig.* files for all the
cloning interfaces.   To that just add

        # PROVIDE: vlan0
and
        # REQUIRE: vlan0

in appropriate places in the ifconfig.vlan0 and ifconfig.bridge1 (in my
case) files, and the order then becomes something that will work for me.

rcorder is smart, and doesn't require any of those lines, so ifconfig.xxx
files that don't have them still work just fine.   rcorder is also known
to exist (it's already been used before the network script was run) so
it is safe to use here (nothing new required to move from /usr/bin to /bin
or anything messy like that.)

There are three potential problems however.  One is that interface shutdown
also should be done in a sane order.  That's currently controlled by the
output of ifconfig -l, which appears to list the hardware interfaces,
followed by the pseudo-interfaces in the order they were created, which
is good, as ideally we would de-configure in reverse order.   Unfortunately
network_stop() in rc.d/network doesn't do that, it processes them in
forward ifconfig -l order, rather than last to first.   I think that
needs fixing, but if so, it needs fixing already (without the changes
I'm asking about, or with them) - so I'll leave that for someone else,
or some other day, and maybe interface destruction order really doesn't
matter anyway.)

Second, it is always possible that someone might have rcorder keywords in
their ifconfig.* files now (for cloner interfaces - nothing I'm suggesting
goes anywhere near hardware interface config files).  If so, and they
are just intended to be comments, and not rcorder directives, then strange
things might happen.   I am not proposing that anything be done about this,
except perhaps a warning in UPDATING, as I think the chances of it actually
affecting anyone are almost zero.

Third, and the reason I am discussing this here, instead of just submitting
a change request PR with the above patch in it, relates to a quirk of the
way rcorder works, which is quite likely to affect people, if the patch
above is applied, and nothing else is done.

That is, rcorder doesn't preserve order of files in the absence of
any keywords - rather, it reverses the order. That's because it
takes the args it is given (filenames), pushes them onto the head of
a list (effectively a stack) then does the ordering (if there are
no directives anywhere, that's a no-op of course), then outputs the
final list, popping them off the stack in order.  That's LIFO, and
causes
        rcorder a b c
to produce
        c b a
if the files contain no rcorder directives.

The effect would be, that if the patch above is applied, and nothing
else is done, then people's cloning interfaces would end up being
configured in the exact opposite oder to they're configured now.
That would not be good.

Fortunately, there is a simple change that could be applied to rcorder
that will completely solve this problem, but isn't completely without
issues itself ...

--- rcorder.c.ORIG      2008-08-03 14:49:46.000000000 +0700
+++ rcorder.c   2013-12-10 09:58:34.000000000 +0700
@@ -564,9 +564,9 @@
 crunch_all_files(void)
 {
        int i;
        
-       for (i = 0; i < file_count; i++)
+       for (i = file_count; --i >= 0; )
                crunch_file(file_list[i]);
        insert_before();
 }
 

That simply pushes the files onto the stack, from last in the arg
list, to first, so if nothing changes the order, they come off again
in the order that they were given to rcorder originally.

With that change, the patch above to rc.d/network is (for practical purposes 
anyway) completely benign, if there are no rcorder keywords in the
ifconfig.ifname files, then rc.d/network configure will setup (and then
later shut down) the interfaces in the exact same order it does now.
On the other hand, if you are like me, and need to change that order,
a couple of "comment" (rcoder directive) lines in the relevant ifconfig.*
files, and problem solved.

Of course, what matters then is the effect this change would have on
other users of rcorder, which I think, is principally just ordering
the scripts in /etc/rc.d (does anyone know of any other uses?)

The specification is that it should not matter, rcorder is defined
as producing the output in arbitrary order, just respecting the order
in the directives is supposed to be all that can be expected.

If we believe that, then the change above can be installed without
any negative effects (the ordering required by rcorder is not affected
at all, if a is required to be before b according to the directives,
then that is the order they will be output, regardless of this
change.)

But ... the ordering produced will change, in ways that are not
supposed to matter. For example, raidframe, cgd ccd and lvm are
all required to be before DISKS but there's no other defined
relationship between them.   The input order (lexical on filenames)
is ccd cgd lvm raidframe so, from what I said above, you'd expect
the output to be raidframe lvm cgd ccd.   It isn't however, as
the process of ordering, traverses the list (in LIFO order again)
moving files that have to be before some other file, pushing them
back onto the list, immediately ahead of the file they must be before.
The effect is that where there are a group of files, which all must
be before some other file, their input ordering will be reversed
again so they go back to their original order (within this small group of
files).   That is the current rcorder configures those four in the
order ccd cgd lvm raidfame.   (Aside: most raidframe devices are
probably configured with "raidctl -A yes" (or "raidctl -A root"), those
are configured much earlier, by the kernel, and are irrelevant here,
the raidframe config that's of issue is just any other raid arrays
that need to be configured with raidctl -c).

The patch I'm suggesting (above) doesn't alter the way rcorder works
at all, so when ordering, the group of files that go together will
be reversed from the order in the original processing list, which
after that patch is applied, is the exact opposite of what it was before.
For that group of scripts it produces raidframe lvm cgd ccd - which
possibly excepting ccd, could be argued to be a more rational order,
but as there is no specified required order, is supposed to be within
rcorder's right to produce.

There are a bunch of similar groups of scripts that would end up
being processed in a different order than they now are.  If any
of that actually matters, it would be a bug in the scripts (incorrect
keyword usage) but while exposing latent bugs is always fun, it
is usually only considered fun, rather than pain, if you're not
adversely affected by it.

So, this is a request for opinions - is this something worth adding/changing
in NetBSD, and is the risk to people's rc.d processing worth it?

Also, if people could review the output on their rc.d directories,
before and after the patch, and see if there are ay obvious errors
in what is produced, and if so, submit PRs against the relevant rc.d
scripts to get them fixed (this patch won't change anything that's
properly specified - so if something changes in a way that it shouldn't,
then it has been working just by chance, and that should be fixed.)

Lastly, for someone who actually understands rcorder's implementation
fully, adding an option to force it to produce stable ordering for
otherwise unordered input (when given, which it would be in rc.d/network)
would be a much better solution than the quick fix patch above.

Thanks for reading all of this...

kre