Jonathan,

> >(2) Make sending broadcasts to 255.255.255.255 work so that things
> >    which want to broadcast don't have to dig through the interface
> >    address configuration to do it in the normal case (support for the
> >    non-interface-specific broadcast address comes for almost for free as
> >    a side effect of proper wire multicast support in any case).
> 
> I don't intuit quite how this is supposed to work on a multi-homed
> host.  To which interface(s) should the all-ones broadcast be sent?

Note that this question is equivalent to, `to which interface should
packets to 224.0.0.9 be sent', so what ever works for that will also
work for the all-one's broadcast.  I ended up maintaining the notion
of a `primary' interface, which is the place you send multicasts and
all-one's broadcasts when no other information is supplied, and adding
a control message to allow the specification of an alternate interface
with sendmsg(), either using the interface index or the name of the
interface.  This is not dissimilar to the multicast code in the current
kernel.  It is imperfect in general, certainly, but does deal with
the (normal?) single-homed case and does almost the right thing in many
multihomed situations.  And it does leave applications that do care
caching interface names/indices instead of interface-specific addresses,
one might argue that the former are less likely to change than the latter,
certainly in the case of DHCP where addresses change but interfaces don't.

> But asynchronous I/O on a readonly routing socket (with a signal
> indicating an interface change is available) could do almost as well.

I can tell you that I'm a big fan of the routing socket for interface-related
stuff, though we did end up making some scary changes to it to make the
messages easier to parse.  You are correct that using the routing socket
for this is more intrusive than it might be (the application really needs
to know what it is doing).

> >(4) Return errors to connected sockets when the address configuration
> >    changes in a way which leaves the sockets orphaned.
> 
> Yup. Though for an app blocked on a read that isn't going to complete,
> a signal would be more useful.  Or is that a case where you'd return
> an error?  Just in passing, there are more cases than `address

Yes, outstanding reads get an error.  The error is sort of the moral
equivalent of EHOSTUNREACH, the application can choose to ignore it if
it just `knows' the removal of the address is transient.

> expired' that want errors.  In a mobile world, having an interface
> physically removed from the machine could, perhaps, be treated the
> same way as an expired address.

This is true, sort of.  Mobile hosts tend to have the opposite problem
of hosts with DHCP-assigned addresses, that is the mobile host usually
knows what his home address is supposed to be but the address isn't always
assigned to any particular interface (and the interfaces can come and
go, sometimes you get assigned interface additional addresses that you
shouldn't get too attached to).  You can make sure the home address
doesn't go away (and TCP sessions don't drop across connectivity changes)
by adding it to the loopback interface.

Dennis Ferguson