So the problems are limited to igmp and in6? For the igmp case, I looked
  at the code and it seems that it is doing the right thing to me. If we
  can come up with a simple test case then it should be easy to debug it.

I would say that I have observed two problems of 'struct ifnet *'
remaining when it shouldn't, and they are 
a) v4 multicast group memberships on sockets
b) someplace in the v6 udp output path (on ripng socket)
I don't know if there are more.

It seems the invariant is that all reachable (in the lisp gc sense)
variables of type 'struct ifnet *' point to a currently allocated,
valid, struct ifnet that is in the master array of struct ifnets.

I found in_pcbpurgeif0, which looks like it is trying to do the right
thing.  I think it's a raw socket used for OSPF for the straggling
IPv4 membership record ifnet pointers.

In reading if_detach, I think I found the problem.  The code only
calls PRU_PURGEIF for protocols belonging to address families that are
on the interfaces address list.  This assumes that multicast
membership records in PF x pointing to the ifp can only exist if the
ifp has an address of PF x.  But, if pppd removes the IP address from
the interface as part of a clean shutdown after an LCP TermReq, there
won't be addresses, but still could be joined groups.

So, I think we need to either

a) call PRU_PURGEIF on all protocols in all families from if_detach

b) make in_purgeaddr delete memberships for all protocols within a
family when the last address of the family is deleted.
and
clear cached routes on all sockets of a protocol family when the last
address of that family is removed from any interface

For the v6 udp case, I think the following in udp6_output

        ip6->ip6_hlim   = in6_selecthlim(in6p,
                                 in6p->in6p_route.ro_rt ?
                                 in6p->in6p_route.ro_rt->rt_ifp : NULL);

may be using a route cached with the socket.  This seems odd, because
ip6_output hasn't been called, and perhaps this cached route may or may
not be appropriate for this packet.  But, I see that udp6_usrreq's
PRU_PURGEIF calls in6_pcbpurgeif, and that seems to clear such routes.

A wrinkle is that ripngd is calling sendmsg with a IP6_PKTINFO control
option with the ifindex.  udp6_output calls ip6_setpktoptions, which
processes the option and validates the ifindex against the
ifindex2ifnet array.  If the resulting ifp is NULL, it returns ENXIO.
Later in processing, the ifp is presumed valid.  But I don't see
splnet() around all this.  I don't believe this is my problem - the
crash has happened too many times to think that pppd took the
interface down during the udp6 output processing.

pppd(8) has code to clean up the link-local address pair it added for
the destination.  It's not clear when/how/if the original link-local
address gets removed, but it might be that this somehow happens before
if_detach.


To repeat, set up ppp between two systems.  Let the answering system
have a getty on the line, and use a PPP user login account that runs
pppd as the shell.  I use modems, but I don't think that's necessary.
Run quagga's ospfd and ripngd on both ends.  Configure ppp to redial
by giving it persist on one end, and put holdoff 60 in the options
file to make it be after a minute.  Then, set up a cron job to do
"pkill -HUP pppd" every 10 minutes.  I would expect a crash well
within an hour.  The igmp crash has been 75% of disconnects on the
called system, and perhaps 25% on the calling.  The in6_selecthlim
crash is rarer, perhaps 10% of disconnects on the calling system.

-- 
        Greg Troxel <gdt@ir.bbn.com>