kern/44647: routed sometimes thinls kernel routes disappear

[kardel%pip.acrys.com@localhost]

>Number:         44647
>Category:       kern
>Synopsis:       routed somes thinls kernel routes disappear
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun Feb 27 21:45:00 +0000 2011
>Originator:     Frank Kardel
>Release:        5.99.47-20110227 i386
>Organization:
        
>Environment:
        
System: NetBSD pip.kardel.name 5.99.47 NetBSD 5.99.47 (PIPGEN) #5: Sat Feb 26 
16:48:16 MET 2011 
kardel%pip.kardel.name@localhost:/fs/raid1a/src/NetBSD/tc-merge/obj.i386/sys/arch/i386/compile/PIPGEN
 i386
Architecture: i386
Machine: i386
>Description:
        After upgrading (kernel+userland) to 5.99.47-20110227 I have static 
default routes randomly disappearing.
        Further investigation shows that routed finds disappearing kernel 
routes.
        Either it claims a missing route for 10.0.3.0/28->10.0.3.1 (interface 
msk0 is 10.0.3.1 netmask 0xfffffff0). routed's 
        attempt to add that route again gets an error EEXIST. Or it claims 
0.0.0.0 -> <some ip> via the pppoe interface
        has disappeared. The pppoe interface is stable over the whole time. 
When routed thinks the default route has
        disappeared is successfully removes the route from the routing table -> 
then the internet connectivity is
        gone until the static default route is manually added again.

        Extract from routed debug log:
        10.0.3.0/28 --> 10.0.3.1 disappeared from kernel
        route disappeared before RTM_DELETE 10.0.3.0/0      -->10.0.3.1        
metric=0 flags=0
        10.0.3.0/28 --> 10.0.3.1 disappeared from kernel
        0.0.0.0 --> 217.0.116.31 disappeared from kernel
        10.0.3.0/28 --> 10.0.3.1 disappeared from kernel
        route disappeared before RTM_DELETE 10.0.3.0/0      -->10.0.3.1        
metric=0 flags=0
        10.0.3.0/28 --> 10.0.3.1 disappeared from kernel
        route disappeared before RTM_DELETE 10.0.3.0/0      -->10.0.3.1        
metric=0 flags=0
        10.0.3.0/28 --> 10.0.3.1 disappeared from kernel
        route disappeared before RTM_DELETE 10.0.3.0/0      -->10.0.3.1        
metric=0 flags=0
        10.0.3.0/28 --> 10.0.3.1 disappeared from kernel
        route disappeared before RTM_DELETE 10.0.3.0/0      -->10.0.3.1        
metric=0 flags=0
        10.0.3.0/28 --> 10.0.3.1 disappeared from kernel
        route disappeared before RTM_DELETE 10.0.3.0/0      -->10.0.3.1        
metric=0 flags=0

        And in syslog:
        2011-02-27T21:17:49.808954+01:00 pip.kardel.name routed 23534 - - 
10.0.3.0/28 --> 10.0.3.1 disappeared from kernel
        2011-02-27T21:18:20.237604+01:00 pip.kardel.name routed 23534 - - 
write(rt_sock) RTM_ADD    10.0.3.0/28     -->10.0.3.1        metric=0 flags=0: 
File exists
        2011-02-27T21:22:49.809101+01:00 pip.kardel.name routed 23534 - - 
10.0.3.0/28 --> 10.0.3.1 disappeared from kernel
        2011-02-27T21:23:20.238990+01:00 pip.kardel.name routed 23534 - - 
write(rt_sock) RTM_ADD    10.0.3.0/28     -->10.0.3.1        metric=0 flags=0: 
File exists
        2011-02-27T21:27:49.803619+01:00 pip.kardel.name routed 23534 - - 
0.0.0.0 --> 217.0.116.31 disappeared from kernel
        2011-02-27T21:27:49.803727+01:00 pip.kardel.name routed 23534 - - 
10.0.3.0/28 --> 10.0.3.1 disappeared from kernel
        2011-02-27T21:28:20.232260+01:00 pip.kardel.name routed 23534 - - 
write(rt_sock) RTM_ADD    10.0.3.0/28     -->10.0.3.1        metric=0 flags=0: 
File exists
        2011-02-27T21:32:49.795658+01:00 pip.kardel.name routed 23534 - - 
10.0.3.0/28 --> 10.0.3.1 disappeared from kernel
        2011-02-27T21:33:19.792873+01:00 pip.kardel.name routed 23534 - - 
write(rt_sock) RTM_ADD    10.0.3.0/28     -->10.0.3.1        metric=0 flags=0: 
File exists
        2011-02-27T21:37:49.813876+01:00 pip.kardel.name routed 23534 - - 
10.0.3.0/28 --> 10.0.3.1 disappeared from kernel
        2011-02-27T21:38:20.241299+01:00 pip.kardel.name routed 23534 - - 
write(rt_sock) RTM_ADD    10.0.3.0/28     -->10.0.3.1        metric=0 flags=0: 
File exists

        BTW: routed reads the routing table via sysctl.

>How-To-Repeat:
        Run 5.99.47 with pppoe internet connection and static default route + 
internal interface + routed.
        Watch routed error messages + disappearing static default route.
>Fix:
        maybe some fallout from 32/64 bit structure ABI stabilisation ?

>Unformatted:
        <and append source date of snapshot, if applicable (one line).>