RFC: nexthop cache separation

[Ryota Ozaki <ozaki-r%netbsd.org@localhost>]

Hi,

# I sent this mail too early to meet AsiaBSDCon
# NetBSD BoF, so I would update later.

This proposal separates Layer 2 nexthop caches (ARP
and NDP entries) from the routing table and instead
stores them in each interface. This change obsoletes
the concept of cloning and cloned routes; nexthop
caches won't be bound to any routes.

Here is a patch (tl;tr):
  http://www.netbsd.org/~ozaki-r/separate-nexthop-caches.diff

[Motivations]

- Reduce complexity of the routing table operations
  - We don't need recursing operations anymore
- Reduce contentions on the routing table
  - Accesses to nexthop caches are distributed to
    a datastore on each interface
- A route data structure (rtentry) can be reduced
  in size
  - Not done yet (to not break backcompat) though

joerg@ also described some thoughts on this change:
  http://mail-index.netbsd.org/tech-kern/2015/07/09/msg019144.html

[Backward compatibility]

An important concern of this kind of big changes
is backward compatibility on NetBSD, I think.
I summarize what are kept and what are not.

- Nexthop caches are not listed route/netstat -r
  - sysctl(NET_RT_DUMP) doesn't return them
  - If RTF_LLDATA is specified, it returns
    nexthop caches
- RTF_CLONING and RTF_CLONED are obsolete
  - Keep the definitions to not break package
    builds
  - RTF_CLONED is no more used in the kernel
  - RTF_CLONING is just renamed to RTF_CONNECTED
  - 'c' flag of route/netstat -r is gone
  - 'C' flag of route/netstat -r remains, but
    the meaning is now that it's a connected
    route
- RTM_RESOLVE and RTF_XSORELVE are obsolete
  - The definitions remain to not break
    package builds (may not be needed)
- RTF_LLINFO is obsolete
  - The definition remains
  - The value is set by the kernel for old
    userland binaries
  - 'L' is gone
- route -xresolve and route -[no]cloned are
  removed
  - -[no]cloning remains because it seems
    there are users
  - -[no]connected is introduced and recommended
    to be used instead of -[no]cloning
- Gateway value of a route of an interface address
  is now not a L2 address but "link#N" like a
  connected (cloning) route
- Proxy ARP "arp -s ... pub" doesn't create a
  route
  - Should be fixed?
- Trivial behavior changes
  - arp/ndp -d don't remove interface addresses
    - They were removed (unexpectedly?)
    - This behavior is the same as FreeBSD
  - ARP entries that are created by arp ... temp
    can be overwritten now
    - This behavior is the same as FreeBSD

You can see what are changed by seeing diffs
under tests/net in the patch.

[ToDo]

- Update manuals
- Separate backcompat codes and put them
  in COMPAT_70
- Reduce obsolete variables, e.g., rt_llinfo
  and rt_parent of rtentry with keeping
  backcompat

Any comments or suggestions are welcome
(eps. backward compatibility things).

Thanks,
  ozaki-r