Re: What should be the abstraction for ethernet switches?

[David Young <dyoung%pobox.com@localhost>]

On Tue, May 10, 2011 at 05:03:22PM -0700, Matt Thomas wrote:
> More and more embedded boards come with ethernet switch switches that attach 
> directly to the MII interface of an ethernet controller.
> 
> So where do we plug these into autoconf?
> 
> How do they interact with the vlan and bridge interfaces?
> 
> Do we reveal the other switch ports as interfaces?
> 
> ifconfig arswitch0.1 media ....

I have been thinking about this.  I don't think that virtual ethernet
devices such as agr(4), bridge(4), vlan(4) are tenable for the future.
We need multiple ethernet forwarding domains, and each driver for an
ethernet adapter needs to split into top and bottom halves: logical
ethernet interface on top, hardware driver below, with logical ethernets
assigned to ports and bridge groups using device locators.

The following is drawn from my imagination, especially some of the
drvctl uses and including a little bit of "future autoconf(9) history,"
so you will be disappointed if you grovel in your dmesg for or type into
a command line any of the following.

Hardware parts will have an interface attribute "ether" where logical
ethernet drivers may attach with ether_attach_args adapted from the
content of struct ethercom and some of struct ifnet.  One or more
interfaces may attach to each ether attribute.  In the case of a
hypothetical 6-port switch chip, at least as many logical ethernets
attach as there are ports:

        sw0 at pci0 dev 13 function 0: 6-port ethernet switch
        swport0 at sw0 port 0 vlan 0 domain 0
        swport0: Ethernet address 00:aa:bb:cc:dd:e0
        swport1 at sw0 port 1 vlan 0 domain 1
        swport1: Ethernet address 00:aa:bb:cc:dd:e1
        swport2 at sw0 port 2 vlan 0 domain 2
        swport2: Ethernet address 00:aa:bb:cc:dd:e2
        swport3 at sw0 port 3 vlan 0 domain 3
        swport3: Ethernet address 00:aa:bb:cc:dd:e3
        swport4 at sw0 port 4 vlan 0 domain 4
        swport4: Ethernet address 00:aa:bb:cc:dd:e4
        swport5 at sw0 port 5 vlan 0 domain 5
        swport5: Ethernet address 00:aa:bb:cc:dd:e5

Note a couple of things about the default port locators: each port is
assigned vlan 0 (VLAN ID 0---i.e., no VLAN tagging) and an independent
domain (i.e., there is no bridging from port to port).

Ordinarily, a single-port ethernet adapter will attach one logical
ethernet.  Example right below.  It's ugly, but for backwards
compatibility I add "hw" to the hardware instances so that I can assign
the logical ethernets their legacy name:

        fxphw0 at pci1 dev 8 function 0: Intel PRO/100 VM Network Controller
            with 82562ET/EZ PHY, rev 2
        fxphw0: interrupting at ioapic0 pin 20
        wmhw0: 1-port ethernet
        inphy0 at fxphw0 phy 1: i82562ET 10/100 media interface, rev. 0
        inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
        fxp0 at fxphw0 port 0 vlan 0 domain 6
        fxp0: Ethernet address 00:0a:0b:0c:0d:0e
        .
        .
        .
        wmhw0 at pci0 dev 12 function 0: Intel i82540EM 1000BASE-T Ethernet,
            rev. 2
        wmhw0: interrupting at ioapic0 pin 18
        wmhw0: 32-bit 33MHz PCI bus
        wmhw0: 256 word (8 address bits) MicroWire EEPROM
        makphy0 at wmhw0 phy 1: Marvell 88E1011 Gigabit PHY, rev. 3
        makphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 
1000baseT-FD
        X, auto
        wmhw0: 1-port ethernet
        wm0 at wmhw0 port 0 vlan 0 domain 7
        wm0: Ethernet address 00:01:02:03:04:05

Let's bridge fxp0 and wm0 to each other.  To do that, we put them
in the same ethernet domain.  We will move wm0 to fxp0's domain,
domain 6:

        # drvctl -d wm0
        wm0: detached

(Note: I am anticipating some changes to autoconf(9). wm0 was detached,
but not forgotten!  We do not get another instance wm0 in the subsequent
discussion, because we haven't used the same locators as the original
had attached with.)

Now, let's attach wmhw0 port 0 to domain 6:

        # drvctl -r -a ether wmhw0 domain 6
        wm1 at wmhw0 port 0 vlan 0 domain 6

Now the two NICs are bridged.  Say that I want for bridged traffic to be
tagged w/ VLAN ID 3 on the fxp, then:

        # drvctl -d fxp0
        fxp0: detached

        # drvctl -r -a ether fxphw0 vlan 3 domain 6
        fxp1 at fxphw0 port 0 vlan 3 domain 6

Finally, I want to bridge fxp1, wm1, and sw0 ports 0..2 together:

        # drvctl -d swport0
        swport0: detached
        # drvctl -d swport1
        swport1: detached
        # drvctl -d swport2
        swport2: detached
        # drvctl -r -a ether sw0 port 0 domain 6
        swport6 at sw0 port 0 vlan 0 domain 6
        swport6: Ethernet address 00:aa:bb:cc:dd:e0
        # drvctl -r -a ether sw0 port 1 domain 6
        swport7 at sw0 port 1 vlan 0 domain 6
        swport7: Ethernet address 00:aa:bb:cc:dd:e1
        # drvctl -r -a ether sw0 port 2 domain 6
        swport8 at sw0 port 2 vlan 0 domain 6
        swport8: Ethernet address 00:aa:bb:cc:dd:e2

Note that sw0 has the opportunity as it detaches/attaches each of
swport0..2 and swport6..8 to update the hardware switch matrix.

Finally, I also want for traffic on sw0 port 3 to be bridged to sw0 port
0 and tagged with VLAN ID 7:

        # drvctl -r -a ether sw0 port 0 vlan 7 domain 3
        swport9 at sw0 port 0 vlan 7 domain 3
        swport9: Ethernet address 00:aa:bb:cc:dd:e0

Note that we now have two logical ethernet instances at sw0 port 0:

        swport6 at sw0 port 0 vlan 0 domain 6
        swport6: Ethernet address 00:aa:bb:cc:dd:e0
        swport9 at sw0 port 0 vlan 7 domain 3
        swport9: Ethernet address 00:aa:bb:cc:dd:e0

Now say that we want to bridge two ethernets with a GRE tunnel.  Imagine
that gre(4) instances have an interface attribute 'ether', too, and we
have a GRE-in-IP instance set up already.  We bridge domain 3 to the
remote site like so:

        # drvctl -r -a ether gre0 domain 3
        greport0 at gre0 port 0 vlan 0 domain 3

It is too bad that locators are currently so primitive, because I would
like to be able to use MAC addresses as locators so that I can stick
a configuration to a particular sw(4) adapter and/or avoid applying a
configuration to the wrong switch if they renumber:

I added a new sw(4) to the chassis, is sw0 the old one?

        # drvctl -r -a ether sw0 macaddr 00:aa:bb:cc:dd:e0 vlan 7 domain 3
        drvctl: No such file or directory

No.  My old sw(4) has a new unit number:

        # drvctl -r -a ether sw1 macaddr 00:aa:bb:cc:dd:e0 vlan 7 domain 3
        swport9 at sw1 port 0 vlan 7 domain 3
        swport9: Ethernet address 00:aa:bb:cc:dd:e0

I think that agr(4) can fit neatly into this framework, too, but I need
to think more about it.

A nice thing (I think) about wiring together a router/bridge using
autoconf(9) is that you can conceivably write much of the network
configuration for a network appliance in the kernel configuration file.
I think that we can imagine reading a lot of a router's configuration
back out of the kernel in the form of, say, a property list that tells
us the current autoconf(9) state such as parent/child relationships,
locators, etc.

BTW, I think that it may help to take a similar approach all of the way
up the stack by, for example, attaching IPv4 & IPv6 instances to logical
ethernets:

ipvf0 at swport9 domain 0: logical IPv4 interface
ipvs0 at swport9 domain 0: logical IPv6 interface
ipvf0 at swport3 domain 1: logical IPv4 interface

Dave

-- 
David Young             OJC Technologies
dyoung%ojctech.com@localhost      Urbana, IL * (217) 344-0444 x24