Marc Horowitz forwarded me a packet trace showing something of a
multicast storm where an ip-level unicast packet kept getting
retransmitted via ethernet multicast on a subnet containing multiple
NetBSD machines acting as routers (one of the MediaOne cable modem
subnets..)

15:09:08.438064 0:0:f4:90:4e:ab 1:e0:bb:23:79:20 ip 60: solaria.ne.mediaone.net.3204 > www-2b.bloomberg.com.www: . [tcp sum ok] ack 587930673 win 17055 (DF) (ttl 128, id 19811)
                         4500 0028 4d63 4000 8006 4f1b 18da 889a
                         ccb3 f029 0c84 0050 0141 8bbf 230b 1c31
                         5010 429f 35d3 0000 2020 2020 2020
15:09:08.438231 0:60:8c:ea:cd:53 1:0:5e:5a:88:1 ip 54: solaria.ne.mediaone.net.3204 > www-2b.bloomberg.com.www: . [tcp sum ok] ack 1 win 17055 (DF) (ttl 127, id 19811)
                         4500 0028 4d63 4000 7f06 501b 18da 889a
                         ccb3 f029 0c84 0050 0141 8bbf 230b 1c31
                         5010 429f 35d3 0000
15:09:08.439502 0:a0:24:1b:34:da 1:0:5e:5a:88:1 ip 60: solaria.ne.mediaone.net.3204 > www-2b.bloomberg.com.www: . [tcp sum ok] ack 1 win 17055 (DF) (ttl 127, id 19811)
                         4500 0028 4d63 4000 7f06 501b 18da 889a
                         ccb3 f029 0c84 0050 0141 8bbf 230b 1c31
                         5010 429f 35d3 0000 0000 0000 0000
15:09:08.439604 0:60:8c:ea:cd:53 1:0:5e:5a:88:1 ip 54: solaria.ne.mediaone.net.3204 > www-2b.bloomberg.com.www: . [tcp sum ok] ack 1 win 17055 (DF) (ttl 126, id 19811)
                         4500 0028 4d63 4000 7e06 511b 18da 889a
                         ccb3 f029 0c84 0050 0141 8bbf 230b 1c31
                         5010 429f 35d3 0000
15:09:08.439906 0:60:8c:cd:37:f4 1:0:5e:5a:88:1 ip 60: solaria.ne.mediaone.net.3204 > www-2b.bloomberg.com.www: . [tcp sum ok] ack 1 win 17055 (DF) (ttl 127, id 19811)
                         4500 0028 4d63 4000 7f06 501b 18da 889a
                         ccb3 f029 0c84 0050 0141 8bbf 230b 1c31
                         5010 429f 35d3 0000 0000 0000 0000

Here's what I think happened:

M_MCAST is set if either:
	- a packet was received as a link-level multicast
	- the output side of the stack wants to send it as a
	  link-level multicast.

If a link-level multicast packet is received which contains a unicast
packet for a non-local destination, we end up in ip_forward() to
forward it to the right destination.

Since M_MCAST is still set by the time the packet hits ether_output,
rather than unicasting to the next-hop router, it hits the following
code in if_ethersubr.c:

		else if (m->m_flags & M_MCAST) {
			ETHER_MAP_IP_MULTICAST(&SIN(dst)->sin_addr,
			    (caddr_t)edst)

and it winds up multicasting the packet to a group address derived
from the next-hop multicast address.

If multiple NetBSD routers on a wire are receiving all link-level
multicasts, you end up with a multicast storm.

Fixing this looks straightforward, and should probably be done in
multiple places.

Most importantly:

Currently, ip_forward() contains the following:

	if (m->m_flags & M_BCAST || in_canforward(ip->ip_dst) == 0) {
		ipstat.ips_cantforward++;
		m_freem(m);
		return;
	}

I'm pretty sure that this check should be:

	if (m->m_flags & (M_BCAST|M_MCAST) || in_canforward(ip->ip_dst) == 0) 

i'll commit this if i don't hear any objections soon..

A second "belt and suspenders" check would be to allow
MAP_IP_MULTICAST to fail, causing if_ethersubr to drop the packet, if
M_MCAST is set but the the next-hop address isn't a multicast address.

					- Bill