RFC: softint-based if_input

[Ryota Ozaki <ozaki-r%netbsd.org@localhost>]

Hi,

This topic was sometimes appeared in some
threads though, I picked it again.

This proposal is to run ifp->if_input
(normally ether_input) and above including
Layer 2 in softint for every network
drivers; we provide a common if_input code
that enqueues received frames and schedules
softint to run rest frame processing. By
doing so, changes to each driver will be small.

The motivation of this change is to ensure
Layer 2 to run in softint, not HW interrupt.
This makes Layer 2 MP-safe work easy; we can
use adaptive mutex and pserialize and reduce
complex handling between HW and SW interrupts.
bridge, vlan and bpf can gain this merit.

One possible objection should be performance
degradation, however I believe we can mitigate
by introducing polling packet processing
like Linux and FreeBSD do. Of course, we also
have to rethink existing sofint (L3 softints
and bridge_forward softint).

Another objection would be fastforward; running
if_input in softint will degrade fastforward
performance. The answer for the issue is...
let's make normal forward fast compared to
fastforward (and throw it away). I think we
can do it :-)

Here is a POC patch:
https://gist.github.com/ozaki-r/4b871fdf16d88ef3f75c
(hmm, I cannot access www.netbsd.org now...)
The patch is designed to minimize drivers other
than wm, so an actual patch would be different.
Anyway if_input part would be almost same.

BTW, this proposal includes another change;
move bpf_mtap to a common place, not doing it
in each driver. This change is also needed by
the above proposal to ensure to always run bpf
in softint.

Any comments or suggestions are welcome.

Thanks,
  ozaki-r