Making pf block DomU <-> DomU traffic

[tg%gmplib.org@localhost (Torbjörn Granlund)]

I have used pf for many years, and also Xen under NetBSD.  I have never
used them in combination.  Now I do, using a custom-built Dom0 kernel
with pf (as loadable kernel modules + Xen is well-known as non-working).

Despite draconian block rules, I fail to block traffic between DomU
guests.  These guests both run NetBSD PV.

From /etc/rc.conf (with some masking):

  ifconfig_wm0="X.Y.Z.A netmask 0xffffff00"
  defaultroute=X.Y.Z.B
  ifconfig_tap0="10.0.0.1 netmask 255.0.0.0"

  auto_ifconfig=NO
  net_interfaces="`/sbin/ifconfig -l` tap0 bridge0"

  pf=YES
  pflogd=YES

My pf.conf looks like this (very simplified, even simpler variant
below):

  ext_if="wm0"
  srv="X.Y.Z.A"           # same address as in rc.conf
  guests="10.0.0.0/8"
  shell="10.0.0.2"
  nsext="10.0.0.5"
  nsint="10.0.0.1"

  set block-policy drop
  scrub in on $ext_if all fragment reassemble

  nat  on $ext_if from $guests   to any -> $srv

  rdr  on $ext_if proto tcp from any to  $srv port 2202   -> $shell
  rdr  on $ext_if proto tcp from any to  $srv port domain -> $nsext
  rdr  on $ext_if proto udp from any to  $srv port domain -> $nsext

  table <norouteaddrs> const { 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 }
  block in  quick on $ext_if from <norouteaddrs> to any
  block out quick on $ext_if from any to <norouteaddrs>

  block log all
  antispoof quick log for lo0

  pass out proto udp all keep state
  pass out proto tcp all modulate state

  block    quick on tap0

  pass in            proto udp from { 127.0.0.1 } to 127.0.0.1                  keep state
  pass in            proto udp from { $guests }   to 127.0.0.1                  keep state
  pass in            proto udp from { 127.0.0.1, $guests } to $nsint            keep state
  pass in on $ext_if proto tcp                   to $shell port 2202 flags S/SA keep state
  pass in on $ext_if proto udp                   to $nsext port domain          keep state

This does *not* block ssh/tcp traffic from e.g., 10.0.0.2 to 10.0.0.5.
I would expect "block log all" to block it.  I would expect "block quick
on tap0" to perhaps also block it (but see text at end of message).

The traffic between these IPs would never $ext_if.  So let's simplify
the pf.conf to not have any $ext_if references:

  guests="10.0.0.0/8"
  nsint="10.0.0.1"

  set block-policy drop

  block log all
  antispoof quick log for lo0

  pass out proto udp all keep state
  pass out proto tcp all modulate state

  block    quick on tap0

  pass in    proto udp from { 127.0.0.1 } to 127.0.0.1        keep state
  pass in    proto udp from { $guests }   to 127.0.0.1        keep state
  pass in    proto udp from { 127.0.0.1, $guests } to $nsint  keep state

Still, pf lets tcp traffic pass from 10.0.0.2 to 10.0.0.5.  Bug?
Confusion?

The way I understand NetBSD bridges is that they act as "level 2"
switches.  The DomU systems I wish to isolate from eachother are
attached to the same bridge, bridge0.  Packet to the rest of the world
go through tap0 as it is also attached to bridge0.

This view explains why the 'block tap0' rule in ineffective; the bridge0
switch will naturally pass packets directly from 10.0.0.2 to 10.0.0.5.
But 'block all' should, er, block it all.

But then, how do I force this blocking?  "block all dammit!".  :-)


-- 
Torbjörn
Please encrypt, key id 0xC8601622