Subject: Re: confusing pf behaviour, block drop still returns messages?
To: Nino Dehne <ndehne@gmail.com>
From: Daniel Hartmeier <daniel@benzedrine.cx>
List: tech-net
Date: 04/14/2006 16:51:12
On Wed, Apr 12, 2006 at 01:57:30AM +0200, Nino Dehne wrote:

> However, when I tcpdump on interface vlan1 on host gw, I see two errors:
> 
> 01:03:00.296138 IP 192.168.0.254 > 192.168.1.1: icmp 36: host 1.0.0.1 unreachable - admin prohibited filter
> 01:03:00.296352 IP 192.168.0.254 > 192.168.1.1: icmp 36: host 1.0.0.1 unreachable
> 
> The first is expected, the second isn't. The second message even persists
> when I remove the return-icmp(13). At that point gw-ext shouldn't emit error
> messages because of the default block-policy.
> 
> I also noticed that, when I try to ping the host 1.0.0.1 directly from gw-ext
> I get a confusing error message:
> 
> # ping 1.0.0.1
> PING 1.0.0.1 (1.0.0.1): 56 data bytes
> ping: sendto: No route to host
> 
> This is plain wrong. First, there is a route to that network in the routing
> table. Second, I even get that message when I tell pf to silently block and
> not return anything.
> 
> What is going on here? To me, this is not consistent behaviour at all. How
> can I get a nice "admin prohibited" error while shutting up pf about the
> normal unreachable which I think shouldn't be there in the first place?

The problem is that you're blocking these packets on their output path
instead of on their input path (block in on $int_if).

Of the two ICMP errors you're seeing, only the code 13 (admin
prohibited) one is actually generated by pf. The plain 'host
unreachable' one is generated by the generic TCP/IP stack.

The packet first comes in through $int_if, ip_input() calls
pfil_run_hooks() to check whether it should be allowed in. You don't
block this, so the packet is processed. That calls ip_forward(), which
calls ip_output().

ip_output(), now on $ext_if, calls pfil_run_hooks() again. This time pf
blocks the packet (block out on $ext_if), returning error ==
EHOSTUNREACH to ip_output() and ip_forward(). Based on that,
ip_forward() generates the generic 'host unreachable' with icmp_error().

The same would happen if you blocked the packet on the output path with
any other pfil filter, not just with pf. Basically, whenever
ip_forward() gets any error from ip_output(), it generates an ICMP
error. I think all BSDs show this behaviour.

The question is, why would you pass those packets in on $int_if, just to
later block them out on $ext_if? Blocking them on the input path is
faster and saves the stack the trouble of trying to forward the packet.

As for the 'No route to host' error you get when trying to ping from the
same host, that's just the errno translation (65 EHOSTUNREACH) that the
userland ping process gets from ip_output() through sendto(). When pfil
blocks an outgoing packet, it has to return SOME error. Since there is
no dedicated errno for this purpose, and we can't just add new ones,
EHOSTUNREACH was used as the best match.

The 'abuse' of that errno causes some confusion sometimes (as you
rightly make the point, you DO have a route to that host, the routing
table isn't relevant in this case at all), but overall it helps to
return an error to userland, as opposed to just silently drop the packet
and return (fake) success from sendto(), that would be more misleading
IMO. The userland process wouldn't immediately notice any problem, try
to retransmit, timeout eventually.

Daniel