On Tue, Apr 04, 2006 at 04:08:08PM -0400, der Mouse wrote:

> But if a connection arrives on 10.100.1.5, things are less good.
> 10.101.0.4 hears about it, and it looks right from its point of view,
> but the response traffic goes out ex0, presumably because that's where
> the default route points, despite coming from 10.100.1.5, despite the
> state table entry and state-policy, despite even the pass line trying
> to send it out rtk0!

I'm picking an example external address as source. The first packet of a
connection is seen

  1) in on rtk0, src 62.65.145.30, dst 10.100.1.5

    a) state lookup, no match

    b) translation rules check
       matches second rdr rule, dst gets replaced with 10.101.0.4

    c) filter rules check
       matches last pass rule, passes, creates state (because
       translations implicitely create state)

       state entry looks like this:
         rtk0, in, ext 62.65.145.30, gwy 10.100.1.5, lan 10.101.0.4

Normal IP forwarding happens, routing table is consulted for destination
10.101.0.4, which is locally reachable on fxp0.

  2) out on fxp0, src 62.65.145.30, dst 10.101.0.4

     a) no matching state
     b) no matching translation rule
     c) matches last pass rule, passes, does not create state

Now 10.101.0.4 gets the packet, and wants to reply to 62.65.145.30.
Assuming its gateway is the pf box, the reply will arrive there

  3) in on fxp0, src 10.101.0.4, dst 62.65.145.30

     a) no matching state
     b) no matching translation rule
     c) matches last pass rule, passes, does not create state

IP forwarding, routing table for destination 62.65.145.30 says not
local, using default gateway 

  4) out on ex0, src 10.101.0.4, dst 62.65.145.30

     a) no matching state (the one created in step 1 is if-bound to
        fxp0!)
     b) no matching translation rule
     c) matches last pass rule, passes, does not create state
        (the other pass rules don't match, because the src address
         hasn't been replaced, because the state didn't match)

So the src 10.101.0.4 is never translated back, and 62.65.145.30 gets a
reply from 10.101.0.4, which it will ignore.

The most common solution is to make the state created from step 1 not
if-bound, but floating, so it matches replies attempting to go out
through ex0.

Then, add an explicit pass rule for the initial packet creating that
state, and add 'reply-to' there.

  rdr on rtk0 inet proto tcp from any to 10.100.1.5 -> 10.101.0.4
  pass in on rtk0 reply-to (rtk0 10.100.1.6) \
      inet proto tcp from any to 10.101.0.4 keep state

Note that the destination address is already translated when filtering
occurs (translation rules apply before filter rules), hence the "to
10.101.0.4" in the pass rule.

The reply will still first be filtered out on ex0, but it now matches
the (floating) state entry, which contains the instruction to reply-to
rtk0, which applies to all packets matching the state flowing in the
reverse direction (compared to the packet that created the state).

Daniel