Hi!

On Tue, 23 Jul 2002 itojun@iijlab.net wrote:

> >	- without IPSEC I can transfer bytes in both directions normally
> >	- with IPSEC enabled transfers to either direction fail for
> >	  bigger packets (one end has ep0, the other has ex0); ie. packets
> >	  that grow over MTU size due to IPSEC overhead
> >	- with IPSEC policies, but Path MTU Discovery disabled
> >	  (sysctl -w net.inet.ip.mtudisc=0) problems disappear

> 	i think, between your nodes, there's some router which is discarding
> 	icmp need fragment message (= generic PMTUD blackhole problem).

That could well be, but I think it is not related to the problems I'm
seeing. As I mentioned in my original post, running tcpdump on both
hosts during a failing session, I can see the same packets at both
dumps, which, to me, says that I'm not loosing any packets over the network.
Thus, no packets dropped there, no ICMP/NF, no ICMP/NF drop, no
PMTUD blackhole found. Please correct me, if my logic fails here.

Also, since both hosts have the interface MTU at 1500, and without
IPSEC, with mtudisc=1, I can get regular traffic with 1500 byte packets
through, I think the problem really isn't "in the network", but rather
at the end points. I have no idea why the IPSEC'ized packets that
need to be fragmented never show up. Is there a setting somewhere
(other than ipfilter) that could cause them to not get sent (nor appear
in the tcpdump output of sending host)?

Also, as the tcpdump output in my original post shows, the server that
should be sending the data *seems* to be the one that is dropping packets,
since the tcpdump on that host shows the sent IPSEC/seq# skipping several
numbers. Here is the output again (from the server 10.2.2.2):

tcpdump -n -i ep0 host 10.1.1.1 (with IPSEC, modified timestamp,IP,spi)
------------------------------------------------------------------------------
03.13645 10.1.1.1 > 10.2.2.2: AH(spi=0x1,seq=0x1): ESP(spi=0x0,seq=0x1) (DF)
03.13767 10.2.2.2 > 10.1.1.1: AH(spi=0x3,seq=0x1): ESP(spi=0x2,seq=0x1) (DF)
03.14633 10.1.1.1 > 10.2.2.2: AH(spi=0x1,seq=0x2): ESP(spi=0x0,seq=0x2) (DF)
23.09855 10.1.1.1 > 10.2.2.2: AH(spi=0x1,seq=0x3): ESP(spi=0x0,seq=0x3) (DF)
23.28964 10.2.2.2 > 10.1.1.1: AH(spi=0x3,seq=0x2): ESP(spi=0x2,seq=0x2) (DF)
24.43422 10.1.1.1 > 10.2.2.2: AH(spi=0x1,seq=0x4): ESP(spi=0x0,seq=0x4) (DF)
24.43878 10.2.2.2 > 10.1.1.1: AH(spi=0x3,seq=0x4): ESP(spi=0x2,seq=0x4) (DF)
24.45133 10.1.1.1 > 10.2.2.2: AH(spi=0x1,seq=0x5): ESP(spi=0x0,seq=0x5) (DF)
47.05617 10.1.1.1 > 10.2.2.2: AH(spi=0x1,seq=0x6): ESP(spi=0x0,seq=0x6) (DF)
47.05724 10.2.2.2 > 10.1.1.1: AH(spi=0x3,seq=0x7): ESP(spi=0x2,seq=0x7)
------------------------------------------------------------------------------

Now, I don't really know where tcpdump gets the data, I'm just assuming
it is the PFIL_HOOKS stage in sys/netinet/ip_output.c. If that is the case
then again I'm guessing the packet loss happens before the driver, so it
would not be (at least directly) a driver issue. Continuing from there,
if my interpretation of ip_ouput (and its relevance here) is correct,
then I should see the oversized IPSEC packet(s) in the tcpdump output
before they get fragmented to fit the interface MTU (or smaller). All
I'm seeing are small packets that don't need fragmentation, and thus
also get through.

Sure, I can live (at least for the time being) with mtudisc=0, using
the small packets that get through. I could even play with the
net.inet.tcp.mssdflt and net.inet.tcp.mss_ifmtu settings to see what
works best for me. If, however, somebody thinks that there may be a bug
related to my problems, please contact me directly, and I'll
try to provide any additional details/testing that might be needed.
Otherwise, I'll just settle for what works for me now, and keep the list
clean.


Artsi
#######======------  http://www.selonen.org/arto/  --------========########
Everstinkuja 5 B 35                               Don't mind doing it.
FIN-02600 Espoo        arto@selonen.org         Don't mind not doing it.
Finland              tel +358 50 560 4826     Don't know anything about it.