Peter Eisch <peter@boku.net> wrote:
[snip]
> 20:59:10.146020 IP (tos 0x0, ttl 115, id 57089, offset 0, flags [DF], \
>      length: 1431) SVR.80 > CLIENT.40033: FP [tcp sum ok] 519:1910(1391) \
>     ack 420 win 65116
> 20:59:10.146067 IP (tos 0x0, ttl 255, id 46179, offset 0, flags [none], \
>      length: 56) FW > SVR: icmp 36: CLIENT unreachable - need to frag for \
>     IP  (tos 0x0, ttl 114, id 57089, offset 0, flags [DF], length: 1431, \
>     bad cksum 99ca (->9aca)!) SVR.80 > CLIENT.40033: [|tcp]

This looks like a sort of problem - wrong checksumming leading to
ICMP message rejected by <SVR>, and as result broken PMTUD.

Could you show output of the following command on both <SVR> and <FW>:

    netstat -s -f inet | \
	egrep -e '^[a-z]+:' -e 'packets (received|sent)$' -e '(frag|sum)'

Also, because you're using wm(4) try to work without hardware
checksumming and fragmentation. The following command may do the
trick:

    ifconfig wm0 -udp4csum -tcp4csum -ip4csum -tso4

> If I raise the mtu on the gif interfaces to 1500, _everything_ works great
> at the application layer.  This obviously introduces fragments, but if it's
> the only way that works I guess I can keep it.

Hm. As far as I understand you have configuration similar to:

    [CLIENT]---[FW]===[SRV]

<FW> and <SRV> are connected through gif(4). <FW> and <CLIENT> are
linked via wm0 on <FW>. Then:

1) Where the packets fragmented? Exact interface on exact host, please...
2) Whom packets fragmented? I.e. who is src? And who is dst?
3) Are all the hosts NetBSD based?
4) Is PMTUD enabled on both <CLIENT> and <SRV>? See ip.mtudisc and
   tcp.mssdflt sysctls.

--
Mishka.