pf has a bug* where if it isn't keeping state then tcp packets with the wscale option set (in the syn, but not in the packet of interest) can be dropped as out-of-window. So make sure you are keeping state and see if that helps. Run tcpdump and look at all the packets, and use pfctl to get counts and stats. This is standard debugging advice but definitely in order here. * 99% sure - really figuring this out and fixing is on my todo list.
Attachment:
pgpKppW0XMuX4.pgp
Description: PGP signature