>Number:         27164
>Category:       kern
>Synopsis:       ipfilter 'keep state' blocking legal connections
>Confidential:   yes
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Oct 06 15:07:01 UTC 2004
>Closed-Date:
>Last-Modified:
>Originator:     Hauke Fath <hf@spg.tu-darmstadt.de>
>Release:        NetBSD 1.6ZG
>Organization:
-- 
/~\  The ASCII Ribbon Campaign                    Hauke Fath
\ /    No HTML/RTF in email	        Institut für Nachrichtentechnik
 X     No Word docs in email	                  TU Darmstadt
/ \  Respect for open standards              Ruf +49-6151-16-3281
>Environment:
	
	
System: NetBSD fifi 1.6ZG NetBSD 1.6ZG (FIFI) #0: Mon Dec 22 14:41:32 CET 2003  hf@heiligenberg:/var/obj/netbsd-builds/i386/obj/sys/arch/i386/compile/FIFI i386

Architecture: i386
Machine: i386

>Description:

Given the setup (4 interfaces, 4 subnets)


                 < DMZ >------+
                              |
                              | 
   <clients>-------------< ipf router >------------------/INTERNET/
                              |
                              |
      <Internal Servers>------+


where <clients> can access a machine in <internal servers> ("IS") for NIS
services and NFS shares.

The NFS server is running NetBSD 2.0_beta, the clients are NetBSD
1.6.2 (upgraded since), NetBSD 2.0_beta, stock RedHat 8 (linux
2.4.18-27.8.0) + 9 (linux 2.4.20-8 / 2.4.27) installations.

The <clients> can access IS via rules like

pass in log quick on wm1 all keep frags head 100
	pass in quick proto tcp from <clients> to any flags S keep state keep frags group 100
	pass in quick proto udp from <clients> to any keep state keep frags group 100
	pass in quick proto icmp from <clients> to any keep state keep frags group 100

<clients> access to IS works fine for YP. NetBSD clients work fine with the IS nfs server.

The five RedHat <clients>, OTOH, work fine *most of the time*. Then,
often after a user login and associated amd mount, a RH machine will
lock up. The ipfilter log on the router will show lots of entries
like

00:03:06.688586 wm0 @0:7 b SERVER[130.83.xx.yy],nfs -> CLIENT[130.83.uu.vv],800 PR tcp len 20 52 -A IN

for nfs3/tcp connections, and

18:01:32.254120 2x wm1 @0:5 b SERVER[130.83.xx.yy] -> CLIENT[130.83.uu.vv] PR udp len 20 (1500) frag +1480@1480 OUT 
18:01:32.254470 13x wm0 @0:7 b SERVER[130.83.xx.yy] -> CLIENT[130.83.uu.vv] PR udp len 20 (1500) frag +1480@1480 IN 

for nfs2/udp mounts. The @0:[57] rules are "block {in,out} log quick"
style head rules.

At this point, rebooting client or server machine
does not change anything. Leaving the client powered off for a night,
OTOH, re-enables client access - until eventually the problem kicks in
again.

Monitoring the connection on the client-side switch (ethereal) shows
repeated attempts to contact the server (nfs3/tcp) that are never
acknowledged (tcpdump protocol available).

A typical ipfstate on the router today gives

 IPv6 packets:          in 50 out 62
 input packets:         blocked 1381539 passed 600800307 nomatch 50 counted 143525206 short 0
output packets:         blocked 14689 passed 600802869 nomatch 62 counted 192702497 short 0
 input packets logged:  blocked 1368453 passed 8678
output packets logged:  blocked 14689 passed 9978
 packets logged:        input 0 output 0
 log failures:          input 37670 output 2937
fragment state(in):     kept 422488     lost 122032
fragment state(out):    kept 422130     lost 125298
packet state(in):       kept 5311041    lost 16173
packet state(out):      kept 2562221    lost 23665
ICMP replies:   92878   TCP RSTs sent:  742704
Invalid source(in):     0
Result cache hits(in):  9550076 (out):  9273810
IN Pullups succeeded:   0       failed: 0
OUT Pullups succeeded:  0       failed: 0
Fastroute successes:    835582  failures:       0
TCP cksum fails(in):    4       (out):  32
Packet log flags set: (0)
        none


Apart from this nfs problem, both netbsd-netbsd nfs connections and
any non-nfs connections involving RH machines have been working fine
during the nine months that the router is running now.

>How-To-Repeat:

	Set up a NetBSD 2 nfs/nis server and RedHat 8/9 client
	machines on two sides of an ipfilter based router like
	pictured above.

	Find that eventually the RH clients wil hang, and rebooting
	them won't help. Find a lot of log entries on the router.

>Fix:

	The workaround I found was to set up vanilla 'pass
	all' rules (2 directions times 2 interfaces) *without* keeping
	state. Anything *with* keep state between RH nfs clients and
	the NetBSD nfs server would eventually trigger the problem.

	While this kludge works for me in the current situation, it is
	not going to suffice in a security-sensitive context.
>Release-Note:
>Audit-Trail:
>Unformatted:
 
 ipfilter version: # ipf -V
 ipf: IP Filter: v3.4.29 (336)
 Kernel: IP Filter: v3.4.29 
 
 [hf@fifi] /sbin # ident ipf
 ipf:
      $IPFilter: parse.c,v 2.8 1999/12/28 10:49:46 darrenr Exp $
      $IPFilter: parse.c,v 2.8 1999/12/28 10:49:46 darrenr Exp $
      $NetBSD: crt0.c,v 1.13 2003/07/26 19:24:27 salo Exp $
 
 [hf@fifi] /sbin # ident /netbsd 
 /netbsd:
      $NetBSD: GENERIC,v 1.583 2003/11/20 13:32:41 fvdl Exp $
      $Revision: 1.1 $
      $NetBSD: std.i386,v 1.24 2003/02/26 21:33:36 fvdl Exp $
 [...]
      $NetBSD: fil.c,v 1.59 2003/08/07 16:33:07 agc Exp $
      $NetBSD: ip_auth.c,v 1.32 2003/08/22 21:53:03 itojun Exp $
      $NetBSD: ip_fil.c,v 1.95 2003/08/22 22:11:44 itojun Exp $
      $NetBSD: ip_frag.c,v 1.34 2002/09/19 08:12:49 martti Exp $
      $NetBSD: ip_log.c,v 1.23 2002/09/25 06:43:21 martti Exp $
      $NetBSD: ip_nat.c,v 1.55 2003/12/04 15:32:01 christos Exp $
      $NetBSD: ip_proxy.c,v 1.36 2002/09/19 08:12:54 martti Exp $
      $NetBSD: ip_ftp_pxy.c,v 1.26 2002/09/19 08:12:50 martti Exp $
      $NetBSD: ip_rcmd_pxy.c,v 1.9 2002/01/24 08:23:45 martti Exp $
      $NetBSD: ip_raudio_pxy.c,v 1.8 2002/01/24 08:23:14 martti Exp $
      $NetBSD: ip_netbios_pxy.c,v 1.4 2002/09/19 08:12:53 martti Exp $
      $NetBSD: ip_h323_pxy.c,v 1.8 2003/06/23 15:20:57 martin Exp $
      $NetBSD: ip_ipsec_pxy.c,v 1.2 2002/04/01 16:47:46 jdolecek Exp $
      $NetBSD: ip_state.c,v 1.42 2002/09/19 08:12:54 martti Exp $
 [...]