Re: IPFilter issue in -current

[Geoff Adams <gadams+netbsd%avernus.com@localhost>]

On Dec 22, 2012, at 11:22 PM, Darren Reed <darrenr%netbsd.org@localhost> wrote:

> The ns_bucketlen issue should be easier to track down with the help
> of a DEBUG build (for ASSERT!) and the patch below. Though it seems
> likely a return path for a failed insert is going to be to blame.

I haven't applied that patch yet, but I do currently have these printfs in my 
running kernel (as wells at other printfs in the places you put the ASSERTs, 
although those don't seem to result in much output):

Index: ip_nat.c
===================================================================
RCS file: /cvsroot/src/sys/external/bsd/ipf/netinet/ip_nat.c,v
retrieving revision 1.7
diff -u -r1.7 ip_nat.c
--- ip_nat.c    20 Dec 2012 21:42:28 -0000      1.7
+++ ip_nat.c    23 Dec 2012 09:20:58 -0000
@@ -2269,14 +2269,22 @@
 
                bkt = nat->nat_hv[0] % softn->ipf_nat_table_sz;
                nss = &softn->ipf_nat_stats.ns_side[0];
                nss->ns_bucketlen[bkt]--;
+               if (nss->ns_bucketlen[bkt] > 100000) {
+                       printf("ns_bucketlen[0:%i] was already 0!\n", bkt);
+                       nss->ns_bucketlen[bkt] = 0;
+               }
                if (nss->ns_bucketlen[bkt] == 0) {
                        nss->ns_inuse--;
                }
 
                bkt = nat->nat_hv[1] % softn->ipf_nat_table_sz;
                nss = &softn->ipf_nat_stats.ns_side[1];
                nss->ns_bucketlen[bkt]--;
+               if (nss->ns_bucketlen[bkt] > 100000) {
+                       printf("ns_bucketlen[1:%i] was already 0!\n", bkt);
+                       nss->ns_bucketlen[bkt] = 0;
+               }
                if (nss->ns_bucketlen[bkt] == 0) {
                        nss->ns_inuse--;
                }

These lines print lots and lots of these:

Dec 23 01:01:34 tho /netbsd: ns_bucketlen[0:5402] was already 0!
Dec 23 01:01:34 tho /netbsd: ns_bucketlen[1:26339] was already 0!
Dec 23 01:01:35 tho /netbsd: ns_bucketlen[0:7336] was already 0!
Dec 23 01:01:35 tho /netbsd: ns_bucketlen[1:26829] was already 0!
Dec 23 01:01:35 tho /netbsd: ns_bucketlen[0:26981] was already 0!
Dec 23 01:01:36 tho /netbsd: ns_bucketlen[0:16564] was already 0!
Dec 23 01:01:36 tho /netbsd: ns_bucketlen[1:19495] was already 0!
Dec 23 01:01:36 tho /netbsd: ns_bucketlen[1:11706] was already 0!
Dec 23 01:01:37 tho /netbsd: ipf_nat_newrdr: mapping already exists!

The slightly good news is that patching ns_bucketlen to 0 after noticing this 
problem keeps everything happy, where before all clients on the network would 
regularly see random connection failures. (Please excuse the extreme hackery in 
that code; I was sleep deprived. Even decent debugging printfs would test for 0 
before decrementing, rather than that ugliness.)

I haven't had time to track down what causes them to be 0 at this point, though.

One final thing I've discovered: The following printf gets hit fairly 
frequently (as seen above):

Index: ip_nat.c
===================================================================
RCS file: /cvsroot/src/sys/external/bsd/ipf/netinet/ip_nat.c,v
retrieving revision 1.7
diff -u -r1.7 ip_nat.c
--- ip_nat.c    20 Dec 2012 21:42:28 -0000      1.7
+++ ip_nat.c    23 Dec 2012 09:20:58 -0000
@@ -2992,6 +3007,7 @@
        if (natl != NULL) {
                DT2(ns_new_xlate_exists, fr_info_t *, fin, nat_t *, natl);
                NBUMPSIDE(0, ns_xlate_exists);
+               printf("ipf_nat_newrdr: mapping already exists!\n");
                return -1;
        }
 
This causes ipf_nat_add to jump to badnat. I also haven't yet spent any time 
looking into why this occurs.

- Geoff