swap space leak in 5.0_STABLE

[Mark Davies <mark%ecs.vuw.ac.nz@localhost>]

I have a bunch of i386 systems running 5.0_STABLE from about 21 July.

They appear to have a problem with swap space exhaustion that is new (only 
started noticing it with this kernel version).

eg.
halswell# pstat -s
Device      1K-blocks     Used    Avail Capacity  Priority
/dev/wd0b     2097648  2092924     4724   100%    0
halswell# ps -laxww
  UID   PID  PPID  CPU PRI NI    VSZ   RSS WCHAN   STAT TTY       TIME 
COMMAND
    0     0     0    0 125  0      0 70188 schedul OKl  ?     38:22.83 
[system]
    0     1     0    0  85  0   2996   200 wait    Is   ?      0:00.05 init
    0    25     1    0  85  0   4180  1148 select  Is   ?      
0:00.00 /usr/pkg/bin/kdm -config /usr/local/lib/kdm/kdmrc
    0   107     1    0  85  0   2968     4 select  IWs  ?      
0:00.46 /sbin/dhclient
    0   150     1    0  85  0   2992   520 kqueue  Is   ?      
0:02.34 /usr/sbin/syslogd -s
    0   181     1    0  85  0   3116   244 select  Ss   ?      
0:01.18 /usr/sbin/rpcbind -l
    0   205    25    0  85  0 312708 62356 select  S    ?      
0:03.30 /usr/pkg/bin/Xorg -lf 256 vt05 
dpms :0 -auth /var/run/xauth/A:0-00025a
    0   227     1    0  85  0   5136  9236 select  Ss   ?      
0:34.49 /usr/sbin/amd -l syslog -x error,noinfo,nostats -c 
900 -r -p -a /am -F /etc/amd.conf
    0   232     1    0  85  0   3024   200 select  Ss   ?      
0:00.79 /usr/sbin/rpc.lockd
    0   251     1 2268  85  0  26540     4 select  IWsl ?      
0:00.27 /usr/sbin/nfsd -6tun 4
    0   253     1    0  85  0   3016   240 select  Ss   ?      
0:00.90 /usr/sbin/mountd
    0   274     1    0  85  0   3024   196 select  Ss   ?      
0:00.81 /usr/sbin/rpc.statd
    0   325     1    0  85  0   5684  5880 pause   Ss   ?      
0:14.36 /usr/sbin/ntpd -A
    0   350     1    0  85  0   6308   640 select  Is   ?      
0:00.33 /usr/sbin/sshd
   96   442     1    0  43  0  31800  2464 parked  Sl   ?      
1:24.81 /usr/pkg/sge/bin/nbsd-i386/sge_execd
    0   457     1    0  85  0   3036     4 kqueue  IWs  ?      
0:00.01 /usr/sbin/inetd -l
32767   473     1    0  85  0   8364   768 select  Ss   ?      
0:07.63 /usr/pkg/sbin/gmond
    0   532     1    0  85  0   2960   504 nanoslp Is   ?      
0:01.17 /usr/sbin/cron
  110   623     1    0  85  0   8232     4 select  IWs  ?      
0:00.41 /usr/pkg/bin/dbus-daemon --system
   96  2808  7943    0  85  0   3012   524 piperd  S    ?      
0:00.97 /bin/sh /usr/pkg/sge/smscs1/common/load.sh
   96  7943   442    0  85  0   3012     4 wait    IW   ?      
0:00.03 /bin/sh -c /usr/pkg/sge/smscs1/common/load.sh
    0 13196 14106    0  85  0  29008 17608 select  S    ?      
0:01.36 /usr/pkg/bin/kdm_greet
    0 14106    25    0  85  0   4276  1620 piperd  I    ?      0:00.00 
kdm: :0
    0 23005     1    0  85  0   5216  2564 select  I    ?      0:00.00 
xconsole
    0 27698   350    0  43  0   6476  3868 -       Ss   ?      0:00.03 
sshd: root@pts/1 (sshd)
    0  7640 27698    0  85  0   2208  1980 pause   Ss   ttyp1  
0:00.02 -tcsh
    0 28836  7640    0  43  0   3024   836 -       O+   ttyp1  0:00.00 
ps -laxww
    0   615     1   11  85  0   2972     4 ttyraw  IWs+ ttyE0  
0:00.00 /usr/libexec/getty Pc ttyE0

This is a machine with 2GB of memory and a 2GB swap partition and normally 
would not use any of the swap at all.  Other machines are similar.

Another machine has gone from:
Device      1K-blocks     Used    Avail Capacity  Priority
/dev/xbd0b    2097152   199860  1897292    10%    0

to

Device      1K-blocks     Used    Avail Capacity  Priority
/dev/xbd0b    2097152   574428  1522724    27%    0

in half an hour but again I don't see any obvious cause in the running 
processes.  It has run some java (linux) processes in that period but not 
much else.

Previously these machines would have been running a 5.0_STABLE from about 
26 June so is there a change in that month that could explain this?

cheers
mark