Weird memory usage, performance problem with 6.1_STABLE

[Mark Davies <mark%ecs.vuw.ac.nz@localhost>]

I have a system that is (sometimes) used as an ftp server to serve g4u 
disk images.  Current machine is a Dell PowerEdge R320 with 16GB 
memory running 6.1_STABLE from yesterday.

If I get 3 ftp clients all reading the same 45GB image from it I 
quickly get into the situation that all memory is used by something 
and the machine becomes very unresponsive.  Below is the output of top 
while in this state:

load averages:  3.38,  1.56,  0.68;               up 0+16:11:05                 
                        
14:51:52
43 processes: 2 runnable, 38 sleeping, 3 on CPU
CPU0 states:  0.0% user,  0.0% nice, 23.6% system,  0.0% interrupt, 
76.4% idle
CPU1 states:  0.0% user,  0.0% nice,  4.1% system,  0.0% interrupt, 
95.9% idle
CPU2 states:  0.0% user,  0.0% nice, 28.5% system,  0.0% interrupt, 
71.5% idle
CPU3 states:  0.0% user,  0.0% nice,  2.1% system,  0.0% interrupt, 
97.9% idle
CPU4 states:  0.0% user,  0.0% nice,  3.8% system,  0.0% interrupt, 
96.2% idle
CPU5 states:  0.0% user,  0.0% nice, 78.9% system,  0.0% interrupt, 
21.1% idle
Memory: 15G Act, 113M Inact, 15M Wired, 29M Exec, 15G File, 112K Free
Swap: 8192M Total, 8192M Free

  PID USERNAME PRI NICE   SIZE   RES STATE      TIME   WCPU    CPU 
COMMAND
    0 root     126    0     0K   29M CPU/1      3:37 99.76% 99.76% 
[system]
 7173 root      79    0    53M 4588K CPU/2      1:52 36.13% 36.13% 
ftpd
 6931 root     113    0    53M 4588K RUN/2      1:13 28.42% 28.42% 
ftpd
 7025 root     115    0    53M 4588K RUN/1      2:03 26.61% 26.61% 
ftpd
 6601 root      43    0    17M 1876K CPU/5      0:01  4.05%  4.05% top


If I just run two ftp clients there seems to be about 5GB of Inactive 
memory and performance is "fine".

Previously I had an i386 box running 5.x or 6.x doing this job and it 
could quite happily have 10-15 clients slurping images at once.

So whats going on?

cheers
mark