Re: Very poor NFS I/O

[MJ <mjose00%optusnet.com.au@localhost>]

On 28/02/2026 8:16 pm, BERTRAND Joël wrote:
> MJ a écrit :
> > On 28/02/2026 8:21 am, BERTRAND Joël wrote:
> > > Michael van Elst a écrit :
> > > > joel.bertrand%systella.fr@localhost (=?UTF-8?Q?BERTRAND_Jo=c3=abl?=) writes:
> > > >
> > > > >      CPU is an i7-4770, main memory 16 GB. This server exports /srv and
> > > > > /home through NFS (V3/TCP, 128 threads, async) and disk I/O from NFS
> > > > > clients are very slow. Server load can raise until 110 or 120 during
> > > > > huge NFS access.
> > > >
> > > > Can you quantify what "slow" means? Any kind of benchmarks?
> > >
> > >      Less than 2 MB/s.
> > >
> > > > The server load just shows that the NFS requests are distributed over
> > > > your server threads. But that is not related to any CPU utilization.
> > >
> > >      I know. NFS process never reaches 35% of one core.
> > As per Michael's email, re: isolating disk and client, can you isolate
> > which system it is slow serving to? I think that is important.
>
> 	Same result with Linux and FreeBSD. If I reduce number of nfs threads
> (currently 128), it seems to run better but I obtain on client side "nfs
> server not responding".
>
> 	Now, I only have two client (a Linux and a FreeBSD). FreeBSD is idle
> and I do some tests on Linux workstation.
>
> 1/ Linux rootfs is on a Raid1 disk on NetBSD server.
> 2/ iftop shows that nfs server is idle too (a few Kbps).
> 3/ apt update && apt dist-upgrade is very slow (-dev packages with a lot
> of little files).
> 4/ make -j1 kicad (9.0), sources on a Raid5 volume shows a nfs mean
> throughput around 40 Mbps. nfs process eats 1 to 2 % od server CPU.
> 5/ now, I start another compilation and load is rising on server side :
>
> load averages:  10.0,  3.17,  2.06;        up 16+01:28:30       10:10:27
> 93 processes: 92 sleeping, 1 on CPU
> CPU states: 0.1% user, 0.0% nice, 0.1% system, 0.0% interrupt, 99.7% idle
> Memory: 7838M Act, 3937M Inact, 49M Wired, 153M Exec, 9617M File, 84M Free
> Swap: 16G Total, 16G Free / Pools: 3708M Used / Network: 2647K In, 8700K Out
>
>    PID USERNAME PRI NICE   SIZE   RES STATE       TIME   WCPU    CPU COMMAND
>   4326 root      85    0   600M 8248K nfsd/4    823:01  7.13%  7.13% nfsd
>   2553 root      85    0    20M 2640K kqueue/3    2:59  4.88%  4.88% syslogd
>      0 root     221    0     0K   64M rfnode/3  803:17  4.10%  4.10% [system]
>    930 root      85    0    12M 1900K select/0    3:26  1.71%  1.71%
> rpc.lockd
>
>
> 6/ dd if=/dev/zero of=test.dd count=10 bs=100M
> nfs  throughput rises until 850 Mbps (iftop)
> but load average on server side until 56 !
>
> hilbert:[~] > dd if=/dev/zero of=test.dd count=10 bs=100M
> 10+0 enregistrements lus
> 10+0 enregistrements écrits
> 1048576000 octets (1,0 GB, 1000 MiB) copiés, 69,9613 s, 15,0 MB/s


If the load is going up but not CPU usage, perhaps something is blocking a process? Lock contention?

If you do a process list (PS) are there any processes in D state on the server when you perform work over NFS?

I would, personally, also look at using ktruss on the NFS server PID. Watch the output fly past but hope to see some form of contention. :-)
If seen, then output to a file for later viewing.

Some other tools to suggest: nfsstat & iostat.

Use fstat on the NetBSD NFS server PID when you do some work to see if there's any ridiculous open file count.

> > I note you use VMS, which is notorious for going out of "tune" if disks
> > fragment or fill etc.
>
> 	Maybe. But it's a VAX (connected through a 100 Mbps) and another
> PWS500a (100 Mbps) that only make a few nfs transactions. If I switch
> off NFS client on both, NFS I/O remain very poor.

OK well that eliminates the VMS box.

Regards,
Max.