Re: kern/55707 (ZFS seems to trigger a lot of xcalls)

[Frank Kardel <kardel%netbsd.org@localhost>]

The following reply was made to PR kern/55707; it has been noted by GNATS.

From: Frank Kardel <kardel%netbsd.org@localhost>
To: gnats-bugs%netbsd.org@localhost
Cc: 
Subject: Re: kern/55707 (ZFS seems to trigger a lot of xcalls)
Date: Thu, 5 May 2022 08:09:44 +0200

 My testcase is a bareos (fork of bacula) incremental backup.
 
 This run has a database (postgres) ohase and then a file system scan phase.
 
 Previously there where high xcall rates.
 
 I see significant improvements.
 
      - xcalls load vanished
 
      - IO seems to have improved
 
      - the incremental backup reduced the run time from 48 minutes to 37 
 minutes
 
 So far very good. But another peculiarity showed up.
 
 pgdaemon gets very busy but does not start to scan.
 
 top
 
 load averages:  2.15,  1.00,  0.44;               up 0+00:06:50 07:28:26
 898 threads: 160 idle, 3 runnable, 725 sleeping, 8 zombie, 2 on CPU
 CPU states:  0.9% user,  0.0% nice,  5.1% system,  0.2% interrupt, 93.6% 
 idle
 Memory: 3732M Act, 1863M Inact, 34M Wired, 140M Exec, 2766M File, 41G Free
 Swap: 544G Total, 544G Free
 
    PID   LID USERNAME PRI STATE      TIME   WCPU    CPU NAME COMMAND
      0   984 root     126 CPU/4      2:28 95.21% 95.21% pgdaemon [system]
   2795  2795 pgsql     77 zfscv/0    1:09 37.30% 37.30% - postgres
      0  8005 root     127 zfscv/0    0:01  4.98%  4.98% zio_write [system]
      0  1733 root      96 zfscv/0    0:02  2.10%  2.10% zfs [system]
      0  7047 root     127 zfscv/1    0:01  1.71%  1.71% zio_write [system]
      0  1638 root     127 zfscv/3    0:01  0.98%  0.98% zio_write [system]
      0  6601 root     126 zfscv/16   0:01  0.88%  0.88% zio_write [system]
      0  6596 root     126 zfscv/18   0:01  0.83%  0.83% zio_write [system]
      0  1632 root     127 zfscv/0    0:02  0.78%  0.78% zio_write [system]
      0  6595 root     126 zfscv/1    0:01  0.73%  0.73% zio_write [system]
      0  1736 root     126 zfscv/0    0:01  0.54%  0.54% zio_write [system]
      0 29247 root     127 zfscv/6    0:00  0.34%  0.34% zio_rea*0 [system]
   3824  3824 clamav    85 poll/3     0:14  0.00%  0.00% - clamd
 
 vmstat -s
 
      ...
 
          0 pages freed by daemon
          0 pages scanned by daemon
          0 anonymous pages scanned by daemon
          0 object pages scanned by daemon
          0 pages reactivated
          0 pages found busy by daemon
          0 total pending pageouts
     949303 pages deactivated
      ...
 
 Looks like pgdaemon gets woken up just to find out that there is nothing 
 to do.
 
 So this is already very goog progress.
 
 Thanks,
 
    Frank
 
 On 05/04/22 18:00, Frank Kardel wrote:
 > The following reply was made to PR kern/55707; it has been noted by GNATS.
 >
 > From: Frank Kardel <kardel%kardel.name@localhost>
 > To: gnats-bugs%netbsd.org@localhost
 > Cc:
 > Subject: Re: kern/55707 (ZFS seems to trigger a lot of xcalls)
 > Date: Wed, 4 May 2022 17:56:49 +0200
 >
 >   Good catch.
 >   
 >   Will test probably on the weekend.
 >   
 >   Frank
 >   
 >   
 >   On 05/04/22 17:52, chs%NetBSD.org@localhost wrote:
 >   > Synopsis: ZFS seems to trigger a lot of xcalls
 >   >
 >   > Responsible-Changed-From-To: kern-bug-people->chs
 >   > Responsible-Changed-By: chs%NetBSD.org@localhost
 >   > Responsible-Changed-When: Wed, 04 May 2022 15:52:04 +0000
 >   > Responsible-Changed-Why:
 >   > take
 >   >
 >   >
 >   > State-Changed-From-To: open->feedback
 >   > State-Changed-By: chs%NetBSD.org@localhost
 >   > State-Changed-When: Wed, 04 May 2022 15:52:04 +0000
 >   > State-Changed-Why:
 >   > this should be fixed by src/external/cddl/osnet/dist/uts/common/fs/zfs/arc.c rev 1.21,
 >   > can you confirm?
 >   >
 >   >
 >   >
 >