Re: pgdaemon high CPU consumption

[Brad Spencer <brad%anduin.eldar.org@localhost>]

"J. Hannken-Illjes" <hannken%mailbox.org@localhost> writes:

>> On 1. Jul 2022, at 07:55, Matthias Petermann <mp%petermann-it.de@localhost> wrote:
>> 
>> Good day,
>> 
>> since some time I noticed that on several of my systems with NetBSD/amd64 9.99.97/98 after longer usage the kernel process pgdaemon completely claims a CPU core for itself, i.e. constantly consumes 100%.
>> The affected systems do not have a shortage of RAM and the problem does not disappear even if all workloads are stopped, and thus no RAM is actually used by application processes.
>> 
>> I noticed this especially in connection with accesses to the ZFS set up on the respective machines - for example after checkout from the local CVS relic hosted on ZFS.
>> 
>> Is there already a known problem or what information would have to be collected to get to the bottom of this?
>> 
>> I currently have such a case online, so I would be happy to pull diagnostic information this evening/afternoon. At the moment all info I have is from top.
>> 
>> Normal view:
>> 
>> ```
>>  PID USERNAME PRI NICE   SIZE   RES STATE       TIME   WCPU    CPU COMMAND
>>    0 root     126    0     0K   34M CPU/0     102:45   100%   100% [system]
>> ```
>> 
>> Thread view:
>> 
>> 
>> ```
>>  PID   LID USERNAME PRI STATE       TIME   WCPU    CPU NAME      COMMAND
>>    0   173 root     126 CPU/1      96:57 98.93% 98.93% pgdaemon  [system]
>> ```
>
> Looks a lot like kern/55707: ZFS seems to trigger a lot of xcalls
>
> Last action proposed was to back out the patch ...
>
> --
> J. Hannken-Illjes - hannken%mailbox.org@localhost


Probably only a slightly related data point, but Ya, if you have a
system / VM / Xen PV that does not have a whole lot of RAM and if you
don't back out that patch your system will become unusable in a very
short order if you do much at all with ZFS (tested with a recent
-current building pkgsrc packages on a Xen PVHVM).  The patch does fix a
real bug, as NetBSD doesn't have the define that it uses, but the effect
of running that code will be needed if you use ZFS at all on a "low" RAM
system.  I personally suspect that the ZFS ARC or some pool is allowed
to consume nearly all available "something" (pools, RAM, etc..) without
limit but have no specific proof (or there is a leak somewhere).  I
mostly run 9.x ZFS right now (which may have other problems), and have
been setting maxvnodes way down for some time.  If I don't do that the
Xen PV will hang itself up after a couple of 'build.sh release' runs
when the source and build artifacts are on ZFS filesets.



-- 
Brad Spencer - brad%anduin.eldar.org@localhost - KC8VKS - http://anduin.eldar.org