Hello, On 01.07.22 12:48, Brad Spencer wrote:
"J. Hannken-Illjes" <hannken%mailbox.org@localhost> writes:On 1. Jul 2022, at 07:55, Matthias Petermann <mp%petermann-it.de@localhost> wrote: Good day, since some time I noticed that on several of my systems with NetBSD/amd64 9.99.97/98 after longer usage the kernel process pgdaemon completely claims a CPU core for itself, i.e. constantly consumes 100%. The affected systems do not have a shortage of RAM and the problem does not disappear even if all workloads are stopped, and thus no RAM is actually used by application processes. I noticed this especially in connection with accesses to the ZFS set up on the respective machines - for example after checkout from the local CVS relic hosted on ZFS. Is there already a known problem or what information would have to be collected to get to the bottom of this? I currently have such a case online, so I would be happy to pull diagnostic information this evening/afternoon. At the moment all info I have is from top. Normal view: ``` PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND 0 root 126 0 0K 34M CPU/0 102:45 100% 100% [system] ``` Thread view: ``` PID LID USERNAME PRI STATE TIME WCPU CPU NAME COMMAND 0 173 root 126 CPU/1 96:57 98.93% 98.93% pgdaemon [system] ```Looks a lot like kern/55707: ZFS seems to trigger a lot of xcalls Last action proposed was to back out the patch ... -- J. Hannken-Illjes - hannken%mailbox.org@localhostProbably only a slightly related data point, but Ya, if you have a system / VM / Xen PV that does not have a whole lot of RAM and if you don't back out that patch your system will become unusable in a very short order if you do much at all with ZFS (tested with a recent -current building pkgsrc packages on a Xen PVHVM). The patch does fix a real bug, as NetBSD doesn't have the define that it uses, but the effect of running that code will be needed if you use ZFS at all on a "low" RAM system. I personally suspect that the ZFS ARC or some pool is allowed to consume nearly all available "something" (pools, RAM, etc..) without limit but have no specific proof (or there is a leak somewhere). I mostly run 9.x ZFS right now (which may have other problems), and have been setting maxvnodes way down for some time. If I don't do that the Xen PV will hang itself up after a couple of 'build.sh release' runs when the source and build artifacts are on ZFS filesets.
Thanks for describing this use case. Apart from the fact that I don't currently use Xen on the affected machine, it performs similiar workload. I use it as pbulk builder with distfiles, build artifacts and CVS / Git mirror stored on ZFS. The builders themself are located in chroot sandboxes on FFS. Anyway, I can trigger the observations by doing a NetBSD src checkout from ZFS backed CVS to the FFS partition.
The maxvnodes trick first led to pgdaemon behave normal again, but the system freezed shortly after with no further evidence.
I am not sure if this thread is the right one for pointing this out, but I experienced further issues with NetBSD current and ZFS when I tried to perform a recursive "zfs send" of a particular snapshot of my data sets. After it initially works, I see the system freeze after a couple of seconds with no chance to recover (could not even enter the kernel debugger). I will come back and need to prepare a dedicated test VM for my cases.
Kind regards Matthias
Attachment:
smime.p7s
Description: S/MIME Cryptographic Signature