Re: port-evbarm/56944: ZFS heavy usage on NetBSD running in Mac M1 VM results in kernel thread running away and filesystem hang

To: port-evbarm-maintainer%netbsd.org@localhost,gnats-admin%netbsd.org@localhost,netbsd-bugs%netbsd.org@localhost,pjledge%me.com@localhost
Subject: Re: port-evbarm/56944: ZFS heavy usage on NetBSD running in Mac M1 VM results in kernel thread running away and filesystem hang
From: Brad Spencer <brad%anduin.eldar.org@localhost>
Date: Sat, 30 Jul 2022 11:20:02 +0000 (UTC)
The following reply was made to PR port-evbarm/56944; it has been noted by GNATS.

From: Brad Spencer <brad%anduin.eldar.org@localhost>
To: gnats-bugs%netbsd.org@localhost
Cc: port-evbarm-maintainer%netbsd.org@localhost, gnats-admin%netbsd.org@localhost,
        netbsd-bugs%netbsd.org@localhost, pjledge%me.com@localhost
Subject: Re: port-evbarm/56944: ZFS heavy usage on NetBSD running in Mac M1
 VM results in kernel thread running away and filesystem hang
Date: Sat, 30 Jul 2022 07:19:41 -0400

 --=-=-=
 Content-Type: text/plain
 
 Chuck Silvers <chuq%chuq.com@localhost> writes:
 
 > The following reply was made to PR port-evbarm/56944; it has been noted by GNATS.
 >
 
 [snip]
 
 >  the "paging=16" indicates that some page-outs are already in progress
 >  and not completing.
 >  
 >  if you do the "show uvmexp" twice with a few seconds in between
 >  and none of these counters are changing, then various threads are
 >  probably stuck on something.  please collect a stack trace with ddb
 >  of the pagedaemon thread ("pgdaemon" in ddb ps) and all of the
 >  "softbio" worker threads (there will be one for each CPU).  you might
 >  as well also include the "km_getwait2" thread that you mentioned above.
 >  please also send me the complete output from ddb "ps" so I can see
 >  if there are any other threads that look interesting.
 >  
 >  you may need to collect stack traces from various zfs kernel threads
 >  as well, but there are probably a lot of them, so hopefully we can
 >  narrow down which ones are interesting so that you don't need to
 >  get stack traces for all of them.
 
 The system was hung up this morning, probably tripped by something
 running out of daily.  I have collected some of the information you
 asked for and attached to this reply as a cleaned up typescript output
 from the guest console.  The Xen guest is a test system for -current and
 is not doing much other than running /etc/daily (when I enable it) right
 now.
 
 I can probably reproduce the zfs receive case if needed.  I left the
 guest in ddb should you be interested in something else right now but
 will probably reboot it at some point.
 
 Another small data point with this hang was complaints on the guest
 console from xennet about no rx buffers.  Another data point, mentioned
 later, but this guest does not have any further limits on the number of
 allowed vnodes.
 
 >  > In the hard hang case, the number of "free" would be much larger, so I
 >  > suspect something else is running out of resources at this point (the
 >  > number for free hints at that perhaps pointing to your free page
 >  > comment).  I also noticed that the pool called "zio_data_buf_51" of size
 >  > 1024 didn't grow much about 16,100 with this patch, as opposed to around
 >  > 30,000 with the hard hang.  Limiting the number of vnodes didn't seem to
 >  > effect the behavior of the softer hang.  I may have also noticed that
 >  > the system was paging to swap even though all that was going on was a
 >  > zfs receive over a ssh connection.
 >  
 >  the previous hang was probably due to running out of kernel virtual space,
 >  whereas this hang is due to running out of free physical pages.
 >  
 >  limiting the number of vnodes does not directly limit ARC memory usage.
 >  
 >  the "zfs receive" is allocating nearly all of the physical memory of
 >  the system to kernel usage (ie. the ZFS ARC), and the VM mechanism to
 >  apply back-pressure on kernel memory allocations is limited.
 >  I'm not sure yet how ZFS is supposed to avoid pushing everything else
 >  out of memory.  there may well be other mechanisms that are not
 >  hooked up properly either, similar to zfs_arc_free_target before
 >  the current patch.
 
 I have been of the personal opinion that there is something with ZFS
 that is leaking (for some definition of the word "leak").  It is clear
 that the "zfs receive" case trips the problem very quickly, especially
 for me if I am receiving a compressed file set (i.e. a zfs send -R where
 one of the file sets is compressed).  However, just using the file set
 (i.e. reading and writing, but even just reading, like the find-the-core
 files check from /etc/daily) will also cause trouble over time.
 
 I have a OS building guest running 9.2 and it uses ZFS a lot, both for
 source and build artifacts.  Without limiting the number of vnodes I can
 run a "build.sh release" 1.5 times before the system hangs up.  With
 limiting the vnodes, a whole lot, I get to run 3 or so "build.sh
 release" runs before a hang.  In that use case, it helps quite a bit.
 
 >  is the behavior with the current patch worse in any way than the behavior
 >  from before my previous change to arc.c on may 4?
 
 I would say that there was an improvement.  With the "zfs receive" test
 for me I was able to receive more before the hang, about 2x or 3x more
 data before a hang up.  I was also able to enable /etc/daily and it did
 succeed in running once and hung up on the second day.  Running
 /etc/daily would trip it every time before this patch which was why I
 disable it.
 
 >  -Chuck
 >  
 
 
 
 -- 
 Brad Spencer - brad%anduin.eldar.org@localhost - KC8VKS - http://anduin.eldar.org
 
 --=-=-=
 Content-Type: application/octet-stream
 Content-Disposition: attachment; filename=ddb_output_2022-07-30_1
 Content-Transfer-Encoding: base64
 Content-Description: cleaned up typescript output from ddb
 
 U2NyaXB0IHN0YXJ0ZWQgb24gU2F0IEp1bCAzMCAwNjozNzo1MSAyMDIyCmRhbGUjIHhsIGNvbnNv
 bGUgdGVzdGN1cnJlbnQKClBJRCAgICBMSUQgUyBDUFUgICAgIEZMQUdTICAgICAgIFNUUlVDVCBM
 V1AgKiAgICAgICAgICAgICAgIE5BTUUgV0FJVAoxMjk2OCAxMjk2OCAzICAgMCAgICAgICAgIDAg
 ICBmZmZmOGYzMDVkODYwYWMwICAgICAgICAgICAgICAgY3JvbiBmbHRfcG1mYWlsMgoxMzkwMyAx
 MzkwMyAzICAgMCAgICAgICAgIDAgICBmZmZmOGYzMDVkMmNiYTgwICAgICAgICAgICAgICAgZmlu
 ZCB6ZnNjdgoxMzc4MiAxMzc4MiAzICAgMCAgICAgICAxODAgICBmZmZmOGYzMDVjNzc1YTQwICAg
 ICAgICAgICBwb3N0ZHJvcCBuZXRpbwoxNDA2MSAxNDA2MSAzICAgMCAgICAgICAxODAgICBmZmZm
 OGYzMDVkMmNiNjQwICAgICAgICAgICBzZW5kbWFpbCBwaXBlX3JkCjEyMTE4IDEyMTE4IDMgICAw
 ICAgICAgIDE4MCAgIGZmZmY4ZjMwNjQxMjI2ODAgICAgICAgICAgICAgICAgdGVlIHBpcGVfcmQK
 MTIxNjkgMTIxNjkgMyAgIDAgICAgICAgMTgwICAgZmZmZjhmMzA1ZDQ2YWI0MCAgICAgICAgICAg
 ICAgICAgc2ggd2FpdAoxMzYwMSAxMzYwMSAzICAgMCAgICAgICAxODAgICBmZmZmOGYzMDVkMmNi
 MjAwICAgICAgICAgICAgICAgICBzaCB3YWl0CjEyMTUyIDEyMTUyIDMgICAwICAgICAgIDE4MCAg
 IGZmZmY4ZjMwNWRkYTkwNDAgICAgICAgICAgICAgICBjcm9uIHBpcGVfcmQKMTMxNzMgMTMxNzMg
 MyAgIDAgICAxMDAwMDQwICAgZmZmZjhmMzA1Yzk4ZTc0MCAgICAgICAgICAgICBwaWNrdXAgZmx0
 X25vcmFtMQoxMDY2ICAxMDY2IDMgICAwICAgICAgICAgMCAgIGZmZmY4ZjMwOTZlYzdhNDAgICAg
 ICAgICAgICAgIGdldHR5IGZsdF9ub3JhbTEKMTE3MCAgMTE3MCAzICAgMCAgICAgICAgNDAgICBm
 ZmZmOGYzMDY0ZjAxYjAwICAgICAgICAgICAgICAgY3JvbiBmbHRfbm9yYW01CjEwNjggIDEwNjgg
 MyAgIDAgICAgICAgMTgwICAgZmZmZjhmMzA2NDEyMmFjMCAgICAgICAgICAgICAgaW5ldGQga3F1
 ZXVlCjEwMTggIDEwMTggMyAgIDAgICAxMDAwMDQwICAgZmZmZjhmMzA1ZWY2ZDljMCAgICAgICAg
 ICAgICAgIHFtZ3IgZmx0X25vcmFtMQoxMjcxICAxMjcxIDMgICAwICAgMTAwMDA0MCAgIGZmZmY4
 ZjMwNjRmMDE2YzAgICAgICAgICAgICAgbWFzdGVyIGZsdF9ub3JhbTEKMTE3OSAgMTE3OSAzICAg
 MCAgICAgICAxODAgICBmZmZmOGYzMDVmNzI4YTQwICAgICAgICAgICAgICAgc3NoZCBwb2xsCjkx
 MCAgICA5MTAgMyAgIDAgICAgICAgICAwICAgZmZmZjhmMzA2MDdiNmE4MCAgICAgICAgICAgICAg
 cndob2QgZmx0X25vcmFtMQo4MTQgICAgODE0IDMgICAwICAgICAgIDE4MCAgIGZmZmY4ZjMwNjA3
 YjY2NDAgICAgICAgICAgICAgcG93ZXJkIGtxdWV1ZQoxMDIzICAxMDIzIDMgICAwICAgMTAwMDAw
 MCAgIGZmZmY4ZjMwNjA3YjYyMDAgICAgICAgICAgICAgICBudHBkIHV2bmZwMQo5MjQgICAgOTI0
 IDMgICAwICAgICAgICA0MCAgIGZmZmY4ZjMwNWY3MjgxYzAgICAgICAgICAgICBycGNiaW5kIHV2
 bmZwMQo2NTYgICAgNjU2IDMgICAwICAgICAgICA0MCAgIGZmZmY4ZjMwOTcwYWZhMDAgICAgICAg
 ICAgICBzeXNsb2dkIGZsdF9ub3JhbTEKNDgzICAgIDQ4MyAzICAgMCAgICAgICAxODAgICBmZmZm
 OGYzMDk3MGFmMTgwICAgICAgICAgICAgIGRoY3BjZCBwb2xsCjQ4MiAgICA0ODIgMyAgIDAgICAg
 ICAgICAwICAgZmZmZjhmMzA1Yjk1ODBjMCAgICAgICAgICAgICBkaGNwY2QgZmx0X25vcmFtMQo0
 ODEgICAgNDgxIDMgICAwICAgICAgIDE4MCAgIGZmZmY4ZjMwOTcwYWY1YzAgICAgICAgICAgICAg
 ZGhjcGNkIHBvbGwKMzUyICAgIDM1MiAzICAgMCAgICAgICAgNDAgICBmZmZmOGYzMDVlZjZkNTgw
 ICAgICAgICAgICAgIGRoY3BjZCBmbHRfbm9yYW0xCjEgICAgICAgIDEgMyAgIDAgICAgICAgMTgw
 ICAgZmZmZjhmMzA1YjEzYjc0MCAgICAgICAgICAgICAgIGluaXQgd2FpdAowICAgICAxNDQ4NSAz
 ICAgMCAgICAgICAyMDAgICBmZmZmOGYzMDc2NmIzMzAwICAgIHppb19yZWFkX2ludHJfNSBwbHBn
 CjAgICAgIDEzODgyIDUgICAwICAgICAgIDI0MCAgIGZmZmY4ZjMwNWM5OGUzMDAgICAgICAgICAg
 ICh6b21iaWUpCjAgICAgIDEzMjQ5IDMgICAwICAgICAgIDIwMCAgIGZmZmY4ZjMwNWI3YzU0YzAg
 ICAgemlvX3dyaXRlX2ludHIvIHRzdGlsZQowICAgICAgMzkwIDMgICAwICAgICAgIDIwMCAgIGZm
 ZmY4ZjMwNWUwMjM5NDAgICAgICAgICAgICAgICAgemZzIHpmc2N2CjAgICAgICAzODkgMyAgIDAg
 ICAgICAgMjAwICAgZmZmZjhmMzA1ZTAyMzUwMCAgICAgICAgICAgICAgICB6ZnMgemZzY3YKMCAg
 ICAgIDMxNyAzICAgMCAgICAgICAyMDAgICBmZmZmOGYzMDVjZWM3NTQwICAgICAgICAgICB2ZGV2
 c3luYyB2ZGV2c3luYwowICAgICAgMzgzIDMgICAwICAgICAgIDIwMCAgIGZmZmY4ZjMwNWNjOGMx
 NDAgICAgICAgICAgIHZkZXZzeW5jIHZkZXZzeW5jCjAgICAgICAxOTAgMyAgIDAgICAgICAgMjAw
 ICAgZmZmZjhmMzA1Y2M4YzU4MCAgICAgICAgICAgdmRldnN5bmMgdmRldnN5bmMKMCAgICAgIDM4
 MSAzICAgMCAgICAgICAyMDAgICBmZmZmOGYzMDVjYzhjOWMwICAgICAgICAgICB2ZGV2c3luYyB2
 ZGV2c3luYwowICAgICAgMTYyIDMgICAwICAgICAgIDI0MCAgIGZmZmY4ZjMwNWNjODg4YzAgICAg
 ICAgICAgICAgICAgemZzIHBscGcKMCAgICAgIDE2MSAzICAgMCAgICAgICAyMDAgICBmZmZmOGYz
 MDVjYzg4NDgwICAgICAgICAgICAgICAgIHpmcyB6ZnNjdgowICAgICAgMjAwIDMgICAwICAgICAg
 IDIwMCAgIGZmZmY4ZjMwNWMzN2NhMDAgICAgICAgICAgIHZkZXZzeW5jIHZkZXZzeW5jCjAgICAg
 ICAxOTkgMyAgIDAgICAgICAgMjAwICAgZmZmZjhmMzA1YzY1ZjY4MCAgICAgICBwb29sZGlzcEAx
 MjYgcG9vbGRpc3AKMCAgICAgIDE5OCAzICAgMCAgICAgICAyMDAgICBmZmZmOGYzMDVjNjVmYWMw
 ICAgICAgIHBvb2xkaXNwQDEyNyBwb29sZGlzcAowICAgICAgMjM4IDMgICAwICAgICAgIDIwMCAg
 IGZmZmY4ZjMwNWM1MzcxNDAgICAgICAgIHNwYV9kZWFkbWFuIHNwYV9kZWFkbWFuCjAgICAgICAx
 NTAgMyAgIDAgICAgICAgMjAwICAgZmZmZjhmMzA1YzI5ZTk4MCAgICAgICAgICAgICAgICB6ZnMg
 emZzY3YKMCAgICAgIDIyNCAzICAgMCAgICAgICAyMDAgICBmZmZmOGYzMDVjMjllNTQwICAgICAg
 ICAgICAgICAgIHpmcyB6ZnNjdgowICAgICAgMTU4IDMgICAwICAgICAgIDIwMCAgIGZmZmY4ZjMw
 NWMyOWUxMDAgICAgICAgIHBvb2xkaXNwQDk2IHBvb2xkaXNwCjAgICAgICAgOTUgMyAgIDAgICAg
 ICAgMjAwICAgZmZmZjhmMzA1Yjk1ODk0MCAgICAgICAgICAgICAgICB6ZnMgemZzY3YKMCAgICAg
 IDE3NSAzICAgMCAgICAgICAyMDAgICBmZmZmOGYzMDViOTU4NTAwICAgICAgICBwb29sZGlzcEA2
 NCBwb29sZGlzcAowICAgICAgIDkzIDMgICAwICAgICAgIDIwMCAgIGZmZmY4ZjMwNWI0ZTczNDAg
 ICAgICAgICAgICBwaHlzaW9kIHBoeXNpb2QKMCAgICAgIDExNCAzICAgMCAgICAgICAyMDAgICBm
 ZmZmOGYzMDViNDdjMDQwICAgICAgICAgIHBvb2xkcmFpbiBwb29sZHJhaW4KMCAgICAgIDExMyAz
 ICAgMCAgICAgICAyNDAgICBmZmZmOGYzMDViNGU3YmMwICAgICAgICAgICAgaW9mbHVzaCB6ZnNj
 dgowICAgICAgMTEyIDMgICAwICAgICAgIDIwMCAgIGZmZmY4ZjMwNWI0ZTc3ODAgICAgICAgICAg
 IHBnZGFlbW9uIHpmc2N2CjAgICAgICAxMDkgMyAgIDAgICAgICAgMjAwICAgZmZmZjhmMzA1YjEz
 YmI4MCAgICAgICAgICAgICBucGZnYzAgbnBmZ2N3CjAgICAgICAxMDggMyAgIDAgICAgICAgMjAw
 ICAgZmZmZjhmMzA1YjEzYjMwMCAgICAgICAgICAgIHJ0X2ZyZWUgcnRfZnJlZQowICAgICAgMTA3
 IDMgICAwICAgICAgIDIwMCAgIGZmZmY4ZjMwNWIxNDZiNDAgICAgICAgICAgICAgIHVucGdjIHVu
 cGdjCjAgICAgICAxMDYgMyAgIDAgICAgICAgMjAwICAgZmZmZjhmMzA1YjAxNzY0MCAgICBrZXlf
 dGltZWhhbmRsZXIga2V5X3RpbWVoYW5kbGVyCgowICAgICAgMTA1IDMgICAwICAgICAgIDIwMCAg
 IGZmZmY4ZjMwNWIwMTdhODAgICAgaWNtcDZfd3FpbnB1dC8wIGljbXA2X3dxaW5wdXQKMCAgICAg
 IDEwNCAzICAgMCAgICAgICAyMDAgICBmZmZmOGYzMDViMTFjMjQwICAgICAgICAgIG5kNl90aW1l
 ciBuZDZfdGltZXIKMCAgICAgIDEwMyAzICAgMCAgICAgICAyMDAgICBmZmZmOGYzMDViMTFjNjgw
 ICAgIGNhcnA2X3dxaW5wdXQvMCBjYXJwNl93cWlucHV0CjAgICAgICAxMDIgMyAgIDAgICAgICAg
 MjAwICAgZmZmZjhmMzA1YjExY2FjMCAgICAgY2FycF93cWlucHV0LzAgY2FycF93cWlucHV0CjAg
 ICAgICAxMDEgMyAgIDAgICAgICAgMjAwICAgZmZmZjhmMzA1YjE0MTI4MCAgICAgaWNtcF93cWlu
 cHV0LzAgaWNtcF93cWlucHV0CjAgICAgICAxMDAgMyAgIDAgICAgICAgMjAwICAgZmZmZjhmMzA1
 YjE0MTZjMCAgICAgICAgICAgcnRfdGltZXIgcnRfdGltZXIKMCAgICAgICA5OSAzICAgMCAgICAg
 ICAyMDAgICBmZmZmOGYzMDViMTQxYjAwICAgICAgICB2bWVtX3JlaGFzaCB2bWVtX3JlaGFzaAow
 ICAgICAgIDk4IDMgICAwICAgICAgIDIwMCAgIGZmZmY4ZjMwNWIxNDY3MDAgICAgICAgICAgICAg
 eGVuYnVzIHhzaW8KMCAgICAgICA5NyAzICAgMCAgICAgICAyMDAgICBmZmZmOGYzMDViMTQ2MmMw
 ICAgICAgICAgICB4ZW53YXRjaCBldnRzcQowICAgICAgIDIzIDMgICAwICAgICAgIDIwMCAgIGZm
 ZmY4ZjMwNWIwMTcyMDAgICAgICAgICAgZW50YnV0bGVyIGVudHJvcHkKMCAgICAgICAyMSAzICAg
 MCAgICAgICAyMDAgICBmZmZmOGYzMDk2ZWM3NjAwICAgICAgICAgICBsbnhzeXN3cSBsbnhzeXN3
 cQowICAgICAgIDIwIDMgICAwICAgICAgIDIwMCAgIGZmZmY4ZjMwOTZlYzcxYzAgICAgICAgICAg
 IGxueHViZHdxIGxueHViZHdxCjAgICAgICAgMTkgMyAgIDAgICAgICAgMjAwICAgZmZmZjhmMzA5
 NmVkNmEwMCAgICAgICAgICAgbG54cHdyd3EgbG54cHdyd3EKMCAgICAgICAxOCAzICAgMCAgICAg
 ICAyMDAgICBmZmZmOGYzMDk2ZWQ2NWMwICAgICAgICAgICBsbnhsbmd3cSBsbnhsbmd3cQowICAg
 ICAgIDE3IDMgICAwICAgICAgIDIwMCAgIGZmZmY4ZjMwOTZlZDYxODAgICAgICAgICAgIGxueGhp
 cHdxIGxueGhpcHdxCjAgICAgICAgMTYgMyAgIDAgICAgICAgMjAwICAgZmZmZjhmMzA5NmVlZDlj
 MCAgICAgICAgICAgbG54cmN1Z2MgbG54cmN1Z2MKMCAgICAgICAxNSAzICAgMCAgICAgICAyMDAg
 ICBmZmZmOGYzMDk2ZWVkNTgwICAgICAgICAgICAgIHN5c21vbiBzbXRhc2txCjAgICAgICAgMTQg
 MyAgIDAgICAgICAgMjAwICAgZmZmZjhmMzA5NmVlZDE0MCAgICAgICAgIHBtZnN1c3BlbmQgcG1m
 c3VzcGVuZAowICAgICAgIDEzIDMgICAwICAgICAgIDIwMCAgIGZmZmY4ZjMwOTZmMDA5ODAgICAg
 ICAgICAgIHBtZmV2ZW50IHBtZmV2ZW50CjAgICAgICAgMTIgMyAgIDAgICAgICAgMjAwICAgZmZm
 ZjhmMzA5NmYwMDU0MCAgICAgICAgIHNvcGVuZGZyZWUgc29wZW5kZnIKMCAgICAgICAxMSAzICAg
 MCAgICAgICAyMDAgICBmZmZmOGYzMDk2ZjAwMTAwICAgICAgICAgICAgaWZsbmtzdCBpZmxua3N0
 CjAgICAgICAgMTAgMyAgIDAgICAgICAgMjAwICAgZmZmZjhmMzA5NzEyNzk0MCAgICAgICAgICAg
 bmZzc2lsbHkgbmZzc2lsbHkKMCAgICAgICAgOSAzICAgMCAgICAgICAyNDAgICBmZmZmOGYzMDk3
 MTI3NTAwICAgICAgICAgICAgIHZkcmFpbiB2ZHJhaW4KMCAgICAgICAgOCAzICAgMCAgICAgICAy
 MDAgICBmZmZmOGYzMDk3MTI3MGMwICAgICAgICAgIG1vZHVubG9hZCBtb2RfdW5sZAowICAgICAg
 ICA3IDMgICAwICAgICAgIDIwMCAgIGZmZmY4ZjMwOTcxNTY5MDAgICAgICAgICAgICB4Y2FsbC8w
 IHhjYWxsCjAgICAgICAgIDYgMSAgIDAgICAgICAgMjAwICAgZmZmZjhmMzA5NzE1NjRjMCAgICAg
 ICAgICBzb2Z0c2VyLzAKMCAgICAgICAgNSAxICAgMCAgICAgICAyMDAgICBmZmZmOGYzMDk3MTU2
 MDgwICAgICAgICAgIHNvZnRjbGsvMAowICAgICAgICA0IDEgICAwICAgICAgIDIwMCAgIGZmZmY4
 ZjMwOTcxN2Y4YzAgICAgICAgICAgc29mdGJpby8wCjAgICAgICAgIDMgMSAgIDAgICAgICAgMjAw
 ICAgZmZmZjhmMzA5NzE3ZjQ4MCAgICAgICAgICBzb2Z0bmV0LzAKMCAgICA+ICAgMiAxICAgMCAg
 ICAgICAyMDEgICBmZmZmOGYzMDk3MTdmMDQwICAgICAgICAgICAgIGlkbGUvMAowICAgICAgICAw
 IDMgICAwICAgICAgIDIwMCAgIGZmZmZmZmZmODE4OGEyODAgICAgICAgICAgICBzd2FwcGVyIHV2
 bQoKZGJ7MH0+IHNob3cgdXZtZXhwCkN1cnJlbnQgVVZNIHN0YXR1czoKICBwYWdlc2l6ZT00MDk2
 ICgweDEwMDApLCBwYWdlbWFzaz0weGZmZiwgcGFnZXNoaWZ0PTEyLCBuY29sb3JzPTE2CiAgMjQ3
 NTM2IFZNIHBhZ2VzOiA0Mjg0IGFjdGl2ZSwgNjc2IGluYWN0aXZlLCA1MTQ0IHdpcmVkLCAxIGZy
 ZWUKICBwYWdlcyAgNDM1OSBhbm9uLCAyODE2IGZpbGUsIDI5NDUgZXhlYwogIGZyZWVtaW49MjU2
 LCBmcmVlLXRhcmdldD0zNDEsIHdpcmVkLW1heD04MjUxMgogIHJlc3YtcGc9MSwgcmVzdi1rZXJu
 ZWw9NQogIGJvb3RwYWdlcz03NzM3LCBwb29scGFnZXM9MjMzOTM1CiAgZmF1bHRzPTEzNjA4MzEs
 IHRyYXBzPTEyOTc0NzQsIGludHJzPTE3MTYxMjM1LCBjdHhzd2l0Y2g9NDU0MzU2OQogICBzb2Z0
 aW50PTM3Mjg3MzcsIHN5c2NhbGxzPTYwNzg1NTI1CiAgZmF1bHQgY291bnRzOgogICAgbm9yYW09
 ODUsIG5vYW5vbj0wLCBwZ3dhaXQ9MTMsIHBncmVsZT0wCiAgICBvayByZWxvY2tzKHRvdGFsKT01
 Njg4KDU2ODkpLCBhbmdldChyZXRyeXMpPTE4NjQzMCgzMTg0KSwgYW1hcGNvcHk9MTY5NTcyCiAg
 ICBuZWlnaGJvciBhbm9uL29iaiBwZz05MjEyMi8yMjM2MDc4LCBnZXRzKGxvY2svdW5sb2NrKT03
 NTEwNzQvMjUwNgogICAgY2FzZXM6IGFub249MTI1OTI4LCBhbm9uY293PTYwNDk0LCBvYmo9NTc5
 NjcxLCBwcmNvcHk9MTcxMzkyLCBwcnplcm89MzkxODgzCiAgZGFlbW9uIGFuZCBzd2FwIGNvdW50
 czoKICAgIHdva2U9MTM3LCByZXZzPTEzNywgc2NhbnM9NTg4NzAsIG9ic2NhbnM9MjgxNDksIGFu
 c2NhbnM9NzAyMAogICAgYnVzeT0yNCwgZnJlZWQ9MzUxNTMsIHJlYWN0aXZhdGU9MTgxNywgZGVh
 Y3RpdmF0ZT02MDIwNAogICAgcGFnZW91dHM9NDU4LCBwZW5kaW5nPTY2MzIsIG5zd2dldD0zMTYx
 CiAgICBuc3dhcGRldj0xLCBzd3BnYXZhaWw9MTA0ODU3NQogICAgc3dwYWdlcz0xMDQ4NTc1LCBz
 d3BnaW51c2U9MzE1OSwgc3dwZ29ubHk9Mjg4NSwgcGFnaW5nPTE2CgojIGZpZ3VyaW5nIG91dCBk
 ZGIKZGJ7MH0+IHRyYWNlL3QgMAp0cmFjZTogcGlkIDAgbGlkIDE0NDg1IGF0IDB4ZmZmZmFkMDA3
 Mjg4MmFiMApzbGVlcHFfYmxvY2soKSBhdCBuZXRic2Q6c2xlZXBxX2Jsb2NrKzB4MTNhCm10c2xl
 ZXAoKSBhdCBuZXRic2Q6bXRzbGVlcCsweDE3Zgp1dm1fa21fa21lbV9hbGxvYygpIGF0IG5ldGJz
 ZDp1dm1fa21fa21lbV9hbGxvYysweDFhNwpwb29sX3BhZ2VfYWxsb2MoKSBhdCBuZXRic2Q6cG9v
 bF9wYWdlX2FsbG9jKzB4MmMKcG9vbF9ncm93KCkgYXQgbmV0YnNkOnBvb2xfZ3JvdysweDM0NApw
 b29sX2dldCgpIGF0IG5ldGJzZDpwb29sX2dldCsweGE4CnBvb2xfY2FjaGVfZ2V0X3Nsb3coKSBh
 dCBuZXRic2Q6cG9vbF9jYWNoZV9nZXRfc2xvdysweDEzOQpwb29sX2NhY2hlX2dldF9wYWRkcigp
 IGF0IG5ldGJzZDpwb29sX2NhY2hlX2dldF9wYWRkcisweDIzNAphcmNfZ2V0X2RhdGFfYnVmLmNv
 bnN0cHJvcC4wKCkgYXQgemZzOmFyY19nZXRfZGF0YV9idWYuY29uc3Rwcm9wLjArMHgxYmIKYXJj
 X2J1Zl9hbGxvY19pbXBsLmlzcmEuMCgpIGF0IHpmczphcmNfYnVmX2FsbG9jX2ltcGwuaXNyYS4w
 KzB4NzcKYXJjX3JlYWRfZG9uZSgpIGF0IHpmczphcmNfcmVhZF9kb25lKzB4ZTUKemlvX2RvbmUo
 KSBhdCB6ZnM6emlvX2RvbmUrMHgzOGIKemlvX2V4ZWN1dGUoKSBhdCB6ZnM6emlvX2V4ZWN1dGUr
 MHhlNgp0YXNrX2V4ZWN1dG9yKCkgYXQgc29sYXJpczp0YXNrX2V4ZWN1dG9yKzB4NjcKdGhyZWFk
 cG9vbF90aHJlYWQoKSBhdCBuZXRic2Q6dGhyZWFkcG9vbF90aHJlYWQrMHgxY2MKCiMgZmlndXJp
 bmcgb3V0IGRkYgpkYnswfT4gdHJhY2UvdC9sIDAKYnJlYWtwb2ludCgpIGF0IG5ldGJzZDpicmVh
 a3BvaW50KzB4NQp4ZW5jb25zX3R0eV9pbnB1dCgpIGF0IG5ldGJzZDp4ZW5jb25zX3R0eV9pbnB1
 dCsweGJjCnhlbmNvbnNfaGFuZGxlcigpIGF0IG5ldGJzZDp4ZW5jb25zX2hhbmRsZXIrMHg2YQp4
 ZW5faW50cl9iaWdsb2NrX3dyYXBwZXIoKSBhdCBuZXRic2Q6eGVuX2ludHJfYmlnbG9ja193cmFw
 cGVyKzB4MjEKZXZ0Y2huX2RvX2V2ZW50KCkgYXQgbmV0YnNkOmV2dGNobl9kb19ldmVudCsweDEx
 Ngpkb19oeXBlcnZpc29yX2NhbGxiYWNrKCkgYXQgbmV0YnNkOmRvX2h5cGVydmlzb3JfY2FsbGJh
 Y2srMHgxN2QKWGhhbmRsZV9oeXBlcnZpc29yX2NhbGxiYWNrKCkgYXQgbmV0YnNkOlhoYW5kbGVf
 aHlwZXJ2aXNvcl9jYWxsYmFjaysweDE5Ci0tLSBpbnRlcnJ1cHQgLS0tCng4Nl9zdGlobHQoKSBh
 dCBuZXRic2Q6eDg2X3N0aWhsdCsweDYKYWNwaWNwdV9jc3RhdGVfaWRsZSgpIGF0IG5ldGJzZDph
 Y3BpY3B1X2NzdGF0ZV9pZGxlKzB4MTliCmlkbGVfbG9vcCgpIGF0IG5ldGJzZDppZGxlX2xvb3Ar
 MHgxNGMKZGJ7MH0+IHRyYWNlL2EgZmZmZjhmMzA1YjRlNzc4MAp0cmFjZTogcGlkIDAgbGlkIDEx
 MiBhdCAweGZmZmZhZDAwNzI1OWJiMDAKc2xlZXBxX2Jsb2NrKCkgYXQgbmV0YnNkOnNsZWVwcV9i
 bG9jaysweDEzYQpjdl93YWl0KCkgYXQgbmV0YnNkOmN2X3dhaXQrMHg0OQp0eGdfd2FpdF9vcGVu
 KCkgYXQgemZzOnR4Z193YWl0X29wZW4rMHg2YwpkbXVfdHhfYXNzaWduKCkgYXQgemZzOmRtdV90
 eF9hc3NpZ24rMHgyYzQKenZvbF9zdHJhdGVneSgpIGF0IHpmczp6dm9sX3N0cmF0ZWd5KzB4MWE5
 CnNwZWNfc3RyYXRlZ3koKSBhdCBuZXRic2Q6c3BlY19zdHJhdGVneSsweDZlClZPUF9TVFJBVEVH
 WSgpIGF0IG5ldGJzZDpWT1BfU1RSQVRFR1krMHgzYwpiZGV2X3N0cmF0ZWd5KCkgYXQgbmV0YnNk
 OmJkZXZfc3RyYXRlZ3krMHhjOApzcGVjX3N0cmF0ZWd5KCkgYXQgbmV0YnNkOnNwZWNfc3RyYXRl
 Z3krMHg2ZQpWT1BfU1RSQVRFR1koKSBhdCBuZXRic2Q6Vk9QX1NUUkFURUdZKzB4M2MKdXZtX3N3
 YXBfaW8oKSBhdCBuZXRic2Q6dXZtX3N3YXBfaW8rMHgyMWYKc3dhcGNsdXN0ZXJfZmx1c2gucGFy
 dC4wKCkgYXQgbmV0YnNkOnN3YXBjbHVzdGVyX2ZsdXNoLnBhcnQuMCsweDg1CnV2bV9wYWdlb3V0
 KCkgYXQgbmV0YnNkOnV2bV9wYWdlb3V0KzB4NDc4CgojIHBnZGFlbW9uCmRiezB9PiB0cmFjZS9h
 IGZmZmY4ZjMwNWI0ZTc3ODAKdHJhY2U6IHBpZCAwIGxpZCAxMTIgYXQgMHhmZmZmYWQwMDcyNTli
 YjAwCnNsZWVwcV9ibG9jaygpIGF0IG5ldGJzZDpzbGVlcHFfYmxvY2srMHgxM2EKY3Zfd2FpdCgp
 IGF0IG5ldGJzZDpjdl93YWl0KzB4NDkKdHhnX3dhaXRfb3BlbigpIGF0IHpmczp0eGdfd2FpdF9v
 cGVuKzB4NmMKZG11X3R4X2Fzc2lnbigpIGF0IHpmczpkbXVfdHhfYXNzaWduKzB4MmM0Cnp2b2xf
 c3RyYXRlZ3koKSBhdCB6ZnM6enZvbF9zdHJhdGVneSsweDFhOQpzcGVjX3N0cmF0ZWd5KCkgYXQg
 bmV0YnNkOnNwZWNfc3RyYXRlZ3krMHg2ZQpWT1BfU1RSQVRFR1koKSBhdCBuZXRic2Q6Vk9QX1NU
 UkFURUdZKzB4M2MKYmRldl9zdHJhdGVneSgpIGF0IG5ldGJzZDpiZGV2X3N0cmF0ZWd5KzB4YzgK
 c3BlY19zdHJhdGVneSgpIGF0IG5ldGJzZDpzcGVjX3N0cmF0ZWd5KzB4NmUKVk9QX1NUUkFURUdZ
 KCkgYXQgbmV0YnNkOlZPUF9TVFJBVEVHWSsweDNjCnV2bV9zd2FwX2lvKCkgYXQgbmV0YnNkOnV2
 bV9zd2FwX2lvKzB4MjFmCnN3YXBjbHVzdGVyX2ZsdXNoLnBhcnQuMCgpIGF0IG5ldGJzZDpzd2Fw
 Y2x1c3Rlcl9mbHVzaC5wYXJ0LjArMHg4NQp1dm1fcGFnZW91dCgpIGF0IG5ldGJzZDp1dm1fcGFn
 ZW91dCsweDQ3OAoKIyBzb2Z0YmlvCmRiezB9PiB0cmFjZS9hIGZmZmY4ZjMwOTcxN2Y4YzAKdHJh
 Y2U6IHBpZCAwIGxpZCA0IGF0IDB4ZmZmZmFkMDA2MWZkNGY2MApYc29mdGludHIoKSBhdCBuZXRi
 c2Q6WHNvZnRpbnRyKzB4NGYKLS0tIGludGVycnVwdCAtLS0KMDoKCiMgdGhlIGZpbmQgZnJvbSBj
 cm9uIHRoYXQgcHJvYmFibHkgdHJpcHBlZCB0aGUgaGFuZwpkYnswfT4gdHJhY2UvYSBmZmZmOGYz
 MDVkMmNiYTgwCnRyYWNlOiBwaWQgMTM5MDMgbGlkIDEzOTAzIGF0IDB4ZmZmZmFkMDA3MmU1Yjdl
 MApzbGVlcHFfYmxvY2soKSBhdCBuZXRic2Q6c2xlZXBxX2Jsb2NrKzB4MTNhCmN2X3dhaXQoKSBh
 dCBuZXRic2Q6Y3Zfd2FpdCsweDQ5Cnppb193YWl0KCkgYXQgemZzOnppb193YWl0KzB4OTcKZG5v
 ZGVfaG9sZF9pbXBsKCkgYXQgemZzOmRub2RlX2hvbGRfaW1wbCsweGQwCmRtdV9ib251c19ob2xk
 KCkgYXQgemZzOmRtdV9ib251c19ob2xkKzB4MWUKemZzX2xvYWR2bm9kZSgpIGF0IHpmczp6ZnNf
 bG9hZHZub2RlKzB4NWYKdmNhY2hlX2dldCgpIGF0IG5ldGJzZDp2Y2FjaGVfZ2V0KzB4MWJlCnpm
 c196Z2V0KCkgYXQgemZzOnpmc196Z2V0KzB4MjUKemZzX2RpcmVudF9sb29rdXAoKSBhdCB6ZnM6
 emZzX2RpcmVudF9sb29rdXArMHhmMgp6ZnNfZGlybG9vaygpIGF0IHpmczp6ZnNfZGlybG9vaysw
 eDVmCnpmc19uZXRic2RfbG9va3VwKCkgYXQgemZzOnpmc19uZXRic2RfbG9va3VwKzB4MmUyClZP
 UF9MT09LVVAoKSBhdCBuZXRic2Q6Vk9QX0xPT0tVUCsweDQ0Cmxvb2t1cF9vbmNlKCkgYXQgbmV0
 YnNkOmxvb2t1cF9vbmNlKzB4MWE2Cm5hbWVpX3RyeWVtdWxyb290KCkgYXQgbmV0YnNkOm5hbWVp
 X3RyeWVtdWxyb290KzB4YjAwCm5hbWVpKCkgYXQgbmV0YnNkOm5hbWVpKzB4MjkKZG9fc3lzX3N0
 YXRhdCgpIGF0IG5ldGJzZDpkb19zeXNfc3RhdGF0KzB4MWRiCnN5c19fX2xzdGF0NTAoKSBhdCBu
 ZXRic2Q6c3lzX19fbHN0YXQ1MCsweDI1CnN5c2NhbGwoKSBhdCBuZXRic2Q6c3lzY2FsbCsweDE5
 NgotLS0gc3lzY2FsbCAobnVtYmVyIDQ0MSkgLS0tCm5ldGJzZDpzeXNjYWxsKzB4MTk2OgoKIyBz
 b21ldGhpbmcgemZzCmRiezB9PiB0cmFjZS9hIGZmZmY4ZjMwNWUwMjM5NDAKdHJhY2U6IHBpZCAw
 IGxpZCAzOTAgYXQgMHhmZmZmYWQwMDcyOTBlZDMwCnNsZWVwcV9ibG9jaygpIGF0IG5ldGJzZDpz
 bGVlcHFfYmxvY2srMHgxM2EKY3Zfd2FpdCgpIGF0IG5ldGJzZDpjdl93YWl0KzB4NDkKemlvX3dh
 aXQoKSBhdCB6ZnM6emlvX3dhaXQrMHg5Nwpkc2xfcG9vbF9zeW5jKCkgYXQgemZzOmRzbF9wb29s
 X3N5bmMrMHgyMmEKc3BhX3N5bmMoKSBhdCB6ZnM6c3BhX3N5bmMrMHgzY2IKdHhnX3N5bmNfdGhy
 ZWFkKCkgYXQgemZzOnR4Z19zeW5jX3RocmVhZCsweDI5YwoKIyBzb21ldGhpbmcgemZzCmRiezB9
 PiB0cmFjZS9hIGZmZmY4ZjMwNWUwMjM1MDAKdHJhY2U6IHBpZCAwIGxpZCAzODkgYXQgMHhmZmZm
 YWQwMDcyOTA3ZjIwCnNsZWVwcV9ibG9jaygpIGF0IG5ldGJzZDpzbGVlcHFfYmxvY2srMHgxM2EK
 Y3Zfd2FpdCgpIGF0IG5ldGJzZDpjdl93YWl0KzB4NDkKdHhnX3F1aWVzY2VfdGhyZWFkKCkgYXQg
 emZzOnR4Z19xdWllc2NlX3RocmVhZCsweGZjCgojIHNvbWV0aGluZyB6ZnMKZGJ7MH0+IHRyYWNl
 L2EgZmZmZjhmMzA1Y2M4ODhjMAp0cmFjZTogcGlkIDAgbGlkIDE2MiBhdCAweGZmZmZhZDAwNzI3
 ZDNjMTAKc2xlZXBxX2Jsb2NrKCkgYXQgbmV0YnNkOnNsZWVwcV9ibG9jaysweDEzYQptdHNsZWVw
 KCkgYXQgbmV0YnNkOm10c2xlZXArMHgxN2YKdXZtX2ttX2ttZW1fYWxsb2MoKSBhdCBuZXRic2Q6
 dXZtX2ttX2ttZW1fYWxsb2MrMHgyMzEKa21lbV9pbnRyX2FsbG9jKCkgYXQgbmV0YnNkOmttZW1f
 aW50cl9hbGxvYysweGQ0CmttZW1femFsbG9jKCkgYXQgbmV0YnNkOmttZW1femFsbG9jKzB4NGUK
 ZGR0X2dldF9kZWR1cF9kc3BhY2UoKSBhdCB6ZnM6ZGR0X2dldF9kZWR1cF9kc3BhY2UrMHgxZQpz
 cGFfdXBkYXRlX2RzcGFjZSgpIGF0IHpmczpzcGFfdXBkYXRlX2RzcGFjZSsweDIxCnNwYV9zeW5j
 KCkgYXQgemZzOnNwYV9zeW5jKzB4NWFlCnR4Z19zeW5jX3RocmVhZCgpIGF0IHpmczp0eGdfc3lu
 Y190aHJlYWQrMHgyOWMKCiMgc29tZXRoaW5nIHpmcwpkYnswfT4gdHJhY2UvYSBmZmZmOGYzMDVj
 Yzg4NDgwCnRyYWNlOiBwaWQgMCBsaWQgMTYxIGF0IDB4ZmZmZmFkMDA3MjdjY2YyMApzbGVlcHFf
 YmxvY2soKSBhdCBuZXRic2Q6c2xlZXBxX2Jsb2NrKzB4MTNhCmN2X3dhaXQoKSBhdCBuZXRic2Q6
 Y3Zfd2FpdCsweDQ5CnR4Z19xdWllc2NlX3RocmVhZCgpIGF0IHpmczp0eGdfcXVpZXNjZV90aHJl
 YWQrMHhmYwoKIyBzb21ldGhpbmcgemZzCmRiezB9PiB0cmFjZS9hIGZmZmY4ZjMwNWMyOWU5ODAK
 dHJhY2U6IHBpZCAwIGxpZCAxNTAgYXQgMHhmZmZmYWQwMDcyNmQ5ZTcwCnNsZWVwcV9ibG9jaygp
 IGF0IG5ldGJzZDpzbGVlcHFfYmxvY2srMHhkNwpjdl90aW1lZHdhaXQoKSBhdCBuZXRic2Q6Y3Zf
 dGltZWR3YWl0KzB4NGIKbDJhcmNfZmVlZF90aHJlYWQoKSBhdCB6ZnM6bDJhcmNfZmVlZF90aHJl
 YWQrMHhiN2IKCiMgc29tZXRoaW5nIHpmcwpkYnswfT4gdHJhY2UvYSBmZmZmOGYzMDVjMjllNTQw
 CnRyYWNlOiBwaWQgMCBsaWQgMjI0IGF0IDB4ZmZmZmFkMDA3MjZkMmY2MApzbGVlcHFfYmxvY2so
 KSBhdCBuZXRic2Q6c2xlZXBxX2Jsb2NrKzB4ZDcKY3ZfdGltZWR3YWl0KCkgYXQgbmV0YnNkOmN2
 X3RpbWVkd2FpdCsweDRiCmRidWZfZXZpY3RfdGhyZWFkKCkgYXQgemZzOmRidWZfZXZpY3RfdGhy
 ZWFkKzB4MTQ2CgojIHNvbWV0aGluZyB6ZnMKZGJ7MH0+IHRyYWNlL2EgZmZmZjhmMzA1Yjk1ODk0
 MAp0cmFjZTogcGlkIDAgbGlkIDk1IGF0IDB4ZmZmZmFkMDA3MjZjNGY0MApzbGVlcHFfYmxvY2so
 KSBhdCBuZXRic2Q6c2xlZXBxX2Jsb2NrKzB4ZDcKY3ZfdGltZWR3YWl0KCkgYXQgbmV0YnNkOmN2
 X3RpbWVkd2FpdCsweDRiCmFyY19yZWNsYWltX3RocmVhZCgpIGF0IHpmczphcmNfcmVjbGFpbV90
 aHJlYWQrMHgyNzMKCiMgc29tZXRoaW5nIHpmcwpkYnswfT4gdHJhY2UvYSBmZmZmOGYzMDViNGU3
 YmMwCnRyYWNlOiBwaWQgMCBsaWQgMTEzIGF0IDB4ZmZmZmFkMDA3MjVhMmUzMApzbGVlcHFfYmxv
 Y2soKSBhdCBuZXRic2Q6c2xlZXBxX2Jsb2NrKzB4MTNhCmN2X3dhaXQoKSBhdCBuZXRic2Q6Y3Zf
 d2FpdCsweDQ5CnR4Z193YWl0X29wZW4oKSBhdCB6ZnM6dHhnX3dhaXRfb3BlbisweDZjCmRtdV90
 eF9hc3NpZ24oKSBhdCB6ZnM6ZG11X3R4X2Fzc2lnbisweDJjNAp6ZnNfbmV0YnNkX3N5bmMoKSBh
 dCB6ZnM6emZzX25ldGJzZF9zeW5jKzB4ZDIKVkZTX1NZTkMoKSBhdCBuZXRic2Q6VkZTX1NZTkMr
 MHgyMgpzY2hlZF9zeW5jKCkgYXQgbmV0YnNkOnNjaGVkX3N5bmMrMHg5MApkYnswfT4KCgpTY3Jp
 cHQgZG9uZSBvbiBTYXQgSnVsIDMwIDA2OjQ5OjA3IDIwMjIK
 --=-=-=--
Prev by Date: Re: port-evbarm/56944: ZFS heavy usage on NetBSD running in Mac M1 VM results in kernel thread running away and filesystem hang
Next by Date: Re: port-evbarm/56944: ZFS heavy usage on NetBSD running in Mac M1 VM results in kernel thread running away and filesystem hang
Previous by Thread: Re: port-evbarm/56944: ZFS heavy usage on NetBSD running in Mac M1 VM results in kernel thread running away and filesystem hang
Next by Thread: Re: port-evbarm/56944: ZFS heavy usage on NetBSD running in Mac M1 VM results in kernel thread running away and filesystem hang
Indexes:
Home | Main Index | Thread Index | Old Index