Re: dtrace and ZFS update

[bch <brad.harder%gmail.com@localhost>]

On 10/13/17, Chuck Silvers <chuq%chuq.com@localhost> wrote:
> hi folks,
>
> I've been working on updating netbsd's copy of the dtrace and zfs code to
> rebase
> from the existing ancient opensolaris version to a recent freebsd version.
> most of the freebsd changes are pretty close to what netbsd needs,
> so that seems like a more useful upstream for us.  I have things working
> well enough now that I want to share the code in preparation for
> committing.
>



Anybody else having userland build failures ?

# ./build.sh -x distribution)

[...]

(cd /usr/src/external/bsd/iscsi/lib && rm -f libiscsi.a   conffile.o
disk.o util.o parameters.o protocol.o storage.o netmask.o md5c.o
md5hl.o uuid.o initiator.o target.o conffile.o.tmp disk.o.tmp
util.o.tmp parameters.o.tmp protocol.o.tmp storage.o.tmp netmask.o.tmp
md5c.o.tmp md5hl.o.tmp uuid.o.tmp initiator.o.tmp target.o.tmp ||
true)
(cd /usr/src/external/bsd/iscsi/lib/obj && rm -f libiscsi_p.a
conffile.po disk.po util.po parameters.po protocol.po storage.po
netmask.po md5c.po md5hl.po uuid.po initiator.po target.po
conffile.po.tmp disk.po.tmp util.po.tmp parameters.po.tmp
protocol.po.tmp storage.po.tmp netmask.po.tmp md5c.po.tmp md5hl.po.tmp
uuid.po.tmp initiator.po.tmp target.po.tmp || true)
(cd /usr/src/external/bsd/iscsi/lib && rm -f libiscsi_p.a conffile.po
disk.po util.po parameters.po protocol.po storage.po netmask.po
md5c.po md5hl.po uuid.po initiator.po target.po  conffile.po.tmp
disk.po.tmp util.po.tmp parameters.po.tmp protocol.po.tmp
storage.po.tmp netmask.po.tmp md5c.po.tmp md5hl.po.tmp uuid.po.tmp
initiator.po.tmp target.po.tmp || true)
(cd /usr/src/external/bsd/iscsi/lib/obj && rm -f libiscsi_g.a    || true)
(cd /usr/src/external/bsd/iscsi/lib && rm -f libiscsi_g.a    || true)
(cd /usr/src/external/bsd/iscsi/lib/obj && rm -f libiscsi_pic.a
libiscsi.so.* libiscsi.so  conffile.pico disk.pico util.pico
parameters.pico protocol.pico storage.pico netmask.pico md5c.pico
md5hl.pico uuid.pico initiator.pico target.pico conffile.pico.tmp
disk.pico.tmp util.pico.tmp parameters.pico.tmp protocol.pico.tmp
storage.pico.tmp netmask.pico.tmp md5c.pico.tmp md5hl.pico.tmp
uuid.pico.tmp initiator.pico.tmp target.pico.tmp || true)
(cd /usr/src/external/bsd/iscsi/lib && rm -f libiscsi_pic.a
libiscsi.so.* libiscsi.so  conffile.pico disk.pico util.pico
parameters.pico protocol.pico storage.pico netmask.pico md5c.pico
md5hl.pico uuid.pico initiator.pico target.pico conffile.pico.tmp
disk.pico.tmp util.pico.tmp parameters.pico.tmp protocol.pico.tmp
storage.pico.tmp netmask.pico.tmp md5c.pico.tmp md5hl.pico.tmp
uuid.pico.tmp initiator.pico.tmp target.pico.tmp || true)
(cd /usr/src/external/bsd/iscsi/lib/obj && rm -f llib-liscsi.ln
conffile.ln disk.ln util.ln parameters.ln protocol.ln storage.ln
netmask.ln md5c.ln md5hl.ln uuid.ln initiator.ln target.ln || true)
(cd /usr/src/external/bsd/iscsi/lib && rm -f llib-liscsi.ln
conffile.ln disk.ln util.ln parameters.ln protocol.ln storage.ln
netmask.ln md5c.ln md5hl.ln uuid.ln initiator.ln target.ln || true)
(cd /usr/src/external/bsd/iscsi/lib/obj && rm -f libiscsi.so.2.map
a.out [Ee]rrs mklog core *.core || true)
(cd /usr/src/external/bsd/iscsi/lib && rm -f libiscsi.so.2.map   a.out
[Ee]rrs mklog core *.core || true)
(cd /usr/src/external/bsd/iscsi/lib/obj && rm -f libiscsi.html3
.depend conffile.d disk.d initiator.d md5c.d md5hl.d netmask.d
parameters.d protocol.d storage.d target.d util.d uuid.d
conffile.d.tmp disk.d.tmp initiator.d.tmp md5c.d.tmp md5hl.d.tmp
netmask.d.tmp parameters.d.tmp protocol.d.tmp storage.d.tmp
target.d.tmp util.d.tmp uuid.d.tmp
/usr/src/external/bsd/iscsi/lib/tags  || true)
(cd /usr/src/external/bsd/iscsi/lib && rm -f libiscsi.html3 .depend
conffile.d disk.d initiator.d md5c.d md5hl.d netmask.d parameters.d
protocol.d storage.d target.d util.d uuid.d conffile.d.tmp disk.d.tmp
initiator.d.tmp md5c.d.tmp md5hl.d.tmp netmask.d.tmp parameters.d.tmp
protocol.d.tmp storage.d.tmp target.d.tmp util.d.tmp uuid.d.tmp
/usr/src/external/bsd/iscsi/lib/tags  || true)
cleandir ===> lib/../external/cddl/osnet/lib/libzfs_core
cd: can't cd to /usr/src/lib/../external/cddl/osnet/lib/libzfs_core

*** Failed target:  cleandir-../external/cddl/osnet/lib/libzfs_core
*** Failed command: _makedirtarget() { dir="$1"; shift; target="$1";
shift; case "${dir}" in /*) this="${dir}/"; real="${dir}" ;; .)
this="lib/"; real="/usr/src/lib" ;; *) this="lib/${dir}/";
real="/usr/src/lib/${dir}" ;; esac; show=${this:-.}; echo "${target}
===> ${show%/}${1:+ (with: $@)}"; cd "${real}" &&
/usr/src/obj/tooldir.NetBSD-8.99.4-amd64/bin/nbmake
_THISDIR_="${this}" "$@" ${target}; }; _makedirtarget
../external/cddl/osnet/lib/libzfs_core cleandir
*** Error code 2

Stop.
nbmake[3]: stopped in /usr/src/lib

*** Failed target:  cleandir-lib
*** Failed command: _makedirtarget() { dir="$1"; shift; target="$1";
shift; case "${dir}" in /*) this="${dir}/"; real="${dir}" ;; .)
this=""; real="/usr/src" ;; *) this="${dir}/"; real="/usr/src/${dir}"
;; esac; show=${this:-.}; echo "${target} ===> ${show%/}${1:+ (with:
$@)}"; cd "${real}" &&
/usr/src/obj/tooldir.NetBSD-8.99.4-amd64/bin/nbmake
_THISDIR_="${this}" "$@" ${target}; }; _makedirtarget lib cleandir
*** Error code 1

Stop.
nbmake[2]: stopped in /usr/src

*** Failed target:  build
*** Failed command: _makedirtarget() { dir="$1"; shift; target="$1";
shift; case "${dir}" in /*) this="${dir}/"; real="${dir}" ;; .)
this=""; real="/usr/src" ;; *) this="${dir}/"; real="/usr/src/${dir}"
;; esac; show=${this:-.}; echo "${target} ===> ${show%/}${1:+ (with:
$@)}"; cd "${real}" &&
/usr/src/obj/tooldir.NetBSD-8.99.4-amd64/bin/nbmake
_THISDIR_="${this}" "$@" ${target}; }; _makedirtarget . cleandir
*** Error code 1

Stop.
nbmake[1]: stopped in /usr/src

*** Failed target:  distribution
*** Failed command: _makedirtarget() { dir="$1"; shift; target="$1";
shift; case "${dir}" in /*) this="${dir}/"; real="${dir}" ;; .)
this=""; real="/usr/src" ;; *) this="${dir}/"; real="/usr/src/${dir}"
;; esac; show=${this:-.}; echo "${target} ===> ${show%/}${1:+ (with:
$@)}"; cd "${real}" &&
/usr/src/obj/tooldir.NetBSD-8.99.4-amd64/bin/nbmake
_THISDIR_="${this}" "$@" ${target}; }; _makedirtarget . build
NOPOSTINSTALL=1
*** Error code 1

Stop.
nbmake: stopped in /usr/src

ERROR: Failed to make distribution
*** BUILD ABORTED ***





> this update improves upon our existing dtrace/zfs code in several ways:
>  - picks up all the upstream zfs fixes and enhancements from the last
> decade
>  - zfs now supports mmap on netbsd, so you can run executables stored in
> zfs
>  - dtrace fbt probes can now be used in kernel modules (such as zfs)
>
> here's the diff, which is based on netbsd HEAD as of
> "Tue Oct 10 20:14:48 UTC 2017":
> http://ftp.netbsd.org/pub/NetBSD/misc/chs/diff.cddl.20171012
>
> note that the patch needs to be applied with "patch -E" since it
> adds and removes various files.
>
> here's a summary of what's in the diff:
>
>  - freebsd's dtrace/zfs code as of r315983 (2017-03-26), adapted for
> netbsd.
>    many netbsd changes in our current version to fix warnings have been
> removed
>    or redone to be reduce the difference from the new upstream.
>  - a few updates to our copy of freebsd's libproc.
>    I plan to do a wholesale resync of libproc to current freebsd
>    rather than just picking up the few bits in this diff.
>  - build system support for using -fno-omit-frame-pointer everywhere
>    and disabling other compiler optimizations that confuse dtrace.
>    I'm ok with leaving these changes disabled by default,
>    or perhaps making them conditional on MKDTRACE.
>  - sample kernel config changes for a couple evbarm configs (the ones I
> tested).
>  - module/ksyms enhancements for dtrace integration.
>  - genfs API enhancements to support zfs.
>  - an option to have mutexes not become no-ops during a panic.
>  - uvm_aobj API change to support 64-bit aobj sizes (eg. for tmpfs).
>
> the plan is that items listed separately above will be committed separately
> but in quick succession, to allow one kernel version increment for all of
> the ones which require that.
>
>
> here's a list of known issues with the patch:
>
>  - unloading the zfs module fails even with no zpools imported
>    if you've done much with zfs since it was loaded.
>    there's some refcounting problem that I haven't tracked down yet.
>
>  - the module refcounting for active fbt probes is bogus.
>    currently module refcounting is protected by kernconfig_lock(),
>    but taking that lock down in the bowels of dtrace seems likely
>    to create deadlocks.  I plan to do something fancier but haven't
>    gotten to it yet.
>
>  - the dtrace uregs[] stuff is probably still wrong.
>
>  - the CTF typeid overflow problem is still there (more on this below).
>
>  - a bunch of features are still not supported:
>    - the ".zfs" virtual directory, eg. ".zfs/snapshot/foo@bar"
>    - zvols
>    - ZFS ACLs (aka. NFSv4 ACLs)
>    - NFS exporting a ZFS file system
>    - setting dtrace probes in application code
>    - using ZFS as the root fs
>    - new crypto hashes SHA512_256, skein, and edonr
>      (the last one is not in freebsd yet either)
>    - zio delay injection (used for testing zfs)
>    - dtrace support for platforms other than x86 and arm
>
>
> I don't intend to fix any of these issues before initial commit of
> these changes, since I don't think any of them are regressions from
> what we have currently.
>
> there a few other bits still in the diff that are just for debugging or
> otherwise need to be cleaned up, but not very many.  I'll clean them up
> before committing.
>
>
> about the CTF typeid overflow issue I mentioned above:
>
> CTF is the Compact Type Format data that dtrace uses to know about
> function argument types and return values.  CTF encodes a type ID
> as a 15-bit value, but unfortunately a netbsd GENERIC kernel has
> more than 15 bits worth of types, so the values overflow the field.
> we don't really want to change this format unilaterally since it is
> currently the same on all dtrace-supporting operating systems, so we
> need to work around the problem for now by removing various subsystems
> from the kernel config used with dtrace, so that the number of types
> is within what CTF supports.
>
> to tell if your kernel will work with dtrace, run
> 	ctdump /path/to/kernel | grep "total number of types"
>
> if the number of types is less than 32768 then you're fine,
> otherwise you'll need to change your kernel config to remove
> some drivers or file systems that you won't use.
>
> this error is another symptom of the typeid overflow problem:
> dtrace: failed to compile script sleep.d: "/usr/lib/dtrace/psinfo.d", line
> 46: failed to copy type of 'pr_arglen': Type information is in parent and
> unavailable
>
> note that dtrace scripts that do not reference any argument or return types
> will work even if the kernel has too many types for CTF (but you'll need to
> remove the psinfo.d file mentioned above to remove that reference).
>
> I hope to integrate a solution for this CTF overflow issue before
> the first netbsd release with this new dtrace code in it (netbsd 9.0),
> but it's not clear what that solution will end up being yet.
>
>
> anyone interested in zfs or dtrace on netbsd please give this a try
> and let me know about any problems that you find.
> any comments on the diff are also welcome.
>
> -Chuck
>