Re: dtrace and ZFS update

[Chuck Silvers <chuq%chuq.com@localhost>]

hi folks,

as taylor requested after the previous mail about this,
I've made a copy of the freebsd files that I started with
and put them in the netbsd source tree layout so that they
can easily be diffed against the merged tree.
I've put several files in
http://ftp.netbsd.org/pub/NetBSD/misc/chs/cddl/20171205/ :

f-n-remap.pl
	the perl script to rearrange the cddl files from a
	freebsd source tree into the netbsd source layout
	(mostly directory moves, a few file renames)

fbsd-r315983.tar.gz
	the freebsd sources that I started with

fbsd-r315983-remapped.tar.gz
	those freebsd sources as rearranged by the script

diff.cddl-fbsd-nbsd
	diff of fbsd-r315983-remapped.tar.gz vs. the merged tree

diff.cddl.20171205
	cvs diff of netbsd -current vs. the merged tree


this version also includes some cleanup of the existing files,
mainly removing about 30 mostly useless glue header files
(which I found in the process of getting the remap script working).
I haven't conditionalized DTRACE_OPTS yet, but otherwise it should be
very close to what I'll commit.

please take a look and let me know if there are any other changes
that are desired before I commit it.

-Chuck


On Fri, Oct 13, 2017 at 09:56:01AM -0700, Chuck Silvers wrote:
> hi folks,
> 
> I've been working on updating netbsd's copy of the dtrace and zfs code to rebase
> from the existing ancient opensolaris version to a recent freebsd version.
> most of the freebsd changes are pretty close to what netbsd needs,
> so that seems like a more useful upstream for us.  I have things working
> well enough now that I want to share the code in preparation for committing.
> 
> this update improves upon our existing dtrace/zfs code in several ways:
>  - picks up all the upstream zfs fixes and enhancements from the last decade
>  - zfs now supports mmap on netbsd, so you can run executables stored in zfs
>  - dtrace fbt probes can now be used in kernel modules (such as zfs)
> 
> here's the diff, which is based on netbsd HEAD as of
> "Tue Oct 10 20:14:48 UTC 2017":
> http://ftp.netbsd.org/pub/NetBSD/misc/chs/diff.cddl.20171012
> 
> note that the patch needs to be applied with "patch -E" since it
> adds and removes various files.
> 
> here's a summary of what's in the diff:
> 
>  - freebsd's dtrace/zfs code as of r315983 (2017-03-26), adapted for netbsd.
>    many netbsd changes in our current version to fix warnings have been removed
>    or redone to be reduce the difference from the new upstream.
>  - a few updates to our copy of freebsd's libproc.
>    I plan to do a wholesale resync of libproc to current freebsd
>    rather than just picking up the few bits in this diff.
>  - build system support for using -fno-omit-frame-pointer everywhere
>    and disabling other compiler optimizations that confuse dtrace.
>    I'm ok with leaving these changes disabled by default,
>    or perhaps making them conditional on MKDTRACE.
>  - sample kernel config changes for a couple evbarm configs (the ones I tested).
>  - module/ksyms enhancements for dtrace integration.
>  - genfs API enhancements to support zfs.
>  - an option to have mutexes not become no-ops during a panic.
>  - uvm_aobj API change to support 64-bit aobj sizes (eg. for tmpfs).
> 
> the plan is that items listed separately above will be committed separately
> but in quick succession, to allow one kernel version increment for all of
> the ones which require that.
> 
> 
> here's a list of known issues with the patch:
> 
>  - unloading the zfs module fails even with no zpools imported
>    if you've done much with zfs since it was loaded.
>    there's some refcounting problem that I haven't tracked down yet.
> 
>  - the module refcounting for active fbt probes is bogus.
>    currently module refcounting is protected by kernconfig_lock(),
>    but taking that lock down in the bowels of dtrace seems likely
>    to create deadlocks.  I plan to do something fancier but haven't
>    gotten to it yet.
> 
>  - the dtrace uregs[] stuff is probably still wrong.
> 
>  - the CTF typeid overflow problem is still there (more on this below).
> 
>  - a bunch of features are still not supported:
>    - the ".zfs" virtual directory, eg. ".zfs/snapshot/foo@bar"
>    - zvols
>    - ZFS ACLs (aka. NFSv4 ACLs)
>    - NFS exporting a ZFS file system
>    - setting dtrace probes in application code
>    - using ZFS as the root fs
>    - new crypto hashes SHA512_256, skein, and edonr
>      (the last one is not in freebsd yet either)
>    - zio delay injection (used for testing zfs)
>    - dtrace support for platforms other than x86 and arm
> 
> 
> I don't intend to fix any of these issues before initial commit of
> these changes, since I don't think any of them are regressions from
> what we have currently.
> 
> there a few other bits still in the diff that are just for debugging or
> otherwise need to be cleaned up, but not very many.  I'll clean them up
> before committing.
> 
> 
> about the CTF typeid overflow issue I mentioned above:
> 
> CTF is the Compact Type Format data that dtrace uses to know about
> function argument types and return values.  CTF encodes a type ID
> as a 15-bit value, but unfortunately a netbsd GENERIC kernel has
> more than 15 bits worth of types, so the values overflow the field.
> we don't really want to change this format unilaterally since it is
> currently the same on all dtrace-supporting operating systems, so we
> need to work around the problem for now by removing various subsystems
> from the kernel config used with dtrace, so that the number of types
> is within what CTF supports.
> 
> to tell if your kernel will work with dtrace, run
> 	ctdump /path/to/kernel | grep "total number of types"
> 
> if the number of types is less than 32768 then you're fine,
> otherwise you'll need to change your kernel config to remove
> some drivers or file systems that you won't use.
> 
> this error is another symptom of the typeid overflow problem:
> dtrace: failed to compile script sleep.d: "/usr/lib/dtrace/psinfo.d", line 46: failed to copy type of 'pr_arglen': Type information is in parent and unavailable
> 
> note that dtrace scripts that do not reference any argument or return types
> will work even if the kernel has too many types for CTF (but you'll need to
> remove the psinfo.d file mentioned above to remove that reference).
> 
> I hope to integrate a solution for this CTF overflow issue before
> the first netbsd release with this new dtrace code in it (netbsd 9.0),
> but it's not clear what that solution will end up being yet.
> 
> 
> anyone interested in zfs or dtrace on netbsd please give this a try
> and let me know about any problems that you find.
> any comments on the diff are also welcome.
> 
> -Chuck