Re: Documentation for dtrace?

[Darran Hunt <darran%NetBSD.org@localhost>]

Hi David,

On 10/09/2011, at 10:38 AM, David Young wrote:

> On Fri, Sep 09, 2011 at 12:03:38PM -0400, Thor Lancelot Simon wrote:
>> On Fri, Sep 09, 2011 at 08:51:23AM -0700, Paul Goyette wrote:
>>> 
>>> 1. Is there any info on performance impact for building the kernel with
>>>   "options KDTRACE_HOOKS" ?
>> 
>> In -- minimal -- testing, I was not able to observe a perfomance impact.
>> 
>>> 2. The wiki article says that DTRACE is only for i386 - does this mean
>>>   that it doesn't work on amd64?
>> 
>> That's correct.  This work was done at Coyote Point, and at that time, we
>> had no immediate need for amd64 support, and many other high priority
>> NetBSD tasks in the pipeline.  However, Darran and I did discuss, at the
>> time, what would be needed to complete the amd64 specific pieces of the
>> puzzle; unfortunately, I do not remember the answers and do not have
>> access to my project planning materials from CP.
>> 
>> However, I am nudging Darran by carbon, and perhaps he'll respond to the
>> list.
> 
> Maybe Darran or somebody can explain how the SDT probes work in
> NetBSD?  
> 
> I'd like to put an SDT probe into assembly language.  It looks to
> me like it works differently in NetBSD than in Solaris.  In Solaris,
> the kernel linker ensures that SDT probes are NOPs until activated,
> when they are changed to an illegal instruction (lock nop) that
> causes a #UD trap
> (http://www.cs.dartmouth.edu/~sergey/cs108/dtrace-internals-x86.pdf).
> NetBSD appears to provide SDT probes more simply but possibly with
> higher overhead:
> 
> #define SDT_PROBE(prov, mod, func, name, arg0, arg1, arg2, arg3, arg4) \
>        if (__predict_false(SDT_NAME(prov, mod, func, name).enabled)) { \
>                (*sdt_probe_func)(SDT_NAME(prov, mod, func, name).id, \
>                    (uintptr_t)(arg0), (uintptr_t)(arg1), (uintptr_t)(arg2), \
>                    (uintptr_t)(arg3), (uintptr_t)(arg4)); \
>        }

the NetBSD Dtrace port is a port of the FreeBSD Dtrace port, and yes the port 
has taken a short-cut with the SDT provider hooks.  As you can see from the 
SDT_PROBE() macro, there's an if test added for each probe which does a look-up 
in a table to see if the SDT probe is enabled.  I don't see why we couldn't 
replace the SDT implementation with one that more closes mimics the Solaris 
approach.  It should be noted that the Solaris implementation is not zero 
overhead either because although the probe function call is replaced by NOPs by 
the linker, I believe that the function call stack setup and tear down code is 
still present.

The NetBSD FBT (Functional Boundary Trace) provider is the real deal with zero 
impact probes that are directly patched into the kernel at run-time when the 
probe is enabled.  When the probe is disabled there is no code present for the 
probe.

Some additional details on DTrace probe overheads in Solaris here: 
http://www.solarisinternals.com/wiki/index.php/DTrace_Topics_Overhead.

I don't think the amd64 work is too involved, the code is reasonably structured 
with the architecture specific stuff self contained.  It would require some low 
level amd64 expertise or a bit of research into the amd64 equivalents for the 
x86 specific code.

Darran.