RE: Performance degradation over time on VAX...

To: Ken Wellsch <kcwellsch%outlook.com@localhost>, port-vax List <port-vax%netbsd.org@localhost>
Subject: RE: Performance degradation over time on VAX...
From: Josh Moyer <JMoyer%nodomain.net@localhost>
Date: Thu, 9 Jul 2026 18:12:36 +0000
Ken, that's thorough work -- I'm sure everybody appreciates it!

All, I want to contribute to this effort, though my time is currently limited.  I am hopeful, also, for the implications to other less cycle rich architectures, such as 68k.  To those ends, I just had the following dialog with Claude Sonnet 5.  It covers tracing and performance profiling tools, with a table of support by NetBSD version.  I learned a lot from reading it, like config -p, kgmon(8) and option GRPOF and different vmstat(8) options, also.  Mostly it seems to come from the man pages and I didn't see any hallucinations (I double-checked only some things.)


Me:
What kind of kernel or userland performance and api call profiling tools exist for UNIX (NetBSD)?

Claude:
Here's the landscape for NetBSD specifically, organized by what you're trying to observe:

**Syscall/API-level tracing**
- **`ktrace(1)`/`kdump(1)`** — the core BSD facility. Traces syscalls, namei (pathname) lookups, signal delivery, and I/O, with per-process or per-descendant scope. This is NetBSD's answer to Linux's `strace` (which isn't native to BSD).
- **`ptrace(2)`** — the underlying primitive `ktrace` and debuggers build on, if you want to write your own tracer.
- **`systrace`** — existed historically for syscall policy enforcement/tracing (shared BSD heritage with OpenBSD), though it's seen less maintenance in recent NetBSD and is largely superseded by DTrace for tracing use cases.

**Userland CPU/call-graph profiling**
- **`gprof(1)`** — classic call-graph profiler; compile with `-pg`, run the binary, it drops `gmon.out`, then `gprof` renders call graphs and time attribution.
- **`prof(1)`** — flat-profile counterpart, older and cruder than gprof.

**Kernel profiling**
- **`kgmon(8)` + kernel `GPROF` config option** — build a kernel with profiling support enabled, then `kgmon` collects and dumps kernel-level call-graph profile data, viewable with `gprof`. This is the direct kernel analog to userland gprof profiling.
- **`DDB`** — the in-kernel debugger; not a profiler per se, but useful for point-in-time inspection of kernel state, stack traces, and lock status during a hang.
- **`kgdb`** — for postmortem analysis of kernel crash dumps.

**Dynamic tracing**
- **DTrace** — NetBSD has a port of DTrace (from the OpenSolaris/CDDL lineage, same family as illumos and FreeBSD's DTrace). Gives you `fbt` (function boundary tracing), `syscall`, `profile`, and `sdt` providers, letting you write D scripts to instrument arbitrary kernel and userland functions without recompiling — the closest thing NetBSD has to a comprehensive, low-overhead profiling framework. Worth checking current support level/build options for your specific NetBSD version since DTrace integration has evolved across releases.

**System-wide statistics/monitoring**
- **`vmstat(8)`** — beyond the basic VM/CPU view, `vmstat -e` gives event counters, `vmstat -i` interrupt counts, `vmstat -m` memory pool usage, and `vmstat -l` lock/mutex contention stats (useful for contention profiling if `LOCKDEBUG` or similar is enabled).
- **`iostat(8)`**, **`netstat(1)`** (especially `-s` for protocol stats), **`systat(1)`** — the traditional BSD monitoring toolkit for disk, network, and general activity.
- **`top(1)`/`ps(1)`** — coarse but useful for per-process CPU/memory snapshotting.
- **`sysctl(8)`** — direct read access to a huge range of `kern.*` and `vm.*` counters and tunables, often the fastest way to pull a specific metric without a dedicated tool.

If your actual goal is profiling something like the FFSv2/WAPBL journaling path or a ZFS port issue, DTrace's `fbt` provider combined with `vmstat -l` for lock contention is probably the most useful combo — happy to go deeper on DTrace script syntax for kernel function tracing if that's the direction you're headed.


Me:
Can you create a table that shows which of the above tools are available in which versions of NetBSD, back to 2.0 if you can and through 11.0 release candidates?

Claude:
Here's the version-by-version picture. A few caveats up front: most of these are inherited straight from 4.3/4.4BSD and have been present essentially unchanged in NetBSD since 1.0, so the interesting variation is really in `systrace` and `DTrace`.

| Tool | 2.0 (2004) | 3.0 (2005) | 4.0 (2007) | 5.0 (2009) | 6.0 (2012) | 7.0 (2015) | 8.0 (2018) | 9.0 (2020) | 10.0 (2024) | 11.0 (RC) |
|---|---|---|---|---|---|---|---|---|---|---|
| `ktrace`/`kdump` | X | X | X | X | X | X | X | X | X | X |
| `ptrace(2)` | X | X | X | X | X | X | X | X | X | X |
| `systrace` | X (new in 2.0) | X | X |   removed |   |   |   |   |   |   |
| `gprof`/`prof` | X | X | X | X | X | X | X | X | X | X |
| `kgmon`+`GPROF` kernel option | X | X | X | X | X | X | X | X | X | X |
| `DDB` | X | X | X | X | X | X | X | X | X | X |
| `kgdb` | X | X | X | X | X | X | X | X | X | X |
| `crash(8)` kernel debugger | — | — | — | — | — | — | — | — | X ported | X |
| DTrace |   |   |   |   (merge was 2010, just after 5.0) | ⚠️ present, x86/some ARM only | X enabled by default | X + DTraceToolkit scripts (`dtruss`, `execsnoop`, `opensnoop`) in base | X | X (ported to MIPS/AArch64 compat) | X |
| `vmstat` | X | X | X | X | X | X | X | X | X | X |
| `iostat` | X | X | X | X | X | X | X | X | X | X |
| `netstat` | X | X | X | X | X | X | X | X | X | X |
| `systat` | X | X | X | X | X | X | X | X | X | X |
| `top`/`ps` | X | X | X | X | X | X | X | X | X | X |
| `sysctl` | X | X | X | X | X | X | X | X | X | X |

Key inflection points worth calling out:

- **`systrace`**: Niels Provos' systrace binary sandboxing was integrated into NetBSD in 2.0, but it was removed in NetBSD 5.0 because it fundamentally didn't work — the systrace(1) removal is documented in the 4.0-to-5.0 changelog, dated December 31, 2007. So it's really only a 2.0–4.x era tool for you.
- **DTrace**: it was merged into NetBSD in 2010 (just missing the 5.0 cut), was still a work-in-progress effort limited to x86 systems and some ARM boards for a while, then was enabled by default in NetBSD starting in 2016, which lines up with the 7.x branch. Starting with NetBSD-8, builds with MKDTRACE=yes install Brendan Gregg's DTrace toolkit scripts in base, including dtruss, execsnoop, and opensnoop. By NetBSD 10, DTrace had been ported to MIPS, and the 10.0 release notes mention porting of dtrace(1) and the crash(8) kernel debugger (crash(8) being new to base around this point).
- Everything else in the list (ktrace/ptrace, gprof/kgmon, DDB/kgdb, vmstat/iostat/netstat/systat/top/ps/sysctl) is foundational BSD tooling that's been continuously present with only incremental feature additions (e.g., `vmstat -l` lock stats, additional sysctl nodes) rather than introduction/removal events — so there's no real "table drama" there, just steady evolution.

If you want, I can go deeper on any single row — e.g. exactly which architectures got DTrace support in which release, since that's the one with genuine per-version texture.

--
Kind regards,
    _____    
   | * * |   Josh Moyer (he/him) <JMoyer%NODOMAIN.NET@localhost>
   |*(*)*|   http://jmoyer.nodomain.net/
    \ - /    http://www.nodomain.net/
     \//     
             Love, Responsibility, Justice
             Liebe, Verantwortung, Gerechtigkeit
             
             Please don't eat the animals.
             Thanks.

-----Original Message-----
From: port-vax-owner%NetBSD.org@localhost <port-vax-owner%NetBSD.org@localhost> On Behalf Of Ken Wellsch
Sent: Thursday, July 9, 2026 9:46 AM
To: port-vax List <port-vax%netbsd.org@localhost>
Subject: Re: Performance degradation over time on VAX...

After seeing Jason's posting, I wanted to continue with my amateur sniffing effort.

On a VAXstation 4000/60 (as I used before) using an emulation SCSI card
(as before) I've run through a whole lot of official release ISO's.

This time I installed each release and ran the following two command
sequences (based on my interpretation of Jason's work).

ntpdate -u pool.ntp.org ; dd if=/dev/zero  of=/dev/null bs=4096 count=100000 ; ntpdate -u pool.ntp.org
ntpdate -u pool.ntp.org ; dd if=/dev/rsd0c of=/dev/null bs=4096 count=30000 ; ntpdate -u pool.ntp.org

I selected count values, as before, so they would give me more than 60 seconds of runtime.

I liked the fact that the dev/zero case bypasses any complex device driver and "real" device.

The ntpdate portion came from Google helping me get a working command for the older releases.

Hopefully these time corrections will show no gross time issues during the brief runtime.

============================================================
---- dd if=/dev/zero  of=/dev/null bs=4096 count=100000 ----
============================================================

2.0.0   409.60 MB in  92.476 secs (4.43 MB/s)   offset 0.014478 sec
2.0.2   409.60 MB in  81.469 secs (5.03 MB/s)   offset -0.017075 sec
2.1.0   409.60 MB in  81.851 secs (5.00 MB/s)   offset -0.010569 sec
3.0.0   409.60 MB in  95.097 secs (4.31 MB/s)   offset 0.017103 sec
3.0.2   409.60 MB in  82.617 secs (4.96 MB/s)   offset -0.020239 sec
3.1.1   409.60 MB in  81.218 secs (5.04 MB/s)   offset -0.002095 sec
4.0.0   409.60 MB in  82.683 secs (4.95 MB/s)   offset 0.000865 sec
4.0.1   409.60 MB in  80.682 secs (5.08 MB/s)   offset -0.002540 sec
5.0.0   409.60 MB in  73.896 secs (5.54 MB/s)   offset -0.006272 sec
5.0.2   409.60 MB in  73.251 secs (5.59 MB/s)   offset -0.007861 sec
5.1.2   409.60 MB in  74.305 secs (5.51 MB/s)   offset 0.002074 sec
5.1.5   409.60 MB in  77.247 secs (5.30 MB/s)   offset -0.002412 sec
5.2.3   409.60 MB in  74.234 secs (5.52 MB/s)   offset 0.007559 sec
6.0.0   409.60 MB in 131.145 secs (3.12 MB/s)   offset -0.008218 sec
6.0.6   409.60 MB in 145.589 secs (2.81 MB/s)   offset 0.008178 sec
6.1.3   409.60 MB in 126.978 secs (3.23 MB/s)   offset -0.013498 sec
7.0.0   409.60 MB in 128.069 secs (3.20 MB/s)   offset 0.162509 sec
7.1.2   409.60 MB in 130.681 secs (3.13 MB/s)   offset 0.886323 sec
7.2.0   409.60 MB in 130.732 secs (3.13 MB/s)   offset 0.094225 sec
8.0.0   409.60 MB in 122.039 secs (3.36 MB/s)   offset 0.969403 sec
8.1.0   409.60 MB in 122.904 secs (3.33 MB/s)   offset 0.209796 sec
8.2.0   409.60 MB in 118.316 secs (3.46 MB/s)   offset 0.834749 sec
9.0.0   409.60 MB in 115.717 secs (3.54 MB/s)   offset 0.657326 sec
9.1.0   409.60 MB in 120.125 secs (3.41 MB/s)   offset 0.359956 sec
9.2.0   409.60 MB in 126.775 secs (3.23 MB/s)   offset 0.434128 sec
9.3.0   409.60 MB in 117.573 secs (3.48 MB/s)   offset 0.408211 sec
10.0.0  409.60 MB in 130.912 secs (3.13 MB/s)   offset +0.305483 sec
10.1.0  409.60 MB in 132.401 secs (3.09 MB/s)   offset +0.021104 sec
11.rc5  409.60 MB in 142.258 secs (2.88 MB/s)   offset +0.071411 sec

===========================================================
---- dd if=/dev/rsd0c of=/dev/null bs=4096 count=30000 ----
===========================================================

2.0.0   122.88 MB in 104.826 secs (1.17 MB/s)   offset 0.006032 sec
2.0.2   122.88 MB in 103.624 secs (1.19 MB/s)   offset 0.012788 sec
2.1.0   122.88 MB in 105.223 secs (1.17 MB/s)   offset 0.008445 sec
3.0.0   122.88 MB in 101.946 secs (1.21 MB/s)   offset 0.006388 sec
3.0.2   122.88 MB in 103.549 secs (1.19 MB/s)   offset -0.010305 sec
3.1.1   122.88 MB in 103.051 secs (1.19 MB/s)   offset -0.255196 sec
4.0.0   122.88 MB in 111.208 secs (1.10 MB/s)   offset 0.000746 sec
4.0.1   122.88 MB in 111.860 secs (1.10 MB/s)   offset 0.001173 sec
5.0.0   122.88 MB in 131.359 secs (0.94 MB/s)   offset 0.001126 sec
5.0.2   122.88 MB in 131.080 secs (0.94 MB/s)   offset -0.004246 sec
5.1.2   122.88 MB in 132.431 secs (0.93 MB/s)   offset -0.000763 sec
5.1.5   122.88 MB in 134.746 secs (0.91 MB/s)   offset -0.000325 sec
5.2.3   122.88 MB in 128.433 secs (0.96 MB/s)   offset -0.013470 sec
6.0.0   122.88 MB in 150.388 secs (0.82 MB/s)   offset 0.005609 sec
6.0.6   122.88 MB in 155.608 secs (0.79 MB/s)   offset -0.005730 sec
6.1.3   122.88 MB in 153.486 secs (0.80 MB/s)   offset 0.004184 sec
7.0.0   122.88 MB in 157.764 secs (0.78 MB/s)   offset 0.173563 sec
7.1.2   122.88 MB in 157.760 secs (0.78 MB/s)   offset 0.224465 sec
7.2.0   122.88 MB in 156.500 secs (0.79 MB/s)   offset 0.099339 sec
8.0.0   122.88 MB in 162.346 secs (0.76 MB/s)   offset 0.070651 sec
8.1.0   122.88 MB in 159.242 secs (0.77 MB/s)   offset 0.221792 sec
8.2.0   122.88 MB in 159.596 secs (0.77 MB/s)   offset 0.229680 sec
9.0.0   122.88 MB in 164.284 secs (0.75 MB/s)   offset 0.613972 sec
9.1.0   122.88 MB in 166.545 secs (0.74 MB/s)   offset 0.379712 sec
9.2.0   122.88 MB in 163.019 secs (0.75 MB/s)   offset 0.438902 sec
9.3.0   122.88 MB in 163.831 secs (0.75 MB/s)   offset 0.413608 sec
10.0.0  122.88 MB in 181.440 secs (0.68 MB/s)   offset +0.190593 sec
10.1.0  122.88 MB in 180.109 secs (0.68 MB/s)   offset -0.015763 sec
11.rc5  122.88 MB in 181.721 secs (0.68 MB/s)   offset -0.007328 sec
References:
- Re: Performance degradation over time on VAX...
  - From: Ken Wellsch
Prev by Date: Re: Performance degradation over time on VAX...
Previous by Thread: Re: Performance degradation over time on VAX...
Next by Thread: SDI Disks
Indexes:
Home | Main Index | Thread Index | Old Index