finding the perpetrator of regular disk I/O

[Taylor R Campbell <campbell+netbsd%mumble.net@localhost>]

(I am not subscribed to this list, so please cc me in replies.)

Summary: I'm observing frequent disk I/O for no apparent reason under
a current kernel, but not under a 5.1_STABLE kernel.  How can I find
what part of the system is doing the disk I/O (iostat doesn't tell me
the process, for instance), and how do I make it stop?  I would like
to know how to find sources of disk I/O in general, and also to fix
this particular case.

I have an Acer running NetBSD 5.1_STABLE, and `atactl wd0 standby'
makes the disk quiet for a while like I expect.  It is running my
desktop environment and applications as well as a number of daemons
including Unbound, Tor, &c., and it has too little RAM, so there's
often swapping.  Nevertheless putting wd0 into standby mode makes the
disk quiet for essentially arbitrary durations, as long as I don't
trigger swapping or do stuff with files.

I have a MacBook running a NetBSD-current kernel (as of a couple weeks
ago), almost completely idle: it is booted into multi-user mode, and
the only daemons running are postfix, syslogd, dhcpcd, sshd, and cron
-- all of which are running on the Acer too.  There's plenty of RAM
and no swapping.  Whenever I run `atactl wd0 standby', the disk stops
spinning for a few seconds, but then always spins back up again within
twenty or thirty seconds or so.

All of my file systems on physical disks are mounted noatime and
either nodev or nodevmtime.  The Acer's file systems are mounted log;
the MacBook's are mounted sync.

If I run `iostat -x wd0 1' on the MacBook, I see that every twenty or
thirty seconds there is a 16KB write to the disk.  I tried running
`touch /tmp/testfile; atactl wd0 standby', waiting for the write to
happen, and then running `find / -newer /tmp/testfile -o -anewer
/tmp/testfile -o -cnewer /tmp/testfile', but the only results were in
/kern and /proc.

I don't think it's just the kernel's file syncer: I've tried running
sync a dozen times in a row and then putting the disk into standby
mode, and it still spins up after a few seconds with a 16KB write.

Any suggestions?