Understanding PR kern/43997 (kernel timing problems / qemu)

[Robert Elz <kre%munnari.OZ.AU@localhost>]

I want to leave /bin/sh to percolate for a while, make sure there are
no issues with it as it is, before starting on the next round of
cleanups and bug fixes, so I was looking for something else to poke
my nose into ...

[Aside: the people I added to the cc of this message are those who have
 added text to PR kern/43997 and so I thought might be interested, if you're
 not, just say...]

kern/43997 is the "qemu is too slow, clock interrupts get lost, timing
gets all messed up" problem that plagues many of the ATF tests that kind
of expect time to be maintained rationally.

Now there's no question that qemu is slow, for example, on my amd64 Xen
DomU test system, the shell arithmetic test of ++x (etc) takes:
	var_preinc: [0.077617s] Passed.
whereas from the latest completed b5 (qemu) test run (as of this e-mail)
	var_preinc   Passed   N/A   6.200489s

That's about 80 times slower (and most of the other tests show similar
factors).   I don't think we can blame qemu for that, given what it is
doing.

So, it is hardly surprising that, to borrow Paul's words from the PR:
	On (at least) amd64 architecture, qemu cannot simulate clock
	interrupts at 100Hz.
nor that
	Therefore, a simple "date ; sleep 5; date" command
	actually requires 10 seconds to complete!

This (aside from the workload it creates on b5) shouldn't even really be
an issue, I don't think we have any ATF NTP tests, and if we did, attempting
those in a qemu emulated environment would be insane.

The problem is really (again from the PR)

	The routines sleep(3), usleep(3), and nanosleep(2) wake-up based on the 
	occurrence of clock ticks.  However, the timer interrupt routine
	determines the actual absolute time.

which means that the NetBSD kernel is getting itself out of sync - it is
not maintaining one consistent view of the time for the system it is running.

Whether its time view internally matches the outside reality is not really
a big issue - obviously it is better if it does, at least as close as possible
(without external time sync mechanisms, nothing is perfect) but internally
it really should be consistent.

What's more, at least from the description of the problem, I see nothing that
would prevent the same issue arising (probably on a much smaller scale) on
any system that happened to suffer an interrupt storm (due to either something
broken, some kind of attack, or just a very heavy workload) that happens to
last more than 10ms (on a 100Hz based tick system, 1ms on an alpha with 1024Hz)
and causes a clock tick to be lost.

So, I think qemu is no more than a good environment for simulating the
underlying problem, and not itself in any material way related to the
problem, which is squarely a NetBSD kernel issue.

If there's no disagreement about this analysis, I plan on digging into the
clock/time handling parts of the kernel, and fixing this (whatever it takes...)

My current guess of the "whatever it takes" is that something along the lines
of
	we know absolute time (the timer interrupt routine uses it)
	we know when the last clock tick happened (we made it happen, we
		can remember when that was)
	we can calculate how many clock ticks should have been generated in
		the intervening period
	tick tick tick...

is needed.   But I am yet to delve into the code (this is mostly just from the
PR.)   Note: this can be optimised so that there's very little (though probably
not zero) extra work in the common case where nothing is being missed.

kre