Re: TSC improvement

[Taylor R Campbell <campbell+netbsd-tech-kern%NetBSD.org@localhost>]

> Date: Wed, 10 Jun 2020 23:11:32 +0000
> From: Andrew Doran <ad%netbsd.org@localhost>
> 
> On Tue, Jun 09, 2020 at 05:16:27PM +0000, Taylor R Campbell wrote:
> 
> > At the same time, I wonder whether we should _also_:
> > 
> > 1. Modify the tsc timecounter so that it uses a global atomic to
> >    ensure that there is a global view of time as counted by the tsc.
> 
> A long time ago we used to do something quite like this for x86 but it
> worked poorly which is somewhat understandable because it's really difficult
> to get something like that right.  (Which reminds me someone posted a patch
> for kern_cctr.c for alpha a couple of years ago.)

It's not hard to get global monotonicity right.  We do that for Xen
these days, in arch/xen/xen/xen_clock.c.  The Alpha patch was probably
this thread:

https://mail-index.netbsd.org/port-alpha/2017/11/01/msg000873.html

What's trickier is synchronizing per-CPU timecounters so that they all
give a reasonably consistent view of absolute wall clock time -- and
so it's not just one CPU that leads while the others play catchup
every time they try to read the clock.  (In other words, adding atomic
catchup logic certainly does not obviate the need to synchronize
per-CPU timecounters!)

But neither synchronization nor global monotonicity is always
necessary -- e.g., for rusage we really only need a local view of time
since we're only measuring relative time durations spent on the
current CPU anyway.

> >    This is what the timecounter(9) API per se expects of timecounters,
> >    and right now tsc (along with various other per-CPU cycle counters)
> >    fails to guarantee that.
> 
> Howso, do you see a bug?  I think it's okay.  The TSC is only used for the
> timecounter where it's known that it's insensitive to core speed variations
> and is driven by PLL related to the bus clock.  Fortunately that means most
> x86 systems, excepting a window of some years from roughly around the time
> of the Pentium 4 onwards.

If tc_get_timecount goes backward by a little, e.g. because you
queried it on cpu0 the first time and on cpu1 the second time,
kern_tc.c will interpret that to mean that it has instead jumped
forward by a lot -- nothing in the timecounter abstraction copes with
a timecounter that goes backwards at all.

(There's also an issue where the `monotonic' clock goes backwards
sometimes, as reported by sched_pstats.  I'm not sure anyone has
tracked down where that's coming from -- it seems unlikely to be
related to cross-CPU tsc synchronization because lwp rtime should
generally be computed from differences between samples on a single CPU
at a time, but I don't know.)