Re: timekeeping regression?

["Mathew, Cherry G.*" <c%bow.st@localhost>]

>>>>> "Greg" == Greg A Woods <woods%planix.ca@localhost> writes:

    Greg> At Tue, 19 Mar 2024 04:03:41 +0000, "Mathew, Cherry G.*" <c%bow.st@localhost> wrote:
    Greg> Subject: Re: timekeeping regression?
    >> 
    >> 
    >> I had a cursory look at the xen time stuff - the per-cpu
    >> tick-count seems to come from an rdtsc (hardware counter),

    Greg> I may be misunderstanding some important concepts here, but I
    Greg> don't think it's that simple (if I understand what you mean by
    Greg> "tick count" here).

    Greg> If I understand correctly the tick counter in Xen parlance is
    Greg> what they call the "Platform Timer" -- i.e. the timer (tick
    Greg> counter) from which all the other timers it uses are
    Greg> driven/derived.

I had a re-look - judging by the comments in:
sys/external/mit/xen-include-public/dist/xen/include/public/xen.h:
struct shared_info;
struct vcpu_time_info;

- The hypervisor (Xen) does keep wallclock time. This makes sense, since
  it needs to provide services such as timer callbacks to domains.

- It seems to rely on "privileged domains" (I imagine that means dom0)
  to keep the wallclock up-to date or in sync (it's unclear to me what
  the exact protocol for this is - but probably explains why our dom0
  continually updates this value)

- Keeps a copy of the TSC per-vCPU, corresponding to its (presumably)
  last scheduled pCPU.

Per comments, the idea seems to be that domUs have some work:
"
 * The intent is that this copy may be mapped (RO) into userspace so
 * that usermode can compute system time using the time info and the
 * tsc.  Usermode will see an array of vcpu_time_info structures, one
 * for each vcpu, and choose the right one by an existing mechanism  
 * which allows it to get the current vcpu number (such as via a     
 * segment limit).  It can then apply the normal algorithm to compute
 * system time from the tsc.
 *
"

This is not what we do currently - we just read off the underlying TSC
and update our understanding of time - without factoring in which vCPU
we're on, etc. as above. Overall, I blame xen for poor
abstraction/documentation for this situation.

    Greg> In x86/amd64 platforms that can be the CPU TSC register via
    Greg> RDTSC, but if, and only if, the CPU(s) have a reliable TSC,
    Greg> i.e. CONSTANT_TSC.

I'm not sure of the mechanics here, but I'm assuming TSC is used for
timekeeping, in the discussion above.


    Greg> On my systems the platform timer is always reported as HPET.

    >> while the "wall clock" seems to come from the hypervisor (via
    >> shared mem).

    Greg> If by "hypervisor" you mean the dom0?  

No, I meant the Xenhypervisor.

    Greg> As I understand it the dom0 periodically updates the Xen
    Greg> kernel's wall clock time (via a hypercall).  (at least when
    Greg> the dom0 is NetBSD)


Yes, I realised that, re-reading, as above.

    Greg> But what I wanted to understand is why Xen ever needs to know
    Greg> about wall clock time, and, most importantly if ever it uses
    Greg> its own tick counter to update its own idea of wall clock
    Greg> time, or worse what happens if the dom0 updates it with a
    Greg> skewed wall clock time because dom0 can't keep time properly.

Likely to have a notion of "absolute time" - needed to make decisions
about timeouts, etc.

    Greg> I suspect the Xen kernel wants to have wall clock time so that
    Greg> it can emulate RTC hardware, and/or tell PV guests directly
    Greg> via a hypercall what the current wall clock time is when they
    Greg> are first booted.

Unsure - and I'm not sure it's related to the problems you're seeing.

    Greg> I think the "xen_system_time" should be a timecounter derived
    Greg> from the Xen platform timer.

I think the problem here is that there's no clarity in the API about
ownership. So we'll probably need to have a look at what linux and/or
FreeBSD does, to understand what the current consensus is - this seems
to be the linux way of documentation - a sort of mob rules situation.

Anyway, at this point I'm personally not going to touch this until I
have my head wrapped around our own (NetBSD) timekeeping. Will report
back after I have any progress there, and am ready to look at this.

-- 
Math/(~cherry)