Re: hardware timestamping of packets

[Dennis Ferguson <dennis.c.ferguson%gmail.com@localhost>]

On 20 Jun, 2012, at 17:47 , Vlad Balan wrote:

> This is a good question. I was planning to follow Linux's lead, where
> the hardware counter and the system counter can be both reported, upon
> request. For each cycle counter in the system Linux maintains a rough
> translate to system time, approximating the skew and the offset based
> on a few measurements and comparisons between the cycle counter and
> system clock. This kind of timestamp translation is not hard to
> implement.
> 
> I will start by providing support for the hardware timestamp and then
> take care of translating the cycle counts to system time as well.
> 
> The rest of the discussion identified the problems present in
> providing a socket interface for timestamping, on the transmission
> path. I will try, for starters, to provide an interface that is
> compatible with the Linux one in order to avoid too much
> differentiation to userland applications. I agree that the current
> form of the interface satisfies the needs of only a handful of
> applications. I will try to maintain the implementation of
> transmit-side timestamps clean in order to make it possible to
> implement additional mechanisms for passing them on to userland.
> 
> I agree that providing hardware timestamps to bpf is a good idea. I
> will try to see how to best implement it, given that bpf takes its
> timestamps through a slightly different mechanism, close to the packet
> arrival time. Here we might have to decide upon an interface again, to
> specify the source of timestamps to be returned.
> 
> Finally, it appears that none of the cards currently available is
> capable of obtaining hardware timestamps for all packets and returning
> them in the packet descriptors. Line capture cards built on FPGAs have
> this capability, however my understanding is that they use different
> interfaces to the host (possibly userspace), making them unsuitable
> for testing the hardware timestamping kernel feature.

After thinking about it for a bit, I think I'd have the following
concerns about and preferences for this.

The reason why I'm interested in the applications which will use
this facility is that different time consumers have different
requirements.  I agree that hardware tick counters need to be
translated into a regular, common time format for presentation to
consumers, but how you do this and where the data best comes from
depends a lot on what the application is using this for.  Roughly
maintaining the hardware clock in synchronization with the system
clock (and hardwiring that relationship) might be appropriate if
what you want from the hardware timestamps is a rough approximation
to system time, but if what you instead want to do with this is an
IEEE 1588 implementation operating at the precision of the hardware
it seems like both the "rough translation" thing and the synchronization
direction end up being exactly wrong.  I think for a good PTP
implementation you want the application which is sending and receiving
PTP packets and having them timestamped with ethernet clock timestamps
is to precisely adjust the ethernet clock into synchronization with
the PTP master (this can be done very precisely since the data you are
getting, referenced to the ethernet clock, is very good) and then to
synchronize the system clock as closely as possible to the now-synchronized
ethernet clock. This suggests to me that this might be more useful
if what was provided was a precise user-space clock adjustment
interface that could be used to discipline any or all clocks in the
system and allow the application using that API to determine what
is the best way to apply it.

The other thing I'd like is for system clock timestamps to be acquired
as close to the packet arrival time as possible (i.e. early in
the receive interrupt).  This would make bpf's normal source of
receive timestamps the same as for regular packets delivered
to sockets; if the timestamp(s) was(/were) attached to the packet's
mbuf by the time bpf looked at it, bpf could just copy the timestamp
out of there.  Being able to timestamp received packets with a system
timestamp very soon after they've arrived will provide moderately
accurate time samples for interfaces with no hardware timestamping
capability, having every packet carry a system clock arrival timestamp
allows an interesting and useful alternative way to benchmark the
kernel's network stack (measuring how long it takes to get individual
packets from where they entered the kernel to where they end up),
and any timestamps unused for other purposes might (or might not)
provide useful entropy for the random number pool.  Implementing the
transmit timestamp interface in terms of system clock timestamps would
complete this so that you could always count on the timestamp mechanism
working to the level of precision available even without hardware
support.

Finally, I have a rudimentary design for a clock adjustment interface
which deals with multiple clocks in a system using common adjustment
code, along with addressing some issues to make it easier to to write
accurate time synchronization applications.  If I haven't made a mistake
it should be available here:

    http://www.mistimed.com/Clock.pdf

This was designed without your application in mind, and its current
drawback for this particular application is that I never considered
trying to use this to maintain an "unreliable" clock whose underlying
hardware rate might change during operation (which is why I was
concerned about this).  It instead thinks the only thing you'll
want to do is avoid using such clocks altogether, but in this case I
can see how trying to deal with this might be useful and with a little
thinking it might be possible to enhance the interface to help track
the state of such clocks.  Beyond this, however, it does address the
issues of how to maintain clock conversions accurately and how to poll
one clock against another so that one can be synchronized to the other,
so it might provide a more generically useful framework for your work
than just hardcoding sampling and conversion stuff in the kernel.

Dennis Ferguson