Re: GSoC 2011 project proposal [packet timestamping]

[Dennis Ferguson <dennis.c.ferguson%gmail.com@localhost>]

On 28 Mar 2011, at 11:54 , Frank Kardel wrote:
> On 03/28/11 19:52, Matt Thomas wrote:
>> On Mar 28, 2011, at 10:08 AM, Joerg Sonnenberger wrote:
>> 
>>   
>>> On Mon, Mar 28, 2011 at 07:52:52PM +0400, Dmitry Cherkassov wrote:
>>>     
>>>>     2. Design and implement SIOCSHTSTAMP ioctl support in one ore more
>>>> drivers;
>>>>       
>>> What speaks against having the timestamping done as part of the normal
>>> ifq operations?
>>>     
>> I think timestamp should be a capability (like jumbogram).  If the
>> driver doesn't support it, it could be generically done in if_input
>> but with a small loss of accuracy.
>> 
>> Also, what happens on cpus/systems with out a highly accurate clock?
>>   
> I agree with the notion that it should be a capability.
> The PTP capable interfaces do the time stamping at PHY
> level using a local counter of the interface and that
> should be transparently supported.

Yes, I've seen those interfaces (though the one's I've seen maintained
the counter in the MAC chip rather than the PHY hardware).

I think the existence of those NICs suggests some arguments about what might
be a good way to implement this functionality.  While I have some reservations
about the current system clock code, how it works is basically sound.  It
assumes the basic time source is some sort of hardware counter that increments
at a fixed frequency (in the worst case it is a variable in memory which gets
incremented at every clock interrupt).  When you want to know what time it is 
you
sample that counter, then do some math to turn it into a time of day.

When using one of those timestamping NICs you end up doing even more math.  The
NIC's counter is normally inappropriate to use for the system clock (it normally
takes a PCI transaction to sample it, and that is way too expensive to use to
find out the time) and, if you have more than one NIC, all those hardware clocks
will be different anyway.  The best way to deal with this is to run a sampling
process for each NIC which maintains calibration constants between each NIC's
counter and the counter being used for the system clock, so to convert a NIC
counter to a time of day you first need to do the math to convert that 
particular
NIC's counter to system count ticks, then do the math to convert system clock 
ticks
into time of day.  If the NIC counter tick-stamp is approximately free then most
of the work of getting a time-of-day stamp is doing that math (as it is when 
your
system clock tick-stamp is cheap to sample).

This suggests that what should be carried in from the interface with packets is
the rawest form of tick-stamp in whatever form it is available (i.e. system 
clock
tick-stamps if done in software, or NIC-specific tick-stamps if done in 
hardware).
Only when it finally arrives at a socket whose listener actually cares to have a
time-of-day stamp should it go to the trouble of doing the math (and figuring 
out what
math needs to be done in the case of NIC tick-stamps) to produce the time of 
day.  Any
tick-stamps which aren't wanted by the socket the packet arrived at can be fed 
(in raw
form) to the pool /dev/random feeds from, providing a significant improvement 
in that
as well.

This would require that the current clock code separate the taking of a 
tick-stamp
from the math to turn it into a time of day, and that it be capable of turning a
tick-stamp taken "some time ago" (for a reasonable definition of that) into the
same time of day that it would have arrived at had it done the conversion 
immediately.

> For the ntpd interleave protocol Dave Mills called for
> timestamping the start and the end of specific packets.
> So this should be controllable/supportable too.
> Also modern interfaces use interrupt moderation, thus
> software only time stamping will not work too well with
> these interfaces.

Certainly, though it is safe to say that in the absence of NIC
timestamp support taking tick-stamps in the driver software is
still much superior to the alternative of taking them in the user-space
application.

Dennis Ferguson