03/02/2011
Matt Sherer, engineer on FSMLabs’ TimeKeeper product, writes:
Having highly accurate time available on the network is great. Getting that time to the application before it’s stale is even better.
Even highly accurate time can get pretty stale by the time it gets to the applications that need it. Lost syncs, network latencies, and other factors can accumulate. Even having local hardware that corrects perfectly for these factors is not a complete solution.
The trouble is, applications don’t run on that card with the accurate time - they’re running under a host operating system. Even if that card provides perfect time, the OS may be delayed in processing it. Or it could mangle the value in an attempt to satisfy conflicting clock requirements.
Within the OS, an application asking for the time on different processors may get different results. Just requesting time from the OS can add significant additional overhead in the application. The result here is that when your application returns with a time sample from the OS, that time may be stale or just wrong.
FSMLabs’ TimeKeeper can remove these ambiguities. TimeKeeper can deliver or consume time over a network (NTP or PTP), but let’s just look at how it can avoid timing ambiguities locally.
TimeKeeper converges on an accurate time and provides it throughout the entire system - not just to specialty applications or to the OS (which may again skew time). Any application or OS component on the system, when it asks for time, will be getting it from TimeKeeper, undiluted and undelayed.
Having a single source of trustworthy time provides a number of benefits:
* The OS is no longer providing different time values to different users depending on their clock.
* The OS has consistent timestamping - all internal state, from network sockets to filesystem updates, are all on a common time base.
* Time drift between processors is gone - every processor has the same time.
* Without per-processor skew, there’s no chance that an application migrating between processors will see time move backwards.
So, TimeKeeper gives the system accurate time directly, whether it’s serving the OS or applications on the OS. Applications can trust the time they get is accurate and not stale.
This is all well and good - but TimeKeeper actually speeds up the process of delivering time too.
It used to be that getting time from the OS involves a system call. That’s expensive - state has to be saved, the application may be switched out, housekeeping overhead may take time, the OS more code has to be run, and the application just isn’t getting real work done. Modern Linux versions have reduced that overhead, and some, like Red Hat Enterprise Linux 6, avoid the
system call entirely. Kernel support for a feature called VDSO allow specially designed services to provide data - like time - without system call overhead.
TimeKeeper leverages this support to further improve performance. Not only is TimeKeeper getting a more accurate and unified time to the entire system, it can actually speed the process of getting time to the caller. Before we show TimeKeeper’s numbers, here are the improvements in using VDSO over system calls in stock Red Hat Enterprise Linux 6:
| Function | Overhead improvement | Speedup |
|---|---|---|
| gettimeofday | 48ns | 54% |
| clock_gettime | 49ns | 60% |
If VDSO is supported, it is selected transparently over a system call. There are no source changes to be made in applications. As you can see, skipping that system call is very worthwhile - it cuts the time spent by more than 50%.
When TimeKeeper starts, if VDSO or vsyscalls are supported, it transparently provides the more accurate time in place of Linux’s data. Again, there are no source changes
needed. Applications don’t even need to be restarted. Once enabled, these numbers get even better:
| Function | TimeKeeper improvement | Additional speedup on RHEL6’s VDSO |
|---|---|---|
| gettimeofday | 9ns | 25% |
| clock_gettime | 10ns | 30% |
TimeKeeper never leaves the processor for data, and it doesn’t enter the OS - so it can take Red Hat 6’s improved performance, and still improve upon that by 30%. (In fact, TimeKeeper also provides an optional direct access function that can reduce overhead by another 15%.) Applications asking for time can have it, accurately, in less than 20 nanoseconds.
What does all this mean? Well, a number of things:
* Applications can trust the time they get from the OS isn’t stale, and doesn’t have built in inaccuracies.
* The whole system - OS and applications - can operate on the same time base without fear of internal drift or time going backwards. Knowing that data from different systems are tagged with the same time base is a huge benefit.
* Having faster access to time means that the application can spend fewer cycles accessing time, and more cycles doing real work. Or, it makes more cycles available to timestamp events that couldn’t be tagged before.
As a software client, TimeKeeper can get accurate time across your network, even where additional hardware is not an option. It can serve PTP (version 1 or 2) or NTP (versions 1-4) or act as a client, to get accurate time to systems that need it. We saw above that TimeKeeper solidly improves on how that time is driven all the way out to the application.
Is your application’s time data stale? Contact .(JavaScript must be enabled to view this email address) and we can quantify how TimeKeeper could improve your system.