path: root/www/gpsd-time-service-howto.txt

= GPSD Time Service HOWTO =
Gary E. Miller <gem@rellim.com>, Eric S. Raymond <esr@thyrsus.com>
v1.4, Dec 2014

This document is mastered in asciidoc format.  If you are reading it in HTML,
you can find the original at the GPSD project website.

== Introduction ==

GPSD, NTP and a GPS supplying 1PPS (one pulse-per-second) output can be
used to set up a high-quality NTP time server. This HOWTO explains the
method and various options you have in setting it up.

Here is the quick-start sequence. The rest of this document goes
into more detail about the steps.

1. Ensure that gpsd and either ntpd or chronyd are installed on your
   system. (Both gpsd and ntpd are preinstalled in many stock Linux
   distributions; chronyd is normally not.) You don't have to choose
   which to use yet if you have easy access to both, but knowing which
   alternatives are readily available to you is a good place to start.

2. Connect a PPS-capable GPS to one of your serial or USB ports.
   A random cheap consumer-grade GPS won't do; you may have to
   do some hunting to find a usable one.

3. Check that it actually emits PPS by pointing GPSD's gpsmon utility
   at the port.  If it has a good (3D-mode) fix, lines marked "PPS"
   should scroll by in the packet-logging window.

4. If you persistently fail to get live PPS, (1) you may have a
   skyview problem, (2) you may have a cabling problem, (3) you may
   have a gpsd or kernel configuration problem, (4) you may have a
   device problem, (5) there may be a bug in the core GPSD code used
   by gpsmon.  These are listed in roughly decreasing probability
   order.  Troubleshoot appropriately.

5. Edit your ntpd or chronyd configuration to tell your NTP daemon to
   listen for time hints. (This step is somewhat tricky.)

6. Start up gpsd.  If you are using ntpd, you can use ipcrm(1) to check that
   verify that the shared-memory segment that gpsd and ntpd want to
   use to communicate has been attached; or you can impatiently skip
   to the next step and look for the segment only if that fails.

7. Use ntpq or the chronyc sources command to verify that your device 
   is feeding time corrections to your NTP daemon.

8. (Optional and challenging.) Hand-tune your installation for the
   best possible performance.

This document is intended to be a comprehensive guide suitable for
people who know little about time service. We encourage others to
contribute additions and corrections.

This document does not attempt to explain all the intricacies of
time service; it is focused on practical advice for one specific
deployment case.  An overview of the protocols can be found at
<<WIKI-NTP>>, and the official NTP FAQ <<NTP-FAQ>> is probably
as gentle an introduction as presently exists.

.Units table
|====================================================
| nSec    | nanoSecond  | 1/1,000,000,000 of a second
| uSec    | microSecond | 1/1,000,000 of a second
| mSec    | milliSecond | 1/1,000 of a second
|====================================================

There are a few important terms we need to define up front.  *Latency*
is delay from a time measurement until a report on it arrives where it
is needed. *Jitter* is short-term variation in latency. *Wobble* is a
jitter-like variation that is long compared to typical measurement
periods.  *Accuracy* is the traceable offset from 'true' time as
defined by a national standard institute.

A good analogy to jitter vs wobble is changes in sea level on a beach.
Jitter is caused by wave action, wobble is the daily effect of tides.
For a time server, the most common causes of wobble are varying GPS
satellite geometries and the effect of daily temperature variations on
the oscillators in your equipment.

== How NTP Works ==

NTP (Network Time Protocol) is a set of protocols that attempts to
synchronize computer clocks across the Internet - ultimately, Universal
Coordinated Time (UTC) as defined by several scale-related atomic clocks
run by national standards institutes.  NTP Version 4 is defined in
<<RFC-5905>>.

NTP service daemons running on each host do clock synchronization in
the presence of variable network latency by monitoring those delays in
real time and passing around messages that say, essentially, "I
received your request at time X, I'm sending the reply at time Y and
the propagation delay to my source of time is Z".  Each daemon then
uses rather complex algorithms that we won't attempt to describe here
to digest incoming messages into a composite "NTP time".

NTP conditions your system clock by noticing how your system clock time
differs from deduced NTP time, then speeding up or slowing down your
clock's tick rate until it is synchronized.  These tick-rate changes
are usually extremely small, much too small for a human or even most
software timing loops to ever notice. But large changes are possible.

Most computers are just NTP clients.  They send NTP requests to a set
of servers and use it to adjust the local clock.  It is generally
expected that NTP clients will have an accuracy (that is, maximum
divergence from the master atomic clock) of at most &plusmn;100 mSec,
possibly less depending upon the quality of your network connection.

Some NTP hosts are time *servers*.  These are known as "chimers".
They respond to NTP clients with time read from high-precision
reference clocks (often abbreviated "refclocks") synchronized to
national time standards.  GPSes, because they distribute a single
atomic-clock time basis in a way that is generally accessible, are
widely used as refclocks.

You will hear time service people speak of "Stratum 0" (the reference
clocks) "Stratum 1" (NTP servers directly connected to reference
clocks) and "Stratum 2" (servers that get time from Stratum
1). Stratum 3 chimers redistribute time from Stratum 2, and so
forth. There are defined higher strata up to 15, but you will probably
never see a public chimer higher than Stratum 3 or 4.

Ordinary client computers are normally configured to get time from one
or more Stratum 2 (or less commonly Stratum 3) servers. However, with GPSD and
a suitable GPS device, you can easily condition your clock to higher accuracy
than what you get from typical Stratum 2; with a little effort you can
do better than you can get from most public Stratum 1 servers.

You can then make your high-quality time available to other systems on
your network, or even run a public NTP server.  Anyone can do this;
there is no official authority, and any NTP client may choose to use
your host as a chimer by requesting time from it. The time-service
network is self-regulating, with NTP daemons constantly pruning
statistical outliers so the timebase cannot be accidentally or
deliberately compromised.

In fact many public and widely-trusted Stratum 1 chimers use GPSes as
their Stratum 0, and a significant fraction of those use GPSD in the
way we will describe here.

== 1PPS quality issues ==

GPSD is useful for precision time service because it can use the 1PPS
pulse delivered by some GPSes to discipline (correct) a local NTP
instance.

It's tempting to think one could use a GPS for time service just by 
timestamping the arrival of the first character in the report on each fix
and correcting for a relatively small fixed latency composed of
fix-processing and RS232 transmission time.

At one character per ten bits (counting framing at stopbits) a
9600-bps serial link introduces about a mSec of latency *per
character*; furthermore, your kernel will normally delay delivery
of characters to your application until the next timer tick, about
every 4 mSec in modern kernels. Both USB and RS232 will incur that
approximately 5mSec-per-char latency overhead.  You'll have to deal
with this latency even on chips like the Venus 6 that claim the
beginning of their reporting burst is synced to PPS.

Unfortunately, fix reports are also delayed in the receiver and on
the link by as much as several hundred mSec, and this delay is not
constant. This latency varies (wobbles) throughout the day.  It may be
stable to 10 mSec for hours and then jump by 200mSec.  Under these
circumstances you can't expect accuracy to UTC much better than 1
second from this method.

For example: SiRF receivers, the make currently most popular in
consumer-grade GPSes, exhibit a wobble of about 170mSec in the offset
between actual top-of-second and the transmission of the first
sentence in each reporting cycle. You can see this graphed at
<<SIRF-WOBBLE>>.

To get accurate time, then, the in-band fix report from the GPS needs
to be supplemented with an out-of-band signal that has a low and
constant or near-constant latency with respect to the time of of the
fix.  GPS satellites deliver a top-of-GPS-second notification that is
nominally accurate to 50nSec; in capable GPses that becomes the 1PPS
signal.

1PPS-capable GPSes use an RS-232 control line to ship the 1PPS
edge of second to the host system (usually Carrier Detect or Ring
Indicator; GPSD will quietly accept either).  Satellite top-of-second
loses some accuracy on the way down due mainly to variable delays in
the ionosphere; processing overhead in the GPS itself adds a bit more
latency, and your local host detecting that pulse adds still more
latency and jitter.  But it's still often accurate to on the order of
1 uSec.

Under most Unixes there are two ways to watch 1PPS; Kernel PPS (KPPS)
and plain PPS latching.  KPPS is an implementation of RFC 2783 <<RFC-2783>>.  
Plain PPS just references the pulse to the system clock as 
measured in user space.  These have different error budgets.  

Kernel PPS uses a kernel function to accurately timestamp the status
change on the PPS line.  Plain PPS has the kernel wake up the GPSD PPS
thread and then the PPS thread reads the current system clock.  As
noted in the GPSD code, having the kernel do the time stamp yields
lower latency and less jitter. Both methods have accuracy degraded by
interrupt-processing latency in the kernel serial layer, but plain
PPS incurs additional context-switching overhead that KPPS does not.

With KPPS it is very doable to get the system clock stable to &plusmn;1
uSec.  Otherwise you are lucky to get &plusmn;5 uSec, and there will be
about 20uSec of jitter. All these figures were observed on
plain-vanilla x86 PCs with clock speeds in the 2GHz range.

All the previous figures assume you're using PPS delivered over
RS232.  USB GPSes that deliver 1PPS are rare, but do exist. Notably,
there's the Navisys GR601-W. In case this device goes out of
production it's worth noting that it's a trivial modification of the
stock two-chip-on-a-miniboard commodity-GPS design of engine plus
USB-to-serial adapter; the GR601-W wires a u-blox 6 to a Prolific Logic
PL23203.  To get 1PPS out, just wire the 1PPS pin from the GPS engine to
the Carrier Detect pin on the USB adapter. (This is known as the "Macx-1
mod".)

With this design, 1PPS from the engine will turn into a USB event that
becomes visible to the host system (and GPSD) the next time the USB
device is polled. USB 1.1 polls 1024 slots ever second.  Each slot is
polled in the same order every second.  When a device is added it is
assigned to one of those 1024 polling slots.  It should then be clear
that the accuracy of a USB 1.1 connected GPS would be about 1 mSec.

As of early 2015 no USB GPS we know of implements the higher
polling-rate options in USB 2 and 3 or the interrupt capability in
USB 3.  When one does, and if it has the Macx-1 mod, higher USB
accuracy will ensue.

.Summary of typical accuracy
|=======================================
| GPS atomic clock    |  &plusmn;50nSec 
| KPPS                |  &plusmn;1uSec  
| PPS                 |  &plusmn;5uSec  
| USB poll interval   |  &plusmn;1mSec  
| Network NTP time    |  &plusmn;10mSec 
|=======================================

Observed variations from the typical figure increase towards the bottom
of the table.  Notably, a heavily loaded host system can reduce PPS
accuracy further, though not KPPS accuracy except in the most extreme
cases.  The USB poll interval tends to be very stable (relative to its
1mSec poll).

Network NTP time accuracy can be degraded below 10mSec by a number of
factors. Almost all have more to do with the quality of your Internet
connection to your chimers than with the accuracy of the chimers
themselves.  Some negatives:

* Having a cable modem.  That is, as opposed to DSL or optical fiber, which
  tend to have less variable latencies.

* Path delay asymmetries due to peering policy.  These can confuse
  NTP's reconciliation algorithms.

With these factors in play, worst-case error can reach up to
&plusmn;100mSec.  Fortunately, errors of over &plusmn;100mSec are
unusual and should occur only if all your network routes to chimers
have serious problems.

== Software Prerequisites ==

gpsd includes support for interpreting 1PPS pulses that is mostly
autoconfiguring and requires no special setup.  If you built GPSD from
sources, make sure the build is with pps=yes and ntpshm=yes (the
default).  The command "gpsd -L" should indicate that
time-service features and PPS are enabled.

If your kernel provides the RFC 2783 KPPS (kernel PPS) API, gpsd will
use that for extra accuracy. Many Linux distributions have a package
called "pps-tools" that will install KPPS support and the time_pps.h
header file.  We recommend you do that.  If your kernel is built in
the normal modular way, this package installation will suffice.

If you are scratch-building a Linux kernel, the configuration 
must include either these two lines, or the same with "y" replaced
by "m" to enable the drivers as modules:

-----------------------------------------------------------------------------
CONFIG_PPS=y
CONFIG_PPS_CLIENT_LDISC=y
-----------------------------------------------------------------------------

Your Linux distribution may ship a file /boot/config-XXX (where XXX is
the name of a kernel) or one called /proc/config.gz (for the running
kernel).  This will have a list of the configuration options that were
used to build the kernel.  You can check if the above options are
set. Usually they will be set to "m", which is sufficient.

NetBSD has included the RFC2783 Pulse Per Second API for real serial
ports by default since 1998, and it works with ntpd.  NetBSD 7
(forthcoming) includes RFC2783 support for USB-serial devices, and
this works (with ntpd) with the GR601-W.  However, gpsd's code
interacts badly with the NetBSD implementation, and gpsd's support for
RFC2783 PPS does not yet work on NetBSD (for serial or USB).

Other OSes have different ways to enable KPPS in their kernels.
When we learn what those are, we'll document them or point
at references.

You will need to have either ntpd or chrony installed. If you are
running a Unix variant with a package system, the packages will
probably be named 'ntpd' and either 'chrony' or 'chronyd'.

Of these two, ntpd is the older and more popular choice - thus, the
one with the best-established peer community if you need help in
unusual situations.  On the other hand, chrony has a reputation for
being easier to set up and configure, and is better in situations
where your machine has to be disconnected from the Internet for long
enough periods of time for the clock to drift significantly.

ntpd and chrony have differing philosophies, with ntpd more interested
in deriving conensus time from multiple sources while chrony tries to
identify a single best source and track it closely.

A feature comparison, part of the chrony documentation, is at
<<CHRONY-COMPARE>>. An informative email thread about the differences
is <<CHRONYDEFAULT>>. If you don't already know enough about time
service to have a preference, the functional differences between them
are unlikely to be significant to you; flip a coin.

== Choice of Hardware ==

To get 1PPS to your NTP daemon, you first need to get it from a
PPS-capable GPS. As of early 2015 this means either the previously
mentioned GR601-W or a serial GPS with 1PPS.

You can find 1PPS-capable devices supported by GPSD at <<HARDWARE>>.
Note that the most popular consumer-grade GPS do not usually deliver
1PPS through USB or even RS232.  The usual run of cheap GPS mice won't
do.  In general, you can't use a USB device for time service unless you
know it has the Macx-1 mod.

In the past, the RS232 variant of the Garmin GPS-18 has been very
commonly used for time service.  While it is still a respectable
choice, newer devices have better sensitivity and signal
discrimination. This makes them superior for indoor use as time
sources.

In general, use a GPS with an RS232 interface for time service if you
can.  The GR601-W was designed (by one of the authors, as it happens)
for deployment with commodity TCP/IP routers that only have USB ports.
RS232 is more fiddly to set up (with older devices like the GPS-18 you
may even have to make your own cables) but it can deliver three orders
of magnitude better accuracy and repeatability - enough to meet
prevailing standards for a public Stratum 1 chimer.

Among newer receiver designs we've found the the u-blox 6 receiver used
in the GR601-W to be particularly good.  Very detailed information on
its timing performance can be found at <<UBLOX-TIMING>>. One of us
(Raymond) has recent experience with an eval kit, the EVK 6H-0-001,
that would make an excellent Stratum 0 device. 

Both the EVK 6H and GR601-W are built around the LEA-6H module, which
is a relatively inexpensive but high-quality navigation GPS. We make a
note of this because u-blox also has a specialized timing variant, the
LEA 6T, which would be expensive overkill for an NTP server.

Unfortunately as of early 2015 the LEA-6H is still hard to find in a
packaged RS232 version, as opposed to a bare OEM module exporting TTL
levels or an eval kit like the EVK 6H-0-001 costing upwards of
US$300. Search the web; you may find a here-today-gone-tomorrow offer
on alibaba.com or somewhere similar.

The LEA-6T, and some other higher-end GPS receivers (but not the
LEA-6H) have a stationary mode which, after you initialize it with the
device's location, can deliver time service with only one good
satellite lock (as opposed to the three required for a fix in its
normal mode). For most reliable service we recommend using stationary
mode if your device has it. GPSD tools don't yet directly support
this, but that capability may be added in a future release.

The design of your host system can also affect time quality.  The
&plusmn;5uSec error bound quoted above is for a dual-core or better
system with clock in the 2GHz range on which the OS can schedule the
long-running PPS thread in GPSD on an otherwise mostly unused
processor (the Linux scheduler, in particular, will do this). On a
single-core system, contention with other processes can pile
on several additional microseconds of error.

If you are super-serious about your time-nuttery, you may want to look
into the newest generation of dedicated Stratum 1 microservers being
built out of open-source SBCs like the Raspberry Pi and Beaglebone, or
sometimes with fully custom designs. A representative build is well
described at <<RPI>>.

These microserver designs avoid load-induced jitter by being fully
dedicated to NTP service.  They are small, low-powered devices and
often surprisingly inexpensive, as in costing less than US$100.  They
tend to favor the LEA-6H, and many of them use preinstalled GPSD on
board.

== Enabling PPS ==

You can determine whether your GPS emits 1PPS, and gpsd is
detecting it, by running the gpsmon utility (giving it the GPS's
serial-device path as argument).  Watch for lines of dashes marked
'PPS' in the packet-logging window; for most GPS types there will also
be a "PPS offset:" field in the data panels above showing the delta
between PPS and your local clock.

If you don't have gpsmon available, or you don't see PPS lines in it,
you can run gpsd at -D 5 and watch for carrier-detect state change
messages in the logfile.

If you don't see evidence of incoming PPS, here are some trouble
sources to check:

1. The skyview of your GPS may be poor.  Suspect this if, when you
   watch it with gpsmon, it wanders in and out of having a good 3D
   fix. Unfortunately, the only fix for this is to re-site your GPS
   where it can see more sky; fortunately, this is not as common
   a problem as it used to be, because modern receivers are often
   capable of getting a solid fix indoors.

2. If you are using an RS232 cable, examine it suspiciously, ideally
   with an RS232 breakout box. Cheap DB9 to DB9 cables such as those
   issued with UPSes often carry TXD/RXD/SG only, omitting handshake
   lines such as DCD, RI, and DSR that are used to carry 1PPS.
   Suspect this especially if the cable jacket looks too skinny to
   hold more than three leads!

3. Verify that your gpsd and kernel were both built with PPS support,
   as previously described in the section on software prerequisites.

4. Verify that the USB or RS232 device driver is accepting the ioctl
   that tells it to wait on a PPS state change from the device.  The
   messages you hope *not* to see look like "KPPS cannot set PPS line
   discipline" and "PPS ioctl(TIOCMIWAIT) failed".  The former
   can probably be corrected by running as root; the latter (which
   should never happen with an RS232 device) probably means your USB
   device driver lacks this wait capability entirely and cannot be
   used for time service.

5. If you have a solid 3D fix, a known-good cable, your software is
   properly configured, the wait ioctl succeeded, but you still get no
   PPS, then you might have a GPS that fails to deliver PPS off the
   chip to the RS232 or USB interface.  You get to become intimate
   with datasheets and pinouts, and might need to acquire a different
   GPS.

== Running GPSD ==

If you're going to use gpsd for time service, you must run in -n mode
so the clock will be updated even when no clients are active.

Note that gpsd assumes that after each fix the GPS will assert
1PPS first and ship sentences reporting time of fix second. Every
GPS we know of does things in this order.  If you ever encounter 
an exception, it should manifest as reported times that look like
they're from the future and require a negative fudge. If this
ever happens, please report the device make and model to the GPSD
maintainers so we can flag it in our GPS hardware database.

There is another possible cause of small negative offsets which
shows up on the GR-601-W: implementation bugs in your USB driver,
combining with quantization by the USB poll interval.  This 
doesn't mean the u-blox 6 inside it is actually emitting PPS 
after the GPS timestamp is shipped.

In order to present the smallest possible attack surface to
privilege-escalation attempts, gpsd, if run as root, drops its root
privileges very soon after startup - just after it has opened any
serial device paths passed on the command line.

Thus, KPPS can only be used with devices passed that way, not with
GPSes that are later presented to gpsd by the hotplug system.  Those
hotplug devices may, however, be able to use plain, non-kernel
PPS. gpsd tries to automatically fall back to this when absence of
root permissions makes KPPS unavailable.

In general, if you start gpsd as other than root, the following things
will happen that degrade the accuracy of reported time:

1. Devices passed on the command line will be unable to use KPPS and
will fall back to the same plain PPS that all hotplug devices must
use, increasing the associated error from ~1 uSec to about ~5 uSec.

2. gpsd will be unable to renice itself to a higher priority.  This
action helps protect it against jitter induced by variable system
load. It's particularly important if your NTP server is a general-use 
computer that's also handling mail or web service or development.

3. The way you have to configure ntpd and chrony will change away
from what we show you here; ntpd will need to be told different
shared-memory segment numbers, and chronyd will need a different
socket location.

You may also find gpsd can't open serial devices at all if your
OS distribution has done "secure" things with the permissions.
 
== Feeding NTPD from GPSD ==

Most Unix systems get their time service through ntpd, a very old and
stable open-source software suite which is the reference
implementation of NTP.  The project home page is <<NTP.ORG>>.

When gpsd receives a sentence with a timestamp, it packages the
received timestamp with current local time and sends it to a
shared-memory segment with an ID known to ntpd, the network time
synchronization daemon.  If ntpd has been properly configured to
receive this message, it will be used to correct the system clock.

When in doubt, the preferred method to start your timekeeping is:

-----------------------------------------------------------------------------
$ su - (or sudo -s )
# killall -9 gpsd ntpd
# ntpd -gN
# sleep 2
# gpsd -n /dev/ttyXX
# sleep 2
# cgps
-----------------------------------------------------------------------------

where /dev/ttyXX is whatever 1PPS-capable device you have.  In a
binary-package-based Linux distribution it is probable that ntpd
will already have been launched at boot time.

If you're using dhcp3-client to configure your system, make sure
you disable /etc/dhcp3/dhclient-exit-hooks.d/ntp, as dhclient would
restart ntpd with an automatically created ntp.conf otherwise - and
gpsd would not be able to talk with ntpd any more.

The rest of these setup instructions will assume that you are starting
gpsd as root, with occasional glances at the non-root case.

Now check to see if gpsd has correctly attached the shared-memory
segments in needs to communicate with ntpd.  ntpd's rules for the
creation of these segments are:

Segments 0 and 1:: 
	 Permissions are 0600 - other programs can only read and
	 write this segment when running as root.

Segments 2 and 3:: 
	 Permissions are 0666 - other programs can read
 	 and write as any user. If ntpd has been
 	 configured to use these segments, any
 	 unprivileged user is allowed to provide data
 	 for synchronisation.

Because gpsd can be started either as root or non-root, it checks and
attaches the more privileged segment pair it can - either 0 and 1 or 2
and 3.

For each GPS receiver that gpsd controls, it will use the attached ntpshm
segments in pairs (for coarse clock and pps source, respectively)
starting from the first found segments.

To debug, try looking at the live segments this way

-----------------------------------------------------------------------------
ipcs -m
-----------------------------------------------------------------------------

If gpsd was started as root, the results  should look like this:

-----------------------------------------------------------------------------
 ------ Shared Memory Segments --------
  key        shmid      owner      perms      bytes      nattch     status
  0x4e545030 0          root       700        96         2
  0x4e545031 32769      root       700        96         2
  0x4e545032 163842     root       666        96         1
  0x4e545033 196611     root       666        96         1
-----------------------------------------------------------------------------

For a bit more data try this:

-----------------------------------------------------------------------------
cat /proc/sysvipc/shm
-----------------------------------------------------------------------------

If gpsd cannot open the segments, check that you are not running SELinux
or apparmor. Either may require you to configure a security exception.

If you see the shared segments (keys 1314148400 -- 1314148403), and
no gpsd or ntpd is running then try removing them like this:

-----------------------------------------------------------------------------
ipcrm -M 0x4e545030
ipcrm -M 0x4e545031
ipcrm -M 0x4e545032
ipcrm -M 0x4e545033
-----------------------------------------------------------------------------

Here is a minimal sample ntp.conf configuration to work with GPSD run
as root, telling ntpd how to read the GPS notifications

-----------------------------------------------------------------------------
pool us.pool.ntp.org iburst

driftfile /var/lib/ntp/ntp.drift
logfile /var/log/ntp.log

restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
restrict 127.0.0.1 mask 255.255.255.0
restrict -6 ::1

# GPS Serial data reference
server 127.127.28.0 
fudge 127.127.28.0 time1 0.9999 refid GPS

# GPS PPS reference
server 127.127.28.1 prefer
fudge 127.127.28.1 refid PPS
-----------------------------------------------------------------------------

The number "0.9999" is a placeholder, to be explained shortly.  It
is *not a number to be used in production* - it's too large. If you
can't replace it with a real value, it would be best to leave out the 
clause entirely so the entry looks like

-----------------------------------------------------------------------------
fudge 127.127.28.0 refid GPS
-----------------------------------------------------------------------------

This is equivalent to declaring a time1 of 0.

The pool statement adds 4 chimers as additional time references needed
by ntpd for redundancy and to give you a reference to see how well
your local GPS is performing.  If you are outside of the USA replace
the pool servers with one in your local area. See <<USE-POOL>> for
further information.

The pool statement, and the driftfile and logfile declarations after it,
will not be strictly necessary if the default ntp.conf that your
distribution supplies gives you a working setup. The two pairs of
server and fudge declarations are the key.

ntpd can be used in Denial of Service (DoS) attacks.  To prevent that,
but still allow clients to requst the local time, be sure the restrict
statements are in your ntpd config file.  For more information see
<<CVE-2009-3563>>.

Users of ntpd versions older than revision ntp-4.2.5p138 should instead use
this ntp.conf, when gpsd is started as root:

-----------------------------------------------------------------------------
pool us.pool.ntp.org iburst

driftfile /var/lib/ntp/ntp.drift
logfile /var/log/ntp.log

restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
restrict 127.0.0.1 mask 255.255.255.0
restrict -6 ::1

# GPS Serial data reference
server 127.127.28.0 minpoll 4 maxpoll 4
fudge 127.127.28.0 time1 0.9999 refid GPS

# GPS PPS reference
server 127.127.28.1 minpoll 4 maxpoll 4 prefer
fudge 127.127.28.1 refid PPS
-----------------------------------------------------------------------------

Users of ntpd versions prior to ntp-4.2.5 do not have the "pool" option.
Alternative configurations exist, but it is recommended that you upgrade
ntpd, if feasible.

The magic pseudo-IP address 127.127.28.0 identifies unit 0 of the ntpd
shared-memory driver; 127.127.28.1 identifies unit 1.  Unit 0 is used
for in-band message timestamps and unit 1 for the (more accurate,
when available) time derived from combining in-band message timestamps
with the out-of-band PPS synchronization pulse.  Splitting these
notifications allows ntpd to use its normal heuristics to weight them.

(Different units - 2 and 3, respectively - must be used when gpsd is not
started as root.)

With this configuration, ntpd will read the timestamp posted by gpsd
every 64 seconds (16 if non-root) and send it to unit 0.  

The number after the parameter time1 (0.9999 in the example above) is a
"fudge", offset in seconds.  It's an estimate of the latency between
the time source and the 'real' time. You can use it to compensate out
some of the fixed delays in the system. An 0.9999 fudge would be
ridiculously large.

You may be able to find a value for the fudge by looking at the entry
for your GPS type on <<HARDWARE>>.  Later in this document we'll
explain methods for estimating a fudge factor on unknown hardware.

There is nothing magic about the refid fields; they are just labels
used for generating reports.  You can name them anything you like.

When you start gpsd, it will wait for a few good fixes before
attempting to process PPS.  You should run gpsmon or cgps to verify
your GPS has a 3D lock before worrying about timekeeping.

After starting (as root) ntpd, then gpsd, a listing similar to the one below
should appear as the output of the command "ntpq -p" (after allowing the
GPS to acquire a 3D fix).  This may take up to 30 minutes if your GPS
has to cold start or has a poor skyview.

-----------------------------------------------------------------------------
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
xtime-a.timefreq .ACTS.           1 u   40   64  377   59.228   -8.503   0.516
-bonehed.lcs.mit 18.26.4.106      2 u   44   64  377   84.259    4.194   0.503
+clock.sjc.he.ne .CDMA.           1 u   41   64  377   23.634   -0.518   0.465
+SHM(0)          .GPS.            0 l   50   64  377    0.000    6.631   5.331
-----------------------------------------------------------------------------

The line with refid ".GPS." represents the in-band time reports from
your GPS.  When you are getting PPS then it may look like this:

-----------------------------------------------------------------------------
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
xtime-a.timefreq .ACTS.           1 u   40   64  377   59.228   -8.503   0.516
-bonehed.lcs.mit 18.26.4.106      2 u   44   64  377   84.259    4.194   0.503
+clock.sjc.he.ne .CDMA.           1 u   41   64  377   23.634   -0.518   0.465
+SHM(0)          .GPS.            0 l   50   64  377    0.000    6.631   5.331
*SHM(1)          .PPS.            0 l   49   64  377    0.000    0.222   0.310
-----------------------------------------------------------------------------

Note the additional ".PPS." line.

If the value under "reach" for the SHM lines remains zero, check that
gpsd is running; cgps reports a 3D fix; and the '-n' option was used.
Some GPSes specialized for time service can report time with signal
lock on only one satellite, but with most devices a 3D fix is
required.

When the SHM(0) line does not appear at all, check your ntp.conf and
the system logs for error messages from ntpd.

Notice the 1st and 3rd servers, stratum 1 servers, disagree by more than
8 mSec.  The 1st and 2nd servers disagree by over 12 mSec.  Our local
PPS reference agrees to the clock.sjc.he.net server within the expected
jitter of the GR-601W in use.

When no other chimers or local reference clocks appear in the NTP
configuration, the system clock will lock onto the GPS clock, but this
is a fragile setup - you can lose your only time reference if the GPS
is temporarily unable to get satellite lock.

You should always have at least two (preferably four) fallback chimers
in your ntpd.conf for proper ntpd operation, in case your GPS fails to
report time.  The 'pool' command makes this happen.  And you'll need
to adjust the offsets (fudges) in your ntp.conf so the SHM(0) time is
consistent with your other chimers (and other local reference clocks,
if you have any). We'll describe how to diagnose and tune your chimer
configuration in a later section.

Also note that after cold-starting ntpd it will calibrate for up to 15
minutes before it starts adjusting the clock. Because the frequency
error estimate ("drift") that NTP uses isn't right when you first
start NTP, there will be a phase error that persists while the
frequency is estimated.  So if your clock is a litle slow, then it
will keep getting behind, and the positive offset will cause NTP to
adjust the phase forward and also increase the frequency offset error.
After a day or so or maybe less the frequency estimate will be very
close and there won't be a persistent offset.

The gpsd developers would like to receive information about the
offsets (fudges) observed by users for each type of receiver. If your
GPS is not present in <<HARDWARE>>, or doesn't have a recommended
fudge, or you see a fudge value very different from what's there,
send us the output of the "ntpq -p" command and the make and type of
receiver.

== Feeding chrony from GPSD ==

chrony is an alternative open-source implementation of NTP service,
originally designed for systems with low-bandwidth or intermittent
TCP/IP service.  It interoperates with ntpd using the same NTP
protocols.  Unlike ntpd which is designed to always be connected to
multiple internet time sources, chrony is designed for long periods
of offline use.  Like ntpd, it can either operate purely as a client
or provide time service. The chrony project has a home page at
<<CHRONY>>. Its documentation includes an instructive feature comparison
with ntpd at <<CHRONY-COMPARE>>.

gpsd, when run as root, feeds reference clock information to chronyd
using a socket named /var/run/chrony.ttyXX.sock (where ttyXX is
replaced by the GPS device name.  This allows multiple GPS to feed one
chronyd.

No gpsd configuration is required to talk to chronyd. chronyd is
configured using the file /etc/chrony.conf or /etc/chrony/chrony.conf.
Check your distributions documentation for the correct location.  To get
chronyd to connect to gpsd using the basic ntpd compatible SHM method
add this to use this basic chrony.conf file:

-----------------------------------------------------------------------------
server 0.us.pool.ntp.org
server 1.us.pool.ntp.org
server 2.us.pool.ntp.org
server 3.us.pool.ntp.org

driftfile /var/lib/chrony/drift

allow

# set larger delay to allow the NMEA source to overlap with
# the other sources and avoid the falseticker status
refclock SHM 0 offset 0.9999 delay 0.2 refid GPS
refclock SHM 1 refid PPS
-----------------------------------------------------------------------------

If you are outside of the USA replace the pool servers with one in your
local area. See <<USE-POOL>> for further information.

The offset option is functionally like ntpd's time1 option, used to 
correct known and constant latency.

The allow option allows anyone on the internet to query your server's
time.

To get chronyd to connect to gpsd using the more precise socket 
method add this to your chrony.conf file (replacing ttyXX
with your device name).  

If running as root:

-----------------------------------------------------------------------------
server 0.us.pool.ntp.org
server 1.us.pool.ntp.org
server 2.us.pool.ntp.org
server 3.us.pool.ntp.org

driftfile /var/lib/chrony/drift

allow

# set larger delay to allow the NMEA source to overlap with
# the other sources and avoid the falseticker status
refclock SHM 0 offset 0.9999 delay 0.2 refid GPS
refclock SOCK /var/run/chrony.ttyXX.sock refid PPS
-----------------------------------------------------------------------------

If not running as root change the "refclock SOCK" line to:

-----------------------------------------------------------------------------
refclock SOCK /tmp/chrony.ttyXX.sock refid PPS
-----------------------------------------------------------------------------

See the chrony man page for more detail on the configuration options
<<CHRONY-MAN>>.

Finally note that chronyd needs to be started before gpsd so the
socket is ready when gpsd starts up.

If running as root, the preferred starting procedure is:

-----------------------------------------------------------------------------
$ su - (or sudo -s )
# killall -9 gpsd chronyd
# chronyd -f /etc/chrony/chrony.conf
# sleep 2
# gpsd -n /dev/ttyXX
# sleep 2
# cgps
-----------------------------------------------------------------------------

After you have verified with cgps that your GPS has a good 3D lock you
can check how chrony is doing by running 'chronyc sources'.  Your output
will look like this:

-----------------------------------------------------------------------------
# chronyc sources

210 Number of sources = 7
MS Name/IP address         Stratum Poll Reach LastRx Last sample
===============================================================================
#- GPS                           0   4   377    12  +3580us[+3580us] +/- 101ms
#* PPS                           0   4   377    10    -86ns[ -157ns] +/- 181ns
^? vimo.dorui.net                3   6   377    23   -123ms[ -125ms] +/- 71ms
^? time.gac.edu                  2   6   377    24   -127ms[ -128ms] +/- 55ms
^? 2001:470:1:24f::2:3           2   6   377    24   -122ms[ -124ms] +/- 44ms
^? 142.54.181.202                2   6   377    22   -126ms[ -128ms] +/- 73ms
-----------------------------------------------------------------------------

The stratum is as in ntpq.  The Poll is how many seconds elapse between samples.
The Reach is as in ntpq. LastRx is the time since the last successful
sample.  Last sample is the offset and jitter of the source.

To keep chronyd from preferring NMEA time over PPS time, you can add an 
overlarge fudge to the NMEA time.  Or add the suffix 'noselect' so it
is never used, just monitored.

== Performance Tuning ==

This section is general and can be used with either ntpd or chronyd.
We'll have more to say about tuning techniques for the specific 
implementations in later sections.

The clock crystals used in consumer electronics have two properties we
are interested in: accuracy and stability.  *Accuracy* is how well the
measured frequency matches the number printed on the can.  *Stability*
is how well the frequency stays the same even if it isn't accurate.
(Long term aging is a third property that is interesting, but ntpd and
chrony both a use a drift history that is relatively short; thus,
this is not a significant cause of error.)

Typical specs for oscillator packages are 20, 50, 100 ppm.  That includes
everything; initial accuracy, temperature, supply voltage, aging, etc.

With a bit of software, you can correct for the inaccuracy.  ntpd and
chrony both call it *drift*.  It just takes some extra low order bits
on the arithmetic doing the time calculations.  In the simplest case,
if you thought you had a 100 MHz crystal, you need to change that to
something like 100.000324. The use of a PPS signal from gpsd
contributes directly to this measurement.

Note that a low drift contributes to stability, not necessarily accuracy.

The major source of instability is temperature.  Ballpark is the drift
changes by 1 PPM per degree C.  This means that the drift does not stay
constant, it may vary with a daily and yearly pattern.  This is why the
value of drift the ntpd uses is calculated over a (relatively) short time.

So how do we calculate the drift?  The general idea is simple.
Measure the time offset every N seconds over a longer window of time
T, plot the graph, and fit a straight line.  The slope of that line is
the drift.  The units cancel out.  Parts-per-million is a handy scale.

How do you turn that handwaving description into code?  One easy way
is to set N=2 and pick the right T, then run the answer through a
low pass filter.  In that context, there are two competing sources of
error.  If T is too small, the errors on each individual measurement
of the offset time will dominate.  If T is too big, the actual drift
will change while you are measuring it.  In the middle is a sweet
spot.  (For an example, see <<ADEV-PLOT>>.)

Both ntpd and chrony use this technique; ntpd also uses a more
esoteric form of estimation called a "PLL/FLL hybrid loop". How T and N are
chosen is beyond the scope of this HOWTO and varies by implementation
and tuning parameters.

If you turn on the right logging level ("statistics loopstats peerstats"
for ntpd, "log measurements tracking" for chronyd) , that will record
both offset, drift, and the polling interval. The ntpd stats are easy to
feed to gnuplot, see the example script in the GPSD contrib directory.
The most important value is the offset reported in the 3rd field in
loopstats and the last field in tracking.log. With gnuplot you can
compare them (after concatenating the rotated logs):

-----------------------------------------------------------------------------
plot "tracking.log" using 7 with lines, "loopstats" using 3 with lines
-----------------------------------------------------------------------------

While your NTP daemon (ntpd or chrony) is adjusting the polling
interval, it is assuming that the drift is not changing.  It gets
confused if your drift changes abruptly, say because you started some
big chunk of work on a machine that's usually idle and that raises the
temperature.

Your NTP daemon writes out the drift every hour or so.  (Less often if
it hasn't changed much to reduce the workload on flash file systems.)
On startup, it reloads the old value.

If you restart the daemon, it should start with a close old drift
value and quickly converge to the newer slightly different value.  If
you reboot, expect it to converge to a new/different drift value and
that may take a while depending on how different the basic calibration
factors are.

== NTP tuning and performance details ==

This section deals specifically with ntpd.  It discusses algorithms
used by the ntpd suite to measure and correct the system time.  It is not
directly applicable to chronyd, althogh some design considerations
may be similar.

=== NTP performance tuning ===

For good time stability, you should always have at least four other
chimers in your ntpd or chrony configuration besides your GPS
- in case, for example, your GPS is temporarily unable to achieve
satellite lock, or has an attack of temporarity insanity. You can find
public NTP chimers to add to your configuration at <<USE-POOL>>. 

To minimize latency variations, use the national and regional
pool domains for your country and/or nearby ones.  Your ntp.conf
configuration line should probably look like this

-----------------------------------------------------------------------------
pool us.pool.ntp.org iburst
-----------------------------------------------------------------------------

where "us" may be replaced by one of the zone/country codes the Pool
project supports (list behind the "Global" link at <<ZONES>>). The
"pool" tag expands to four randomly chosen servers by default.  "iburst"
implements a fast start algorithm that also weeds out bad servers.

Note that a chimer can be a poor performer for any of three
reasons. It may be shipping bad time, or the best routes between you
and it have large latency variations (jitter), or it may have an asymetric
route, confusing NTP.

The standard tool for tuning ntpd is "ntpq" ("NTP query program"). To
show a list of all servers declared in ntp.conf and their statistics,
invoke it with the "-p" option. On a sample system configured with 7
servers from the NTP pool project and one PPS GPS attached via RS232,
this is the output:

------------------------------------------------------------------------
$ ntpq -p
 remote          refid         st t when poll reach delay offset jitter
========================================================================
-arthur.testserv 162.23.41.56   2 u 62     64 377  5.835 -1.129   8.921
-ntppublic.uzh.c 130.60.159.7   3 u 62     64 377  6.121 -4.102   6.336
-smtp.irtech.ch  162.23.41.56   2 u 35     64 377 15.521 -1.677   8.678
+time2.ethz.ch   .PPS.          1 u 27     64 377  5.938 -1.713  16.404 
-callisto.mysnip 192.53.103.104 2 u 53     64 377 49.357 -0.274   5.125 
-shore.naturalne 122.135.113.81 3 u 22     64 377 14.676 -0.528   2.601 
-ntp.univ-angers 195.220.94.163 2 u 41     64 377 40.678 -1.847  13.391 
+SHM(0)          .GPS.          0 l  4     64 377  0.000 34.682   7.952 
*SHM(1)          .PPS.          0 l  3     64 377  0.000 -2.664   0.457
------------------------------------------------------------------------

The interesting columns are "remote", "st", "reach" and "offset".

"remote" is the name of the remote NTP server. The character in its
first column shows its current state: "-" or "x" for out-of-tolerance
servers, "+" for good servers ("truechimers"), and "*" for the one good
server currently used as the primary reference. The calculations used to
determine a server's state are outside the scope of this document;
details are available in NTPv4 RFC 5905.

"st" shows the remote server's stratum.

"reach" is the octal representation of the remote server's reachability.
A bit is set if the corresponding poll of the server was successful,
i.e. the server returned a reply. New poll results are shifted in from
the least significant bit; results older than 8 polls are discarded. In
the absence of network problems, this should show "377".

"offset" shows the mean offset in the times reported between this local
host and the remote server in milliseconds. This is the value that can
be fudged with the "time1" parameter of the GPS server line in ntp.conf.
If the offset is positive, reduce the time1 value and vice versa.

The asterisk in this example indicates that ntpd has correctly
preferred .PPS.  over .GPS., as it should.  If for some reason it
locks on to GPS time as a preferred source, you can add an overlarge
fudge to the NMEA time to discourage it.  Or add the suffix 'noselect'
so GPS time is never used, just monitored.

A more detailed description of the output is available at
<<NTPQ-OUTPUT>>.

In order to determine the correct GPS offset, do the following:

[start=1]
1. Add these lines to ntp.conf:

-----------------------------------------------------------------------------
statsdir /var/log/ntpstats/
statistics peerstats
filegen peerstats file peerstats type day enable
-----------------------------------------------------------------------------

This enables logging of the peer server statistics.

[start=2]
2. Make sure the directory exists properly.  For ntpd as root do:

-----------------------------------------------------------------------------
   # mkdir -p /var/log/ntpstats
   # chown ntp:ntp /var/log/ntpstats
-----------------------------------------------------------------------------

[start=3]
3. Start ntpd and let it run for at least four hours.
Periodically check progress with "ntpq -p" and wait
until change has settled out.

[start=4]
4. Calculate the average GPS offset using this script (a copy is
included as contrib/ntpoffset in the GPSD distribution):

-----------------------------------------------------------------------------
awk '
     /127\.127\.28\.0/ { sum += $5 * 1000; cnt++; }
     END { print sum / cnt; }
' </var/log/ntpstats/peerstats
-----------------------------------------------------------------------------

This prints the average offset.

[start=5]
5. Adjust the "time1" value for unit 0 of your ntp.conf (the non-PPS
   channel) by subtracting the average offset from step 4.

6. Restart ntpd.

== Chrony peformance tuning

The easiest way to determine the offset with chronyd is probably to
configure the source whose offset should be measured with the noselect
option and a long poll, let chronyd run for at least 4 hours and
observe the offset reported in the chronyc sourcestats output.  If the
offset is unstable, wait longer.  For example:

SHM 0 configured as:
refclock SHM 0 poll 8 filter 1000 noselect

-----------------------------------------------------------------------------
# chronyc sourcestats
210 Number of sources = 6
Name/IP Address            NP  NR  Span  Frequency  Freq Skew  Offset  Std Dev
==============================================================================
SHM0                       21   9   85m      4.278      4.713   +495ms  8896us
SHM1                       20   8   307      0.000      0.002     +0ns   202ns
mort.cihar.com             21   8   72m      0.148      0.798   +668us   490us
vps2.martinpoljak.net       6   4   17m    -53.200    141.596    -24ms    15ms
ntp1.kajot.cz              25  16   91m     -0.774      1.494    -11ms  1859us
ntp1.karneval.cz           17  10   89m      0.127      0.539  -4131us   574us

In this case (Garmin 18x) the offset specified in the config for the
SHM 0 source should be around 0.495.
-----------------------------------------------------------------------------

//FIXME: What more can we say about chronyd tuning?

== Providing public NTP service ==

<<NTP-FAQ>> has good advice on things to be sure you have done - and
are ready to do - before becoming a public chimer. One detail it
doesn't mention is that you'll need to un-firewall UDP port 123.  The
NTP protocol does not use TCP, so no need to unblock TCCp port 123.

If and when you are ready to go public, see <<JOIN-POOL>>. 

== Acknowledgments ==
Beat Bolli <bbolli@ewanet.ch> wrote much of the section on NTP
performance tuning. Hal Murray <hmurray@megapathdsl.net> wrote
much of the section on NTP working and performance details.
Sanjeev Gupta <ghane0@gmail.com> assisted with editing.
Shawn Kohlsmith <skohlsmith@gmail.com> tweaked the Bibliography.

== References ==

[bibliography]
- [[[WIKI-NTP]]] http://en.wikipedia.org/wiki/Network_Time_Protocol[Network Time Protocol]

- [[[NTP-FAQ]]] http://www.ntp.org/ntpfaq/[NTP FAQ]

- [[[RFC-5905]]] http://www.ietf.org/rfc/rfc5905.txt[Network Time Protocol Version 4: Protocol and Algorithms Specification]

- [[[SIRF-WOBBLE]]] http://www.megapathdsl.net/~hmurray/ntp/GPSSiRF-off.gif[Peer Offset of SiRF units]

- [[[RFC-2783]]] http://tools.ietf.org/html/rfc2783[RFC 2783]

- [[[CHRONY-COMPARE]]] http://chrony.tuxfamily.org/manual.html#Comparison-with-ntpd[ntpd (comparison with chrony)]

- [[[CHRONYDEFAULT]]] https://lists.fedoraproject.org/pipermail/devel/2010-May/135679.html

- [[[HARDWARE]]] http://catb.org/gpsd/hardware.html[Compatible Hardware]

- [[[UBLOX-TIMING]]] http://www.u-blox.com/images/downloads/Product_Docs/Timing_AppNote_%28GPS.G6-X-11007%29.pdf[GPS-based timing considerations with u-blox 6 receivers]

- [[[RPI]]] http://www.satsignal.eu/ntp/Raspberry-Pi-NTP.html[The Raspberry Pi as a Stratum-1 NTP Server]

- [[[NTP.ORG]]] http://www.ntp.org/[Home of the Network Time Protocol project]

- [[[USE-POOL]]] http://www.pool.ntp.org/en/use.html[How do I use pool.ntp.org?]

- [[[CVE-2009-3563]]] http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2009-3563

- [[[CHRONY]]] http://chrony.tuxfamily.org/[Chrony Home]

- [[[CHRONY-MAN]]] http://chrony.tuxfamily.org/manual.html

- [[[ADEV-PLOT]]] http://www.leapsecond.com/pages/adev-avg/[Allan deviation and Averaging]

- [[[ZONES]]] http://www.pool.ntp.org/zone

- [[[NTPQ-OUTPUT]]] http://nlug.ml1.co.uk/2012/01/ntpq-p-output/831[ntpq output description]

- [[[JOIN-POOL]]] http://www.pool.ntp.org/en/join.html[How do I join pool.ntp.org?]

== Changelog ==

1.1, Nov 2013: Initial release.

1.2, Aug 2014: Note that NetBSD now has PPS support.

1.3, Aug 2014: Add a note about the GR-601W.

1.4, Dec 2014: Cleaned up Bibliography

Repository: More about troubleshooting PPS delivery.