Monitoring the accuracy of the system clock on a real-time data acquisition system provides useful information about the performance of the system. Hence this long discussion.
Data System Clock, NTP and GPS
The turbulence tower data system at the Manitou Forest Observatory (aka, the DSM) uses a GPS receiver and the NTP (Network Time Protocol) software to set the system clock, which, in addition to the normal uses of a system clock, is used to time-tag the data samples.
The serial messages from the GPS are received on serial port 3, /dev/ttyS3. The pulse-per-second square-wave signal (PPS) from the GPS is also connected to the CTS DCD line of that serial port. A patch has been added to the Linux kernel on the data system so that an interrupt function can be registered to run in response to the CTS DCD interrupts. This interrupt function will be called immediately after the rising edge of the PPS signal has been detected by the serial port hardware.
...
NTP on the DSM is configured to log its status in a "loopstats" file. See http://www.eecis.udel.edu/~mills/ntp/html/monopt.html
for information on the NTP monitoring options. The loopstats file includes these variables, which have been merged into the Manitou data archive:
- NTPClockOffset: the estimated offset of the GPS time from the data system time. A positive value indicates that NTP is estimating has determined that the GPS clock is ahead of the system clock, i.e. the GPS is showing a later time than the system clock. The maximum, minimum and mean values of NTPClockOffset in each 5 minute period are computed and written to the NetCDF files and plotted as NTPClockOffset_max, NTPClockOffset_min and NTPClockOffset.
- NTPFreqOffset: the correction applied to the system clock frequency in parts-per-million, a . A positive value indicates that NTP has determined that the system clock oscillator is slow and the frequency offset is NTPFreqOffset PPM values are being added periodically to the system clock valuescounter. The NetCDF files and plots contain 5 minute means of NTPFreqOffset.
...
On April 12, 2011 the old Garmin GPS 25-HVS at the tower was replaced with a much newer Garmin 18x-LVC model. The model numbers are shown in the $PGRMT messages in the archive, where the time is UTC:
| Code Block |
|---|
data_dump -i 1,30 -A manitou_20110412_120000.bz2 | fgrep GPSPGRMT ... 2011 04 12 16:41:39.6568 0.15 49 $PGRMT,GPS 25-HVS VER 2.50 ,P,P,R,R,P,,23,R*08\r\n 2011 04 12 16:42:50.4248 0.1249 51 $PGRMT,GPS 18x-LVC software ver. 3.10,,,,,,,,*6D\r\ |
...
The following plot is for the old 25-HVS model for 3 days prior to the swap:
The NTPClockOffset ranges from approximately shows spikes between -100000 to 50000 microseconds during this period, which is much worse than expected for a GPS/NTP reference clock. The spikes in NTPClockOffset are simultaneous with positive jumps in GPSdiff_max, up to as much as 2.5 seconds. These jumps in GPSdiff_max events seem to happen when the number of tracked satellites changes, indicating which indicates that internal processing lags in the 25-HVS cause it to report late. It is unknown if , causing large values of GPSdiff. The extent of this effect on the timing of the PPS signal is effected by these eventsunknown.
The following plot shows a close up of one of the clock offset spikes using un-averaged data:
At these moments, NTP estimates The sudden downward jump in NTPClockOffset causes NTP to think that the GPS clock is earlier than the system clock, and . NTP starts to correct for the error offset by slowing down the system clock, due to as seen in the negative spikes in NTPClockOffsetvalues for NTPFreqOffset. When the GPS recovers from its delayed reporting, NTP then sees positive values for NTPClockOffset and adjusts the system clock ahead.
After installing 18x-LVC, the NTPClockOffset is in a much improved range, from -70 to 25 microseconds:
. NTPFreqOffset is also in a much tighter range, indicating that NTP is applying smaller corrections to the system clock. GPSdiff is also much better behaved, ranging from a minimum of 0.5 to 1.1 seconds. The number of satellites tracked by the new GPS is also generally higher.
Temperature Effects
As expected, the The frequency offset shows a temperature dependence in the system clock oscillator. We do not measure the temperature inside the data system boxenclosure, which is at the base of the tower. The nearest temperature measurement is of the ambient air at 2 meters on up the tower. The top panel in the plot below shows a time series of the air temperature, along with NTPFreqOffset, for a cool 3 day period in April, after the installation of the new GPS. It appears that when the air temperature is below 5 deg C, the system clock oscillator does not show an obvious temperature relation.
The bottom panel shows a close relationship between the NTPClockOffset and the time derivative of NTPFreqOffset, which, I believe, indicates how NTP adjusts the system clock based on the measured offset. It also enforces the obvious conclusion that we could improve the time-keeping by insulating the CPU from temperature changes.
On a warmer 3 day period in July, where the temperatures were all above 5 degC, the temperature effect on the system clock oscillator is very evident.
...
The periodic spikes in GPSdiff_max up to 1 second that occur at 23:00 local time and last about an hour, are simultaneous with the network transfer of the day's data files from the DSM to the RAL server. These suggest that increased sampling sample buffering and latency is happening at these times, which needs to be investigated and improved.
...
In July, the clock behaviour during the file transfer is similar, but the initial increase in NTPClockOffset and a rising slope in NTPFreqOffset , might be due to increased heating of the system clock oscillator, due to increased CPU load during the file transfers. Wild conjecture? After a quick scan of the web plots of 5 minute averages, I think these positive bumps in NTPFreqOffset seem to occur during file transfers when the outside air temperatures are above 0 C, and don't occur in colder conditions.
ppstest and ntpq
On the DSM, the ppstest program is helpful for gaining an understanding of the system and GPS clocks. It displays the system clock value when the interrupt function is called at the time of the assertion and the clear of the PPS signal. Do ctrl-C to terminate ppstest.
...
The above sequence shows that the system clock is behind the GPS. The system time when the interrupt function is being called on the PPS assert is 5 microseconds before the exact second (1.0 - 0.999995). This corresponds to a NTPClockOffset of a positive 5 microseconds. This ss is confirmed with the ntpq program (which reports its offset in milliseconds):
| Code Block |
|---|
ntpd -p
root@manitou root# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
xral 38.229.71.1 3 u 34 64 377 0.320 3.804 0.031
LOCAL(0) .LOCL. 10 l 93d 64 0 0.000 0.000 0.000
oGPS_NMEA(0) .GPS. 2 l 6 16 377 0.000 0.005 0.031
|
...