...
- nc_server
- cabl_flab_statsproc@qc_geo_notiltcor: computes statistics from the 300m tower for the qc_geo_notiltcor dataset, i.e. the files in netcdf_geo_notiltcor
- cabl_flab_statsproc@noqc_instrument: computes statistics from the 300m tower for the noqc_instrument dataset, i.e. the files in netcdf_noqc_instrument
- cabl_flab_statsproc2@qc_geo_notiltcor: computes statistics from the bao and ehs flux stations for the qc_geo_notiltcor dataset, i.e. the files in netcdf_geo_notiltcor
- cabl_flab_statsproc2@noqc_instrument: computes statistics from the bao and ehs flux stations for the noqc_instrument dataset, i.e. the files in netcdf_noqc_instrument
- rsync_flab: runs rsync_loop_flab.sh script, which wakes up periodically and rsync's files from flux, then does merge_nightly_flab.sh to merge and reprocess the previous day's files.
- R_derived
- proc_restarter: runs every 10 seconds to check if a user has requested to restart any services
sstat will also show rsync_loop and statsproc@trh_test services on porter2. Those are running in support of the CentNet project.
Note that NetCDF files of 5 minute statistics are being created independently on flux and on porter2. The files on flux may not be really needed, in which case the statsproc and R_derived services on flux could be disabled. The files created by porter2 are used by Ncharts and are rsync'd periodically to ftp://ftp.eol.ucar.edu/pub/archive/isff/projects/cabl.
Restart real-time service (restart_service, restart_statsproc)
...
On EOL systems, the default value of the DATADIR environment variable should be "merge", in which case statsproc will process all files on /scr/isfs/projects/CABL/merge. If you want to process a different set of files, for example you can pass the list of files instead of the start and end time. For example, the 50m files:
cd /scr/isfs/projects/CABL/raw_data
...
rm $ISFF/projects/CABL/ISFF/logs/R_derived_last.txt
Time keeping
The DSMs each have a GPS with a pulse-per-second signal. Using NTP reference clock software, each DSM is then a stratum 1 time server. NTP on the DSM uses the GPS reference clock to adjust the CPU system clock, and generally reports that the GPS reference clock has less than a 50 micro-second offset from the system clock.
To query the system clock on a tower DSM from flux, use the ntpq -p command, for example 50m:
ntpq -p 50m
remote refid st t when poll reach delay offset jitter
==============================================================================
LOCAL(0) .LOCL. 10 l 20d 64 0 0.000 0.000 0.000
oGPS_NMEA(0) .GPS. 0 l 5 16 377 0.000 -0.001 0.031
The above shows the GPS reference clock is offset from system CPU clock by -0.001 milliseconds. The "reach" value for GPS_NMEA should be 377 (octal value of all 1's). The reach for the LOCAL clock is always 0.
flux is configured to use all 6 DSMs as network time servers, using chrony, a NTP client. To display the current chrony status, use chronyc sources:
chronyc sources
210 Number of sources = 6
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^+ 50m 1 10 377 558 -298ns[+5000ns] +/- 742us
^+ 100m 1 10 377 179 +52us[ +52us] +/- 662us
^+ 150m 1 10 377 219 +2000ns[+2000ns] +/- 739us
^* 200m 1 10 377 338 +9656ns[ +15us] +/- 671us
^+ 250m 1 10 377 439 +5677ns[ +11us] +/- 732us
^+ 300m 1 10 377 95 +11us[ +11us] +/- 794us
The above shows that the clock on flux agrees to within a maximum 52 microseconds with each DSM, as indicated by the "Last sample" value in brackets, which is the last calculated offset of the reference clock (in this case the DSM) from the system time on flux. A positive offset means the reference clock is ahead of the system clock.
The second character, under "S" should be '*' (indicating chrony on flux is sync'd to this server) or '+' (good server). You may also see '-' (recently on 100m for some reason) indicating chrony does not have a high opinion of its time information, relative to the others.
The "reach" values should again be 377. If not, it means the DSM is not on the network, or its NTP server is not responding or sync'd to its GPS.
One can also ssh into bao or ehs and check their clocks with "ntpq -p".