On each DSM, the lsu command displays the last 10 archive files. On the flux system, the lsu command runs lsu on each of the tower DSMs in succession. After 0Z, the rsync scripts copy and remove the previous day's files, so you will generally see only files for the current day. The modification time of the last file shown for each DSM should be the current time in UTC. If run a second time the size of the file should have grown.
On flux, you can run the cockpit command from a terminal window, to monitor the real-time data. A popup window will appear. Select "Search" to connect to the server on the localhost host, at port 30005.
cockpit will then create a set of tabbed pages, each containing small time-series plots for each sampled variable of a DSM. Each second a vertical line is drawn from the minimum to the maximum value of the variable over that second. When the trace reaches the right-hand-side of the plot, the trace is greyed out, and a new trace begun. If a sensor is not reporting, RIP will be displayed on the plot. The greyed-out traces provide a history of the data. The history can be cleared by selecting GlobalSetup -> Color -> Cleanup History.
cockpit can be configured to cycle through the tabbed pages via GlobalSetup -> AutoCycleTabs.
The post processing of CABL data is done on the flux laptop at the BAO tower, and on porter2 at EOL. The systemd service manager of RedHat Linux is used to start and monitor the services which run the various processing steps. On flux, the processes are running under the aster userid. On porter2 they are running under user maclean. They are started automatically by systemd at bootup.
To check the status of the services, use the sstat command. It displays a tree of the various services, followed by an indication of "all services seem to be running", or it will list the missing services.
If a process of a service isn't running, look at the system log file, /var/log/isfs/isfs.log to help track down the problem. Many of the scripts run by the services listed below also write to log files on $ISFF/projects/CABL/ISFF/logs.
On flux, the services are:
On porter2:
sstat will also show rsync_loop and statsproc@trh_test services on porter2. Those are running in support of the CentNet project.
If you make a change to the XML or a calibration file, you will usually want to restart the real-time statsproc processes. Only if an XML change effects the archive of the raw data do you need to restart dsm_server on flux.
To restart the statsproc processes on flux or porter2, use the restart_statsproc command. On flux it does a systemctl --user restart of the two statsproc services.
On porter2 the processes are running under the maclean login, and only that user has permission to restart the services. As a work-around, restart_statsproc writes a string to the file $ISFF/projects/$PROJECT/ISFF/logs/restart_proc.txt. The proc_restarter service wakes up every 10 seconds, checks that file, and if it contains the string "statsproc", does a systemctl --user restart on the four statsproc services.
Or you can use the command restart_service to restart any service running for CABL. You will be prompted to choose a service, by number, and that service will be restarted in a similar way to restart_statsproc.
To recalculate the statistics for the whole project, run this command on an EOL server (porter2, barolo, tikal), after setting your project to CABL:
statsproc -S qc_geo_notiltcor -B "2015 feb 18 00:00" -E "2015 jun 1 00:00"
If you want to recalculate the noqc_instrument dataset, set the -S option accordingly.
The value of the NC_SERVER environment variable should be "porter2" so that the data is sent to nc_server on porter2.
On EOL systems, the default value of the DATADIR environment variable should be "merge", in which case statsproc will process all files on /scr/isfs/projects/CABL/merge. If you want to process a different set of files, for example the 50m files:
cd /scr/isfs/projects/CABL/raw_data
statsproc -S qc_geo_notiltcor 50m*
To have the R_derived service recalculate the derived variables for the entire project period the next time it runs, remove this file:
rm $ISFF/projects/CABL/ISFF/logs/R_derived_last.txt
.