...
To check the running experiments:
| Code Block | ||||
|---|---|---|---|---|
| ||||
ssh -Y <user.name>@precision-tower-7875.nrit.ucar.edu sudo -iu obsbench # Talk to Ashley, Evan, or Travis if you need to be added to the user group source ~/nrtdev/setup.sh cylc tui # Use the arrow keys to open the experiments and navigate |
Monitoring is set up under the obsbench user to check the current running workflows for task failures. If a task fails, a slack message will be sent to the mvp channel. This script runs via cron after the daily experiments complete. Monitoring scripts and logs can be found at /home/obsbench/nrtdev/monitoring