Page History

...

• Device: JCSDA-PRECISION-TOWER-7875-U1
  • OS: Ubuntu 24.04 
  • SSH Address: 192.168.19.115 (must be on UCAR VPN)
  • Physical location:  MLDC - cabinet S5 - POD 2

AKA the ML (or Mesa Lab) Computer

We are using this machine, as of fall '25, to run an experimental Near-Real Time (NRT) system called "obsbench". This system contains 3 main components: ingest, model experiments, and post-processing. This section briefly documents those components and how they come together for our product. We are using a shared obsbench  user to create and run the NRT experiments. Files and configuration are stored in obsbench@ucar-precision-7875-tower:~/nrtdev , you can set up the correct environment by running source ~/nrtdev/setup.sh which will include cylc installed in your venv.

Ingest - observations and model ingest

• Observation ingest
  • Obs: gdas_prepbufr_ingest, ssec_amv_ingest, and gdas_amsua_ingest.yaml
  • Configuration location: /home/obsbench/nrtdev/configs/ingest-observations.yaml 
  • Current workflow engine: cylc
  • Runs continuously on a day delay for 6 hour cycles
• NGCM Backgrounds (F32 and F64)
  • Files: F32 and F64 Analysis
  • Configuration location(s): /home/obsbench/nrtdev/configs/ngcm-backgroundF32.yaml and /home/obsbench/nrtdev/configs/ngcm-backgroundF64.yaml
  • Current workflow engine: cylc
  • Runs continuously on a 7 day delay, once daily for a 1 day cycle
• Re-ingest of prepbufr
  • Obs: gdas_prepbufr_ingest
  • Configuration location: /home/obsbench/nrtdev/configs/reingestPrepbufr.yaml 
  • Current workflow engine: cylc
  • Runs continuously on a two day delay for 6 hour cycles
  • Note, this was added to get the updated 48h prepburf files that contains a lot more data than the NRT prepbufr files

HofX/ NGCM - Two experiments for F32 and F64

• Resolutions: low and medium
• Observations: radiosonde, aircraft, satwinds, amsua metop-b, amsua metop-c
• Configuration location(s): /home/obsbench/nrtdev/configs/ngcm-obsbenchF32.yaml and /home/obsbench/nrtdev/configs/ngcm-obsbenchF64.yaml
• Current workflow engine: cylc
• Runs continuously on a 8 day delay (plus one hour delay for F32 and two hour delay for F64 due to memory constraints) for a 7 day forecast, once daily 

To check the running experiments: 

ssh -Y <user.name>@precision-tower-7875.nrit.ucar.edu
sudo -iu obsbench     # Talk to Ashley, Evan, or Travis if you need to be added to the user group
source ~/nrtdev/setup.sh
cylc tui              # Use the arrow keys to open the experiments and navigate

Monitoring is set up under the obsbench  user to check the current running workflows for task failures. If a task fails, a slack message will be sent to the mvp channel. This script runs via cron after the daily experiments complete. Monitoring scripts and logs can be found at /home/obsbench/nrtdev/monitoring