This is the landing page for the Infrastructure team's knowledge base. The goal is to provide one centralized location for procedures, information, templates, troubleshooting, and FAQ. For now, we will try and sort information based on overall subject.

Contents:


Misc Short Topics

Content here is not extensive enough to have a dedicated topical page.

UCAR Hosted Physical Machines

Right now we have one physical machine hosted by UCAR in the Mesa Lab Data Center. See ticket HELP-36848 for more information on the setup process. This ticket can be re-opened to address fixes or permission changes on the device. Device passwords, user accounts, and SSH keys are managed by the JEDI Infra team.

Device power administration is done via the Morocco web tool. This must be accessed on the UCAR VPN and you need to be granted permission for each machine. Use the "Sites" tab and navigate by datacenter → cabinet → pod to find the machine's control dashboard.

AKA the ML (or Mesa Lab) Computer

We are using this machine, as of fall '25, to run an experimental Near-Real Time (NRT) system called "obsbench". This system contains 3 main components: ingest, model experiments, and post-processing. This section briefly documents those components and how they come together for our product. We are using a shared obsbench  user to create and run the NRT experiments. Files and configuration are stored in obsbench@ucar-precision-7875-tower:~/nrtdev , you can set up the correct environment by running source ~/nrtdev/setup.sh which will include cylc installed in your venv.

Ingest - observations and model ingest

HofX/ NGCM - Two experiments for F32 and F64

To check the running experiments: 

ssh -Y <user.name>@precision-tower-7875.nrit.ucar.edu
sudo -iu obsbench     # Talk to Ashley, Evan, or Travis if you need to be added to the user group
source ~/nrtdev/setup.sh
cylc tui              # Use the arrow keys to open the experiments and navigate