...

• Make sure your code is up to date.
• Try deleting your old venv and starting with fresh installs of solo, r2d2, ewok, and simobs.
• Rebuild jedi-bundle using the scripts available in jedi-tools' build_skylab.sh.
• Make sure your environment is set up correctly. Protip: use jedi-tools' setup.sh. We keep the HPC setup scripts up to date with the most recent release of spack-stack.
• Did you restart the ecflow server?

Previous Issues

Many many ctests failing after new build

A lot of ctests failing after a fresh JEDI build is an indication that there is something very wrong with your build. One thing to check first before panicking is if git lfs  is working properly. You might have just failed to download the correct files needed to for the ctest. Check your ~/.gitconfig for the section:

[filter "lfs"]
    clean = git-lfs clean -- %f
    smudge = git-lfs smudge -- %f
    process = git-lfs filter-process
    required = true

If you are on an HPC, some require you to load the git-lfs module. This is the case for Hercules and Discover. More notes on git lfs can be found in JEDI docs git-lfs on HPCs and JEDI docs for developer git lfs information. 

module show git-lfs
module load git-lfs

FATAL: At least one pe in pelist is not used by any tile in the mosaic

The following error was received during the variational task in an EWOK experiment:

FATAL from PE    44: mpp_domains_define.inc: At least one pe in pelist is not used by any tile in the mosaic

Cause: Input from the ALGO team explained that this usually implies a mismatch between the number of MPI tasks used in the run vs the fv3 layout, i.e. how the fv3 tiles are split up in the geometry yaml section. 

Solution: Check the layout and mpi config. For this case, it needed to have layout1 x layout2 x 6 = nodes x tasks per node.

Plots and figures not showing up on experiments.jcsda.org

The user was able to see in the log that media files were being copied to r2d2-experiments-jcsda-noaa-aws-us-east-1 bucket, but there were not being populated on the website. 

Cause: The username for the experiment was blank. Publishing media uses a username harvested from the github credentials. It does not matter what the username is.

Solution: Update the ~/.gitconfig where you are running to include a section that looks like the following. Note, that R2D2 will strip out the <your_email> portion before the "@" symbol, so you can add a fake email to customize the name listed on the website. 

[user]
	name = <github_user_name>
	email = <your_email>@gmail.com

Updating CMakeLists.txt to use TAG

...