Github repository: https://github.com/JCSDA/spack-stack

Documentation: https://spack-stack.readthedocs.io/en/latest/

Table of Contents

Third party dependencies: licenses

https://docs.google.com/spreadsheets/d/1k3aMu8gkEsAZGJRUqYTx1iQG4xFnJGqe/edit#gid=220490890

spack-stack Development

If you are going to contribute to the spack-stack repository, development is done by forking the spack-stack repository. This section contains steps to fork spack-stack from https://github.com/JCSDA/spack-stack, make changes, and issue a PR.

Create a fork of https://github.com/JCSDA/spack-stack by clicking the "Fork" button on the top right side of the page to "Fork your own copy of JCSDA/spack-stack".
Navigate to your repositories and click on the spack-stack repo. Clicking the green "Code" button will show you options to clone this repository. One option is by using https and running git clone https://github.com/<your_username>/spack-stack.git
Rename your fork to your name in order to help keep track of multiple user and the authoritative forks. Therefore after you clone the repository run git remote rename origin <your_name> . This can be verified by running git remote -v
You can now create a branch and start developing. Note, make sure you pull the authoritative develop brand if your fork is out of date.
Once your branch is ready for review, you will need to create a PR inside https://github.com/JCSDA/spack-stack pointing to your fork and branch. Push your PR to your fork by running git push <your_name> feature/<your_branch> . Then open the PR request form in a bowser adding the necessary information and setting the base repository to JCSDA/spack-stack.

For further reading on using github forks, checkout this site and this one too.

spack-stack Testing

JCSDA is currently responsible for testing spack-stack environments on S4 and Discover. The documented instructions for building spack-stack on HPC platforms are very good but there are two things to take care of before starting (as noted in the instructions).

Make sure your environment is set properly. Follow the instructions in the spack-stack ReadTheDocs for setting up an environment. Note that the first section of module commands are to be used for both building new spack-environments (which is this task) and for using an existing spack-stack environment to build jedi-bundle and run skylab. You should skip the additional moudle commands that are only for building jedi-bundle and running skylab.

HPC Platform	Instructions for setting up the environment
S4	https://spack-stack.readthedocs.io/en/latest/PreConfiguredSites.html#uw-univ-of-wisconsin-s4
Discover - SCU16	https://spack-stack.readthedocs.io/en/latest/PreConfiguredSites.html#nasa-discover-scu16
Discover - SCU17	https://spack-stack.readthedocs.io/en/latest/PreConfiguredSites.html#nasa-discover-scu17

Make sure you have checked out the branches you need to test before running the setup.sh script

First, clone the JCSDA/spack-stack repo as shown in the instructions for creating a local spack-stack environment. It's important to use the "--recurse-submodules" option to the "git clone" command as shown in these instructions.

Typically you will be testing feature branches that could have come from various developers. Each of these developers has their own fork of which will be marked on the PR, and you will need to gain access to those forks in your local clone. The way to do this is to add their forks to your remotes in your local clone. This can be done using a series of the "git remote add ..." command. Here's an example:

# Alex Richert (NOAA) and Dom (NRL) are two of the more active PR contributors
git remote add alex	https://github.com/AlexanderRichert-NOAA/spack-stack 
git remote add dom	https://github.com/climbfuji/spack-stack
...

# Check that you have these entered
git remote -v

# Update references to the other forks. Running the remote update
# command below will pull in all the metadata to the other forks.
git remote update -p

# Check that you got the references to the other forks (note the use of
# the -a option on the git branch command)
git branch -avv

Once you have the necessary remotes added, then you can checkout feature branches from other repos

# Using the remote link entered above, checkout the feature/cool-spack-thing from Dom's spack-stack fork
git checkout dom/feature/cool-spack-thing

# Make sure to update the submodules
git submodule update

Once the above steps are done, follow the documented instructions here: building spack-stack on HPC platforms. The ideal goal with the testing is to successfully complete the following steps:

Build spack-stack, for each compiler on S4 (intel) and Discover (intel, gnu) both SCU16 and SCU17
Using the environment from step 1, build jedi-bundle
Set up and try some skylab experiments

Typically, getting all of these steps done is way too much work. I would settle for one of S4 or Discover SCU16 or Discover SCU17, with just one compiler. All of these possibilities are listed for your awareness with the idea that you can round robin between then when testing different feature branches.

Another way to trim down work, is to just verify the spack-stack build, and occasionally carry on through with the jedi-bundle and skylab testing.

Note that you don't need jedipara access to do this testing. You can build everything in your user area on S4 or Discover. To help manage that, both HPC platforms supply commands to show how much of your quota (disk space and number of files) is free.

HPC Platform	quota command
S4	myquota
Discover	showquota -h

spack-stack Add-on Environment

Occasionally, there is a need to add a handful of upgraded packages to an existing spack-stack release. spack-stack contains a feature called "chaining environments" which allows the rapid construction of such an "add-on" environment. The idea is to build a new environment that points to an existing environment of which you only have to add in the upgraded packages. The module files in the add-on environment then utilize the base environment's installation and module files for the packages that remain unchanged.

Recently the need for this came up for spack-stack-1.6.0 with the g2@3.5.1 and g2tmpl@1.13.0 upgraded packages. Note that spack-stack-1.6.0 was shipped with g2@3.4.5 and g2tmpl@1.12.0. Here is the spack-stack issue describing the desired upgrade: https://github.com/JCSDA/spack-stack/issues/1180. Note under this comment, a method for handling this particular request (on S4) was described here: https://github.com/JCSDA/spack-stack/issues/1180#issuecomment-2251378587. In this case, an add-on environment already existed so it wasn't necessary to create it and thus it was possible to simply add in the new packages, concretize, install, and update modules (ie, skip some of the steps in the chaining environments recipe).

Here are the steps taken to accomplish this task.

Log into S4 and switch to the jedipara account
1. sudo -iu jedipara
2. We will need to get more of us the jedipara account access
module load miniconda/3.8-s4
1. This satisfies the requirement to be pointing to python3.8+ before sourcing the spack-stack setup.sh script
cd /data/prod/jedi/spack-stack/spack-stack-1.6.0
source setup.sh
cd envs/upp-addon-env
spack env activate .

edit spack.yaml

Make the following modifications in the specs: section

specs:
  - upp-env %intel ^g2tmpl@1.12.0 ^g2@3.4.5
  - prod-util@2.1.1 %intel
  - ip %intel

specs:
  - upp-env %intel ^g2tmpl@1.12.0 ^g2@3.4.5
  - upp-env %intel ^g2tmpl@1.13.0 ^g2@3.5.1
  - grib-util@1.3.0 %intel ^g2@3.4.5
  - grib-util@1.3.0 %intel ^g2@3.5.1
  - prod-util@2.1.1 %intel
  - ip %intel
  - ufs-weather-model-env %intel ^g2tmpl@1.12.0 ^g5@3.4.5

Note that
1. grib-util is dependent on g2, and we want two versions of grib-util: one built with g2@3.4.5 and the other with g2@3.5.1
2. ufs-weather-model-env is dependent on g2tmpl@1.12.0 and g2@3.4.5, and we want to preserve this
3. We want two versions of upp-env: one with g2tmpl@1.12.0/g2@3.4.5 and the other with g2tmpl@1.13.0/g2@3.5.1
4. Hopefully the spec changes indicated above make sense in the context of what we want

Update envrepo/packages following what was done on Orion
1. The idea here is to add the new versions of g2 and g2tmpl to the packages under envrepo . envrepo is a special place used by the chaining environment mechanism to hold extra methods for building the new versions we are trying to add in.
2. There are probably many ways to accomplish this, but since there was a model of what needed to be done already on Orion, I did the following:
  1. Tar'd on Orion the spack-stack-1.6.0/envs/upp-addon-env/envrepo files.
  2. Unpacked the tar on S4 in a temporary directory
  3. Updated the `/data/prod/jedi/spack-stack/spack-stack-1.6.0/envs/upp-addon-env/envrepo` directory by comparing it with the unpacked tar files.

edit common/modules.yaml

Under the modules.default.lmod.hierarchy section, add in entries for g2virt and g2tmplvirt

modules:
  default:
    ...
    lmod:
      ...
      hierarchy:
        - mpi
        - g2virt
        - g2tmplvirt

spack concretize 2>&1 | tee log.concretize

Check that the only things being concretized are those related to the new specs in the spack.yaml file above

grep "==> Concretized " /data/prod/jedi/spack-stack/spack-stack-1.6.0/envs/upp-addon-env/log.concretize 
==> Concretized grib-util@1.3.0%intel ^g2@3.4.5
==> Concretized grib-util@1.3.0%intel ^g2@3.5.1
==> Concretized ufs-weather-model-env%intel ^g2@3.4.5 ^g2tmpl@1.12.0
==> Concretized upp-env%intel ^g2@3.4.5 ^g2tmpl@1.12.0
==> Concretized upp-env%intel ^g2@3.5.1 ^g2tmpl@1.13.0

space install -v 2>&1 | tee log.install
1. Again this should run quickly and only build the new package versions and associated environment updates
spack module lmod refresh --upstream-modules
spack stack setup-meta-modules

spack-stack Upgrade a package to a new version

This tasks typically comes about when a center wants to pull in bugfixes/updates in a spack-stack package. Ideally, there is enough time to follow the preferred steps which are:

Create forks of the JCSDA spack-stack and JCSDA spack repos.
Create a PR in the authoritative (Lawrence Livermore National Labs) repo for spack which adds in the new version(s) of the package of interest.
Create a similar PR in the JCSDA spack fork that is based on cherry picking the commits from the PR in the authoritative spack repo.
Create a PR in the JCSDA spack-stack repo that references the feature branch in the JCSDA spack PR (for testing purposes).

Spack has a handy command (checksum) that will grab the tar file for a particular package version and return the SHA256 value by running a checksum on the tar file. The output is in the form of the python command needed to configure that package version in the package.py script. Here's an example of running the checksum command on the ecmwf-atlas package:

MacBook-Pro-5:unified-env.mymacos steveherbener$ spack checksum ecmwf-atlas 0.38.0
==> Found 1 version of ecmwf-atlas
==> Fetching https://github.com/ecmwf/atlas/archive/0.38.0.tar.gz

    version("0.38.0", sha256="befe3bfc045bc0783126efb72ed55db9f205eaf176e1b8a2059eaaaaacc4880a")

MacBook-Pro-5:unified-env.mymacos steveherbener$ spack checksum ecmwf-atlas 0.38.1
==> Found 1 version of ecmwf-atlas
==> Fetching https://github.com/ecmwf/atlas/archive/0.38.1.tar.gz

    version("0.38.1", sha256="c6868deb483c1d6c241aae92f8af63f3351062c2611c9163e8a9bbf6c97a9798")

Note that before running these commands you will need to check out the JCSDA spack-stack develop branch (as explained below) first and then run "source setup.sh" to get the environment set up properly.

To illustrate the above steps for upgrading a package version, an example of a prior upgrade to the ecmwf-atlas packages is used. These steps are captured in the following 3 PRs:

Authoritative spack PR: https://github.com/spack/spack/pull/41547
JCSDA spack PR: https://github.com/JCSDA/spack/pull/378
JCSDA spack-stack PR: https://github.com/JCSDA/spack-stack/pull/897

Walking through the upgrade steps from above:

Create forks of JCSDA/spack-stack and JCSDA/spack repos

Do the usual steps of bringing up the repos in github and hit the Create Fork button.

Once you have your fork of JCSDA/spack, rename origin to your name and add the remote URLs for the JCSDA fork and the LLNL authoritative repos.

# Set up remote references
git remote rename origin <your-user-name>
git remote add auth_spack	https://github.com/spack/spack
git remote add jcsda	    https://github.com/JCSDA/spack

# Check that you have these entered
git remote -v

# Update references to the other forks. Running the remote update
# command below will pull in all the metadata to the other forks.
git remote update -p

# Check that you got the references to the other forks (note the use of
# the -a option on the git branch command)
git branch -avv

Ditto for your spack-stack repo (note that the JCSDA repo is the authoritative repo).

# Set up remote references
git remote rename origin <your-user-name>
git remote add jcsda	    https://github.com/JCSDA/spack-stack

# Check that you have these entered
git remote -v

# Update references to the other forks. Running the remote update
# command below will pull in all the metadata to the other forks.
git remote update -p

# Check that you got the references to the other forks (note the use of
# the -a option on the git branch command)
git branch -avv

Create PR in LLNL spack repo with the new ecmwf-atlas version(s)

When creating your feature branch, make sure it is based on the LLNL repo develop branch:

# Create feature branch for ecmwf-atlas upgrade
cd <spack-fork>
git checkout auth_spack/develop

# take note of the commit hash for the authoritative branch
git branch -vv

# Create your feature branch, and check that it's commit hash matches
# the authoritative develop branch commit hash
git checkout -b feature/upgrade-atlas-0.38.x
git branch -vv

Utilize the spack checksum command to get the python commands needed for the new versions. See the example above.
1. Run the checksum command and get the version() lines needed for the atlas package.py script.
2. The package.py script can be found in: "var/spack/repos/builtin/packages/ecmwf-atlas/package.py"
3. Here are the changes in the example PR from the recent atlas upgrade: https://github.com/spack/spack/pull/41547/files#diff-97e21a41aae31e448128cc350cb856d4c8d1679b48adfa6a556e24c42710216d
Once the version commands have been inserted into the package.py script, go through the usual steps to create and submit the PR.
Note the commit hash(s) for the changes for the next step.

Create PR in the JCSDA spack repo

Make sure this feature branch is based on the JCSDA spack-stack-dev branch

# Create feature branch for ecmwf-atlas upgrade
cd <spack-fork>
git checkout jcsda/spack-stack-dev

# take note of the commit hash for the authoritative branch
git branch -vv

# Create your feature branch, and check that it's commit hash matches
# the authoritative develop branch commit hash
git checkout -b feature/upgrade-atlas-0.38.x
git branch -vv

Use the git cherry pick command to copy the commit from the LLNL PR feature branch to the new feature branch based on the JCSDA spack repo.
1. Here's the documentation for the cherry pick command: https://git-scm.com/docs/git-cherry-pick
2. Use the commit hash form with the commit hash you noted from the LLNL repo PR.
3. This should place the same changes in your hew feature branch based on the JCSDA repo.
4. Here are the changes in the example PR from the recent atlas upgrade: https://github.com/JCSDA/spack/pull/378/files#diff-97e21a41aae31e448128cc350cb856d4c8d1679b48adfa6a556e24c42710216d

Create PR in the JCSDA spack-stack repo

The idea here is to create a feature branch in this PR that points to your feature branch in the JCSDA spack PR. This will cause the CI test suite in the JCSDA spack-stack PR to run using the feature branch in the JCSDA spack PR.
1. The submodule configuration needs to be updated in the spack-stack ".gitmodules" file
2. The example PR doesn't show this, but here is another example of these changes. You want to switch the JCSDA spack-stack-dev branch to your fork's feature branch.
4. Note that the corresponding update to the spack submodule commit hash has also been updated to point to your JCSDA feature branch commit hash (as well as the .gitmodules updates to point to your JCSDA spack feature branch). Git will take care of updating your spack submodule commit hash by simply doing the git add and git commit commands in your spack-stack feature branch while your spack feature branch is checked out.

Make sure this branch is based on the JCSDA spack-stack develop branch

# Create feature branch for ecmwf-atlas upgrade
cd <spack-stack-fork>
git checkout jcsda/develop

# take note of the commit hash for the authoritative branch
git branch -vv

# Create your feature branch, and check that it's commit hash matches
# the authoritative develop branch commit hash
git checkout -b feature/upgrade-atlas-0.38.x
git branch -vv

Also, make sure the ecmwf-atlas version in the common/packages.yaml file is updated.
1. Here is the corresponding change (line 52) in the example PR: https://github.com/JCSDA/spack-stack/pull/897/files#diff-25897cb1dfa4895bceaba58be9d4490b43b1f3236953f749bd22a27b804adf5a
Create and submit the spack-stack PR in the usual manner.

Creating and using a build cache

If you install spack-stack as a release candidate (RC) on an HPC, it will save time to create a build cache to use when installing the actual release.

Prerequisites:

The path of the final spack-stack install must be the same length or shorter than the original install path

Procedure:

With the environment(s) activated, run the following. Note, you can then deactivate the environment and move on to the next one. You can push to the same directory from all installations (ie for discover: scu16 or scu17, intel or gnu).
spack buildcache push -u /path/to/tmp-build-cache

When you do the actual installs for the release, you run everything up to and including the spack concretize command as usual. After that you add the binary cache and use it. Note the notation (number of slashes etc) in the first line.