...

1. Log into S4 and switch to the jedipara account
   1. sudo -iu jedipara 
   2. We will need to get more of us the jedipara account access
2. module load miniconda/3.8-s4 
   1. This satisfies the requirement to be pointing to python3.8+ before sourcing the spack-stack setup.sh script
3. cd /data/prod/jedi/spack-stack/spack-stack-1.6.0 
4. source setup.sh 
5. cd envs/upp-addon-env 
6. spack env activate . 
7. edit spack.yaml
   1. Make the following modifications in the specs:  section

      Code Block
      language yml theme Midnight firstline 1 title Change From This
      specs: - upp-env %intel ^g2tmpl@1.12.0 ^g2@3.4.5 - prod-util@2.1.1 %intel - ip %intel

      Code Block
      language yml theme Midnight firstline 1 title Change To This
      specs: - upp-env %intel ^g2tmpl@1.12.0 ^g2@3.4.5 - upp-env %intel ^g2tmpl@1.13.0 ^g2@3.5.1 - grib-util@1.3.0 %intel ^g2@3.4.5 - grib-util@1.3.0 %intel ^g2@3.5.1 - prod-util@2.1.1 %intel - ip %intel - ufs-weather-model-env %intel ^g2tmpl@1.12.0 ^g5@3.4.5

   2. Note that
      1. grib-util is dependent on g2, and we want two versions of grib-util: one built with g2@3.4.5 and the other with g2@3.5.1
      2. ufs-weather-model-env is dependent on g2tmpl@1.12.0 and g2@3.4.5, and we want to preserve this
      3. We want two versions of upp-env: one with g2tmpl@1.12.0/g2@3.4.5 and the other with g2tmpl@1.13.0/g2@3.5.1
      4. Hopefully the spec changes indicated above make sense in the context of what we want
8. Update envrepo/packages  following what was done on Orion
   1. The idea here is to add the new versions of g2 and g2tmpl to the packages under envrepo . envrepo  is a special place used by the chaining environment mechanism to hold extra methods for building the new versions we are trying to add in.
   2. There are probably many ways to accomplish this, but since there was a model of what needed to be done already on Orion, I did the following:
      1. Tar'd on Orion the spack-stack-1.6.0/envs/upp-addon-env/envrepo  files.
      2. Unpacked the tar on S4 in a temporary directory
      3. Updated the `/data/prod/jedi/spack-stack/spack-stack-1.6.0/envs/upp-addon-env/envrepo` directory by comparing it with the unpacked tar files.
9. edit common/modules.yaml 
   1. Under the modules.default.lmod.hierarchy section, add in entries for g2virt  and g2tmplvirt

      Code Block
      language yml theme Midnight firstline 1 title Add in 'g2virt' and 'g2tmplvirt'
      modules: default: ... lmod: ... hierarchy: - mpi - g2virt - g2tmplvirt

10. spack concretize 2>&1 | tee log.concretize 
    1. Check that the only things being concretized are those related to the new specs in the spack.yaml  file above

       Code Block
       language bash theme Midnight firstline 1 title Check what was concretized
       grep "==> Concretized " /data/prod/jedi/spack-stack/spack-stack-1.6.0/envs/upp-addon-env/log.concretize ==> Concretized grib-util@1.3.0%intel ^g2@3.4.5 ==> Concretized grib-util@1.3.0%intel ^g2@3.5.1 ==> Concretized ufs-weather-model-env%intel ^g2@3.4.5 ^g2tmpl@1.12.0 ==> Concretized upp-env%intel ^g2@3.4.5 ^g2tmpl@1.12.0 ==> Concretized upp-env%intel ^g2@3.5.1 ^g2tmpl@1.13.0

11. space install -v 2>&1 | tee log.install 
    1. Again this should run quickly and only build the new package versions and associated environment updates
12. spack module lmod refresh --upstream-modules 
13. spack stack setup-meta-modules 

spack-stack Upgrade a package to a new version

This tasks typically comes about when a center wants to pull in bugfixes/updates in a spack-stack package. Ideally, there is enough time to follow the preferred steps which are:

1. Create forks of the JCSDA spack-stack and JCSDA spack repos.
2. Create a PR in the authoritative (Lawrence Livermore National Labs) repo for spack which adds in the new version(s) of the package of interest.
3. Create a similar PR in the JCSDA spack fork that is based on cherry picking the commits from the PR in the authoritative spack repo.
4. Create a PR in the JCSDA spack-stack repo that references the feature branch in the JCSDA spack PR (for testing purposes).

Spack has a handy command (checksum) that will grab the tar file for a particular package version and return the SHA256 value by running a checksum on the tar file. The output is in the form of the python command needed to configure that package version in the package.py script. Here's an example of running the checksum command on the ecmwf-atlas package:

MacBook-Pro-5:unified-env.mymacos steveherbener$ spack checksum ecmwf-atlas 0.38.0
==> Found 1 version of ecmwf-atlas
==> Fetching https://github.com/ecmwf/atlas/archive/0.38.0.tar.gz

    version("0.38.0", sha256="befe3bfc045bc0783126efb72ed55db9f205eaf176e1b8a2059eaaaaacc4880a")

MacBook-Pro-5:unified-env.mymacos steveherbener$ spack checksum ecmwf-atlas 0.38.1
==> Found 1 version of ecmwf-atlas
==> Fetching https://github.com/ecmwf/atlas/archive/0.38.1.tar.gz

    version("0.38.1", sha256="c6868deb483c1d6c241aae92f8af63f3351062c2611c9163e8a9bbf6c97a9798")

To illustrate the above steps for upgrading a package version, an example of a prior upgrade to the ecmwf-atlas packages is used. These steps are captured in the following 3 PRs:

1. Authoritative spack PR: https://github.com/spack/spack/pull/41547
2. JCSDA spack PR: https://github.com/JCSDA/spack/pull/378
3. JCSDA spack-stack PR: https://github.com/JCSDA/spack-stack/pull/897

Walking through the upgrade steps from above:

1. Create forks of JCSDA/spack-stack and JCSDA/spack repos
   1. Do the usual steps of bringing up the repos in github and hit the Create Fork button. 
   2. Once you have your fork of JCSDA/spack, rename origin to your name and add the remote URLs for the JCSDA fork and the LLNL authoritative repos.

      Code Block
      language bash theme Midnight firstline 1 title Add remote repos to your local spack fork
      # Set up remote references git remote rename origin <your-user-name> git remote add auth_spack https://github.com/spack/spack git remote add jcsda https://github.com/JCSDA/spack # Check that you have these entered git remote -v # Update references to the other forks. Running the remote update # command below will pull in all the metadata to the other forks. git remote update -p # Check that you got the references to the other forks (note the use of # the -a option on the git branch command) git branch -avv

   3. Ditto for your spack-stack repo (note that the JCSDA repo is the authoritative repo).

      Code Block
      language bash theme Midnight firstline 1 title Add remote repos to your local spack-stack fork
      # Set up remote references git remote rename origin <your-user-name> git remote add jcsda https://github.com/JCSDA/spack-stack # Check that you have these entered git remote -v # Update references to the other forks. Running the remote update # command below will pull in all the metadata to the other forks. git remote update -p # Check that you got the references to the other forks (note the use of # the -a option on the git branch command) git branch -avv

2. Create PR in LLNL spack repo with the new ecmwf-atlas version(s)
   1. When creating your feature branch, make sure it is based on the LLNL repo develop branch:

      Code Block
      language bash theme Midnight firstline 1 title Create a feature branch based on the LLNL spack repo
      # Create feature branch for ecmwf-atlas upgrade cd <spack-fork> git checkout auth_spack/develop # take note of the commit hash for the authoritative branch git branch -vv # Create your feature branch, and check that it's commit hash matches # the authoritative develop branch commit hash git checkout -b feature/upgrade-atlas-0.38.x git branch -vv

   2. Utilize the spack checksum  command to get the python commands needed for the new versions. See the example above.
      1. Run the checksum command and get the version() lines needed for the atlas package.py script.
      2. The package.py script can be found in: "var/spack/repos/builtin/packages/ecmwf-atlas/package.py"
      3. Here are the changes in the example PR from the recent atlas upgrade: https://github.com/spack/spack/pull/41547/files#diff-97e21a41aae31e448128cc350cb856d4c8d1679b48adfa6a556e24c42710216d
   3. Once the version commands have been inserted into the package.py script, go through the usual steps to create and submit the PR.
   4. Note the commit hash(s) for the changes for the next step.
3. Create PR in the JCSDA spack repo
   1. Make sure this feature branch is based on the JCSDA spack-stack-dev branch

      Code Block
      language bash theme Midnight firstline 1 title Create a feature branch based on the JCSDA spack repo
      # Create feature branch for ecmwf-atlas upgrade cd <spack-fork> git checkout jcsda/spack-stack-dev # take note of the commit hash for the authoritative branch git branch -vv # Create your feature branch, and check that it's commit hash matches # the authoritative develop branch commit hash git checkout -b feature/upgrade-atlas-0.38.x git branch -vv

   2. Use the git cherry pick command to copy the commit from the LLNL PR feature branch to the new feature branch based on the JCSDA spack repo.
      1. Here's the documentation for the cherry pick command: https://git-scm.com/docs/git-cherry-pick
      2. Use the commit hash form with the commit hash you noted from the LLNL repo PR.
      3. This should place the same changes in your hew feature branch based on the JCSDA repo.
      4. Here are the changes in the example PR from the recent atlas upgrade: https://github.com/JCSDA/spack/pull/378/files#diff-97e21a41aae31e448128cc350cb856d4c8d1679b48adfa6a556e24c42710216d
4. Create PR in the JCSDA spack-stack repo 
   1. The idea here is to create a feature branch in this PR that points to your feature branch in the JCSDA spack PR. This will cause the CI test suite in the JCSDA spack-stack PR to run using the feature branch in the JCSDA spack PR.
      1. The submodule configuration needs to be updated in the spack-stack ".gitmodules" file
      2. The example PR doesn't show this, but here is another example of these changes. You want to switch the JCSDA spack-stack-dev branch to your fork's feature branch.
      3. Image Added
      4. Note that the corresponding update to the spack submodule commit hash has also been updated to point to your JCSDA feature branch commit hash (as well as the .gitmodules updates to point to your JCSDA spack feature branch). Git will take care of updating your spack submodule commit hash by simply doing the git add  and git commit  commands in your spack-stack feature branch while your spack feature branch is checked out.
   2. Make sure this branch is based on the JCSDA spack-stack develop branch

      Code Block
      language bash theme Midnight firstline 1 title Create a feature branch based on the JCSDA spack-stack repo
      # Create feature branch for ecmwf-atlas upgrade cd <spack-stack-fork> git checkout jcsda/develop # take note of the commit hash for the authoritative branch git branch -vv # Create your feature branch, and check that it's commit hash matches # the authoritative develop branch commit hash git checkout -b feature/upgrade-atlas-0.38.x git branch -vv

   3. Also, make sure the ecmwf-atlas version in the common/packages.yaml  file is updated.
      1. Here is the corresponding change (line 52) in the example PR: https://github.com/JCSDA/spack-stack/pull/897/files#diff-25897cb1dfa4895bceaba58be9d4490b43b1f3236953f749bd22a27b804adf5a
   4. Create and submit the spack-stack PR in the usual manner. 

Pages

• Setting up CI (spack-stack)
• Chaining environments