Versions Compared

Old Version 14

changes.mady.by.user Stephen Herbener

Saved on

compared with

New Version Current

changes.mady.by.user Ashley Griffin

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: instructions for build cache

...

Documentation: https://spack-stack.readthedocs.io/en/latest/

Table of Contents

Table of Contents

Third party dependencies: licenses

...

JCSDA is currently responsible for testing spack-stack environments on S4 and Discover. The documented instructions for building spack-stack on HPC platforms are very good but there are two things to take care of before starting (as noted in the instructions).

  1. Make sure you are pointing to a python that is version 3.8 or higher

    HPC Platformpython3.8+ environmentS4

    module load miniconda/3.8-s4

    Discover

    module load python/GEOSpyD/Min4.10.3_py3.9

    your environment is set properly. Follow the instructions in the spack-stack ReadTheDocs for setting up an environment. Note that the first section of module commands are to be used for both building new spack-environments (which is this task) and for using an existing spack-stack environment to build jedi-bundle and run skylab. You should skip the additional moudle commands that are only for building jedi-bundle and running skylab.

    HPC PlatformInstructions for setting up the environment
    S4

    https://spack-stack.readthedocs.io/en/latest/PreConfiguredSites.html#uw-univ-of-wisconsin-s4

    Discover - SCU16

    https://spack-stack.readthedocs.io/en/latest/PreConfiguredSites.html#nasa-discover-scu16

    Discover - SCU17

    https://spack-stack.readthedocs.io/en/latest/PreConfiguredSites.html#nasa-discover-scu17


  2. Make sure you have checked out the branches you need to test before running the setup.sh script
    1. First, clone the JCSDA/spack-stack repo as shown in the instructions for creating a local spack-stack environment. It's important to use the "--recurse-submodules" option to the "git clone" command as shown in these instructions.

    2. Typically you

    Make sure you have checked out the branches you need to test before running the setup.sh script

    1. Typically you will be testing feature branches that could have come from various developers. Each of these developers has their own fork of which will be marked on the PR, and you will need to gain access to check those out forks in your local clone. One The way to do this is to add their forks to your remotes in your local clone. This can be done using a series of the "git remote add ..."  command. Here's an example:

      Code Block
      languagebash
      themeMidnight
      firstline1
      titleAdd remote repos to your local clone
      # Alex Richert (NOAA) and Dom (NRL) are two of the more active PR contributors
git remote add alex	https://github.com/AlexanderRichert-NOAA/spack-stack 
git remote add dom	https://github.com/climbfuji/spack-stack
...

# Check that you have these entered
git remote -v

# Update references to the other forks. Running the remote update
# command below will pull in all the metadata to the other forks.
git remote update -p

# Check that you got the references to the other forks (note the use of
# the -a option on the git branch command)
git branch -avv


    2. Once you have the necessary remotes added, then you can checkout feature branches from other repos

      Code Block
      languagebash
      themeMidnight
      firstline1
      titleCheck out feature branches from other forks
      # Using the remote link entered above, checkout the feature/cool-spack-thing from Dom's spack-stack forkstack fork
git checkout dom/feature/cool-spack-thing

# Make sure to update the submodules
git checkout dom/feature/cool-spack-thingsubmodule update


Once the above steps are done, follow the documented instructions here: building spack-stack on HPC platforms. The ideal goal with the testing is to successfully complete the following steps:

...

  1. Create forks of JCSDA/spack-stack and JCSDA/spack repos
    1. Do the usual steps of bringing up the repos in github and hit the Create Fork button. 
    2. Once you have your fork of JCSDA/spack, rename origin to your name and add the remote URLs for the JCSDA fork and the LLNL authoritative repos.
      Code Block
      languagebash
      themeMidnight
      firstline1
      titleAdd remote repos to your local spack fork
      # Set up remote references
git remote rename origin <your-user-name>
git remote add auth_spack	https://github.com/spack/spack
git remote add jcsda	    https://github.com/JCSDA/spack

# Check that you have these entered
git remote -v

# Update references to the other forks. Running the remote update
# command below will pull in all the metadata to the other forks.
git remote update -p

# Check that you got the references to the other forks (note the use of
# the -a option on the git branch command)
git branch -avv
    3. Ditto for your spack-stack repo (note that the JCSDA repo is the authoritative repo).
      Code Block
      languagebash
      themeMidnight
      firstline1
      titleAdd remote repos to your local spack-stack fork
      # Set up remote references
git remote rename origin <your-user-name>
git remote add jcsda	    https://github.com/JCSDA/spack-stack

# Check that you have these entered
git remote -v

# Update references to the other forks. Running the remote update
# command below will pull in all the metadata to the other forks.
git remote update -p

# Check that you got the references to the other forks (note the use of
# the -a option on the git branch command)
git branch -avv
  2. Create PR in LLNL spack repo with the new ecmwf-atlas version(s)
    1. When creating your feature branch, make sure it is based on the LLNL repo develop branch:
      Code Block
      languagebash
      themeMidnight
      firstline1
      titleCreate a feature branch based on the LLNL spack repo
      # Create feature branch for ecmwf-atlas upgrade
cd <spack-fork>
git checkout auth_spack/develop

# take note of the commit hash for the authoritative branch
git branch -vv

# Create your feature branch, and check that it's commit hash matches
# the authoritative develop branch commit hash
git checkout -b feature/upgrade-atlas-0.38.x
git branch -vv
    2. Utilize the spack checksum  command to get the python commands needed for the new versions. See the example above.
      1. Run the checksum command and get the version() lines needed for the atlas package.py script.
      2. The package.py script can be found in: "var/spack/repos/builtin/packages/ecmwf-atlas/package.py"
      3. Here are the changes in the example PR from the recent atlas upgrade: https://github.com/spack/spack/pull/41547/files#diff-97e21a41aae31e448128cc350cb856d4c8d1679b48adfa6a556e24c42710216d
    3. Once the version commands have been inserted into the package.py script, go through the usual steps to create and submit the PR.
    4. Note the commit hash(s) for the changes for the next step.
  3. Create PR in the JCSDA spack repo
    1. Make sure this feature branch is based on the JCSDA spack-stack-dev branch
      Code Block
      languagebash
      themeMidnight
      firstline1
      titleCreate a feature branch based on the JCSDA spack repo
      # Create feature branch for ecmwf-atlas upgrade
cd <spack-fork>
git checkout jcsda/spack-stack-dev

# take note of the commit hash for the authoritative branch
git branch -vv

# Create your feature branch, and check that it's commit hash matches
# the authoritative develop branch commit hash
git checkout -b feature/upgrade-atlas-0.38.x
git branch -vv
    2. Use the git cherry pick command to copy the commit from the LLNL PR feature branch to the new feature branch based on the JCSDA spack repo.
      1. Here's the documentation for the cherry pick command: https://git-scm.com/docs/git-cherry-pick
      2. Use the commit hash form with the commit hash you noted from the LLNL repo PR.
      3. This should place the same changes in your hew feature branch based on the JCSDA repo.
      4. Here are the changes in the example PR from the recent atlas upgrade: https://github.com/JCSDA/spack/pull/378/files#diff-97e21a41aae31e448128cc350cb856d4c8d1679b48adfa6a556e24c42710216d
    Create PR in the JCSDA spack-stack repo 
  4. The idea here is to create a feature branch in this PR that points to your feature branch in the JCSDA spack PR. This will cause the CI test suite in the JCSDA spack-stack PR to run using the feature branch in the JCSDA spack PR.
    1. The submodule configuration needs to be updated in the spack-stack ".gitmodules" file
    2. The example PR doesn't show this, but here is another example of these changes. You want to switch the JCSDA spack-stack-dev branch to your fork's feature branch.
    3. Image Removed
    4. Note that the corresponding update to the spack submodule commit hash has also been updated to point to your JCSDA feature branch commit hash (as well as the .gitmodules updates to point to your JCSDA spack feature branch). Git will take care of updating your spack submodule commit hash by simply doing the git add  and git commit  commands in your spack-stack feature branch while your spack feature branch is checked out.
  5. Make sure this branch is based on the JCSDA spack-stack develop branch
    Code Block
    languagebash
    themeMidnight
    firstline1
    titleCreate a feature branch based on the JCSDA spack-stack repo
    # Create feature branch for ecmwf-atlas upgrade
cd <spack-stack-fork>
git checkout jcsda/develop

# take note of the commit hash for the authoritative branch
git branch -vv

# Create your feature branch, and check that it's commit hash matches
# the authoritative develop branch commit hash
git checkout -b feature/upgrade-atlas-0.38.x
git branch -vv
  6. Also, make sure the ecmwf-atlas version in the common/packages.yaml  file is updated.
    1. Here is the corresponding change (line 52) in the example PR: https://github.com/JCSDA/spack-stack/pull/897/files#diff-25897cb1dfa4895bceaba58be9d4490b43b1f3236953f749bd22a27b804adf5a
      1. repo PR.
      2. This should place the same changes in your hew feature branch based on the JCSDA repo.
      3. Here are the changes in the example PR from the recent atlas upgrade: https://github.com/JCSDA/spack/pull/378/files#diff-97e21a41aae31e448128cc350cb856d4c8d1679b48adfa6a556e24c42710216d
  7. Create PR in the JCSDA spack-stack repo 
    1. The idea here is to create a feature branch in this PR that points to your feature branch in the JCSDA spack PR. This will cause the CI test suite in the JCSDA spack-stack PR to run using the feature branch in the JCSDA spack PR.
      1. The submodule configuration needs to be updated in the spack-stack ".gitmodules" file
      2. The example PR doesn't show this, but here is another example of these changes. You want to switch the JCSDA spack-stack-dev branch to your fork's feature branch.
      3. Image Added
      4. Note that the corresponding update to the spack submodule commit hash has also been updated to point to your JCSDA feature branch commit hash (as well as the .gitmodules updates to point to your JCSDA spack feature branch). Git will take care of updating your spack submodule commit hash by simply doing the git add  and git commit  commands in your spack-stack feature branch while your spack feature branch is checked out.
    2. Make sure this branch is based on the JCSDA spack-stack develop branch
      Code Block
      languagebash
      themeMidnight
      firstline1
      titleCreate a feature branch based on the JCSDA spack-stack repo
      # Create feature branch for ecmwf-atlas upgrade
cd <spack-stack-fork>
git checkout jcsda/develop

# take note of the commit hash for the authoritative branch
git branch -vv

# Create your feature branch, and check that it's commit hash matches
# the authoritative develop branch commit hash
git checkout -b feature/upgrade-atlas-0.38.x
git branch -vv
    3. Also, make sure the ecmwf-atlas version in the common/packages.yaml  file is updated.
      1. Here is the corresponding change (line 52) in the example PR: https://github.com/JCSDA/spack-stack/pull/897/files#diff-25897cb1dfa4895bceaba58be9d4490b43b1f3236953f749bd22a27b804adf5a
    4. Create and submit the spack-stack PR in the usual manner. 

Creating and using a build cache

If you install spack-stack as a release candidate (RC) on an HPC, it will save time to create a build cache to use when installing the actual release. 

Prerequisites:

  • The path of the final spack-stack install must be the same length or shorter than the original install path

Procedure:

  1. With the environment(s) activated, run the following. Note, you can then deactivate the environment and move on to the next one. You can push to the same directory from all installations (ie for discover: scu16 or scu17, intel or gnu).
    Code Block
    languagebash
    themeMidnight
     spack buildcache push -u /path/to/tmp-build-cache
  2. When you do the actual installs for the release, you run everything up to and including the spack concretize command as usual. After that you add the binary cache and use it. Note the notation (number of slashes etc) in the first line.
    Code Block
    languagebash
    themeMidnight
    spack mirror add local-binary file:///path/to/tmp-build-cache/
spack buildcache update-index local-binary
spack buildcache list
spack install --verbose --no-check-signature 2>&1 | tee log.install.env-name.001
    Create and submit the spack-stack PR in the usual manner. 
Pages