Page History

...

1. Activate your Skylab virtual environment, if you haven't already. 
   

   Code Block
   language shell
   source $JEDI_ROOT/venv/bin/activate

2. Force install cylc-flow. Note, cylc is installed in spack-stack but it has some compatibility issues so it is easiest at the point to re-install in your venv. 
   

   Code Block
   language shell
   pip install cylc-flow --force-reinstall

Check cylc location and test with skylab/experiments/workflow-engine-test.yaml: 
3. (*Optional - as needeed) rsync  is required for the workflow.  If a which rsync  does not return this application (if one is on a brand new OrbStack machine):
   

   Code Block
   language shell

which

4. sudo su apt install -y rsync exit

5. Check cylc location and test with skylab/experiments/workflow-engine-test.yaml: 
   

   Code Block
   language shell
   which cylc create_cylc create_experiment.py skylab/experiments/workflow-engine-test.yaml

...

In order to run Skylab with the correct virtual environment, since cylc ignores PYTHONPATH, you need to add a global.cylc file to run an init-script before runtime that will activate the JEDI venv. This should go in ~/.cylc/flow/global.cylc. Replace <JEDI_ROOT> with your path to your JEDI venv. The install The install block is optional for now, but it sets your cylc work directory and run directory to mimic ecflow. Note, these will automatically put a cylc-run directory under the parent EWOK_WORKDIR and EWOK_FLOWDIR directories.

vi ~/.cylc/flow/global.cylc

[install]
    [[symlink dirs]]  
        [[[localhost]]]
            work = ${EWOK_WORKDIR}
            run = ${EWOK_FLOWDIR}

[platforms]
    [[localhost]]
        hosts = localhost
        job runner = background
        global init-script = source ${JEDI_ROOT}/venv/bin/activate

Setting up cylc HPC configuration:

Similar to the localhost setup, you will need to add or update ~/.cylc/flow/global.cylc. The install block is optional for now, but it sets your cylc work directory and run directory to mimic ecflow. Note, these will automatically put a cylc-run directory under the parent EWOK_WORKDIR and EWOK_FLOWDIR directories. Example of global.cylc file for HPCs that use slurm/sbatch for jobs:

vi ~/.cylc/flow/global.cylc

[install]
    [[symlink dirs]]  
        [[[localhost]]]
            work = ${EWOK_WORKDIR}
            run = ${EWOK_FLOWDIR}

[platforms[install]
    [[symlink dirslocalhost]]
  
      hosts = [[[localhost]]]
        job runner =  workbackground
	global init-script = source ${EWOKJEDI_WORKDIR}ROOT}venv/bin/activate 

    [[compute]]
	hosts = localhost
	job runner = slurm
	install target  run= localhost
	global init-script = ${EWOK_FLOWDIR}

[platforms]
"""
	     [[localhost]]
source ${JEDI_ROOT}/venv/bin/activate
	    export SLURM_EXPORT_ENV=ALL
	    hosts = localhost
export HDF5_USE_FILE_LOCKING=FALSE
	    ulimit -s   job runner = background
unlimited || true
	    ulimit -v unlimited || global init-script = source <JEDI_ROOT>/venv/bin/activatetrue
	    """	

Discover via spack-stack:

1. Load spack-stack modules
   

   Code Block
   language shell
   #!/usr/bin/env bash # Initialize modules source $MODULESHOME/init/bash # Load python dependencies echo "Using SLES15 modules" module use /discover/swdev/jcsda/spack-stack/scu17/modulefiles module use /gpfsm/dswdev/jcsda/spack-stack/scu17/spack-stack-1.9.0/envs/ue-intel-2021.10.0/install/modulefiles/Core module load stack-intel/2021.10.0 module load stack-intel-oneapi-mpi/2021.10.0 module load stack-python/3.11.7 module load py-pip/23.1.2

   
   
2. Load cylc module and test
   

   Code Block
   language shell
   # Load the cylc module module use -a /discover/nobackup/projects/gmao/advda/swell/dev/modulefiles/core/ module load cylc/sles15_8.4.0 # Run cylc command cylc "$@"

   
   
3. You might need to create a file called $HOME/bin/cylc, and make sure it is executable in order to run locally: chmod +x $HOME/bin/cylc
1. Note, I did not have to do this when setting up and running on discover-mil with spack-stack 1.9.0 intel
4. Add the example of ~/.cylc/flow/global.cylc see above for HPCs that use slurm/sbatch for job submission.

FAQ

How to remove an experiment from ecflow?

...