You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 4 Next »

Experiments and Workflows Orchestration Kit

Github repository: https://github.com/JCSDA-internal/ewok

Documentation: https://github.com/JCSDA-internal/ewok/blob/develop/README.md

EWOK Developer Section

EWOK currently uses the ECMWSF's ecFlow software system for implementing workflows. This section will describe how EWOK and ecFlow interact to run a Skylab experiment. Instructions for setting up your environment to can be found in the JEDI Documentation.

EWOK contains 3 important components to the workflow:

  1. Suites - reference tasks and task dependencies. The scripts used for suites is located at {JEDI_SRC}/ewok/src/ewok/suites/.

    A suite will typically build and set up JEDI, define a cycle and loop, and control tasks and triggers for the experiment. All of our suites are written in python. Within a suite file, you can identify the task it calls by the "suite.addTask()" function. A standard function will be passed a task and the configuration. Additional inputs to tasks are used as triggers. An example of this is:

    rawfile = suite.addTask(ewok.fetchObservations, obsconf)
iodafiles = suite.addTask(ewok.convertObservations, obsconf, rawfile=rawfile)

    Here "rawfile" will be an output from the "fetchObservations" task. Since "rawfile" is used as an input for the "convertObservation" task, then "convertObservation" will not run until after the "fetchObservation" task completes and is what we are referring to as a "trigger".

  2. Tasks - select the runtime script and set up configuration needed at runtime. The task fils are kept at {JEDI_SRC}/ewok/src/ewok/tasks. These are all python based scripts. When adding a new task file, you must update ewok/src/ewok/__init__.py. Inside the task file, you can setup variables to be used at runtime. It is important to note that if your runtime script is a bash script then use "self.RUNTIME_ENV". If your runtime script is a python script then use "self.RUNTIME_YAML". The following line is how you tell the workflow which runtime file to execute:

    self.command = os.path.join(os.environ.get("JEDI_SRC"),
                                      "ewok/src/runtime/getObservationsRun.py")
  3. Runtime - script executed during experiment. Runtime files are located at {JEDI_SRC}/ewok/src/runtime. These are primarily written in bash or python and will be executed when all of the triggers are satisfied. 

ecFlow UI and directories

As part of EWOK's set up, you will notice two variables that pertain to ecFlow which are needed to run an experiment. They are EWOK_WORKDIR and EWOK_FLOWDIR. 


  • No labels