You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 6 Next »

Experiments and Workflows Orchestration Kit

Github repository: https://github.com/JCSDA-internal/ewok

Documentation: https://github.com/JCSDA-internal/ewok/blob/develop/README.md

EWOK Developer Section

EWOK currently uses the ECMWSF's ecFlow software system for implementing workflows. This section will describe how EWOK and ecFlow interact to run a Skylab experiment. Instructions for setting up your environment to can be found in the JEDI Documentation.

EWOK contains 3 important components to the workflow:

  1. Suites - reference tasks and task dependencies. The scripts used for suites is located at {JEDI_SRC}/ewok/src/ewok/suites/.

    A suite will typically build and set up JEDI, define a cycle and loop, and control tasks and triggers for the experiment. All of our suites are written in python. Within a suite file, you can identify the task it calls by the "suite.addTask()" function. A standard function will be passed a task and the configuration. Additional inputs to tasks are used as triggers. An example of this is:

    rawfile = suite.addTask(ewok.fetchObservations, obsconf)
iodafiles = suite.addTask(ewok.convertObservations, obsconf, rawfile=rawfile)

    Here "rawfile" will be an output from the "fetchObservations" task. Since "rawfile" is used as an input for the "convertObservation" task, then "convertObservation" will not run until after the "fetchObservation" task completes and is what we are referring to as a "trigger".

  2. Tasks - select the runtime script and set up configuration needed at runtime. The task fils are kept at {JEDI_SRC}/ewok/src/ewok/tasks. These are all python based scripts. When adding a new task file, you must update ewok/src/ewok/__init__.py. Inside the task file, you can setup variables to be used at runtime. It is important to note that if your runtime script is a bash script then use "self.RUNTIME_ENV". If your runtime script is a python script then use "self.RUNTIME_YAML". The following line is how you tell the workflow which runtime file to execute:

    self.command = os.path.join(os.environ.get("JEDI_SRC"),
                                      "ewok/src/runtime/getObservationsRun.py")
  3. Runtime - script executed during experiment. Runtime files are located at {JEDI_SRC}/ewok/src/runtime. These are primarily written in bash or python and will be executed when all of the triggers are satisfied. 

ecFlow UI 

The ecFlow UI is user friendly and you can follow instructions in the JEDI Documentation to get started. After creating an experiment you can use the UI to suspend and rerun tasks, view log files and configuration, and much more. In order to execute a command on a task, you must click on the task name in the UI. The top bar will then show you which task you have clicked by saying "your_host → exp_id → task_name". You can right click on the task to pull up a menu of options and select what you want to run or view. Some helpful hot keys to use inside the UI are:

  • Suspend: command S (or ctrl S)
  • Rerun: command U (or ctrl U)
  • Execute: command E (or ctrl E)

The UI will also color the boxes to the left of a task to show the status of that task. The UI will update every 60 seconds. If you want to see the most recent status of the tasks, then click the green refresh arrow at the top left of the screen. The colors of the status boxes mean:

  • Red: aborted
  • Green: active
  • Yellow: completed
  • Blue: queued
  • Cyan: submitted
  • Orange: suspended
  • Grey: unknown status

ecFlow Directories

As part of EWOK's set up, you will notice two variables that pertain to ecFlow which are needed to run an experiment. They are EWOK_WORKDIR and EWOK_FLOWDIR. The EWOK_WORKDIR is where all of your experiment files will be saved that are generated by the workflow - such as feedback files, background files, observations, and your forecasts. The EWOK_FLOWDIR will contain configuration files and the runtime files that get executed. Tip: for testing small on the fly changes, after kicking off an experiment you can edit the runtime files in EWOK_FLOWDIR and then restart the task. Although the runtime file in the EWOK repository will not be updated, this method is useful if you need to force something to work or if you want to troubleshoot without touching the repo. 


  • No labels