This page describes one of the layers of the pyHWRF system: the Scripting Layer.  The job of the Scripting Layer is to set the environment variables needed to properly run the HWRF on the supercomputer or desktop machine chosen.  Strictly speaking, this layer is optional.  It is possible to run the entire system manually through a few batch jobs.  However, in practice, it is easier to have some sort of scripting system sitting on top of the Experiment Layer. 

Duties of the Scripting Layer

The next layer below this one, the Experiment Layer, merely provides a module that produces set of Python objects, which each know how to run part of the HWRF workflow.  The Experiment layer also specifies paths, and the experiment configuration.  However, it does not actually run anything: that is the purpose of the Scripting Layer.  The scripting layer must separate the system into runnable parts, and actually execute each portion of the HWRF system.  Generally, this also involves setting the $PATH and other environment variables so that the compiled executables will actually be able to run.  Of course, this Scripting Layer could be a human, running an interactive Python shell, manually executing each Task's run method.  However, that is laborious and error-prone, so some form of Scripting Layer is recommended.

NCEP Jobs/Scripts/USH Scripting Layer

For historical reasons, the National Centers for Environmental Prediction have used a three tier system to implement their Scripting Layer, known as the NCEP Vertical Structure.  This model has worked well, and so NCEP will continue using it for the foreseeable future.  This section describes the scripting portions of the NCEP Vertical Structure, and how the Python HWRF uses it.

General Meaning of Each Tier

This three tier system assigns one directory to each forecasting model, with several subdirectories, each of which has a name and purpose.  It is referred to as the NCEP Vertical Structure:

  • exec – binary executables
  • parm – small read-only parameter files.  This is where the Experiment Layer hwrf.conf file is kept.
  • fix – large read-only parameter files.
  • lib – binary libraries
  • sorc – source code for executables
  • lib/sorc – source code for libraries
  • jobs – contains the J-Jobs (detailed below)
  • scripts – contains the ex-Scripts (detailed below)
  • ush – contains the utility scripts (detailed below)

Note that non-binary libraries, such as Perl, Python and Ruby libraries, do not go in lib/: lib/ is only for binary libraries such as from Fortran or C.  This is for historical reasons: when these directories were originally assigned meanings many years ago, scripting language libraries had not yet been thought up.  Hence the proper location of a Python library is in somewhat of a limbo status right now.  For now, we store them in the ush directory, along with the utility scripts.

J-Jobs

The Jobs, usually called J-Jobs, are simple scripts that set environment variables to ensure the lower-level scripts are able to run their programs.  They lie within the jobs/ subdirectory of each model.  In addition, it configures delivery locations and delivery methods, as well as logging methods.  These are written in whatever native shell language is most useful for the supercomputer used by NCEP: presently this is ksh.  Generally, one does not have much of a choice of languages, due to the use of the unix module command on all NOAA clusters, as well as other dirty tricks, which only exists for certain shells.

Note that these J-Jobs are not necessarily appropriate for all machines: generally users will want to replace them with files specific to their cluster.  Also, some users may find it easier to configure the $PATH and other environment variables in the next higher layer: the Workflow Layer.  The operational system is unique in that it needs to configure quite a lot of environment variables in order for NCEP's workflow system to work properly, hence the complexity of the HWRF J-Jobs.

ex-Scripts

The Scripts files, sometimes called ex-Scripts, are within each model's scripts/ subdirectory.  These files contain the high-level logic of the Scripting Layer.  In the pyHWRF, these are Python scripts which load the Experiment Layer's hwrf_expt.py and hwrf_alerts.py and tell each hwrf.hwrftask.WRFTask in the hwrf_expt module to run itself. 

Utility Scripts (USH)

This layer contains simple utilities to support the work of the ex-Scripts.  In the pyHWRF, the Scripting Layer does not store anything here.  This is where the HWRF Layer hwrf package and Platform Compatibility Layer produtil packages reside, along with some utility scripts from the Experiment Layer, and other layers if needed.

 

  • No labels