Upon starting the Python rewrite of the HWRF system, we discovered an alarming amount of missing functionality in the Python standard library, as well as many bugs in Python and its underlying libraries, and deficiencies in the design of the language itself. Furthermore, Python and its standard library are terrible at backward compatibility. The produtil
package is our workaround for these problems: it adds functionality missing in the standard libraries, and provides a layer of safety between the hwrf
package and Python itself, to let us adapt to backward-incompatible upgrades in the Python libraries. Furthermore, it adds a number of utilities needed for working in production environments. This package is not written to be specific to the HWRF: it is designed to be an entirely independent layer to work around Python issues, and provide additional functionality needed in operational environments.
As with all Python packages, this package is split into several modules, each of which provides functions, classes, properties and the like. This page gives an overview of those modules, what is in them, and how they interact. It also describes dependencies between modules in the produtil
package. This page is not thorough by design: separate documentation automatically generated from the Python docblocks is more detailed and up-to-date. This page aims to provide only a high level overview.
Module produtil.datastore
: Sharing Information Between Jobs
One of the main goals of the produtil
package, aside being a "Python issues workaround," is to add a way for batch jobs, cron jobs, and user-run scripts to communicate with each other within the same supercomputer. That allows us to avoid unnecessary filesystem or batch system operations to indirectly monitor other jobs. Instead, when, for example, a post processor has produced an output GRIB file, a regribbing job knows immediately that it is available without having to do a filesystem "stat" operation. It can immediately pick up the file and process it. The produtil.datastore
package implements this inter-job communication.
The produtil.datastore
uses sqlite3 on-disk database files to share information between jobs, but it does so in an abstracted manner so the database file could be replaced with a proper database server, a flat text file, or some other communication mechanism at a future date, if the need arises. Furthermore, the classes implemented support having multiple database files within the same workflow, which allows them to avoid database lock contention by splitting the workflow up in logical parts. For example, each ensemble member of a hurricane forecast ensemble could have its own database file, with one more shared database file for collective input and output products such as the ensemble mean hurricane track.
Classes
Datastore
– this is the class that performs the actual data storage. It maps strings of the form category::id
to a Datum
object, which has a string type
, a string location
, an integer availability
, and optional string metadata values. The type, location, availability and category::id
pair are stored in a PRODUCTS
table, while the metadata is stored in a METADATA
table. The Datastore
supports transactions via the python "with
" construct. Transactions allow the underlying database to have several modifications made to it
Datum
– the abstract base class of anything that can be stored in a Datastore
. This class is generally not accessed directly.
Product
– the direct subclass of Datum
that represents any product, such as a file, produced by the workflow. It has a location, and it treats its Datastore available
integer as a logical true/false value: is the product there (True) or not (False)?
Task
– the other direct subclass of Datum: it represents anything that produces and consumes products. It has a task name, which defaults to the id
part of category.id
but can be anything. It adds a Python logging.Logger
object accessible via self.log()
with a logging domain set to its task name. It uses the Datastore available
integer to define its state
: an integer which can have several values: FAILED
, UNSTARTED
, RUNNING
or COMPLETED
.
UpstreamFile
– a subclass of Product
that represents a file produced by a workflow outside this one. For example, if one is running a regional forecasting model that requires input and boundary conditions from a global model, then the global model's output would be an UpstreamFile
. This class has a check
class method that checks to see if its file exists and has non-zero size, setting self.state
accordingly.
FileProduct
– a subclass of Product
that represents a file created by this workflow. Has a deliver
class method that delivers the file, setting self.state
to true.
Data Delivery
The Product class, and its subclasses, use two mechanisms to deliver data:
produtil.fileop.deliver_file – documented below, this is a generalized file copy or move operation that FileProduct
uses to copy a file from one place to another in FileProduct.deliver
.
callbacks – the Product class maintains a list of functions or callable objects to call after the file is delivered, in the deliver
class method. One can add to the list by call_callbacks
, and this is how the NCEP DBN alerts are implemented in the HWRF. It is the responsibility of the subclasses' implementation of deliver
to ensure that Product.call_callbacks
is called after the file is copied.
Dependencies
Direct dependencies: produtil.fileop
, produtil.locking
, produtil.sigsafety
Initialization requirements: for safety, call produtil.setup.setup
first. Otherwise, database locks may not be released when using Lustre if the process receives a terminal signal.
Note: if one wants to signal a DBN alert after file delivery, see the produtil.dbnalert
package below.
Module produtil.run
: Running Programs
One of the most grievous omissions from the Python standard library is a way to launch complex shell commands without either calling a shell, or reimplementing the entire shell yourself. The Python subprocess
module provides some means by which this can be done, but lacks critical functionality and requires overly verbose coding. The produtil.run
provides a simple, shell-like syntax to launch jobs, via the use of operator overloading, and adds to that a cross-platform means by which to start OpenMP and MPI programs. The actual implementation of produtil.run
is actually in the produtil.prog
and produtil.mpiprog
modules, as well as the produtil.mpi_impl
subpackage, none of which should generally be accessed directly.
Executing programs is a two step process:
- Construct a Python object that knows how to run the command.
- Execute the command
Command Construction
The first step to running a command is creating an object that knows what command you want to run. Internally in the implementation, this is the creation of produtil.prog.Runner
objects.
Finding the Program
exe('myprog')
– creates an object that will search your $PATH
for program "myprog".
bigexe('myprog')
– same as above, but indicates the program should be run on compute nodes. This is only relevant on systems like Cray where batch job scripts run on different machines than the programs they execute.
mpi('myprog')
– same as above, but specifies that the program is an MPI program.
Arguments and Environment Variables
exe('ls')['-l','/usr/bin']
– the [arg1,arg2,arg3,...]
syntax adds arguments to a program before it is started
exe('myprog').env(ENSID="03",COLOR="blue")
– adds a list of environment variables to be set before a program is started. Note that this does not change the environment variables of the current process: it only modifies the list that will be sent in to the child process when it is executed.
MPI Programs and MPI Execution of Serial Programs
mpi('coupler')+mpi('ocean')*9+mpi('atmos')*200
– specifies an MPI world that will have one rank of a program called "coupler", nine ranks of a program called "ocean" and two hundred ranks of a program called "atmos"
mpirun(mpi('coupler')+mpi('ocean')*9+mpi('atmos')*200)
– the "mpirun" subroutine converts an MPI specification to a simple serial command that will start the MPI program on your local platform.
mpirun(mpi('post'),allranks=True)
– runs the program "post" on all MPI ranks available to the current batch job
mpirun(mpiserial(exe('pwd')),allranks=True)
– generates an MPI command that will run the serial program "pwd" on all ranks
mpirun(mpiserial(exe('hostname')),allranks=True) | exe('sort')['-u'] > hostlist
– generates an MPI command that will run the serial program "hostname" on all MPI ranks, pipe the result through the sort command (with argument -u) and store the result in a file "hostlist"
OpenMP Specification
openmp(bigexe('myprog'),threads=15)
– specifies a program "myprog" to be run with 15 threads. For compatibility with Cray, you must always use "bigexe" when selecting non-MPI OpenMP programs.
openmp(bigexe('myprog'))
– same as above, but uses the maximum number of threads available to this batch job.
openmp(mpirun(mpi('myprog')*32), threads=15)
– specifies an MPI environment with 32 ranks of "myprog" and sets the number of OpenMP threads to 15.
Piping and Redirection
exe('ls') | exe('sort')
– the pipe ("|") specifies piping the stdout output from a previous command into the stdin of the next
( exe('sort') < "infile" ) > outfile
– a less than sign ("<") specifies the stdin input file while a greater than sign (">") specifies the stdout. Note the parentheses, which are needed due to the order of precedence of Python operators.
exe('myprog') >= 'my.log'
– a greater than or equal to (">=") redirects both stdout and stderr to the same file
Storing a Command for Later Reuse
my_ls = alias( exe('ls')['-l','--color=none'] )
– the "alias" function creates a produtil.prog.ImmutableRunner
object that knows how to run the specified command. This object can be reused later, without modifying the original object.
my_ls['/usr/local'].env(LOCALE='C')
– creates a new produtil.prog.Runner
object based on "my_ls" but appends the argument "/usr/local" to the end of the argument list, and sets its environment variable $LOCALE to "C".
Command Execution
The following commands will run a produtil.prog.Runner
object created by the commands in the above sections:
errcode = run(exe('echo')['hello','world'])
– runs the command and returns the integer exit status. If the command died due to a signal, then returns 128 plus the signal number.
checkrun(exe('echo')['hello','world'])
– runs the command and raises ExitStatusException
if the command exited with non-zero status or died due to a signal
s = runstr(exe('ls')['-l'])
– runs the command, raises ExitStatusException if it exited with non-zero status, and stores the stdout from the command in variable "s"
Note that there is no way to background a command. That is because that functionality has not been needed yet in the HWRF Python scripts.
Dependencies
Direct Dependencies: produtil.prog
, produtil.mpiprog
, produtil.mpi_impl
, produtil.sigsafety
Note: you must run produtil.setup.setup
to initialize the produtil.sigsafety
module, otherwise this module may not properly clean up when receiving a fatal signal.
Module produtil.fileop
: Filesystem Operations
The produtil.fileop
module is a collection of functions that implement many common filesystem operations, working around bugs in Python and the underlying filesystem or operating system. Most functions accept an optional "logger" argument, which takes a Python logger.Logger
object that accepts log messages. The user is directed towards the docblock documentation for details an the various functions, as there are quite a few. However, two notable functions are mentioned here due to how critical they are to the entire HWRF workflow.
Function produtil.fileop.deliver_file: File Delivery
One of the problems with the various file copy or move operations in the Python standard library and POSIX shell commands is that they can leave a partially completed file if the file operation fails, or during file operation. This can wreak havoc on an operational workflow, causing unexpected problems that can be hard to track down. The deliver_file
function copies or moves a file, without allowing a partial file to exist at the destination filename during the copy operation. This is done one of two ways:
- If the source and destination are on the same filesystem, and the source is no longer needed, then
deliver_file
will simply move the source to the target. A move is guaranteed to be a unit operation in a POSIX-compliant operating sytem. - If the source and destination are on different filesystems, or if the source is still needed, then
deliver_file
will copy the file to a randomly-generated name in the same directory as the target, with the same permissions as the target, and then do a move (rename) to replace the target, if it exists.
All of this is done with as few filesystem operations as possible, to reduce the filesystem load. There is extensive logging available through the optional logging
argument to deliver_file
Function produtil.fileop.check_file: Checking Upstream Files
The produtil.datastore
class provides an excellent means to avoid having to "stat" a file to determine whether it is availabe when the file is produced by this workflow. Unfortunately, if an external workflow generates a file we need, there is no way to query its sqlite3 database to see if the file is available: instead, we must check the file some other way. The check_file
function provides heuristics to perform this operation. It knows how to check the file size and modification time, and compare times to the local current time to get the file age. This is not a perfect method of determining if a file is "available" but it is the best available when no other information is present, which is often the case
Dependencies
Direct Dependencies: none within the produtil
package. This is a standalone module.
Module produtil.retry
: Retrying Operations
This module provides a produtil.retry.retry
function that runs a Python function or callable object over and over until it works. It has support for logging, it can sleep between retries, and has a pseudrandom exponential backoff algorithm to avoid collisions in cases where multiple threads or processes are retrying a conflicting operation at the same time. It can also call additional functions or callable objects upon failure, and upon giving up after the maximum number of tries has been reached.
Dependencies
No dependencies within the produtil
package: this is a standalone module.
Module produtil.dbnalert
: NCEP DBN Alerts
This module is a simple wrapper around a call to the "dbn_alert" program. However, it is smart enough to not call dbn_alert, and instead log that it would be called, if SENDDBN=NO
or PARAFLAG=YES
. A DBN alert is requested in code by sending a produtil.dbnalert.DBNAlert
object into the add_callback
class method of a produtil.datastore.Product
object. This allows a high level script to add DBN alerts without any knowledge of the underlying code, just by finding a product object, and giving it a DBNAlert object to call upon delivery.
Dependencies
Direct Dependencies: produtil.run
, produtil.prog
Note: The produtil.prog
module is only used in an a debug assertion while checking arguments to the DBNAlert
constructor for type correctness.
Module produtil.locking
: Safe File Locking
The produtil.locking
module provides safer wrappers around the POSIX locking functions via use of Python with LockFile('/path/to/lock/file')
blocks. The blocks ensure the file is only locked inside the block, and will unlock and close the file (in that order) before the block exits, even if the program is terminated via a fatal signal. This is needed to work around bugs in some parallel filesystem locking implementations.
To avoid deadlocks during a fatal signal, the signal safety module python.sigsafety
disables all locking after a fatal signal is received, by calling produtil.locking.disable_locking
. When that happens, attempts to lock a file will raise the produtil.locking.LockingDisabled
exception. This speeds up exits during a fatal signal, and also allows one to trace where the code was when it attempted to lock a file during a fatal signal.
Dependencies
Direct Dependencies: produtil.locking
, produtil.retry
Note: You must call produtil.setup.setup
to initialize the produtil.sigsafety
module or the produtil.locking
module may not properly respond to a fatal signal. That causes problems on some filesystems, including Lustre.
Module produtil.log
: Logging
The Python standard library has excellent logging functionalities in the Python logging
module. This module is simply a wrapper around that standard module, which provides default logging formats to stdout, stderr and a "jlogfile." The "jlogfile" is a file used in NOAA NCEP operations to track high-level logging messages from all production jobs. The format used in stdout and stderr is slightly more verbose than that in the jlogfile, and the jlogfile does not contain stack traces. In addition, these are the logging levels in each stream:
stdout – INFO, WARNING, ERROR and CRITICAL. Stack trace included.
stderr – WARNING, ERROR and CRITICAL. Stack trace included.
jlogfile – ERROR and CRITICAL. Stack trace NOT included.
Note that the DEBUG level is not sent anywhere by default. These settings are configurable via either produtil.setup
, which is the preferred method, or the lower-level produtil.log.configurelogging
subroutine.
Dependencies
Direct Dependency: produtil.batchsystem
for determining the job name for logging to the jlogfile.
Note: you should call produtil.setup.setup
before using this module.
Module produtil.sigsafety
: Signal Handling
The default behavior of the CPython interpreter is to exit the program immediately when receiving any fatal signal other than SIGINT. A SIGINT causes KeyboardInterrupt. This causes problems, especially with file locking on Lustre. The produtil.sigsafety
module implements a similar exception raising for additional signals: it raises the FatalSignal or HangupSignal exceptions, both of which derive indirectly from KeyboardInterrupt (for compatibility with other Python libraries). This allows cleanup via "with" or "finally" blocks, while bypassing most "except" blocks since the KeyboardInterrupt does not derive from Exception, but rather BaseException. By default, FatalSignal is raised for SIGINT, SIGTERM and SIGQUIT, while HangupSignal is raised for SIGHUP, but this is configurable.
Note that there is an intentional side-effect of raising a fatal exception upon receiving a fatal signal: if the signal is not caught, one receives a stack trace to the point where the code was when receiving the signal. This allows one to track down deadlocks, or unsatisfied wait loops that lead to the scripts hitting their wallclock limit.
Dependencies
Direct Dependency: calls produtil.locking
's disable_locking
upon receiving a fatal signal to ensure no more file locks are made after a fatal signal.
Module produtil.tempdir
: Directories
This module provides two classes that are intended to be used in Python with
blocks: TempDir
and NamedDir
. The TempDir
creates a randomly-named directory, cd's to it during the with
block, and then cd's out and optionally deletes the directory after the with
block completes. It will never delete the directory during a fatal signal. The NamedDir
cd's to a specified directory, but does not delete it (by default) after the with
block exits. This behavior is configurable via the constructor.
Dependencies
Direct Dependency: uses functions from produtil.fileop
to implement some filesystem operations.
Note: will not delete the directory, even if told to do so, after a produtil.sigsafety.FatalSignal
is raised (or anything else that does not derive from Exception
or GeneratorExit
). This is to avoid the potentially long process of deleting a large directory tree during a signal exit or SystemExit
, which should be a nearly instantaneous operation. However, it will cd
to the prior directory at the end of the with
block, regardless of what exception, if any, occurs during the block.
Other Modules
There are several other modules in the produtil
package that are intended only for internal use by the package itself, or exist only for testing future functionality. This is a list, and their purpose:
produtil.prog
– contains the Runner
and Pipeline
classes that are the underlying implementation of most of the produtil.run
module. Also knows how to convert a Runner
object to a Pipeline
object, or to a POSIX sh command (if possible). This module generally should not be directly accessed, except for checking the correct type of input arguments if a produtil.prog.Runner
is expected as an argument.
produtil.mpiprog
– contains classes for creating MPI programs for input to the produtil.run.mpirun
function. This is the underlying implementation of produtil.run.mpi
.
produtil.mpi_impl
– this is actually a subpackage with several modules inside it, each of which contains the implementation for a particular MPI+OpenMP imlementation pair. It replaces itself with one of those modules upon loading of the produtil.run
module. This makes the produtil.mpi_impl
appear to be a module rather than a package, and it can be treated as such. You should never load this module (or package) directly: it must be loaded in a particular way in order to ensure that it is setup correctly.
produtil.batchsystem
– has a limited ability to provide information from the batch system that launched the current batch job. At present, this is used only to initialize the produtil.log
jlogfile functionality, which needs a batch system job name when the environment variable $job
is unspecified.
produtil.whereami
– specifies which NOAA cluster the job is running on. If it is running on WCOSS, it tells whether it is the production or backup (development) machine. This module is only used by the experimental hwrf.input
module for deciding which directory to use to obtain input data. The produtil.whereami
module will be replaced by the more general produtil.cluster
module at some future date.