Upon starting the Python rewrite of the HWRF system, we discovered an alarming amount of missing functionality in the Python standard library, as well as many bugs in Python and its underlying libraries, and deficiencies in the design of the language itself. Furthermore, Python and its standard library are terrible at backward compatibility. The produtil package is our workaround for these problems: it adds functionality missing in the standard libraries, and provides a layer of safety between the hwrf package and Python itself, to let us adapt to backward-incompatible upgrades in the Python libraries. Furthermore, it adds a number of utilities needed for working in production environments. This package is not written to be specific to the HWRF: it is designed to be an entirely independent layer to work around Python issues, and provide additional functionality needed in operational environments.
As with all Python packages, this package is split into several modules, each of which provides functions, classes, properties and the like. This page gives an overview of those modules, what is in them, and how they interact. It also describes dependencies between modules in the produtil package. This page is not thorough by design: separate documentation automatically generated from the Python docblocks is more detailed and up-to-date. This page aims to provide only a high level overview.
Module produtil.datastore: Sharing Information Between Jobs
One of the main goals of the produtil package, aside being a "Python issues workaround," is to add a way for batch jobs, cron jobs, and user-run scripts to communicate with each other within the same supercomputer. That allows us to avoid unnecessary filesystem or batch system operations to indirectly monitor other jobs. Instead, when, for example, a post processor has produced an output GRIB file, a regribbing job knows immediately that it is available without having to do a filesystem "stat" operation. It can immediately pick up the file and process it. The produtil.datastore package implements this inter-job communication.
The produtil.datastore uses sqlite3 on-disk database files to share information between jobs, but it does so in an abstracted manner so the database file could be replaced with a proper database server, a flat text file, or some other communication mechanism at a future date, if the need arises. Furthermore, the classes implemented support having multiple database files within the same workflow, which allows them to avoid database lock contention by splitting the workflow up in logical parts. For example, each ensemble member of a hurricane forecast ensemble could have its own database file, with one more shared database file for collective input and output products such as the ensemble mean hurricane track.
Classes
Datastore – this is the class that performs the actual data storage. It maps strings of the form category::id to a Datum object, which has a string type, a string location, an integer availability, and optional string metadata values. The type, location, availability and category::id pair are stored in a PRODUCTS table, while the metadata is stored in a METADATA table. The Datastore supports transactions via the python "with" construct. Transactions allow the underlying database to have several modifications made to it
Datum – the abstract base class of anything that can be stored in a Datastore. This class is generally not accessed directly.
Product – the direct subclass of Datum that represents any product, such as a file, produced by the workflow. It has a location, and it treats its Datastore available integer as a logical true/false value: is the product there (True) or not (False)?
Task – the other direct subclass of Datum: it represents anything that produces and consumes products. It has a task name, which defaults to the id part of category.id but can be anything. It adds a Python logging.Logger object accessible via self.log() with a logging domain set to its task name. It uses the Datastore available integer to define its state: an integer which can have several values: FAILED, UNSTARTED, RUNNING or COMPLETED.
UpstreamFile – a subclass of Product that represents a file produced by a workflow outside this one. For example, if one is running a regional forecasting model that requires input and boundary conditions from a global model, then the global model's output would be an UpstreamFile. This class has a check class method that checks to see if its file exists and has non-zero size, setting self.state accordingly.
FileProduct – a subclass of Product that represents a file created by this workflow. Has a deliver class method that delivers the file, setting self.state to true.
Data Delivery
The Product class, and its subclasses, use two mechanisms to deliver data:
produtil.fileop.deliver_file – documented below, this is a generalized file copy or move operation that FileProduct uses to copy a file from one place to another in FileProduct.deliver.
callbacks – the Product class maintains a list of functions or callable objects to call after the file is delivered, in the deliver class method. One can add to the list by call_callbacks, and this is how the NCEP DBN alerts are implemented in the HWRF. It is the responsibility of the subclasses' implementation of deliver to ensure that Product.call_callbacks is called after the file is copied.
Dependencies
Direct dependencies: produtil.fileop, produtil.locking, produtil.sigsafety
Initialization requirements: for safety, call produtil.setup.setup first. Otherwise, database locks may not be released when using Lustre if the process receives a terminal signal.
Note: if one wants to signal a DBN alert after file delivery, see the produtil.dbnalert package below.
Module produtil.run: Running Programs
One of the most grievous omissions from the Python standard library is a way to launch complex shell commands without either calling a shell, or reimplementing the entire shell yourself. The Python subprocess module provides some means by which this can be done, but lacks critical functionality and requires overly verbose coding. The produtil.run provides a simple, shell-like syntax to launch jobs, via the use of operator overloading, and adds to that a cross-platform means by which to start OpenMP and MPI programs. The actual implementation of produtil.run is actually in the produtil.prog and produtil.mpiprog modules, as well as the produtil.mpi_impl subpackage, none of which should generally be accessed directly.
Executing programs is a two step process:
- Construct a Python object that knows how to run the command.
- Execute the command
Command Construction
The first step to running a command is creating an object that knows what command you want to run. Internally in the implementation, this is the creation of produtil.prog.Runner objects.
Finding the Program
exe('myprog') – creates an object that will search your $PATH for program "myprog".
bigexe('myprog') – same as above, but indicates the program should be run on compute nodes. This is only relevant on systems like Cray where batch job scripts run on different machines than the programs they execute.
mpi('myprog') – same as above, but specifies that the program is an MPI program.
Arguments and Environment Variables
exe('ls')['-l','/usr/bin'] – the [arg1,arg2,arg3,...] syntax adds arguments to a program before it is started
exe('myprog').env(ENSID="03",COLOR="blue") – adds a list of environment variables to be set before a program is started. Note that this does not change the environment variables of the current process: it only modifies the list that will be sent in to the child process when it is executed.
MPI Programs and MPI Execution of Serial Programs
mpi('coupler')+mpi('ocean')*9+mpi('atmos')*200 – specifies an MPI world that will have one rank of a program called "coupler", nine ranks of a program called "ocean" and two hundred ranks of a program called "atmos"
mpirun(mpi('coupler')+mpi('ocean')*9+mpi('atmos')*200) – the "mpirun" subroutine converts an MPI specification to a simple serial command that will start the MPI program on your local platform.
mpirun(mpi('post'),allranks=True) – runs the program "post" on all MPI ranks available to the current batch job
mpirun(mpiserial(exe('pwd')),allranks=True) – generates an MPI command that will run the serial program "pwd" on all ranks
mpirun(mpiserial(exe('hostname')),allranks=True) | exe('sort')['-u'] > hostlist – generates an MPI command that will run the serial program "hostname" on all MPI ranks, pipe the result through the sort command (with argument -u) and store the result in a file "hostlist"
OpenMP Specification
openmp(bigexe('myprog'),threads=15) – specifies a program "myprog" to be run with 15 threads. For compatibility with Cray, you must always use "bigexe" when selecting non-MPI OpenMP programs.
openmp(bigexe('myprog')) – same as above, but uses the maximum number of threads available to this batch job.
openmp(mpirun(mpi('myprog')*32), threads=15) – specifies an MPI environment with 32 ranks of "myprog" and sets the number of OpenMP threads to 15.
Piping and Redirection
exe('ls') | exe('sort') – the pipe ("|") specifies piping the stdout output from a previous command into the stdin of the next
( exe('sort') < "infile" ) > outfile – a less than sign ("<") specifies the stdin input file while a greater than sign (">") specifies the stdout. Note the parentheses, which are needed due to the order of precedence of Python operators.
exe('myprog') >= 'my.log' – a greater than or equal to (">=") redirects both stdout and stderr to the same file
Storing a Command for Later Reuse
my_ls = alias( exe('ls')['-l','--color=none'] ) – the "alias" function creates a produtil.prog.ImmutableRunner object that knows how to run the specified command. This object can be reused later, without modifying the original object.
my_ls['/usr/local'].env(LOCALE='C') – creates a new produtil.prog.Runner object based on "my_ls" but appends the argument "/usr/local" to the end of the argument list, and sets its environment variable $LOCALE to "C".
Command Execution
The following commands will run a produtil.prog.Runner object created by the commands in the above sections:
errcode = run(exe('echo')['hello','world']) – runs the command and returns the integer exit status. If the command died due to a signal, then returns 128 plus the signal number.
checkrun(exe('echo')['hello','world']) – runs the command and raises ExitStatusException if the command exited with non-zero status or died due to a signal
s = runstr(exe('ls')['-l']) – runs the command, raises ExitStatusException if it exited with non-zero status, and stores the stdout from the command in variable "s"
Note that there is no way to background a command. That is because that functionality has not been needed yet in the HWRF Python scripts.
Dependencies
Direct Dependencies: produtil.prog, produtil.mpiprog, produtil.mpi_impl, produtil.sigsafety
Note: you must run produtil.setup.setup to initialize the produtil.sigsafety module, otherwise this module may not properly clean up when receiving a fatal signal.
Module produtil.fileop: Filesystem Operations
The produtil.fileop module is a collection of functions that implement many common filesystem operations, working around bugs in Python and the underlying filesystem or operating system. Most functions accept an optional "logger" argument, which takes a Python logger.Logger object that accepts log messages. The user is directed towards the docblock documentation for details an the various functions, as there are quite a few. However, two notable functions are mentioned here due to how critical they are to the entire HWRF workflow.
Function produtil.fileop.deliver_file: File Delivery
One of the problems with the various file copy or move operations in the Python standard library and POSIX shell commands is that they can leave a partially completed file if the file operation fails, or during file operation. This can wreak havoc on an operational workflow, causing unexpected problems that can be hard to track down. The deliver_file function copies or moves a file, without allowing a partial file to exist at the destination filename during the copy operation. This is done one of two ways:
- If the source and destination are on the same filesystem, and the source is no longer needed, then
deliver_filewill simply move the source to the target. A move is guaranteed to be a unit operation in a POSIX-compliant operating sytem. - If the source and destination are on different filesystems, or if the source is still needed, then
deliver_filewill copy the file to a randomly-generated name in the same directory as the target, with the same permissions as the target, and then do a move (rename) to replace the target, if it exists.
All of this is done with as few filesystem operations as possible, to reduce the filesystem load. There is extensive logging available through the optional logging argument to deliver_file
Function produtil.fileop.check_file: Checking Upstream Files
The produtil.datastore class provides an excellent means to avoid having to "stat" a file to determine whether it is availabe when the file is produced by this workflow. Unfortunately, if an external workflow generates a file we need, there is no way to query its sqlite3 database to see if the file is available: instead, we must check the file some other way. The check_file function provides heuristics to perform this operation. It knows how to check the file size and modification time, and compare times to the local current time to get the file age. This is not a perfect method of determining if a file is "available" but it is the best available when no other information is present, which is often the case
Dependencies
Direct Dependencies: none within the produtil package. This is a standalone module.
Module produtil.retry: Retrying Operations
This module provides a produtil.retry.retry function that runs a Python function or callable object over and over until it works. It has support for logging, it can sleep between retries, and has a pseudrandom exponential backoff algorithm to avoid collisions in cases where multiple threads or processes are retrying a conflicting operation at the same time. It can also call additional functions or callable objects upon failure, and upon giving up after the maximum number of tries has been reached.
Dependencies
No dependencies within the produtil package: this is a standalone module.
Module produtil.dbnalert: NCEP DBN Alerts
This module is a simple wrapper around a call to the "dbn_alert" program. However, it is smart enough to not call dbn_alert, and instead log that it would be called, if SENDDBN=NO or PARAFLAG=YES. A DBN alert is requested in code by sending a produtil.dbnalert.DBNAlert object into the add_callback class method of a produtil.datastore.Product object. This allows a high level script to add DBN alerts without any knowledge of the underlying code, just by finding a product object, and giving it a DBNAlert object to call upon delivery.
Dependencies
Direct Dependencies: produtil.run, produtil.prog
Note: The produtil.prog module is only used in an a debug assertion while checking arguments to the DBNAlert constructor for type correctness.
Module produtil.locking: Safe File Locking
The produtil.locking module provides safer wrappers around the POSIX locking functions via use of Python with LockFile('/path/to/lock/file') blocks. The blocks ensure the file is only locked inside the block, and will unlock and close the file (in that order) before the block exits, even if the program is terminated via a fatal signal. This is needed to work around bugs in some parallel filesystem locking implementations.
To avoid deadlocks during a fatal signal, the signal safety module python.sigsafety disables all locking after a fatal signal is received, by calling produtil.locking.disable_locking. When that happens, attempts to lock a file will raise the produtil.locking.LockingDisabled exception. This speeds up exits during a fatal signal, and also allows one to trace where the code was when it attempted to lock a file during a fatal signal.
Dependencies
Direct Dependencies: produtil.locking, produtil.retry
Note: You must call produtil.setup.setup to initialize the produtil.sigsafety module or the produtil.locking module may not properly respond to a fatal signal. That causes problems on some filesystems, including Lustre.
Module produtil.log: Logging
The Python standard library has excellent logging functionalities in the Python logging module. This module is simply a wrapper around that standard module, which provides default logging formats to stdout, stderr and a "jlogfile." The "jlogfile" is a file used in NOAA NCEP operations to track high-level logging messages from all production jobs. The format used in stdout and stderr is slightly more verbose than that in the jlogfile, and the jlogfile does not contain stack traces. In addition, these are the logging levels in each stream:
stdout – INFO, WARNING, ERROR and CRITICAL. Stack trace included.
stderr – WARNING, ERROR and CRITICAL. Stack trace included.
jlogfile – ERROR and CRITICAL. Stack trace NOT included.
Note that the DEBUG level is not sent anywhere by default. These settings are configurable via either produtil.setup, which is the preferred method, or the lower-level produtil.log.configurelogging subroutine.
Dependencies
Direct Dependency: produtil.batchsystem for determining the job name for logging to the jlogfile.
Note: you should call produtil.setup.setup before using this module.
Module produtil.sigsafety: Signal Handling
The default behavior of the CPython interpreter is to exit the program immediately when receiving any fatal signal other than SIGINT. A SIGINT causes KeyboardInterrupt. This causes problems, especially with file locking on Lustre. The produtil.sigsafety module implements a similar exception raising for additional signals: it raises the FatalSignal or HangupSignal exceptions, both of which derive indirectly from KeyboardInterrupt (for compatibility with other Python libraries). This allows cleanup via "with" or "finally" blocks, while bypassing most "except" blocks since the KeyboardInterrupt does not derive from Exception, but rather BaseException. By default, FatalSignal is raised for SIGINT, SIGTERM and SIGQUIT, while HangupSignal is raised for SIGHUP, but this is configurable.
Note that there is an intentional side-effect of raising a fatal exception upon receiving a fatal signal: if the signal is not caught, one receives a stack trace to the point where the code was when receiving the signal. This allows one to track down deadlocks, or unsatisfied wait loops that lead to the scripts hitting their wallclock limit.
Dependencies
Direct Dependency: calls produtil.locking's disable_locking upon receiving a fatal signal to ensure no more file locks are made after a fatal signal.
Module produtil.tempdir: Directories
This module provides two classes that are intended to be used in Python with blocks: TempDir and NamedDir. The TempDir creates a randomly-named directory, cd's to it during the with block, and then cd's out and optionally deletes the directory after the with block completes. It will never delete the directory during a fatal signal. The NamedDir cd's to a specified directory, but does not delete it (by default) after the with block exits. This behavior is configurable via the constructor.
Dependencies
Direct Dependency: uses functions from produtil.fileop to implement some filesystem operations.
Note: will not delete the directory, even if told to do so, after a produtil.sigsafety.FatalSignal is raised (or anything else that does not derive from Exception or GeneratorExit). This is to avoid the potentially long process of deleting a large directory tree during a signal exit or SystemExit, which should be a nearly instantaneous operation. However, it will cd to the prior directory at the end of the with block, regardless of what exception, if any, occurs during the block.
Other Modules
There are several other modules in the produtil package that are intended only for internal use by the package itself, or exist only for testing future functionality. This is a list, and their purpose:
produtil.prog – contains the Runner and Pipeline classes that are the underlying implementation of most of the produtil.run module. Also knows how to convert a Runner object to a Pipeline object, or to a POSIX sh command (if possible). This module generally should not be directly accessed, except for checking the correct type of input arguments if a produtil.prog.Runner is expected as an argument.
produtil.mpiprog – contains classes for creating MPI programs for input to the produtil.run.mpirun function. This is the underlying implementation of produtil.run.mpi.
produtil.mpi_impl – this is actually a subpackage with several modules inside it, each of which contains the implementation for a particular MPI+OpenMP imlementation pair. It replaces itself with one of those modules upon loading of the produtil.run module. This makes the produtil.mpi_impl appear to be a module rather than a package, and it can be treated as such. You should never load this module (or package) directly: it must be loaded in a particular way in order to ensure that it is setup correctly.
produtil.batchsystem – has a limited ability to provide information from the batch system that launched the current batch job. At present, this is used only to initialize the produtil.log jlogfile functionality, which needs a batch system job name when the environment variable $job is unspecified.
produtil.whereami – specifies which NOAA cluster the job is running on. If it is running on WCOSS, it tells whether it is the production or backup (development) machine. This module is only used by the experimental hwrf.input module for deciding which directory to use to obtain input data. The produtil.whereami module will be replaced by the more general produtil.cluster module at some future date.