This webpage documents the project to rewrite the HWRF automation scripts in Python. At the time this project started, there were a total of four different scripting systems within NCEP and DTC alone, which independently implemented the HWRF, using a combination of ksh, Perl, awk, sed, grep and other languages. The systems had insufficient fault tolerance and error detection, and were a major drain on manpower. Management and developers came to the decision that the scripts need to be replaced.
We document here both the ongoing work to add functionality to the Python-based HWRF, hereafter called the pyHWRF, and also provide high-level documentation of the existing functionality and design of the system. Low-level documentation will be provided automatically generated from the Python docstrings within the source code itself. This website only attempts to provide a high-level description of the modules, packages, scripts and how to use them and modify them. Detailed descriptions of every single function are available in the docstrings.
Presentations and Workgroup Notes
Original Project Proposal (September 19, 2013)
This is the original project proposal, given to management in the NCEP Environmental Modeling Center (EMC), NCEP Central Operations (NCO) and the Developmental Testbed Center (DTC) on September 19, 2013. It was at that meeting that management decided on Python for the rewrite (alternatives being Ruby or 1993 ksh). Their decision was primarly due to prior existing knowledge of Python throughout the atmosphere and ocean modeling community, especially within NOAA.
Notes PyHWRF Telecon Friday, June 20, 2014
Python Post-Processing Presentation from EMC to NCO (April 4, 2014)
This presentation was given to describe the Python post-processing and delivery systems that are expected to be in the operational HWRF in May 2014. This presentation was given to the NCEP Central Operations developers that would be implementing HWRF in operations, as well as Environmental Modeling Center researchers that would be supporting them. The purpose was to describe the product delivery system, the implementation of the HWRF components, and how they link to one another.
Repository
Due to the rapid development of the 2014 HWRF system over the past weeks, this project is presently advancing in three branches. Soon, all three will be merged into the HWRF trunk. For now, relocation work is ongoing in the following branch, which runs the 2013 HWRF system:
The production 2014 system is in the "H214" branch, which only runs pyHWRF post-processing. This branch will be deleted once the WRF and UPP updates for 2014 are merged to the trunk:
The pyH214 branch runs the 2014 system, and has additional experimental code added in to run most of the initialization and the forecast in Python. It will soon be merged to the pyHWRF branch, and the pyH214 will be deleted:
Documentation
The documentation of the pyHWRF is split into several pages due to the large size of the available documentation. New readers are directed to the Technical Overview which provides an overview of the entire system, and links to additional documentation.
FIXME: add the missing pages.
- Technical Overview – an overview of the pyHWRF system
- Intercycle and Workflow Layers – describes the available and planned implementations of the optional top two layers of the system, which implements the interaction with the batch system
- ecflow HWRF – the automation system used by NCEP Central Operations to run the operational HWRF. It is based on the ECMWF ecflow automation system.
- HHS and kick_scripts – the system used by the NCEP Environmental Modeling Center (EMC) to automate EMC HWRF simulations for the past few years.
- Rocoto HWRF – Rocoto is a NOAA-originated workflow management system with capabilities similar to ecflow. EMC plans on using this instead of the HHS+kick_scripts combination.
- Scripting Layer – sets the pyHWRF data locations, and controls when and where each HWRF component is run
- NCEP jobs/scripts/ush – the three tier implementation used by NCEP
- Experiment Layer – sets the experiment configuration
- HWRF Layer – runs the HWRF components. This is the
hwrfPython package. - Platform Compatibility Layer – utilities that sit between the
hwrfpackage and Python itself, providing a platform-independent environment.
- Intercycle and Workflow Layers – describes the available and planned implementations of the optional top two layers of the system, which implements the interaction with the batch system
- Reasons for Design Decisions – explanations about why certain decisions were made in the design of the pyHWRF
- Detailed documentation – automatically generated from Python docstrings. This details every function, class and property in the Python code.
UML
This section is a placeholder for additional future documentation of the project structure.
Tasks
In order to complete the re-write to obtain a system that will produce results equivalent to the 2014 HWRF implementation the following needs to be achieved.
- High level model/system flow
- Low level components
The high level system.
The flow is defined within ush/hwrf_expt.py, while the system configuration is defined within parm/hwrf.conf.
To run an experiment/storm you need to modify the initial kick script pyhwrf_driver.py and then execute for the storm.
fe2$ cd kick fe2$ $EDITOR pyhwrf_driver.py fe2$ ./pyhwrf_driver.py 2012102806 18L HISTORY
The items to edit within pyhwrf_driver.py are
| Variable | Value (examples) |
|---|---|
| diskgroup | 'dtc-hurr' |
| rungroup | 'dtc-hurr' |
| projdir | '/pan2/projects/dtc-hurr' |
Note: That the values have to be quoted.
Low level components
- EDIT THE CALENDAR
Customize the different types of events you'd like to manage in this calendar.
#legIndex/#totalLegs - RESTRICT THE CALENDAR
Optionally, restrict who can view or add events to the team calendar.
#legIndex/#totalLegs - SHARE WITH YOUR TEAM
Grab the calendar's URL and email it to your team, or paste it on a page to embed the calendar.
#legIndex/#totalLegs - ADD AN EVENT
The calendar is ready to go! Click any day on the calendar to add an event or use the Add event button.
#legIndex/#totalLegs - SUBSCRIBE
Subscribe to calendars using your favorite calendar client.
#legIndex/#totalLegs