There were several critical reasons, of technical, personnel, or organizational manners, that drove the design decisions behind most of pyHWRF. This page attempts to explain some of the more complex, or commonly queried, decisions. Before delving into details, we would like to mention that these statements are not the official position of NOAA, nor any of the contractors or contracting organizations involved. They are merely observations that we have made, and problems we have had to work around, during the design and implementation of pyHWRF.
The largest problem we have dealt with has been the lack of availability of modern, bug-fixed software. Most NOAA machines use a Linux distribution designed for enterprise server tasks, and is excellent for that purpose. However, it has proven largely unsuitable for most other NOAA work. It lacks many common software packages, forcing us to compile and maintain the software ourselves. That is a major problem in an operational environment where we need 24/7 support and 100% reliability. Worse, the software this distribution does provide, is typically 3-5 years behind on feature upgrades and bug fixes the vendor views as non-critical. That forces us to re-implement existing functionality and work around bugs that were fixed months or years ago by the upstream distribution.
This problem is compounded by the lack of software engineering expertise within the organizations involved in HWRF development. The NCEP Environmental Modeling Center has had a long-standing policy of not hiring software engineers since the management views the organization as a "science organization." The other main organization involved in the pyHWRF construction is the Developmental Testbed Center, which does hire software engineers, but has few personnel assigned to the HWRF project. One more organization is involved: the NCEP Central Operations, which takes the developmental version, and runs it operationally. NCO has substantial software engineering expertise, but unfortunately, it is not involved in the design or creation of NCEP models on a heavy coding, day-to-day basis, with the exception of certain libraries. NCO mainly is involves itself in the last few weeks, plugging the final HWRF scripting system into the NCO scripting system.
These limitations necessitated picking a language that is easy for non-programmers to learn and maintain. Preferably, it should already be known within the NOAA and general geospatial modeling community. In addition, the language should have some semblance of backward compatibility, and an international standard that describes it, from a recognized standards organization. Five options were considered:
The Ruby and Python languages are well-known for their ease of use. Indeed, they are often used for "first programming languages" in grade school programming courses. The 1993 ksh is certainly not a contender for that prize, but its 1988 version is already well known within all organizations involved, so learning the 1993 version was considered easy.
Perl stands out as a terrible language for maintainability when written by non-programmers. While careful coding can result in simple, elegant, highly maintainable Perl code, Perl code written by non-programmers can appear to have come from a random number generator. This is unfortunate, since the language is suitable for the task of implementing HWRF, and is already used in several parts of the system. However, it is that experience with Perl that drove the designers of the HWRF to decide against using it in the HWRF scripting rewrite.
Python is already used extensively in the general atmospheric modeling community, and many people in NOAA have been using it for years. The organization that actually runs the production HWRF, NCEP Central Operations (NCO) has limited knowledge of Python, but does have some. This is because the European ECMWF software that NCEP uses to run its models, the ecFlow, has Python modules to provide additional functionality to ecFlow users. Hence, even though NCO's Python experience is limited, it will be getting much larger as time goes on regardless of the language chosen for the HWRF scripting rewrite.
It is important in an operational environment for the languages used to provide backward compatibility so that an update to the language does not break operational scripts. We cannot afford to have to rewrite our entire scripting system every time someone does a commit. Furthermore, there should be multiple vendors available for any language, allowing competition in contracts to support NOAA work. Lastly, it is ideal for a language to have a standard in an international standards organization in order to fully describe how it works, allowing such competition and backward compatibility to be provable in the first place.
The final decision made for what language to use rested on a few hard facts. NCEP has used ksh for many years and has a long list of problems it causes in implementations. Future use of ksh was ruled out immediately due to its portability and scalability issues. Furthermore, any shell language requires the use of numerous spawned processes and temporary files, which causes problems on NOAA's already stressed filesystems. Perl was ruled out quickly due to the negative experiences we have had with non-programmers programming in Perl. That left the decision between Ruby and Python. The final decision to use Python was made almost entirely on the basis of familiarity with the language. Of course, that leaves a decision between Python versions.
Python 2.x was chosen over Python 3.x for several reasons:
Many other organizations have come to the same conclusions, so the use of Python 2.7.x should be safe for the foreseeable future. However, it is likely that advantages to Python 3 will develop, and so a future upgrade to Python 3 may one day be on the horizon. For that reason, we are trying to maintain forward compatibility to Python 3 to the greatest extent possible.
For the present release of pyHWRF, we were forced to use Python 2.6.6, the only version supported by NOAA's Linux vendor. However, our system is designed with planned forward compatibility to the latest 2.6 and 2.7 series (2.6.8 and 2.7.6, at the time of writing).
This leaves one seemingly dubious decision made during the design process: we do not use any Python modules outside the Python 2.6.6 standard library, except the ones that we write. That is because the only Python modules available on the NCEP operational supercomputers, WCOSS, was the core Python 2.6.6. There was significant political resistance to adding more Python functionality with operational support from anyone other than EMC. Python modules written by EMC have operational support from EMC, but what happens if we provide modules we did not write, from the outside community, and those modules break? We cannot wait days to weeks for volunteer developers in the open source community to come up with a fix. We need a fix within hours of a failure. Hence, we went with only the standard library, and wrote our own implementations of anything that is needed beyond that limited functionality.
One consequence of this decision is that many things that could have been done in Python scripts, are instead done in C or Fortran programs. In some cases, this is work that would be much simpler if done from Python, with access to the rest of the functionality of the HWRF scripting system. It is for this reason that we look forward to a future upgrade to the operationally supported version of Python.
FIXME: Insert text here