There were several critical reasons, of technical, personnel, or organizational manners, that drove the design decisions behind most of pyHWRF.  This page attempts to explain some of the more complex, or commonly queried, decisions.  Before delving into details, we would like to mention that these statements are not the official position of NOAA, nor any of the contractors or contracting organizations involved.  They are merely observations that we have made, and problems we have had to work around, during the design and implementation of pyHWRF.

Language and Language Version Decision

The largest problem we have dealt with has been the lack of availability of modern, bug-fixed software.  Most NOAA machines use a Linux distribution designed for enterprise server tasks, and is excellent for that purpose.  However, it has proven largely unsuitable for most other NOAA work.  It lacks many common software packages, forcing us to compile and maintain the software ourselves.  That is a major problem in an operational environment where we need 24/7 support and 100% reliability.  Worse, the software this distribution does provide, is typically 3-5 years behind on feature upgrades and bug fixes the vendor views as non-critical.  That forces us to re-implement existing functionality and work around bugs that were fixed months or years ago by the upstream distribution.

This problem is compounded by the lack of software engineering expertise within the organizations involved in HWRF development.  The NCEP Environmental Modeling Center has had a long-standing policy of not hiring software engineers since the management views the organization as a "science organization."  The other main organization involved in the pyHWRF construction is the Developmental Testbed Center, which does hire software engineers, but has few personnel assigned to the HWRF project.  One more organization is involved: the NCEP Central Operations, which takes the developmental version, and runs it operationally. NCO has substantial software engineering expertise, but unfortunately, it is not involved in the design or creation of NCEP models on a heavy coding, day-to-day basis, with the exception of certain libraries.  NCO mainly is involves itself in the last few weeks, plugging the final HWRF scripting system into the NCO scripting system. 

These limitations necessitated picking a language that is easy for non-programmers to learn and maintain.  Preferably, it should already be known within the NOAA and general geospatial modeling community.  In addition, the language should have some semblance of backward compatibility, and an international standard that describes it, from a recognized standards organization.  Five options were considered:

  • 1993 ksh – most NCEP models used 1988 ksh, which is the only version supported on the old AIX-based machines until a 2013 upgrade to Intel/Linux, which allows 1993 ksh.
  • Perl
  • Ruby
  • Python 2.x
  • Python 3.x

Ease of Use

The Ruby and Python languages are well-known for their ease of use.  Indeed, they are often used for "first programming languages" in grade school programming courses.  The 1993 ksh is certainly not a contender for that prize, but its 1988 version is already well known within all organizations involved, so learning the 1993 version was considered easy. 

Perl stands out as a terrible language for maintainability when written by non-programmers.  While careful coding can result in simple, elegant, highly maintainable Perl code, Perl code written by non-programmers can appear to have come from a random number generator.  This is unfortunate, since the language is suitable for the task of implementing HWRF, and is already used in several parts of the system.  However, it is that experience with Perl that drove the designers of the HWRF to decide against using it in the HWRF scripting rewrite.

Python is already used extensively in the general atmospheric modeling community, and many people in NOAA have been using it for years.  The organization that actually runs the production HWRF, NCEP Central Operations (NCO) has limited knowledge of Python, but does have some.  This is because the European ECMWF software that NCEP uses to run its models, the ecFlow, has Python modules to provide additional functionality to ecFlow users.  Hence, even though NCO's Python experience is limited, it will be getting much larger as time goes on regardless of the language chosen for the HWRF scripting rewrite.

Backward Compatibility and Standards Conformance

It is important in an operational environment for the languages used to provide backward compatibility so that an update to the language does not break operational scripts.  We cannot afford to have to rewrite our entire scripting system every time someone does a commit.  Furthermore, there should be multiple vendors available for any language, allowing competition in contracts to support NOAA work.  Lastly, it is ideal for a language to have a standard in an international standards organization in order to fully describe how it works, allowing such competition and backward compatibility to be provable in the first place.

  • 1993 ksh: This language is in the POSIX 2 standard, and three implementations do exist for it.  However, since the original AT&T implementation by David Korn is open source (as of 2000), and is ported to nearly all unix distributions, this is a non-issue.  Unfortunately, ksh provides no functionality beyond a simple shell: any actual work (making directories, performing computations, etc.) must be done using programs provided elsewhere in the system.  This adds a whole host of other portability concerns, making ksh a dubious option for future portability.  Indeed, this caused many problems in the move from AIX to Linux.
  • Perl: There is only one implementation of Perl, and there are no standards describing it.  However, the language has been quite excellent at backward compatibility, and it has been ported to most unix flavors. 
  • Ruby: There is a standard document for Ruby: ISO/IEC 30170, which is for Ruby 1.8.7, the only version available on NOAA machines.  There is a project to provide new standard documents for later Ruby versions.  Furthermore, there are several implementations of Ruby, and at least the standard Ruby distribution has been ported to multiple operating systems, though not quite to the extent as Perl or ksh. Lastly, although Ruby is not 100% backward compatible, it is nearly so.
  • Python: Python has no standards document, and is instead governed by the Python Software Foundation, an organization dedicated to maintaining Python.  It has a terrible record of backward compatibility, as evidenced by the upgrade to Python 3, which requires a major overhaul of all Python code.  However, it does provide a tool to automate most of the work of these version upgrades, and warns about what features will be modified or removed in the next planned Python upgrade. 
    • Python 2.7.x: this is the long-term support series of Python 2.x.  It will receive only bug fixes and security patches.  Also, there are many interpreters of this version of Python, and several libraries specific to atmospheric and oceanic forecasting, for use with Python 2.x.
    • Python 2.6.6: this is the only version of Python available on the Linux distribution used by NOAA.  It has many bugs and critical missing functionality.  However, upon examining the situation, the developers concluded the problems can be worked around, and an upgrade to the 2.7.x versions will be easy, if pyHWRF carefully avoids deprecated functionality.
    • Python 3.x: This is a major reworking of the Python system, largely to fix inconsistencies in internationalization, and also to rework other aspects of the Python language.  However, this upgrade completely broke backward compatibility, and requires a major overhaul of all Python code.  For this reason, the Python developers created the long-term support series Python 2.7.x, which will continue to receive bug fixes and security patches.  

Final Language and Language Version Decision

The final decision made for what language to use rested on a few hard facts.  NCEP has used ksh for many years and has a long list of problems it causes in implementations.  Future use of ksh was ruled out immediately due to its portability and scalability issues.  Furthermore, any shell language requires the use of numerous spawned processes and temporary files, which causes problems on NOAA's already stressed filesystems.  Perl was ruled out quickly due to the negative experiences we have had with non-programmers programming in Perl.  That left the decision between Ruby and Python.  The final decision to use Python was made almost entirely on the basis of familiarity with the language.  Of course, that leaves a decision between Python versions.

Python 2.x was chosen over Python 3.x for several reasons:

  1. We see no advantage, whatsoever, to this project in using Python 3.  
  2. Several libraries we would like to use in the future do not exist for Python 3 yet.
  3. Python 3 is not available on NOAA machines.
  4. Python 2.7.x will not have backward-incompatible upgrades, unlike future releases of Python 3, since Python 2.7.x will have no upgrades whatsoever, beyond bug and security fixes.

Many other organizations have come to the same conclusions, so the use of Python 2.7.x should be safe for the foreseeable future.  However, it is likely that advantages to Python 3 will develop, and so a future upgrade to Python 3 may one day be on the horizon.  For that reason, we are trying to maintain forward compatibility to Python 3 to the greatest extent possible.

For the present release of pyHWRF, we were forced to use Python 2.6.6, the only version supported by NOAA's Linux vendor.  However, our system is designed with planned forward compatibility to the latest 2.6 and 2.7 series (2.6.8 and 2.7.6, at the time of writing). 

2014 pyHWRF: Python 2.6.6 Standard Library with NO Outside Modules

This leaves one seemingly dubious decision made during the design process: we do not use any Python modules outside the Python 2.6.6 standard library, except the ones that we write.  That is because the only Python modules available on the NCEP operational supercomputers, WCOSS, was the core Python 2.6.6.  There was significant political resistance to adding more Python functionality with operational support from anyone other than EMC.  Python modules written by EMC have operational support from EMC, but what happens if we provide modules we did not write, from the outside community, and those modules break?  We cannot wait days to weeks for volunteer developers in the open source community to come up with a fix.  We need a fix within hours of a failure.  Hence, we went with only the standard library, and wrote our own implementations of anything that is needed beyond that limited functionality. 

One consequence of this decision is that many things that could have been done in Python scripts, are instead done in C or Fortran programs.  In some cases, this is work that would be much simpler if done from Python, with access to the rest of the functionality of the HWRF scripting system.  It is for this reason that we look forward to a future upgrade to the operationally supported version of Python.

Looking Forward: Python 2.7.x with a Full SciPy Stack

FIXME: Insert text here

 

  • No labels