Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Jianjun then pointed out that, in practice, radiance data often has many locations and hundreds or even thousands of channels.  He asked if ioda converters are fast enough to process such data in an operational setting.  Steve explained that there is continuing work to optimize ioda-converters.  This includes efforts to move selected converters from python to a compiled language (C++), parallelization of the data processing with MPI, and processing the files line by line in chunks so all data isn't necessarily read into memory simultaneously.

Ling then asked about the limitations of specifying data types for ioda-engines.  Steve responded that ioda engines has a flexible interface that can handle essentially any of all the fundamental data types defined by C++, plus std::string.   And, the data can have an arbitrary number of dimensions.  On a related note, Wojceich later asked whether it's possible to store vectors of variable length.  Steve said they had considered this capability but decided that this is most relevant for conventional data like radiosondes which is only a small fraction of the total data.  It is more important to optimize the format to handle radiance data which dominates the observation data volume and which does tend to have vectors of equal length.  So, they decided to handle radiance and other conventional obs by storing the data in contiguous arrays of locations with marks to note missing data.

...

Steve V asked if the source code for today's examples is available.  Steve H said that the slides will be made available (above).  There are more examples , that are located in the ioda-engines repo (see the slides for the paths to the example code described in the presentation).

Chris S asked if anyone was maintaining a list of who is writing converters for what.  This would be helpful to avoid redundant work.  Yannick said such a list was not being maintained in JEDI and asked Dick about whether the OBS group is keeping track of this.  Dick said not at the moment but it is a good idea and he made a note to address this in the future.

JJ asked if the current converters will become obsolete.  Steve said yes, eventually they will be re-written based on a new API provided by the application "pybind11" which provides inter-operability between python and C++.  So, the new python API will look much like the C++ API that Steve was describing today.  So, the old nc-writer NcWriter class will become obsolete soon.  The new common ground for the converters and ioda will be the structure ObsGroup class now provided by ioda-engines.

...