David Simonin from the Met Office gave the following presentation.

In summary, the current ObsSpace in-memory storage is imposing structure (ie, dimensionality) in places where it is not needed (and undesired). The problem with this is that a significant number of client operations (ie, UFO filters, QC, operators) don't need that structure and can be made more generic if the structure is not applied. For example, a QC check for bad data just needs to run through the obs values and reject the ones that are outside a range of reasonable values. There is no need to know if the underlying data is 1D, 2D, 3D etc. If the data can be retrieved out of the ObsSpace as an unstructured list then the implementation of the bad data check can be simplified (since it doesn't have to query what the structure is and then access accordingly).

Some operators, however, do want the structure so the idea with this proposal is to allow clients to access data in an unstructured manner, and also access data in a structured manner when there is an underlying structure to that data AND the client needs to view it as structured data.

A discussion ensued with some points (not all points unfortunately) captured here:

  • Only talking about changing the in-memory piece of IODA (the files will remain as they are)
  • Make precision a template and people can choose
  • Hernan G
    • Radar data has radial components (eg, radial velocity)
      • Just pointing out a different kind of structure that needs to be supported
  • Tom
    • Will unstructured scheme hurt us in performance?
    • Needs to scale up to greater numbers of obs types
  • Cory
    • DataFrame library will be difficult to get approval from NCO
    • Perhaps need to pull in DataFrame and make into a JCSDA repo (develop ourselves)?
  • Ben J
    • Sounds like this will impact clients that do need structure
      • will clients need to create new interfaces to add structure?
    • Can DataFrame or something like DataFrame provide this?

The design and implementation of this idea both need further development, and we are completely open to other ideas of how to solve this issue.


  • No labels