2021-02-11

We had a meeting yesterday of the JEDI 1 core team (Steve, Maryam, Mark O, Mark M).  The main agenda item was a presentation by Mark O on his work profiling the fv3-jedi Hofx application on S4.  Profiling tools that he is assessing include Vtune, Callgrind (an offshoot of valgrind that tracks where, when, and how often functions and subroutines are called), and Massif (another valgrind offshoot that tracks heap memory usage).  These are useful tools for deep dives into performance for moderate runs with a limited number of MPI tasks.  But, he is also assessing other potential tools that can be scaled to thousands of MPI tasks and that are portable across HPC systems.  We will have a focused topic discussion at some point in the future on this optimization work.

In other updates, Mark, Maryam, and Claude (maybe others?) have been attending the ECMWF Workshop on Weather and Climate in the Cloud this week.

Maryam is also working on how best to implement the new handling of test data through the ioda-data and ufo-data repositories.  At issue is whether to do this with explicit git calls in get_test_data scripts or to handle it through cmake in the bundles.  We'll discuss this further and make a decision on how to proceed at the next JEDI 1 meeting.

Steve continues to work on performance issues with the ioda-engines integration, as covered in the JEDI 2 update.

The issue and pull request templates in GitHub/ZenHub are broken.
New templates have been defined for the JCSDA-internal GitHub organization.  The issue templates include explicit fields to be filled out when you create a general issue or report a bug.  You'll see these if you create an issue from GitHub.  However, ZenHub does not recognize organization-wide issue templates using the new GitHub format.  The issue has been raised with ZenHub, but we don't expect this to be fixed any time soon.  So, we may have to copy these issue templates into each repository for them to become effective.

Since pull requests are typically created from GitHub rather than ZenHub, the organization templates *should* work.  But, it seems they do not, so Mark M will look into debugging this.

JEDI2 met yesterday. Meeting notes are at [1]. There were two parts to this meeting. First, Steve presented draft guidelines for writing IODA converters [2]. The guidelines focus mostly on technical details, such as how to best write strings, appropriate data types for variables, and setting appropriate dimensions for data. This should eventually become part of the schema defining the organization of a valid ioda file.

For the remainder of the meeting, we looked at ZenHub roadmaps to see where we are in terms of the ioda release and to see which issues are best moved to AOP21.

Several group members reported contributions, which are summarized here:
        • Anna created a converter to turn gsi's satbias_in files into ioda files. This gets rid of iostream code in ufo and allows us to refer to bias predictors by their proper names. Previously, names were hardcoded whether they were valid or not.
        • Anna also has propagated the bias correction code inside of the ObsOperator::simulateObs function. This means that our HofX output fields now contain the result of a particular observation operator plus any bias correction. Formerly, the raw output and bias were computed and stored separately.
        • Steve has submitted a draft PR linking ObsSpaces with ioda engines, and is halfway through optimizing the code. He has also produced bug fixes for the Halo obs distribution and other parts of ioda.
        • August has been working on option-parsing code for the ODB engine. He will look at the EMC bufr code in ioda_converters and see if the parsing code can be adapted for his purpose. If so, then this code could be generalized and moved into ioda.
        • Chris has two pull requests for profile vertical averaging. One is merged, and one is in review.
        • Neill's PR on profile background checks is now merged.
        • Wojciech has added a new action to the UFO QC code. This action, "accept", is the opposite of "reject". He also added support in the ObsDiagnostics code for storing background error estimates, and also created a new, composite, observation operator that acts as a wrapper for multiple operators.

Links:
[1] - https://docs.google.com/document/d/10nhcaKYcYM9_B6yO6c2rJQUINmfH9f7-5aoeOdyyiGQ
[2] - https://docs.google.com/document/d/1pLV5P5R_AtEDgwNDWQcWmoTk9ksd9J9SQiiMt4CgA_U2 Attachments

Benjamin: working on a way to parallelize vertical balance estimation for large ensembles for long periods of time. Has PRs in saber and model repositories introducing barycentric Lagrange interpolation

BJ: issued PR for linear variable change in mpas-jedi, converting pseudo RH to specific humidity (so pseudoRH could be used for covariances). Nancy added that NRL uses pseudoRH as the humidity variable, and they are open to discussion on this. Nancy also noted that the disadvantage of using pseudoRH is that it introduces background dependency.
BJ: has a PR in oops adding an option to skip the nonlinear Jb evaluation.

Clementine: working on some minimizer bug fixes, and diagnostics for block Lanczos minimizer.

JJ: running experiments with EDA with mpas-jedi.

Marek: has added time interpolation in um-jedi. Plans to add a generic time interpolation in oops; probably not until late March-April.

Nancy and Sarah: work on neptune refactoring, and catching up to changes in JEDI interfaces.

Olly: works on code for transfering atlas fields from a functionspace partitioned in one way to a functionspace partitioned in the other way.

Sergey: discussed plans for a reanalysis configuration; the initial runs would use JEDI in the ocean, LETKF-JEDI in the atmosphere and JEDI land data assimilation. Continuing to work on Halo distribution update in oops.
Wei and Sergey are also looking at LETKF performance and are profiling LETKF code.

The Travis-CI build/test flow is broken. The GSL library is not being found. One thing to note about the SOCA build is that the components ufo and ioda are built in separate bundles and this issue is related to that fact. Because of this, the other model CI build/test flows are not impacted. (Note, since this meeting, this issue has been resolved.)

Travis is working on an EWOK for SOCA flow. He is anticipating results related to reanalysis by the end of March.

EMC is helping with GEOS CICE interface development.

Hamideh is working on CRTM related development for ocean obs.

UFO QC and Filter flows are being developed for ocean obs.

The Water Model interface is making good progress with activities including B-Matrix localization, grid simulation, Travis-CI configuration, and ioda converters for land obs ingest.

Youlong has developed land obs ioda converters (Python scripts).

Sergey has helped with the integration of surface elevation into land obs, primarily for downstream LETKF analysis.

Next steps include UFO operators for snow pack and UFO QC for snow depth.

Three weeks ago, results from the large OBS code sprint effort were presented in the Thursday special topics meeting. (See https://wiki.ucar.edu/display/JEDI/2021-01-21%3A+OBS+Update for details.) These results show ability to replicate what GSI does starting with GSI ncdiag output. The next step is to move upstream and ingest obs provider data directly (bypassing the ncdiag path). This will involve ioda converter development.

Work is under way related to the NOAA Data Lake project. This is a project that provides public facing data sets. Different cloud storage options are being investigated, including Amazon, Microsoft and Google clouds. R2D2 will be able to connect to this, and will provide obs data. A method of grabbing data and converting on the fly is being investigated.

A new hire, Rachel Schweiker, is working on diagnostic tools. She will be consolidating various existing solutions into a Jupyter notebook based diagnostics system.

Greg is working on ioda converters for conventional obs. Greg acknowledged that Ron McLaren (EMC) has provided a solid foundation for a common BUFR converter. Ron has focused on radiance obs types, and Greg will expand on Ron's converter to add in conventional obs types. Nan is working on adding in more radiance obs types to Ron's converter.