Good progress being made on the coupled H(x) application

Not much to report this week I'm afraid - folks out on vacation

• On-going work tuning and testing R-localization for snow depth assimilation in FV3-JEDI 
• On-going work tuning and testing B-localization for snow water equivalence assimilation in NWM-JEDI

Greg is testing and validating conventional data. This work includes exercising composite operators and variable changes.

Wei Han is working on GIIRS converter development.

Work is under way to compare obs errors (methods and results) from different operational centers (NOAA, UKMO).

A pull request is under review in ioda-converters (https://github.com/JCSDA-internal/ioda-converters/pull/585) that introduces a new MODIS AOD converter.

Work is in progress for ozone QC and nadir profiles.

Migration of all data to ioda v2 format is under way (maybe completed?)

New obs types from EMC have been configured into the JEDI GDAS experiments.

Work is in progress for prepBUFR conversion of satwind data.

The UK Met Office has added a lot of new ufo features

• Cloud detection
• RTTOV surface types
• Ground GNSS obs operator
• TLAD operators for both ground and satellite based GPSRO
• Applied bias and bias correction for surface obs

- Updating data read-in for diagnostics using ioda python package. Will be compatible with iodaV2, more maintainable, better memory usage.
- Met with jedi-godas folks, starting to think about how to generalize diagnostics to work with atmospheric models beyond jedi-gdas

The OBS group was quite busy with preparations for academies, sprints, and releases over the past month. The OBS team presented at their first JEDI academy and contributed six UFO-focused lectures and two practical exercises. There was quite a crowd of people who wanted to attend the academy, so an overflow academy will be held in early October.

Observation data:

The variable and IODA file conventions meetings have resumed. We are aiming to release our conventions document this month. The document will detail how IODA data are structured, and will describe variable names, units, and dimensions that should be used within the JEDI system. This deliverable is a good milestone for starting a round of upgrades to ioda-converters, ufo, and the model interfaces. Now that IODA version 2 is released, we want to combine its required changes to the converters with our changes to variable names, and then we want to make these newer files work with the rest of JEDI. Several groups have expressed interest in the IODA-v2 upgrade.

Besides this issue, there are several IODA improvements:

        • Cory and Kevin at EMC are expanding the ioda-Python interface to make it more intuitive to Python developers. This includes adding convenience functions and following Python conventions. We do have a PR in review to remove the need to set LD_LIBRARY_PATH before loading the Python library - we are waiting on Mac testing before merging.
        • Cory has also been testing Python ioda-converter library improvements. Basically, to have all of the Python-based converters natively write ioda-v2.
        • Nick, Praveen, Emily, and Ron are still working on the BUFR reader and have been drafting tests and YAMLs for prepBUFR to IODA conversions.
        • The UKMO ODB group's PR to IODA is in review and should reach develop soon. This PR adds a basic capability to read in ODB data and translate it into IODA.

For UFO improvements:
        • Wojciech created several pull requests enabling the ability to assimilate derived variables. This work is reported in both JEDI3 and OBS3. Derived variables are variables that are not located in our initial observation files. They are produced by operators and filters. This made assimilation difficult because our ObsSpaces were divided into ObsVectors and ObsDataVectors before the new variables were created. This new capability significantly expands what we can do with our QC filters. Now we can transform variables and use these products in assimilation.
        • Wojciech has also added code to import CSV and NetCDF-based external data tables into UFO. He has also added a new interpolation feature to interpolate based on these tables.
        • Wojciech has also expanded ObsFunctions to return more data types than a float. ObsFunctions are now templated in UFO.
        • Chris Thomas has been working on the ObsSpace extension for averaging profiles onto model levels. Several pull requests were made in ioda and ufo, which are almost all merged.
        • Following this, Chris' work needs an expansion of how we store QC data. For assimilation purposes, we care about accepting data, rejecting it, and manipulating estimates of observation error. However, we also want to tag data for downstream use by both JEDI and non-JEDI developers. This is a broad issue that affects multiple agencies, so Yannick has asked for a special topic discussion with this group on August 19. Until then, we are exploring various strategies for doing this.

Release 2.4.1 is coming soon (one to two weeks from now). The plan is to release first to just the partners (internal) and follow later on with a public release.

In collaboration with CSU, a new hyperspectral ATMS-NG instrument has been added.

IASI coefficients have been updated.

Documentation is being developed to describe the workflow in CRTM for adding a new obs type (instrument/platform). This documentation is targeted for a new developer coming on board to contribute a new obs type.

Steve H is starting the effort to replace the older ObsSpace get_db/put_db interface with the full featured ioda-engines interface.

Maryam is working on organizing and managing tier 2 and 3 testing. She will run a special topics discussion (Thursday meeting) for those interested in working out the definitions of the higher tiers in the testing scheme. See https://github.com/JCSDA-internal/jedi-discussions/issues/28 for more details.  

There was no JEDI 2 meeting in this period due to the July 4th holiday so updates are only for work I'm involved with:

The first version of VADER has been finalized and a PR has been issued. We've made a plan to start issuing a sequence of pull requests that will affect the models. The first PR will switch the variable changes from factory to trait instantiation and change the interface to help eliminate state copies. The second PRs will introduce VADER to the models. The third PR will eliminate the inverse method.

We are working to clean up the model to GeoVaLs variable changes. Greg is cleaning up the surface height variables to make clear differentiation between geophysical height, geopotential height and geopotential height multiplied by gravity. Dan is adding new cloud effective radius calculations to match what's done in the GSI for NCEP and GMAO. Made some bug fixes.

Steve V has been able to compile UM-JEDI and is starting to work on adding CI for UM-JEDI.

Participating in the code sprint to clean up the use of the FMS library in FV3-JEDI. This will result in changes to the config files so anyone running FV3-JEDI develop with their own config will need to update things.

We've advertised a new position to work on 4DVar in the JEDI 2 team.

JEDI3 (DA Algorithms) update:
- Wojciech's series of pull requests for assimilating derived variables was merged (oops#1259, ioda#316, ufo#1304).
- Anna added an implementation of observation error covariance with cross-variable (or cross-channel) correlations: ufo#1327.
- Clementine added an option for computing and saving eigenvectors in DRPLanczos minimizer: oops#1276
- Anna opened a PR to refactor LETKF-type filters to use Observers instead of CalcHofX class: oops#1270

For the purpose of enabling EWOK/R2D2 on different HPC platforms, a significant amount of work was done with HPC IT groups to get the proper software stacks installed. EWOK/R2D2 is now in good shape (in regard to various platforms) which enables the development of using different models in the workflow.

Work is under way to develop coupled H(x) capability at the workflow level.