JEDI1 update (Mark):

There was a JEDI 1 meeting last week led by Steve H, since Mark was away.

There is one topic that received some discussion then that we want to pose to this larger group.  We would like to designate one infrastructure representative from each site/organization.  Dan has already offered to be the representative for NASA/Discover.

The motivation for this is to foster better communication about changes to the jedi stack and the environment modules.  The JEDI 1 team already has a vetting process for proposed changes to the jedi-stack but this typically proceeds without any input from those outside the JEDI core team.  So, the main responsibilities of the infrastructure reps would be:

* To be included as a reviewer on PRs to jedi-stack and other repos that effect JEDI dependencies and environment modules

* To participate in biweekly JEDI 1 meetings, particularly those in which we discuss stack changes as part of the vetting procedure

So, a request to all participants of this meeting: please identify an infrastructure representative from your organization who can serve as a point of contact for stack/module related issues.

In other news, Maryam and Mark have continued to help out with the JEDI-MPAS release scheduled for this Friday.  Everything is on schedule.  We have defined internal tags for oops, saber, ioda, ufo, mpas-jedi, and the associated data repos that we'll continue to test this week.  Meanwhile, the MPAS team is finalizing the documentation and tutorials.  Maryam will also implement new CI containers this week, the main update being a move to gnu 10.3 (along with a minor modification needed for CodeBuild).

Steve H continues to work on changing the representation of DateTime in ioda files in order to improve IO efficiency.  We have decided to store the DateTime as integers rather than strings to avoid time-consuming string conversions.  But, there is an additional question of what integers?  Should they be identical to those stored in oops DateTime objects, or should they be more human-readable for users viewing the data with tools like ncdump or hdfview.  The latter would be more efficient than strings but would still require a conversion.  Steve is looking into whether the time of this conversion is significant enough to worry about.

Steve also issued a recent fix to properly handle missing latlon values for GOES16 (and other) data and continues to resolve conflicts between how netcdf and hdf5 define and implement fill values.

Another issue that has come up recently is an apparent incompatibility between python conda/miniconda and the ioda python bindings https://github.com/JCSDA-internal/ioda/issues/463.  Mark M has has some difficulty in getting these to work together in the container and Rick has run into similar problem s on Hera.

Kat, Anna, Yannick, and Mark have also begun our collaboration with Peter van Leeuwen, Steve Fletcher, and Chih Chi Hu from Colorado State University on a JEDI-EDU application that will be used for teaching DA.

OBS1 update (Hui):

• Naming conventions: OBS team is getting close to publish its IODA convention. Greg and Ryan were invited to give a intro last week to the UFO team meeting. The IODA convention will be released followed up by IODA/UFO changes in the following month. 
• UFO team made an update for all conventional yaml for JEDI_GDAS (operators, QC, data files). PR is in review. They will be used for JEDI_GDAS testebed together with data files update. JEDI_GDAS radiance UFO have been gone through series of testing using the latest data files provided by EMC, which were generated by using GSI v16. Since the UFO currently follows GSI v15.3 implementations, such tests will make sure the consistency of the UFO with the latest ops GSI at EMC.
• Bufr2IODA converters (EMC) are in progress, particularly for conventional data (aircraft). Associated with NCEP BUFR lib updates.
• Several PR in reviews regarding the UKMet capabilities for conventional data assimilation. UKMet is working on wrapping up ATMS radiance data assimilation with PR on RTTOV changes.
• AERONET AOP data assimilation was added to develop. It is being tested for AOP data assimilation (GSL/OU).
• NRL has also been working on their data flow and setting up workflow for the NETUNE applications.  

OBS2 update (Francois):

• Replicate SageMaker interface on JCSDA's SSO account (Imported the SageMaker/IODA2 ECR from JCSDA-NOAA account (#109))
• Review PR #6 for BESPIN (binning functions for diagnostics plots). Merged by Travis.
• Develop plotting scripts for the Data Lake project.

CRTM update (Ben):

• GEMS-1/2 Coefficients delivered to OMS and NRL, undergoing testing.
• GOES-U (to be 19) is the next coefficient on the priority stack, GOES-18 coefficient still under evaluation.  
• Andy Tangborn working on NASA coefficient consistency,  broader collaboration with GMAO on aerosol updates for CRTM to enable consistency.   Had a meeting with Arlindo at GMAO, encouraging use of NASA table in CRTM.
• Cheng to work with UFO on piping radiance as a obs type (in addition to brightness temperature), and will work on adding downwelling radiance to the CRTM to enable upward-looking instrument support (longer term plan). She will also be exploring methods for improving vis/IR computations in general.  
• Nick Nalli has provided an updated IRSSE coefficient, improves temperature dependence in the mid-wave IR regions, publication in preparation.
• Assisting OBS team with transition to CRTM v2.4.  Improvements to Unit/Regression and application tests.  Coordination with EMC plans for v2.4 transition, will have broader impact on OBS evaluations of JEDI-GDAS.  
• Patrick worked with Maryam on CI/CD implemented in CRTM Repository, thanks Maryam!

JEDI2 update (Dan):

There was no JEDI2 meeting on 13 Sep due to the WCRP-WWRP symposium on Data Assimilation. The following update is only for core team work.

Jedi 2.1 - Improved interpolation

• Reported under JEDI3: Marek added adjoint interpolation to Atlas that will allow for interpolation to observation locations using Atlas.

Jedi 2.2 - Generalized locations

• GSI code for producing the FOV ellipses and grid within the ellipse was successfully ported to UFO. Work has focused on designing the interfaces by which to communicate the additional locations to the GetValues class used by the model.

Jedi 2.3 - Use of NUOPC driver with FV3-JEDI

• Work to make sure the FV3 version used by EMC is compatible with JEDI. Containerizing of UFS also underway. 

Jedi 2.7 - Background error model validation

• Successfully ran a series of single observation tests with both GSI and JEDI to compare static background error structures.

Jedi 2.15 - VADER

• Pull request removing the variable change factory from OOPS has been completed and is under review.

Jedi 2.18 - Locations refactoring

• Pull request changing the locations class to use ObsGroup for holding the lat and lon data completed.

JEDI3 update (Anna):

• Marek presented his work on adjoint interpolation in the previous JEDI meeting; see presentation attached.
• Anna has PRs in oops and ufo refactoring AnalyticInit classes. All model interfaces that implement the GetValues test will need to adapt to this change. See description of required changes here: https://github.com/orgs/JCSDA-internal/teams/jedi-models/discussions/52. The oops and ufo PRs will be merged on Tuesday Sep 28.
• Olly reported that Lorenzo Milazzo added a PR in oops with FFT implementations: https://github.com/JCSDA-internal/oops/pull/1377
• Benjamin is working on adding peaked functions (in addition to Gaussian-like Gaspari-Cohn) to BUMP.

JEDI4 update (Yannick):

Rick and Eric focusing on data lake project (due by end of month); also maintaining the rest of the system. Working on ingesting goes16, goes17 data for the data lake (pilot project for NOAA).

SOCA update (Guillaume):

- Diagnostic tools (Travis)

- Rossby radius based localization for the LETKF in soca (Travis)

- GEOS and UFS h(x) with GMI and SMAP (Hamideh, Guillaume)

- Cleanup and sanity check of the marine observations. Trying to manage the database with github tools (pr/issues). Please contact Guillaume if you have ideas on improving this process.

LAND update (Andy):

GFS snow DA work, folks at PSL and EMC are aiming to get things ready for the reanalysis work as soon as possible, hopefully 1 November

• Update letkf_snow tests to include all of the capabilities required by the snow LETKF-OI as is being used for the UFS, to ensure that none of this capability is lost going forwar
• Clara is developing some workflow aspects for bringing in JEDI increment with UFS_UTILS global cycle
• We’re still working on updating to Noah-MP from Noah, so changing the land surface model. For us this might be as simple as changing some FieldIONames in fieldsets, but we do need to be careful about changes in units and even what variables actually are and their dimensionality. 
• I believe this will have implications for the geovals used by CRTM - so someone, not sure who, might need to look at this 

Water Model work

• Profiling of WRF-Hydro H(x) - not surprisingly, much of the time is in interpolate, so we’re looking to improve this

Questions and other updates:

Q: David Simonin: what is the timeline for the refactoring of how datetime is stored in files in ioda?

A: Steve Herbener: expect PR by the end of this week; merge likely next week or so; depends on other changes in ioda.

Q: Guillaume Vernieres: will this change be backward compatible for ioda files, or will we need to update the files?

A: Steve Herbener: yes, it is backward-compatible. We’ll try to introduce the change in the files during the ioda-converters code sprint.

Adjoint interpolation presentation from Marek