Mark M opened the meeting by announcing that we are now alternating agendas for this meeting between general communication and discussion of specific topics. This week is general communications, and next week will be a discussion of QC filtering. We then collected updates from everyone.
EMC
Cory is working on a wind observation operator, and reported seeing performance issues on Theia. This issue appears to be related to certain nodes on Theia, and it was noted that GSI is also being affected. A ticket has been submitted to the IT group managing Theia, and they are working on the problem. MPI communication came up in the discussion and it was suggested to try different MPI implementations. Chris H offered to talk with the IT team (since he knows some of the members) if desired.
Hamideh is investigating large H(x) differences with observations (SMAP, Salinity). Improvements have been made and this is work in progress.
Jong reported progress with the JEDI CICE6 interface. The global array scheme in CICE6 is proving to be troublesome, but this is not a showstopper.
Boulder
Mark M announced that we are close to updating repos, containers, and jedi-stack to use the latest eckit (version 1.1.0) from ECMWF along with the latest version of the JCSDA fckit fork.
Mark O added a status of the eckit/fckit update. Everything is working with GNU compilers, and there are some issues with the Intel compilers (which are being debugged now). After the issues, when compiling with Intel, are fixed the plan is to rebuild the containers, merge the related updates into the repos (OOPS, models, jedi-stack and jedi-tools), and update the AWS and HPC environments. An announcement in the JEDI github group (which will email everyone) will be made when this update occurs, plus instructions on how to proceed with your work.
Mark M noted that we are doing the eckit/fckit update due to the dependence of the new ODC code on eckit 1.1.0, and the dependence of several updates that we have placed in our fork of fckit. Mark mentioned that he is also working on getting the S4 (which only provides Intel compilers) environment in sync with the eckit/fckit update. Mark added that Mark O has brought the JCSDA/fckit fork up to date with the latest fckit version (0.6.3) from ECMWF. This will facilitate merging in our fckit fork back into ECMWF's repository down the road.
Hui reported that she cannot run in Singularity on Gateway due to running out of memory. Mark M will help fix this.
Steve S (UKMO) asked if the eckit/fckit update will impact them. Mark O responded yes and they will need to modify include statements for some eckit header (.h) files. Mark O will help with this. Marek noted that the changes for LFRic should be similar to changes that were made to get MPAS updated so NCAR is another resource to help with this.
Mark M mentioned another issue related to the eckit/fckit update. It is possible to access configuration (YAML) information at the Fortran level using a module provided by OOPS. This happens to be redundant to a module provided by fckit for this purpose. Everyone needs to switch to the fckit module. Benjamin is going to submit PRs soon that take care of this for the core repos (UFO and OOPS have PRs now).
Steve H reported that his PR to switch IODA from the netcdf C API to the netcdf C++ API is having trouble on Theia with Intel compilers. This PR works on Discover, in the containers and on Mac systems, but crashes occur on Theia. Dan was helping Steve, but Dan is on vacation now. Steve asked for assistance (since he doesn't have access to Theia) if he gets a fix he wants to test before Dan returns. Ryan and Chris H both volunteered to help.
Anna announced two UFO PRs that will impact others. The first is JJ's work to add absorbers to CRTM which will require YAML file changes. The second is Cory's work that updates obs and geovals files to netcdf-CF naming conventions, and adds units to the obs files. This is done on all GSI generated data, and will require some variable name changes. Anna would like to merge these PRs in the next few days. It was noted that the marine obs and geovals data remains unchanged for these PRs. Marek will test tomorrow, and Anna will wait for Marek's feedback before merging.
Hailing is working on implementing a GSI-style RO background check for bending angle in JEDI. This will be selectable from a YAML configuration.
Chris H got the MakeObs function working for his shallow water (SW) model. This allows for the generation of synthetic observations. This functionality has been merged into the develop branches. Chris now has H(x) and H(x) 3D running to completion, and he is verifying the output. Chris asked for help with this verification from a DA expert. Chris may add in a speed obs operator before submitting this work in a PR. Chris will be moving to 3DVar next. He should have this working in the near future, and he may need help with interpolations. Chris reminded everyone that he will be out of the office for the next couple weeks (workshop, then vacation).
Steve S asked if the SW model is available through Singularity. Yes, it is in a JCSDA repo along with a bundle repo (JCSDA/shallow-water and JCSDA/sw-bundle). Chris has run sw-bundle successfully with GNU compilers, and Marek has run it inside the container.
Met Office
Marek is working on time interpolation, and would like someone to review his code (UFO). He also requested some help with an issue that came up in regard to locations. Marek will create a "draft" PR, and Anna will review and assist with resolving the locations issue.