Yannick opened the meeting announcing that this will be a general round-table update.
Our next focused topic meeting will be in two weeks, on Thursday, Feb 18, when we will have a presentation led by Yannick, Claude, and Rick on ewok.
JEDI 1
Mark M started with a JEDI 1 update. Our most recent JEDI 1 meeting was yesterday.
We discussed the status of the python ncepbufr module because this was causing a CI test failure in ioda-converters. Rahul had previously integrated the JCSDA-internal/py-ncepbufr into the new NCEPLIBS-bufr repo that is now built with the jedi-stack. It was suspected that the CI failure was due to a conflict between these two implementations, which were both being installed in the containers.
We agreed to remove JCSDA-internal/py-ncepbufr from the jedi-stack. Henry mentioned that they are still using it for filtering bufr files before making them available on the cloud. We agreed that the py-ncepbufr repositority will not be removed from JCSDA-internal but we encouraged Henry and colleagues to use the JCSDA-internal fork of NCEPLIBS-bufr which will maintained moving forward.
Other than that, the new intel OneAPI container is ready for use with CI and we have begun to implement it in fv3-jedi, ufo, oops, and other repos. There is an active PR in the "containers" repo with instructions on how you can build your own Intel OneAPI development containers (Singularity, Charliecloud and/or Docker), if interested.
We also discussed the new plan for handling test data in ioda and ufo. Currently the tagged release data is provided via UCAR DASH but the develop and other branch data is provided through public buckets on AWS S3 and requires manual uploading by a member of the JEDI core team every time it needs to be updated. Improving this situation is one of the main JEDI 1 tasks in preparation for the new release.
Our short-term strategy is to return to git-lfs. However, this time we have created separate repos for the test data: ioda-data and ufo-data. There was some discussion about what data should go into these repos. It was decided that they should be independent, so ufo tests should not depend on test files from ioda-data. Currently this would mean the duplication of about a dozen test files that are used in both ioda and ufo tests. But, this will allow the freedom to, for example, replace some of these with reduced versions in ioda that do not require as much information. However, it was decided that the crtm coefficients required by ufo tests should not go into ufo-data. In most cases the latest release of the crtm coefficients should be sufficient so we can get them from UCAR DASH, facilitating maintenance and saving on egress costs.
Maryam will update the test wrappers to download the test data from ioda-data and ufo-data instead of AWS S3. We will experiment with this for a while. If it works well, we may do the same for other repos like saber and fv3-jedi.
Maryam and Mark are also continuing their work on updating the compare tests. This is taking a bit longer than expected due to refactoring inspired by code reviews, including encapsulating the changes in a new test class. It is expected to be ready soon.
Mark O is also proceeding with JEDI profiling. We agreed to focus on profiling the 3D Hofx fv3-gfs application on S4 with Vtune to start. After that we can compare with fv3-geos and eventually move on to other platforms and applications.
Steve continues to focus on the IODA refactoring. There is a PR in now but it is still in a draft mode until performance issues can be resolved. This is all discussed more in the JEDI 2 update.
Henry reported that they are now running weakly coupled 3DVar experiments and have UFS and soca-science running smoothly on the AWS cloud.
Jian will be working on ewok workflows with UFS.
Update from Ryan as of Wednesday: GitHub access on RDHPCS machines (Hera, Jet) is broken again. Hopefully the NOAA security team can resolve this fast. On Thursday morning he thought it might be fixed but Chris H reported that it was still broken.
David S (Met Office) put in the request to record the special topics discussions that we have every other Thursday. It was agreed that this was a good idea and the core team will look into getting this in place for the next special topic discussion (in two weeks).
JEDI 2
We had two JEDI2-related meetings over the past three weeks. One general meeting, where we discussed the upcoming JEDI release and IODA version 2, and a special topic meeting on QC flags.
The special topic meeting was well attended and had a good amount of open discussion. UFO contributions have grown beyond what was originally intended for QC flags, Filters, and ObsFunctions.
In summary, we need to improve how we pass ancillary data around UFO. QC, in particular, should be expressed as more than just pass/fail flags or adjustments to observation error. Flags can also exist for monitoring, multiple QC variables for each filter should be possible, and these variables should support different data types like floats, integers, bitfields or bitmaps, and strings. The UFO filters are written to be generic and customizable, so when we do filter an observation we need to differentiate between QC steps that invoke the same filter but are configured differently. We do not want to incorporate all of this information into the ObsSpace, since we do not want it to act as a global variable store, and so the discussion also touched on ObsDiagnostics, which is used to propagate Jacobians and other large data. We agreed that we need to plan these changes before diving into the UFO code, so a smaller group of Emily Liu, Chris Thomas, and Ryan Honeyager will meet biweekly to sketch out ideas and changes. It is likely that these changes will start after the next JEDI release.
The general JEDI2 meeting mostly discussed the upcoming JEDI release and the IODA changes that we would like to implement before then. Generally, this includes the ioda-engines integration, the Fortran interface, ObsBias enhancements, many bug fixes, and the reworking of multidimensional data storage. This is all bring tracked with a release on ZenHub.
Besides for that, several contributions were noted in the meeting minutes.
• Steve Herbener and Ryan have begun working with David Davies and August Weinbren from the UKMO to add an ODB engine to IODA. To start, we have a PR adding an empty "demonstration" engine that highlights the functions that need to be customized when adding a new backend. Ryan also has added a PR refactoring the HDF5 engine, improving documentation and making the code easier to read.
• Steve Herbener finished the ioda grouping enhancements requested by the obs team. You can now group data using multiple variables.
• August has been working on a PR to remap NetCDF variable names to those used within JEDI. We think that this can be tied into our data layout policy code to transparently ensure that Greg Thompson's variable conventions are obeyed.
• Wojciech Śmigaj contributed code to select observations according to regular expressions and wildcards, which adds a very powerful search capability to our filters.
• Wojciech also added an obs operator that calculates background error estimates and stores them in ObsDiagnostics.
• Chris Thomas is improving how we vertically average atmospheric profiles, and is working on drift calculations for sondes with incomplete 3D position data.
JEDI 3 had a meeting on Monday, led by Dan who is partially back from paternity leave. Much of the meeting was a general update on what model groups have been working on while he was away. Yannick and Mark asked if any of the model groups wished to elaborate on the meeting or any other updates.
Marek noted that he has an active PR in UM-jedi that implements time interpolation. It was agreed that eventually this functionality should be moved into oops but it can stay here for now.
There were no further updates so the meeting was adjourned.