Yannick opened the meeting by announcing the agenda, which is a general round-table update.
Steve H started the roundtable by announcing that he and Ryan are now working on a refactoring of ioda that will clean up duplicated code and improve the interface for accessing data in ioda. This will require changes in how ioda data is handled, with particular implications for ioda-converters. This will require coordination of changes to the various ioda converters. This will not happen immediately - Steve was just announcing this as a change that is coming. A prototype for the new interface should become available in about 3-4 weeks. Yannick asked if all the ioda converters need to be updated simultaneously or whether they can be updated one at a time. Steve responded that they could be updated one at a time but it may be better to coordinate a mini-code sprint to implement these changes.
Hui then reported on a collaboration with the UK Met office to get their GNSSRO operator into the JEDI system. They are currently working on the forward operator part. However, the UKmet GNSSRO DA also needs to use a 1DVar system for the QC. She asked Yannick whether this capability is currently available in JEDI or whether there are plans to add it. Yannick mentioned that Mike Cooke of UKMO is now working on this. He has already had several PRs implementing a 1D cost function inside oops but this is still a work in progress.
Mark M then announced changes in the way we plan to manage the JCSDA forks of ECMWF repos, namely ecbuild, eckit, fckit, atlas, odc, and pyodc. Our new strategy is to synchronize our develop and master branches with upstream every night. So, if you want to make changes to any of these repositories, the new workflow would be to create a feature or bugfix branch from our develop repo (or hotfix from master) and then to do a pull request to ecmwf. So, the only changes to our develop and master branches would be through ecmwf.
This issue was also discussed later when Rahul joined. Yannick emphasized that there are advantages and disadvantages to this and we will try it out to see how it works. Where the balance shifts between advantages and disadvantages will depend on how fast ECMWF responds to our pull requests. The main advantage is to prevent our forks from diverging from upstream. When our forks diverge from upstream, it makes merges from upstream more difficult because there are often conflicts. This has already happened with fckit. The main disadvantage is a potential time delay as we work with ecmwf to implement our changes. Rahul expressed concern about the delay and argued that this approach inhibits our agile workflow. Mark M suggested that this can be handled by tagging particular feature branches that have urgent, necessary changes. Then the bundles, environment modules, and containers could use the tagged versions. We have already started this approach by defining tagged (patched) versions of ecbuild (3.3.0.jcsda3) and eckit (1.10.1.jcsda1). However, this approach works best for a few targeted changes. if the frequency of our changes exceeds the rate at which we can get them reviewed and merged upstream, then this approach might not be sustainable. On upstream PRs, it might help to explain that you are part of the JEDI project, particularly if you contribute infrequently to upstream and the developers at ECMWF might not recognize your GitHub user name.
Rick then reported on progress implementing this plan with CI/CD. The odc, pyodc, and atlas forks are already ready for nightly synching with ECMWF because their develop branches already match upstream. The other forks, namely eckit, fckit, and ecbuild are not yet ready but we are now working with ecmwf to resolve differences and conflicts. Rick has looked into setting up a nightly merge with Travis-CI and said it should be straightforward to do by spinning up a docker container and doing the required clones and pushes. Yannick asked if tests could be run before or after the merges and Rick agreed that this can and should be done.
A discussion then ensued about how to securely provide the CI server with a github token that has permission to push to the develop and master branches on these repositories. Rick mentioned that one way to do this would be to purchase a workstation for JCSDA general use which could carry out the cron jobs instead of Travis-CI. This has been discussed within JCSDA previously and would have other uses, for example as a node where we could host a DDT/MAP debugger/profiler single-node license. Travis S mentioned that Travis-CI is reasonably secure. He defines a GitHub token that only has permission to push to the SOCA nightly release branch. Furthermore, this token is encrypted and only decrypted by Travis-CI for the exact matching repo. Maryam said that AWS CodeBuild is also secure. There one can store Github tokens that only specified users (administrators/managers) have permission to access.
Rick also mentioned that he now has access to NASA Discover, as well as Orion. Mark M also mentioned that he now has access to Orion and Hera.
Following this discussion, Ming presented the following slides demonstrating recent progress in comparing JEDI Hofx to GSI Observer using WRF.
Overall, the comparison was very promising. The results were similar but some differences were identified, particularly for low elevations and wind variables. The temperature in particular is a good match but exhibited a small cool bias. Ming and team are now working on understanding the differences and improving the agreement further. Ming suspected that the discrepancy in the wind variables might be due to a rotation problem having to do with the grid and Greg T agreed that this was a plausible place to look. Ming also asked about the units used for humidity and whether a unit for temperature has been adopted for ufo.
Yannick thanked Ming for his presentation and added that it is great to see this progress and these comparisons.
The round-table then moved to Mark O who announced the first stage of what will be a substantial effort to bring JEDI up to date with the most recent versions of CMake and ecbuild. He has a PR now in oops to get this going. It should be transparent to most users and it will not require most users to make changes to their repos. But, Mark warned that there may be growing pains during this update so he encouraged users to test the current (oops) and upcoming PRs to make sure they don't break your builds. He will proceed to modify the other JCSDA repos one by one. Phase 1 of this update will be limited to changes that are backward compatible with existing code. Phase 2 will implement required changes.
The main consequence of this update to most users is that JEDI will soon require CMake version 3.12 or later. The choice of this as a minimum version was made based on its improved support for Fortran and on atlas, which requires CMake 3.12 or later for its OpenMP support. All JEDI containers and environment modules are already compatible with this requirement, as are the build scripts in jedi-stack. If you need to update your CMake version you can try a package install (e.g. brew or apt or yum). If a sufficiently new version of CMake is not available for your operating system then it is easy to build the CMake version of your choice from source. Instructions are available on the CMake web site. The JCSDA jedi-stack repo also has a build script available for CMake. Mark O recommended to install the most recent version if you can - it's likely to have bugfixes that earlier versions don't.
Rahul then described a PR he has in now to ecbuild that exploits the git submodule tool and recursive clone capability to facilitate the maintenance of third-party libraries (consult the PR for further details). This will help the SOCA team better maintain the integration of third-party applications such as MOM6. The discussion about how to handle jcsda forks of ecmwf repos was then revisited - see above for a summary.
Yannick then closed the meeting by announcing that next week's focused topic discussion will be led by Mark O and will cover the new near-real-time H(x) workflow within JEDI.