2021-08-26

JEDI 1 update from Mark

Mark M was away on PTO last week so Steve H led the JEDI 1 meeting. Ryan gave a presentation on how to build the JEDI stack using Spack. Spack is used by several HPC centers and platforms including Hera, WCOSS, and Orion. It can be used to create both lua/lmod modules and tcl modules that can be used transparently by users in a similar way that the current modules are used. Ryan has a feature branch in the jedi-stack repo that includes a recipe to build the entire jedi stack and fills in the gaps in spack package availability with custom build recipes for components like ecbuild and eckit.

Ryan's presentation was followed by a ioda design discussion (Steve, Mark, and Maryam discussed this further in yesterday's JEDI 1 core team meeting). The main issue being discussed was the interfaces to ObsSpace and ObsGroup. The plan had been to replace the current get_db/put_db API with a more direct access to the the data through the hdf5-inspired interface to ioda-engines. However, Yannick and David S expressed concern that this might make the data access more complicated than it is now, with more room for error.

A promising middle ground is to expose the ioda-engines API in ObsGroup but also provide utilites in ObsSpace to simplify the data access, similar to the current get_db/put_db but more flexible. So, developers familiar with the ioda-engines API (for example by working with the similar ioda-engines API in python) would be able to implement similar access patterns in C++ while more common operations (like fetching radiance data) could be done with simpler function calls.

Here are Steve's more detailed notes from the meeting:

In addition to the C++ API, Steve H has also been making good progress with the Fortran API for ioda-engines. This will be set up similarly to the C++ and python APIs and he will include similar examples in the ioda repo.

There is another ioda issue that came up recently that Steve H is addressing as well. It arises because both netcdf4 its underlying hdf5 substrate have different definitions for fill values. This raises the possibility that these two values can be inconsistent. And, this has occurred in certain procedures for creating ioda v2 files. One solution would be to abort when these values are different and require the user to regenerate the file. However, in practice, a better option might be to use the netcdf value and print out a warning that this is not the same as the hdf5 value.

Yongang implemented a new mpi gather function in fckit and is using it to make the unstructured interpolation in oops more efficient. The simple oops interpolation test passes and now he plans to test it with fv3 and mpas and assess potential performance gains.

Maryam and Mark M have also started working with the MPAS team on a planned release of MPAS next month but I can let others give more details on that. Maryam has also made good progress running mpas-ewok on Cheyenne.

Maryam is also working on modifying ecbuild to be able to do a shallow clone of a specific branch of a repo. The main motivation is to reduce wait time when building crtm.

Rick reports that a role account is now available on Orion. So, we will begin building

Mark M has been working with Steve V and the UKMO team to produce prototype docker and singularity containers for ops-um-jedi. Steve verified that um-bundle builds in the container and the ops-um-jedi tests pass. The next step is for Maryam to implement the CI. As noted in previous meetings, this is somewhat of a novelty compared to other CI implementations because the container includes private code repos. So, special care was taken to build the binary libraries securely without exposing the source code in the final containers and without making the containers themselves public.

JEDI 2 update from Dan

Jedi 2.2 - Generalized locations (EMC liaison, Emily)

• Francois is spinning up. Learning the code base and figuring out how the GSI is doing the FOV calculations.

Jedi 2.3 - Use of NUOPC driver with FV3-JEDI (Mark Potts)

• Exchange of states is working for dynamics and physics. Able to run h(x) in-core for FV3-JEDI and UFS. Will work on putting together some tests for running that in the next month or so.

Jedi 2.6 - Ensemble DA validation (Cory, Sergey)

• Seeing some differences in increment for temperature. Looking into differences between sens temperature and virt temperature. JEDI-GETKF seems to be rejecting too many observations.

Jedi 2.7 - Background error model validation (Benjamin, Dan, Cathy, Ricardo)

• Model PRs merged for the major upgrade to BUMP that will allow static B methodology to be applied at full resolution.
• Dan, Benjamin and Cathy are working on the validation. Putting together a full resolution case study. GSI versus FV3-JEDI with BUMP and FV3-JEDI with recursive filter.

Jedi 2.8 - Regional DA (Ting, Ming)

• Continuing the evaluation comparing JEDI with GSI. Pure ensemble. Currently tuning the QC to make the comparison more exact. Using GSI useflag to make sure the same observations are assimilated by both systems.

Jedi 2.10 & Jedi 2.11 - LFRic, UM general updates (Kristin, Steve, Steve)

• API changes because of OOPS
• Making progress on the containers (UM). LFRic has a number of dependencies including the source code for the model.

Jedi 2.14 - Cubed sphere grid into Atlas (Dan, Marek, Olly, Steve V)

• Multi PE mesh is working. Currently working on the halo. Challenging because of the topology of cube in x-y space, lots of bookkeeping. Needed for the structured interpolation, hopefully quite easy. Need our own interpolation because of the data being in the cell centers.

Jedi 2.15 - VADER (Steve V, Dan)

• Working on refactoring the linear variable change part in OOPS. Moving from factory to traits.

Jedi 2.18 - Localizations refactoring (Dan, Ryan)

• Dan, Francois, Anna and Ryan will meet this week to decide on how to refactor locations for allowing generalized locations.

JEDI 3 update from Anna

Benjamin Menetrier: merged major update to BUMP saber#99. Also merged: a follow-up PR for improving CPU cost and memory use in NICAS setup saber#118. Open PR in saber on BUMP subsampling for regional models with small domains saber#120.
Benjamin also added an option to compute full inverse (including variable changes) for ModelSpaceCovariance: oops#1345 . This option is useful for cases when there is no inverse defined for some of the variable changes included in ModelSpaceCovariance. The inverse is only used for the diagnostics.

Yannick Tremolet’s PR changing all applications to run change of variables from state variables to obs operator variables only once is merged: oops#1255

JJ Guerrette updated the code to pass bias correction as an argument to ObsOperator::simulateObs and ObsFilter::postFilter (oops#1322 and ufo#1404). Filters can access the current iteration ObsBiasData group through ObsFilterData as an ObsDataVector, which gets associated during postFilter. The iteration-specific ObsBiasN, where N is the iteration, can also be optionally stored in the ObsSpace for diagnostic purposes. That iteration-specific value can also be accessed from filters through the ObsFilterData.

Wei Huang changed LocalEnsembleDA applications (LETKF-type filters) to use QC flags and obs errors from H(ensemble mean) (instead of previously used H(last ensemble member)): oops#1323.

Sergey Frolov added an option to the vertical localization in GETKF to read in different localization at each gridpoint: oops#1325

Interfaces refactor and documentation code sprint follow-up PRs merged (Dan Holdaway, Clementine Gas, Wei Huang): oops#1321, oops#1339, oops#1338, oops#1350.

Two bugfixes for the QG model (Benjamin Menetrier): oops#1348, oops#1320.

JEDI 4 update from Rick

Much of the ewok work in the past few weeks has been transitioning the environment on various systems from Claude's hosted directories to a more general setup.  

Rick has now installed ewok on S4 and is working on installing it in a container for use in the cloud.  He was able to build a prototype container (single-node) and is now testing it. The next step is to get a multi-node container working.

Other ewok development is on hold for now.  Coming up soon is to work with the UKMO on cylc integration.

Eric has been setting up R2D2 to work on a number of HPC platforms, transferring files as needed between them.

Orion has been down for the last few days and there is currently no estimate on when it will be back up

Hui and Ryan are away this week on PTO so Ben gave the OBS 1 update.

He first asked Anna if the Obs Bias correction that Anna mentioned had been merged to develop.  Anna confirmed that it had.

The OBS 1 team has been working with EMC (Cory, Andrew) to understand differences between GSI versions 15 and 16 and the implications these changes have for ongoing comparisons with JEDI, particularly with regard to BC.

The release of the conventions document is on hold until Ryan and Greg return from leave. The document will be ready for release once some VarBC details are clarified.

OBS 2 update from Francois

-Rachel and Eric are re-writing the diagnostics scripts from the Notebooks into object oriented Python (#4).
-Rachel is working with AWS support to create cloud formation templates that can be used by users to launch the Notebooks.
-Fabio submitted a PR to add the analysis increment test for FV3 GEOS & GFS (now merged).
-Hailing is working on fixing the ROPP ctest that recently broke (#266).

CRTM update from Ben

The 2.4.1 release is almost ready. They are lacking some coefficient data and the release will be ready once these coefficients are integrated into the system. 

The team is working with Mark Liu to merge version 3 code into their work.

Regarding the JEDI 1 issue mentioned above, Ben said that he had asked the eccode/ecbuild team back in October about the possibility of doing a shallow clone of a repo.  They confirmed that this functionality is not yet implemented but they were receptive to the idea of including it, either through their own development work or through external PRs.  But, they apparently have made no progress on it since. Ben is happy to help Maryam on this as needed.

The CRTM team continues to work on a distributed (for parallel efficiency) method for allowing users/developers to compute coefficients offline.  And, they are working on a pipeline interface to help users/developers more easily engage in this process. The CRTM team is providing a Python interface to assist developers with coefficient generation.

Cheng is working with NASA to integrate the NASA aerosol tables (IR and Visible channels) into crtm

LAND update from Andy

Lots of people have been out on PTO, myself included, so minimal updates this week.

Water model: James has been continuing experiments on how to best apply increments in the multilayer snowpack model, and we’ve been keeping up with minor OOPS updates, in particularly the new ATLAS interface in OOPS which we don’t have implemented, but still need small changes to pass the tests, and that’s Soren’s first PR, so welcome to him.

UFS: Folks have been preoccupied with observations and observation preprocessing, as well as some model testing, so nothing very relevant for this call.

SOCA update from Guillaume

Travis continues to work on ewok-based 3DVar but he is away this week on PTO.

The soca team is also working on merging a PR in ufo that implements passive observations (obs are included in O-B, but not included in minimization).

Cameron added a coupled UFS capability in cycling soca-science applications

Guillaume has started to use Singularity on Orion and was working on determining which directories need to be mounted in the container when Orion went down indefinitely.

Benjamin M contributed a bugfix in saber that was much appreciated by the soca team

Progress has been made with FV3-MOM6 coupled H(x). Thanks to Yannick and Dan for their help with a 2-model H(x) application.