JEDI weekly meeting 26 June 2025
INFRA
Ashley cylc integration underway, basic core infrastructure underway adding this backend in addition to ecflow
Quieter overall changes to CI, these will be rolled out shortly and more detail in future meetings
Please file issues for any test instabilities that are noticed
ALGO
In SABER, BUMP is now optional and can be disabled via CMAKE flag, saves time fulfill request from Dan H (NOAA EMC).
Small IO change in Quench (FYI to SABER users)
Change for how 2D variables are handled, what does model top mean for top down versus bottom up in ocean vs atmosphere? New PR and related PRs for clarifying this abiguity will be incorporated, please be on the lookout for these.
Running multi-scale spectral B experiments, there plugging along, there are some observation in early September that needed further examination but this is JCSDA exclusive issue contacting the OBS team.
JEDI (interface)
PR under review that adds utilities that abstract common for-loops over atlas fields and add OpenMP; we'll use this to add OpenMP across saber, vader, and other generic code over time. It also can help optimization for GPU as well in the future.
PR undergoing initial testing that cleans up the interface to the LocalInterpolator used by GetValues; please take a look if your model interface provides a custom interpolator for GetValues.
OBS
IODA converters there is a new converter being developed for the Madrigal database that hosts public facing TEC this will be a nice addition for open access files and those wanting to look into Ionospheric data assimilation.
There is a PR from Wei Gu (UFO PR3687) where there has been limited progress but need to follow up there.
A couple bug fixes of note are UFO PR3699 it was discovered as a runtime problem in the TL, and also corrects a unit problem.
Further UKMO finished implementation of a refractivity filter used by their 1D-Var.
Lastly, an emissivity jacobian fix was merged with UFO PR3664. This brings CRTM v2.4 in line with CRTM v3.x so now they reproduce the same results from the unit tests and there is a subsequent PR UFO3728 that will now remove the duplicate unit test results that had been required.
CRTM
Release CRTM 3.1.2 this was designed to address an issue Russ Treadon found, specifically with newer compilers (Intel IFX) with spack stack v1.9.1. Found the issue with OpenMP and is now operating as expected. Still some issues remaining with OpenMP issues, need a computer scientist to dive into these and potentially look for other issues particularly with channel subsetting.
IFX issues may still remain, if you are finding issues please share.
Lingering issue discovered by NASA code 613, oxygen layer absorption in MW is being potentially done incorrectly working with NASA to dive into and clarify and resolve this issue.
NOAA EMC (Dan H)
Continuing to try to see if git-lfs dependency can be removed from the source code. git-lfs outages cause very large impact beyond just the JEDI user base
Can source code be separated and the test data be separated so one can clone code (and build) however the data is separated out. SABER and VADER have all the test data in the same repo, these could be moved into a separate data directory which the main code build does not rely upon.
Would it be desirable to have all the data in a single data repository, or are these data repository by data repository. Consensus is repository by repository.
GFSv17 getting running to run a real time parallel full resolution coupled atm/ocn land and snow and sea ice … were planning to have aerosols but this is going to be a separate application (JEDI-based). Working through all of the things needed for operational compliance of the JEDI. Trying for zero difference, the JEDI implementation needs a different MPI layout. The JEDI implementation is in general working by truncating the increments coming out of JEDI. Generic interpolation has some irreproducibility when halving the grid, Anna S. found this, and has implemented a fix.
On the regional side, testing a generic interpolation for regional GSIbec, some updates needed to SABER (MGBF being added also). Regional version of MGBF and GSIbec can replicate localization length scales, this is coming along nicely.
GNSS-RO for single atmospheric cycle, better increment for surface pressure once the 3D pressure PR was merged.
Work on atmospheric observations and aircraft, surface and radiosonde, turning off a filter in GSI which is now believed to be not needed (incorrect?) getting adequate matches.
Compilers and optimization, getting closer v1.9.2, which is needed for v17.
Profiling the LETKF, turning to redistribution of the observation, which is the primary bottleneck.
HAFS regional hurricane system, using bump for localization, having lots of problems, trying to track down a potential memory leak. Looking at BUMP and diffusion operators potentially adding OpenMP threading.
NASA GMAO (Ricardo T)
Steady progress comparing JEDI with parallel GMAO GSI based system. Put JEDI to the most rigorous test, comparing directly a side-by-side 12 km resolution with GEOS-FP this system includes updates not in the production real time system. Examining the consistency of O-B results. There are small differences in how in particular the JEDI system was initialized, but this run is being used to watch the timing, and not only the computational results.
Moving run JEDI to solid state disc rather than regular disc. Timing are much more reasonable, not as fast as the GSI based system, but the time taken is comparable and feasible. Began with downgraded background error in JEDI. Added the dual-resolution B, added some time but not too bad. Overall results are very encouraging that this is running at this resolution.
This impact of the GNSS-RO was great, and once the 3D pressure fix was implemented, this allowed a massive improvement to the system and brought the JEDI based system much more closely in line with the GEOS-FP system so large benefit when this fix, and subsequently GNSS-RO was enabled.
Analysis variable in GSIbec should be virtualTemperature but what is being passed by JEDI is airTemperature.
Would like to add a middle loop, (second outer loop). Hadn't been trusted due to 3D pressure fix, the update that the FV3 interface is providing there must be consistency in the pressure.
Analysis are being written serially, but could be improved by writing these in parallel.
NRL (Sarah K)
We are looking into noise in the upper atmosphere in the 3DVar Solver.
We are continuing to develop our 4DVar implementation it will still be a while before we look at timings.
Fei Lu has been looking at a single precision implementation of OOPS. There is an OOPS issue, and now an OOPS branch also exists. Would a meeting be warranted to discuss this? Main goal is to pass in single precision model fields. The dot products must be maintained in double precision (comment from Ricardo T. - NASA GMAO). Chris S. offers that there is a single precision implementation of the MPAS interface, though this is likely different from what Sarah is discussing. Hernan G Arango commented that the tangent linear and adjoint kernels can become unstable in single precision.
It was agreed that a meeting is warranted to more broadly discuss this feature, and Sarah K will help to coordinate this discussion and what is being developed with this contribution.
Dom H. is working towards having certain JEDI components installable via spack-stack. This subset of the JEDI components in the spack-stack will lessen burden on downloading and the need to rebuild these. Working also to develop a method to rebuild the spack-ified components, that still uses the spack framework.
To use the latest Intel compiler, IFX there is still work to be done. NEPTUNE-bundle is being built with IFX. Need to distinguish compiler flags for Intel Classic and Intel LLVM. IPO gives big performance improvements with IFX, but there is a bug in the linker step for certain applications - Intel developers are looking at this.
Working on re-establishing the ROPP repository once we’re happy will reach out to the INFRA team (Evan) to add this back.