2021-03-11

Last week the main event was the ioda mini-sprint that several of us were involved in.  Ryan will give us
the latest on that.

Yesterday we ran an AMS (virtual) short course on JEDI.  It was a condensed version of the JEDI academy
with a mix of lectures and practicals.  We had 102 participants, each with their own AWS node to do the
practicals.  We're waiting on the results of the post-event survey but we think it went pretty well.

On Tuesday we installed a JEDI stack on the Summit machine at Oak Ridge National Lab (ORNL).  This is part
of a JCSDA collaboration with the Air Force.  One of the projects the ORNL/USAF/JCSDA team is interested
in is implementing GPU support in JEDI, and particularly SABER, to improve efficiency and to enage in B
matrix modeling.  Summit has a few new challenges for JEDI, including the IBM XL compiler family and IBM's
Spectrum MPI library.  As a first attempt to get JEDI working, we built the stack with gnu 9.3 compilers
and the spectrum MPI library.  Everything built fine but there are test failures that we need to sort out
before.

Maryam and Mark O have a PR in oops (#1081) that will likely be merged soon.  We have mentioned this
before - it changes the way the (text) compare tests are done.  Now they are done in C++ using IO streams
that are processed by a new TestReference class.  Currently it's backward compatible so it shouldn't break
existing tests.  But, as time goes on, implementing this new framework will require adding a section to
the test yaml files and re-defining the reference files.  More PRs are expected in the future that will
develop this framework more.

We had a code sprint last week to add multidimensional variable support into ioda and ufo. There were several related tasks that we wanted to do, such as convert all of the ioda files to the new format, ensure that all the ioda tests passed, debug some performance issues, and then make the minimal necessary changes to ufo to use these features. The ioda file-upgrading program was finished shortly before the sprint began, but these new files were the first true multidimensional files in jedi. We encountered a few bugs in the ioda interface and these somewhat blocked downstream changes. The sprint itself was only three days, and in retrospect, more lead time before the sprint would have helped greatly. So, since we addressed over half of our goals, I'll consider this a partial success. As of yesterday, all of the ioda tests pass, and the ufo integration can proceed over the coming weeks. Thanks to everyone who participated in this sprint.

All of this work is occurring in the feature/ioda-v2 branches of ioda and ufo. These branches track develop closely and when merged users should not notice any anomalous builds or testing results. The new, upgraded files are located in the ioda-data and ufo-data repositories.These repositories will go live when we merge in ioda-v2. Examples of the old vs the new ioda file formats were discussed at [yesterday's JEDI2 meeting](https://docs.google.com/document/d/1vNLNDIP9Ge6Q2VSTRrg1k8AsghTysi38KNmuU_GqfZo)

Besides the code sprint, a few group members noted progress on their tasks:

        • August has written string manipulation functions for the ioda odb backend. These will be used in converting ODB's strings, which are really separate ODB fields of eight characters each, into longer strings that the rest of us can use.
        • Ron has been working through issues with NCEPLIB-bufr's missing support for introspection. This means that users of this library cannot query the library about the presence or sizes of certain objects, and instead have to hardcode a bunch of assumptions. This produces very fragile code and really needs to be addressed upstream.
        • Wojciech has opened several PRs adding support for bias correction of multiple single-channel variables and support for static bias correction.

FV3-JEDI

- Ctests added showing the complete static B matrix workflow.
- Working on 3DEnVar benchmarking for Azure and validation of localization, LETKF.
- Continued work on the regional FV3SAR meteorology system. Working towards the 3DEnVar system.
- FV3-JEDI with NUOPC driver work continues. Can propagate forecasts and exchange states back to FV3-JEDI.
- Ming gave an update on the OAR plans for the next few years with the intended implementation goals.

Met Office
- Working with LFRIC group to determine requirements for Atlas integration.
- Using Atlas point cloud class to create fields. Working on adding required grid staggering.
- Added redistribution for going to Gaussian grid ahead of using spectral transforms.

NCAR
- Working on multi-resolution data assimilation capability. Testing with 30km background and 120km increment. Seems to have little impact on resulting analysis state.
- Testing the static B setup with 30km mesh. Working to resolve issues in the linearized psi/chi transforms. Adding CTests. 

NRL

- Working on LETKF and looking at how to implement the required vertical coordinate.

JEDI4 meeting - March 8, 2021

Updates:

Benjamin: working with Dan on a PR for full demonstration on how B matrix should be built (including vertical balance, variances, correlations). Diagnostics would be calculated for different cycles, then the diagnostics could be averaged. Next: running with fv3-jedi full resolution ensemble.

Questions:

• Dan: how straightforward is it to use the static B (estimated on high resolution ensemble) with lower resolution ensemble.

Benjamin: should be possible for vertical balance, but not for correlations currently.

BJ: running experiments with the multivariate 3DVar at higher resolution (previously ran at 120km; now at 30 km resolution). Problem with B statistics: for some hydrometeor variables NICAS fails. 

Benjamin: can work to find a fix.

Questions:

• Sergey: assimilating full suite of obs? How long does it take to run?

BJ: assimilating most of the conventional obs and clear sky AMSUA. 30 km resolution 512 proc, takes about 20-30 min.

Cathy: planned with Dan for GSI-like B-matrix testing with fv3-jedi. Starting to work on recursive filters. Coordinated with Benjamin on using lagged pairs.

Comments:

• Dan: also working on lagged pairs for GEOS.

Clementine: worked on PR for monitoring observations (compute H(x) but not assimilate the data) - only works for a full ObsSpace. Also working on the diagnostics for the minimizers.

Dan: working with Benjamin on static B. Adding code to oops to run the conversion for static B lagged pairs. Changes may be needed to IncrementEnsemble class to handle lagged pairs.

Eric: this week there’s a code sprint to prepare neptune interface for the bundle.

JJ: EDA experiments: moving towards higher resolution deterministic system with a low resolution larger ensemble. Working on getting more ensemble members in (using GEFS lagged ensembles). EDA executable slows down by a factor of 1.5 when increasing 20->40 members (299s -> 439s for 120km MPAS mesh, 36 PE per member, conventional+clear-sky AMSUA).

Marek: working on spectral B. There is a capability In um-jedi to run with trans through atlas. Running some test cases currently, and need to check on MPI capabilities. Once it’s working with um-jedi, the work can start on moving it up to saber.

Also working with cubed sphere in atlas with Dan. Focus is on equiangular projection, but would like to make it generic (lfric/fv3 have different choices for what’s included in the edges).

Olly: finished repartitioner class (atlas). Now have a way to take a lat-lon grid, interpolate to gaussian grid, and repartition to use in trans for spectral transform. Currently repartitions atlas structured grid, but is written in a way that can be extended. 

Nancy: Sarah and John M working on getting refactored neptune into neptune-jedi. Primary goal is to quickly get to cycling 3DVar. Sarah is working on using random draws from NRL static B to generate ensemble. Also working on implementing interfaces for LETKF. 

Questions:

• Sergey: Daley covariance is now implemented in JEDI?

Nancy: yes, Sarah recoded it to be used in real space, and it’s now model-agnostic. Next part is to take parameters and make them tunable.

Sergey: Breakthrough on using Halo distribution in LETKF, test for fv3-jedi is now passing. This will require more testing, Wei will work more on Halo distribution.

Land DA: need some improvements in how localization is done, to enable a new type of localization for land DA. Anna and Sergey will organize a mini code sprint to address this.

Experiments for fv3-jedi LETKF with full obs: Dan is setting up a full resolution full obs 3DEnVar test (for profiling) that may be useful for Sergey’s LETKF experiments.

Profiling: Wei profiled LETKF. ½ deg res, 200K obs. The core of the solver scales well. All the obs processing doesn’t. Dan: we’re aware of bottlenecks, plan to be fixing those in the next AOP. At the next JEDI3 meeting will discuss the strategy. Yannick: looking at the code I’ve noticed specific issues, which could be addressed quickly to improve performance. Yannick will open issues for that.

Junmei: Working on EDA experiments using raw data (conventional obs and amsua) and QC filters + MPAS BC function on 2020 period.

We had a successful AMS short course on JEDI this week.

We conducted a a successful internal EWOK and R2D2 tutorial, demonstration and planning session this week. Good progress has been made and the team feels encouraged that EWOK will be a nice tool for those running DA flows.

• No technical updates this week.

• We are just starting a new project coupling the recently developed EMC offline land-only driver with JEDI. We will be relying heavily on the existing Wrf-Hydro-NWM-JEDI work to do this, will update you as this work progresses. Also see the new UCLDAS repo (the “UFS CCPP Land Data Assimilation System'')

The issues with BUMP in high-resolution experiments has been fixed, thanks Benjamin! The group is now able to execute one year DA runs.

- Started running weekly experiments using EWOK/R2D2/DIAG for advancing the UFO development for JEDI-GDAS. Each experiment is based on latest develop, running H(x) (3D, no cycling) for a month. Weekly briefings jointly with EMC to review results, including comparison with GSI, and plan next steps.
- Work on SSMI/S bias correction and QC
- Short codesprint with UKMO/NRL on ZTD forward operators for ground-based GNSS
- Work on BUFR to IODA converters with EMC; addition of global attributes to IODA.

I also mentioned that it would be worthwhile to have a separate update in the JEDI meetings from UKMO on their work on observations.