2021-04-08

JEDI1 update

Regarding IODA the version 2 (v2) release, things are proceeding well.  The main issue that has
been holding up the merge of the ioda-v2 feature branches over the past few months has been
concern about performance degradation; back in February, the ioda and ufo test times for
ioda-v2 were much slower than in develop (https://github.com/JCSDA-internal/ioda/issues/116).  
Thanks to the most recent changes by Ryan (and by both Ryan and Steve over the last month),
these appear to be resolved.  Ryan has a feature branch (PR soon, after more cleanup and
testing) where the test times are comparable in the ioda-v2 and develop branches.  Ryan can
give further details.

So, we are close to being ready with ioda-v2.  After the performance issues are resolved, the
main things left to do for the ioda-v2 release are:
* implement a ioda exceptions class for better error handling
* implement obsspace extensions
* Documentation & a tutorial

The ioda v2 code sprint has been cancelled for now.  We found that by creating a test set on
AWS S3 with the current ioda test data set converted to v2 format, we can make the transition
easier for the models.  See https://github.com/orgs/JCSDA-internal/teams/jedi-models/
discussions/44.

After the release, there is potential for two ioda related code sprints:

1) Update the ioda converters to write ioda-v2 format files directly (by switching to the ioda engines Python API)

2) Update the repos that use the AWS ioda test set, or their own copy of ioda files, to the scheme being adopted by the ufo and ioda repos. This scheme consists of creating a test data repo which is git lfs enabled, and using git lfs for version management of the test data. In addition the idea is to place all files necessary for your repo's tests in your test data repo such that the test data repo is independent of the other test data repos and is thus freed up to evolve on its own.

Another thing we wanted to do for the next JEDI release is to move to the latest ecmwf branches
of ecbuild, eckit, fckit, and atlas.  There are "feature/ecbuild" PRs in many of the fv3-bundle
repos that implement this.  These should be backward compatible (they are on my Mac), but there
are some remaining issues that need to be investigated.  In particular, the CI test on fv3-jedi
is failing and I'm finding the oops and ufo parameters tests are failing with the updates
(possibly due to eckit changes?).

You can test the new ecbuild yourself.  "ecbuild35" versions of the environment modules have
been installed on Hera, Orion, Discover, Cheyenne, and S4.  There is also a
gnu-openmpi-dev:ecbuild35 singularity container ready for testing.  We decided to include fckit
and atlas in these modules as well, since we are now using tagged version.  So, by default,
they will be included in the modules and containers and should not be built with the bundle (to
change this you can unload the particular module, e.g. atlas, and turn off the BUNDLE_SKIP_ATLAS
flag).

Note that there has been another release of ecbuild since with started this work, so these
modules contain ecbuild 3.6.1, not 3.5 (despite their name).  There is also a new release of
atlas, 0.24.1.  This has been included in all the modules except for the intel-impi stack on
Cheyenne.  There the build failed for the atlas 0.24.1 release so the stack includes the
previous release, 0.23.0.  Mark is looking into it.

Also for the release, Maryam is working on implementing a git-lfs fv3-jedi-data repository
similar to ufo-data and ioda-data.  And, meeting participants will be glad to hear that she is
also working on publicly posting the CodeBuild results to cdash so all PR authors will be
able to see the build logs and determine why tests failed, if they did.

Mark O, Mark M, Yannick, and Jacques have been assessing next steps with regard to
optimization.  Mark O plans to create a new jedi-profiling repo to host the profiling tools he
has been developing.  We currently have two benchmarks we plan to work with, namely an NRT Hofx
benchmark and a 3denvar benchmark that Dan put together.  We have discussed how to organize and
monitor these benchmarks and track their performance over time using hierarchical S3 buckets
and fv3-bundle tagged with commit hashes.  We plan to focus first on how we might improve the
performance of the unstructured grid interpolation in oops.

Things are also progressing on the cloud front.  We are in the process of migrating our AWS
account to UCAR and we are working with a team from Microsoft to get JEDI working on Azure.  We
have simple tests working on Azure but we're having problems running a larger benchmark with
the new intel oneapi application HPC container.

JEDI2:

- Steve Vahl has joined the JEDI2 team as the Met Office Liaison. He will be working on generic developments that will benefit the Met Office as well as working on the Met Office model interfaces. Initially Steve will work on the software engineering design for a generic variable transform repository called VADER.
- MPAS and SOCA groups are working to implement the variable change for the GeoVaLs so that they can take advantage of the faster h(x) application that was made available.
- We had some discussion about refactoring the GeoVaLs needed by the UFO where the variable could not be interpolated using transitional interpolation techniques. JJ (NCAR) plans to change the surface wind GeoVaLs for the CRTM.
- Most of the meeting was spent cleaning up tickets ahead of the next AOP.

• Benjamin and Dan: working on training for fv3 background error covariances.
• Clementine: working on diagnostics for block Lanczos minimizer.
• JJ: working on EDA for MPAS. Using leave-one-out approach, running 60 members at 120km resolution stably.
• Nancy: working on Var, main issue currently is vertical interpolation for moisture variables (in observation operator).
• Anna: working on observation-space localization for LETKF-type filters. Draft PR for moving local observation search out of ObsSpace (to ObsLocalization).
• Yannick: working on cost function and observer refactoring (series of PRs in oops).

• Weekly experiments are running H(x) application using EWOK and R2D2
  • Running on Orion and work is under way to get this flow working on other HPC systems
  • Primary purpose of these experiments are to validate UFO filtering, QC and H(x)
• R2D2 has one month of obs data (which is being used in the weekly experiments)
  • Rick G reported that he successfully converted all of these files to ioda-v2 format
• Work is under way on next steps
  • Run the weekly experiment using the new ioda-v2 code and files to evaluate performance
  • Run UFS forecasts in EWOK

OBS3 / JEDI2 summary for Apr. 8:

We have had one week under both the old and new AOPs since I gave an update. JEDI2 had its last meeting yesterday to wrap up AOP20, and the first OBS3 meeting will occur in two weeks. I described how the new subtasks are organized, and the new epics are available on ZenHub. We may move the time slightly due to a persistent conflict with an EMC meeting, and I am going to poll the new group to get their feedback on a few options.

We then talked a bit about the IODA-v2 release. Steve Herbener and I are both working on performance enhancements to the code. We are taking a tiered approach, and are first addressing the more egregious issues that appear in ioda, ufo, and fv3 unit tests before running the H(x) testbed on Orion to evaluate a larger data set. Thanks to Rick Grubin for converting those testbed files. There are three issues that we have noted at the unit test level, and we have solved two of them.
        • First, we addressed an issue with UFO's GeoVaLs performing an incredibly inefficient string comparison on Intel compilers, and with a small fix we have halved the UFO unit test times. We committed that fix directly to UFO's develop branch.
        • Second, I rewrote part of the HDF5's dimension scale code to allow us to attach many dimensions to many variables in a collective, batch call. With this change, ioda-v2 and develop unit tests now take the same time overall. The longer tests, such as for IASI, CrIS and GIIRS, run 2-4 times faster.
        • Conversely, many tests still take longer due to an issue with data selection during file reads. That fix is half done. Once complete, the affected tests will likely run faster than their develop-branch equivalents. From here, we are comfortable with progressing to EWOK testing, which we could run starting later today.

Besides the benchmarking, we want to improve IODA's error handling and add additional documentation. Then, we are good to release. April is still viable.

Besides the IODA work, we made progress on other tasks:

        • Wojciech Smigaj has added support for bias correction of single channel variables and for static bias correction. He is also working on a bias predictor that interpolates arrays loaded from NetCDF or CSV resource files.
        • Jianjun Jin has been expanding UFO's CRTM interface to process GMI's scan angles correctly. Unlike many other instruments, GMI has scan angles that vary by channel. The CRTM interface can now handle this correctly, but we need to expand how we encode this information in our IODA files.
        • Ron McLaren is continuing his expansion of NCEPLIB-bufr to better handle sequence types. This bufr library is missing functions to examine the structure of bufr files related to conventional observations. Users have to hardcode structures and object sizes manually, and these vary with the input bufr data. This is incredibly inflexible and error-prone, and Ron's work will address this.
        • Anna Shlyaeva has issued a fix for UFO's TestFilters executable and the IODA distribution's inner product codes. We use different missing value markers for different data types in JEDI, and these missing values were not updated upon type conversion. This affected our calculation of RMS errors, but only in our UFO filter unit tests. Thanks to Greg Thompson for pointing out this bug.
        • August Weinbren and David Davies have been working on the ODB IODA engine. August submitted pull requests related to mapping ODB variable names to IODA equivalents. These mappings are customizable using YAML, and now uses the regular OOPS Parameters classes.

Here's what the great soca people did:
- NG-GODAS release tag in soca-science (ocean sea ice reanalysis for NOAA)
- Better BUMP configuration for soca
- First ocean surface increment from direct radiance assimilation
- Debugging the regional soca configuration

• Work continues on the weekly H(x) experiments
  • EMC is working on providing updated backgrounds for improving comparisons between JEDI and GSI
  • Work is under way to assimilate the SSMIS obs data and perform VarBC for this instrument

• Work is in progress for VarBC integration
  • GNSS obs types, eg.
• Biggest challenge is super-obbing/averaging
  • Need to be able to execute compatible changes in the geovals file