2021-02-25

At  last week's meeting of the broader JEDI 1 team Mark O gave a presentation on his recent work to profile the FV3 3D H(x) application and, more broadly to define a set of tools and metrics that we will use for JEDI profiling and optimization going forward.  He's working in particular with Intel Vtune, Arm MAP (part of FORGE, along with DDT), GPTL, and callgrind.  All can potentially offer different insights.    His presentation highlighted MPI inefficiencies in GetValues (interpolation), variable changes, and the FV3 Geometry constructor.  Further work will clarify these inefficiencies and formulate a path forward.  Mark O will lead a future topic discussion on these profiling tools and results.  And, in the meantime, he will write up some documentation to guide others on how to use and interpret these tools.

In preparation for the next release of IODA and JEDI, Maryam is implementing new CMake functionality for bundles to use the new ioda-data and ufo-data repositories for the test data.  She has a PR in ioda now for those who want to take a look.  This is an important improvement over the current test data management and will allow developers to add their own test data without having to rely on a JEDI team member to upload it.  She and Mark O are also finalizing their work on improving compare tests in oops.

In our new process of vetting proposed additions to the JEDI stack, we also discussed Rick's PR to add ecflow as an optional component.  Given the growing usage of ewok, it was agreed to approve this addition.

The core JEDI 1 team also discussed the possibility of adding a capability to ecbuild to do a shallow clone of a particular repo branch.  This would help mitigate slow clone times on platforms such as S4 and Cheyenne for repos such as crtm where unneeded branches may be significantly larger.  Maryam created a zenhub issue for this.

Mark M and David Hahn participated in a meeting on Monday morning with representatives from Microsoft's Azure cloud platform.  They are interested in working with us on running JEDI applications on Azure.  This would be experimental for now but could lead to a larger collaboration and more resources.  To support this collaboration, Mark M created an HPC Intel OneAPI application container with infiniband drivers for Azure and the release 1.0.0 version of fv3-bundle.  Mark O and Dan provided input files for an H(x) application that we can use as an initial benchmark.

In the JEDI 1 and JEDI 2 meetings this week, we made plans to do a mini code sprint next week for the IODA release.  Steve described this in the meeting.  The main changes needed for the release can be roughly divided into two stages: Stage 1 implements the ioda-engines refactoring and Stage 2 implements support for multi-dimensional variables.  The issue that is holding up Stage 1 has been performance: tests with particularly large data input files are running substantially more slowly than in the current ioda develop branch.   We believe that these performance problems will be solved by Stage 2.  However, we do not wish to merge Stage 1 until Stage 2 is ready because we do not want to harm the performance of current applications.  Furthermore, in order to avoid performance degradation, current applications that use ioda version 1 data files will have to be converted to ioda version 2.

So, we decided that the best way to handle this situation is to create `feature/ioda-v2` branches in ioda, ufo, and ioda-converters.  This has the Stage 1 code in it now (with ioda-engines).  And, the Stage 2 PRs will be issued to these branches instead of develop when they are ready - likely in 2-3 weeks.   Eventually, these feature branches will be merged into develop to create the ioda 2.0 release.

To accelerate this work we are planning a mini code sprint for next Wed-Fri (March 3-5).  The purpose of this will be to implement the multi-D capability in ufo and to convert existing (v1) ioda data files into the multi-D (v2) format.  Meanwhile, we will also implement the new ioda-data and ufo-data repositories to hold the tier 1 test data.

We need your help with this: anyone available to participate in this mini code sprint is invited to contact Steve, Ryan, or Mark M. 

JEDI 2 Update from Ryan:

JEDI2 met yesterday, and we focused on release planning, roundtable
discussion, and progress updates.

• Steve, Mark, Yannick, and I are organizing parts of the IODA version 2
  release into code sprints. We have two sprints in mind, both which would
  occur in March with dates TBD. The first should be a mini-sprint with
  the JEDI1 team that will produce the pull requests needed in IODA and
  UFO for multidimensional radiance variables. The second sprint will be
  larger and should have several groups work to convert their IODA files
  over to the new format. Again, details will be hashed out in the near
  future.
• The next release-based issue is a question for everyone. What
  documentation would you like to see in the IODA release? We are planning
  both API documentation and usage examples to cover areas like: "How can
  we use the new ObsGroup object in a filter?", and "Best practices for
  writing a IODA converter". If you have suggestions, please email me.

Several group members reported contributions and issues:

• Ron is working to support sequences in the bufr parser, and this will
  cause large changes in the BUFR parser's YAML file format. He is running
  into several limitations in NCEPLib-bufr, and has made several changes
  to its code so that he can extract type information for mnemonic fields.
  These changes will be picked up in the jcsda-internal form of
  NCEPLib-bufr so that they will be usable in the stacks and by a wider group.
• Steve worked on a performance fix with the ObsStore-refactoring PR.
  This performance issue is caused by hyperspectral data and is difficult
  to fully address, so he plans on redirecting his efforts to changing the
  ioda file format to remove the cause of the issue in the ioda-converted
  files.
• Jianjun Jin identified an issue with converted GMI files: GMI is a
  conical-scanning-mode instrument where scan angle depends on channel
  number. This information needs to be propagated into the new IODA file
  format.
• Wojciech is expanding the parameters validation classes to handle the
  "where" clauses in ufo filters. He plans to work on static bias
  correction in the near future.
• Chris is continuing work on profile vertical averaging.
• Ben Ruston is working on ground-based GNSS. This includes: a prototype
  ground-based GNSS forward operator, bias computation code, and defining
  the IODA format for this data.
• I have recently added test tiering options to UFO. Test tiers are
  intended to work the same way as in fv3-jedi, where unit tests always
  run to check functions and code coverage, but more expensive tests are
  run less frequently in a higher testing tier. Over the upcoming weeks,
  we will classify which tests belong in each tier.

JEDI 3 update from Dan

We didn't have a JEDI 3 meeting in the last two weeks due to the President's Day holiday so there's no project summary. For FV3 models:

- Working on the full resolution static B. Have a workflow to generate the entire static B model from a high resolution large ensemble.
- Continuing to work on the UFS interface within FV3-JEDI and have PR that will allow forecasts to run.
- Developing the infrastructure to be able to generate the Static B model from an ensemble of perturbations from the GEOS model.

Dan then invited other model group to contribute updates

Sarah gave an update from NRL

They upgraded the JEDI-Neptune code to use a more recent Neptune release.  This will enable them to implement JEDI ensemble applications and they particularly want to implement ensemble applications that use their static B.  Sarah is also working on implementing an LETKF application for Neptune.

Anna was unavailable to attend so Yannick gave the summary for JEDI 4 & 5

The JEDI 4 team has been working on some BC cleanup.  And, the main focus of the JEDI 5 team is an upcoming AMS short course (a mini JEDI Academy) that will be held for 4 hours on March 10.  We already have reached the limit of 100 participants.

Yannick also gave an update on ewok.  The team is now running reference experiments that compute H(x) for one month of data.  This has now been run twice and it will be used to implement diagnostics.

Dick, Greg, Wei, and Francois then gave an update on recent OBS work, summarized here:

• Progress on comparing FV3 H(x) vs GSI H(x) using the UFO configurations that replicate GSI (Wei, Fabio)
• Working on BUFR -> IODA (Greg, Nan, Ron)
• First demo of obs diagnostics in Jupyter notebooks on AWS, fetching IODA from R2D2 (Rachel)

Wei reported in particular that the time interpolation was a notable factor for the JEDI - GSI comparisons, leading to significant systematic differences, particularly for radiances.   This should be a priority for the future.    Fabio clarified that the comparisons were between GSI FGAT with time interpolation and JEDI FGAT without time interpolation.

Yannick commented that work on time interpolation is proceeding with JEDI , currently in the U-jedi repo.  But, these differences may not require time interpolation - they may be solved when we move to 4D H(x).  This should make a big difference even without time interpolation.

Francois reported that the diagnostics work with Jupyter notebooks and R2D2 (led by Rachel) is going well.  They are now looking at ways to speed up the data transfer from S3 into the notebooks on Sagemaker.

Andy then contributed this summary from the LAND project:

• Nothing very much from Land this week as I’ve been focused on annual operation plan development

• Have a draft pull request for adding covariance localization mask based on Gaspari-Cohn 99 function to the NWM code

• Dealt with adding config to linear getvalues (JCSDA-internal/oops/pull/1044) to the NWM code

Guillaume then contributed this update from SOCA

• Welcome Kriti, just joined the soca project from UMD and will be working on regional ocean DA
• Progress on sea ice freeboard assimilation with GEOS (Min-Jeong)
• Activation of Travis-CI for "soca-science" (Travis)
• Addition of a SOCA-EWOK 3DVAR cycling ctest in "soca-science" (Travis)
• Documentation of SMAP OSE's (Hamideh)
• Complained about the oops people breaking soca (Guillaume & Travis)
• Testing of the 3DVAR/3DEnVAR for the GODAS implementation at EMC/CPC