2021-05-27

JEDI 1 report

* Mark O departure
  - Mark O has decided to pursue another job opportunity and will be leaving JCSDA.  His last day will be Friday, June 4.  We are grateful for his substantial contributions to the JEDI project over the last two years and his broad and deep software engineering expertise will be greatly missed.  We wish him the best as he moves to this next stage of his career.

* Update on Hera and Orion
   - The intel compilers get their C++-14 support from the underlying gcc compiler headers and libraries.
   - Last week, beginning Friday, May 14, the Hera administrators changed the gcc library support for the intel/2020.2 modules.  This broke the jedi stack; eckit and other packages that were compiled with a different glibc now could not find some of the symbols they were linked to. 

 
   - We suspect something similar occurred on Orion
   - In the long term this is a good thing because it provides more reliable C++-14 support for JEDI.  But, it caused much inconvenience when it happened.
   - Rick, Ryan, and Mark rebuilt the intel modules on Hera and Orion.  For Hera you should now use the "jedipara" modules (we have not yet updated the jedi documentation to reflect this change but we will do so):

   export JEDI_OPT=/scratch1/NCEPDEV/jcsda/jedipara/opt/modules
   module use $JEDI_OPT/modulefiles/core
   module purge
   module load jedi/intel-impi/2020.2
   (or)
   module load jedi/gnu-openmpi

   - the procedure for loading modules on Orion has not changed
   - As far as we know everything is working.  Let us know if it is not.

   - See https://github.com/orgs/JCSDA-internal/teams/jedi/discussions/70 for more details about this issue.

* JEDI 1.1.0 Release timeline
    - June 1: creation of release branches
    - June 4: freeze code
    - June 11: Public release

* New containers
   - all Singularity, charliecloud, and docker (CI) containers have been updated with the new versions of ecbuild, eckit, fckit, atlas, odc, and NCEP-bufr

* Academy preparation has begun
  - Virtual Academy: June 21-25
  - will highlight the new release and in particular the changes that have been made to ioda and ufo

Further report from yesterday's meeting (from Steve):

ZenHub issues tagged with the release
* rttov ecbuild failure - fixed by PR from Mike Cooke
* obs space destructor crashes - PRs in oops and ioda to address this
    - moved save to file out of the ObsSpace destructor into a function that is called by the application right before destroying the ObsSpace
* fv3-jedi ctests changing fv3-data files - no update
    - Maryam will help debug this
* We removed the JEDI 1.1.0 release tag from an issue related to creating tutorials based on the ioda-engines examples since, for the release, we have not exposed that interface fully yet. The exposure of the ioda-engines interface is slated for after the release.  We will define another tutorial for the release.
* Discussion about the performance issued raised by JJ; JJ advocates for a fix before the release

ZenHub issues tagged with the current sprint
* Fairly large list, some progress being made, and one or two issues got closed

Round table
* Wojciech has several PRs under review in ioda and ufo that fix issues with the ioda Distribution classes
* Mark & Maryam have built new CI containers (with the ecbuild 3.6 compatibility) and are currently testing them
* There was some debate about the CRTM version to use
  - We settled on release/crtm_jedi, but this does not auto-download the coefficients from UCAR DASH
  - Maryam will fix it so that release/crtm_jedi will trigger downloading the coefficients from UCAR DASH

* Mark O found an issue with building eckit in jedi-stack related to using a static library for LAPACK
  - Mark has a fix in a new docker container and is testing; see PR

* Jaques just returned from vacation and is getting caught up
   - He mentioned concern about the recent messages about the trouble on Hera and Orion, but we determined that he should be okay with these since he is using EWOK (GNU and openMPI) and the issues were related to the intel compilers

JEDI2:

Jedi 2.2 - Generalized locations (EMC liaison, Emily)

• Brief discussion on obtaining temperature in field of view

Jedi 2.3 - Use of NUOPC driver with FV3-JEDI (Mark Potts)

• Finished review of FV3 Cap. Dusan already changed the Cap to expose atmospheric states. 

• Simple to add more variables in the future. 

• Merged into develop on UFS side.

• Testing out with JEDI running a forecast from JEDI and writes netCDF.

• A handful of variables are not the same so some debugging needed.

Jedi 2.6 - Ensemble DA validation (Cory, Sergey)

• Translated 3DEnVar benchmark to LETKF.

• Some problems assimilating conventional data with LETKF.

• Plan to run GSI and start comparing things.

Jedi 2.7 - Background error model validation (Benjamin, Dan, Cathy, Ricardo)

• Benjamin, still struggling to get smooth fields out of psi/chi.

• Looking at the interpolation step in vertical balance.

• Dan working on new wind transforms.

Jedi 2.8 - Regional DA (Ting, Ming)

• Basic version of testbed based on GSI workflow. Run JEDI analysis at the same time.

• Run FV3 hybrid cycles. Then produces JEDI analysis. 

• Uses conventional data.

• Working on a setup of py-gsi to validate and compare diag files. 

• Run GSI and JEDI with the same conventional observations and compare.

• Moving to soon letting JEDI control the cycling.

Jedi 2.10 - LFRic general updates (Kristin, Steve, Steve)

• Fixing some small bugs in the moisture control variable transforms.

Jedi 2.14 - Cubed sphere grid into Atlas (Dan, Marek, Olly, Steve V)

• Just about ready with sending back the changes to ECMWF.

Jedi 2.15 - VADER (Steve V, Dan)

• Fully working prototype ready to be worked with.

JEDI3 (DA Algorithms) update:

Clementine has added a capability to plot convergence for a block minimizer: fv3-jedi-tools#11
Clementine has PRs adding a diagonal observation error covariance to UFO: oops#1230 and ufo#1228. This is a step establishing all the wiring for adding observation error covariance implementations to ufo (a first step towards adding observation error covariances with cross-channel correlations). We would also like to make the UFO diagonal error covariance more flexible than the OOPS one. It would be good to make a link with Carwyn’s work on assigning observation errors (ufo#843).
Clementine met with Andrea Piacentini from CERFACS to discuss improvements to MPI communications in 4D variational algorithms.

Benjamin is working on a PR that would allow BUMP parameter estimation and background error/localization application to be run on different MPI distributions.

Yannick has added improved logging of memory usage in JEDI (statistics for some objects are now printed out at the end of the application run): oops#1214 and oops#1215

Wojciech has extended a static bias correction to use input data from files: ufo#991 and linked PRs.
Wojciech has proposed an approach for assimilation of “derived variables”: ufo#1187

Wei Zhang is working on profiling the benchmark case to identify bottlenecks and spots for the GPU parallelization in the future.

R2D2:

• we are adding ioda v2 observations for the jedi-gdas experiments for users on Orion

• we are in the process of adding backgrounds for other models in order to prepare to run HofX applications for SOCA, MPAS etc...

EWOK:

• we have been working on the ioda converter for GOES-16 data, to update it to convert files to ioda v2 files.

• we have has been working on archiving the logs of all the jobs running in an experiment

• we have been working on a tool for reporting the cost in node / hour for experiments

• we are working on changing the format of experiment configuration yaml files to work with includes so that a particular model can be plugged into an experiment. Plugging in more than one model would eventually result in coupling models.

C3PO (Data Lake)

• we have been working on a generic framework for queues and lambdas for different data sources with integration of ioda converters.

The team has been preparing for the upcoming academy.

Cory is setting up a data store for the JEDI GDAS experiments focusing on conventional obs data.

Emily is verifying radiance H(x) experiments, and is investigating into a bias correction issue that was uncovered.

Fabio is adding in ozone handling to the experiments.

A concatenation (of the ioda output files) issue has surfaced with the ioda-v2 file formats. Eric is working on a solution.

Rachel is making good progress with the diagnostics performance enhancements. She also is making progress on the SageMaker flow.

IODA-v2 merged on Monday. This included branches in ioda, ufo, ioda-converters, several models, and the bundles. Most importantly for end users, this changed how we manage test data, which is now stored in the ioda-data and ufo-data repositories in IODA-v2 format. New PRs that alter test data should upload new files to a feature branch there, and should mention this in their main PR text. The default pull request template will be updated to reflect this soon.

The ioda-v2 merges in all of the repositories means that we are getting close to the release, which should occur during the JCSDA workshop, which is June 7 - 11. To prepare for the release, the OBS team is performing a small UFO documentation sprint today and tomorrow. Our focus is on ensuring that UFO is well documented on both Doxygen and ReadTheDocs. We are also preparing our talks and practical sessions for the next JEDI academy, which is June 21 - 25.

Due to both the workshop and the academy, we are pushing back the next two OBS3 general meetings by a week, to June 16 and June 30.

We had a general meeting yesterday, and we discussed plans for the new variable naming conventions. We would like for the conventions themselves to be ready within June. Greg recently found a set of eccodes tables that have very similar names to what we were considering, so we would like to review those names and unify what we can. We plan on hosting a document on ReadTheDocs that describes how data are organized in IODA, and JEDI users can find and use this data.

Once the conventions are finalized, we will need to coordinate changes throughout ioda-converters, UFO, and every repository that has a hard dependence on specific variable names. Test data and R2D2 data will also need updates. This is rather large, and we only want to do this once. Because of this, we are considering a code sprint in July to tackle these changes. Details will be hashed out in the coming weeks.

The group is also considering how we want to store derived data that is generated by filters and operators. We all want our input data to be immutable, but where do we put new data generated by variable transformations or other filters? The main difficulty is because filters are closely coupled with HofX, and we need to carefully plan changes to the code so that we do not need to recompute GeoVaLs.

Several group members reported recent progress:

        • Anna reports that the RadarReflectivity ObsOperator has been retired and replaced with a more flexible version of the VertInterp operator.
        • She also has been working on removing the explicit mpi communicator from the Locations code, as this is now encapsulated by the ioda::Distribution classes.
        • Steve Herbener is back from vacation and is fixing a file-closure bug that we have observed in the experiments. C and C++ code are interacting badly, and upon application exit output files are occasionally being closed before data are fully written.
        • Ron is continuing his work on extending NCEPLIB-bufr with a better query interface.
        • Nick is enhancing the ioda converter tables for BUFR ADPSFC and AIRCFT data, and Praveen is creating corresponding ioda-v2 files for this data to validate bufr2ioda output.
        • Neill has worked on two PRs. The first unifies the Met Office's refractivity code for GNSS-RO and ground-based GNSS. The second adds the radio occultation refractivity forward operator. TL/AD code will be next.
        • Chris has implemented a categorical observation operator that can run several component observation operators and will pick the value of H(x) depending on a categorical variable.
        • He has also completed an operator for profile vertical averaging, and has helped enhance GeoVaLs with a PutAtLocation function that complements the GetAtLocation function introduced in February.

- Things are working again!

- EWOK for soca in the soca-workflow internal repo (Travis)

- Conversion of ioda to iodav2 marine obs (Kriti, Travis)
- Coupled h(x) with GEOS (Hamideh/Min-Jeong)
- Regional high res reanalysis restarted (Kriti)
- Trying to organize a short 1/4 deg global reanalysis in colaboration with EMC and JEDI to provide MOM6 and CICE6 background for coupled h(x) application for the period of Dec 2020 to Feb 2021

Land update, not much to report here as; 

• Team LAND has been very involved in a UFS land workshop this week, where there was a (slightly surprisingly) very high level of interest in DA in general, and JEDI specifically, so I think the JEDI Academy website will see an uptick in visits this week. One take home was a strong desire for a streamlined, testbed application-type package accessible to the university research community - although maybe that’s a challenge for our new friends at Raytheon.

• We will be merging existing branches in the NWM repo today/tomorrow that will bring Develop up-to-date with the recent ecbuild, IODA and OOPS changes, except adding the variableSizes function to Geometry, which I’ll get to today/tomorrow.

There will be several presentations on CRTM and RTTOV at the upcoming ITSC-23 conference (virtual conference, June 24 - 30).

Antenna handling code fix is working.

A collaborative effort with CSU has started with the goal of improving matching between model microphysics and CRTM microphysics.

Development of operators for satellite radar and lidar is work in progress, and the plan is to add in ground radar next.

CRTM v3 development is focusing on the integration of surface, aerosol and cloud related radiative characteristics and effects.

CRTM v2.4.1 release will occur soon (possibly July). This will be the final bug fix before CRTM v3 is released.

CRMT monthly discussion meetings are now taking place. Please send email to Ben if you are interested in joining.