2021-01-07

JEDI1 general announcement:

Update on Intel:  The good news is that intel is replacing Parallel Studio with the new Intel OneAPI (Gold).  OneAPI includes the intel compilers, intel MPI, MKL, Vtune, etc and, unlike Parallel Studio, is now free for download without any need for a license.  The bad news is that, after consultation with Intel's legal department, we have confirmed that, even though it is free, we are not allowed to distribute OneAPI compile-time resources in containers.

So, we are still unable to provide intel development containers for general use.  Instead, we plan to provide these containers:

• gnu-openmpi and clang development containers
• HPC-ready intel application containers for releases (compiled code, with no compilers)
• Intel OneAPI Dockerfiles + instructions so determined users/developers can build their own intel development containers

JEDI1 project meeting (yesterday):

Since this is the beginning of Quarter 4 in AOP 2020, we took this meeting as an opportunity to review in some detail the status of each of the JEDI 1 Epics/AOP Tasks.  Overall, the JEDI 1 project is On Target.

Some tasks are already complete, including JEDI 1.6 (Infrastructure for JEDI-based NRT suites), JEDI 1.14 (Support for OpenMP), and JEDI 1.15 (Code sprint: improve and speed up tests).  Others have a more open nature, with ongoing tasks that will continue into the next AOP in some, potentially modified form. These include JEDI 1.7 (Provide access to data to support UFO development), JEDI 1.8 (Provide access to NRT data to support DA experiments), JEDI 1.9 (Running and monitoring NRT systems), JEDI 1.10 (Automated testing with all models), JEDI 1.12 (user and developer containers), and JEDI 1.13: (support for HPC modules).

Yannick reported that the workflow-related tasks JEDI 1.1 (Suites and scripts for single DA cycle), JEDI 1.2 (Suites and scripts for cycling deterministic DA), and JEDI 1.3 (Suites and scripts for cycling DA with ensemble update) are on track to be completed by the end of Q4.  Given the nature of ewok development, this work is going on in parallel (as opposed to sequential), and we expect them all to be completed around the same time.  However, JEDI 1.4 (User interface for running scientific experiments with JEDI) is behind schedule and will likely get pushed into AOP21.  JEDI 1.5 (Support for integrating models into suites and scripts) is a more open-ended task that is well underway and that will continue indefinitely.

We spent some time discussing JEDI 1.11 (Tools to measure and report computational performance).  There has been substantial progress on this, including the integration of the GPTL library, the integration of cdash, and the implementation of run time monitoring tests in oops and fv3-jedi.  However, tests to monitor and alert us to sudden performance changes have not yet been incorporated into our standard CI workflow.  This is largely due to the variable nature of run times even on dedicated, homogeneous platforms like AWS CodeBuild.  Any performance monitoring tests will require careful consideration of statistics, relatively compute-intensive tests, and may not be amenable to automation.  It was agreed that further progress on this should await the implementation of a more systematic multi-tier testing structure, which will be a priority item in AOP21.  A similar conclusion was reached for incorporating valgrind tests into our regular CI system.

We then spent the remaining time reviewing the November, December, and January milestones.

We had a JEDI2 meeting on December 16 [meeting notes](https://docs.google.com/document/d/1Nyj6Y3XwVM1CXgY2425M29QBq70zur5WexARJQffdKw/edit?usp=sharing). The next meeting will occur on Wednesday, January 13, where we will assess the overall state of JEDI2 and go through the ongoing task list.

At the December meeting, we discussed progress in UFO. We noted updates on the C++ refactor of the locations class, progress on the stuck check and relative humidity qc routines, and on the expansion of the parameters classes for the existing qc filter code. However, most group members were occupied by the code sprint, academies, other meetings, or vacation, so this was a short meeting.

There are two things worth noting:

First, there will be a special topic discussion on extensions to QC flags and actions in UFO, tentatively scheduled for Tuesday, January 19 at 10-11 ET. I am coordinating schedules with Emily and Wojciech on this, and will send out the general invite soon. Please contact me if you are not already in the JEDI2 email group and wish to be invited.

Second, ioda-engines will be merged into ioda's develop branch today [JCSDA-internal/ioda#44](https://github.com/JCSDA-internal/ioda/pull/44). This PR represents months of work in the ioda-engines repository, and many thanks to everyone who helped develop, review, and test this code. The integration has been tested extensively across fourteen compilers. It has been tested on Discover, Hera, Orion, and S4, in containers, and on the core team's macs. We have fully-working C++, C, and Python interfaces. There are almost a hundred new unit tests. When we click the merge button, we are reasonably confident that end users will not have issues, but if you do, please email me.

This PR is the first in a series that integrates new functionality into ioda and ufo. It adds the code, and later PRs enable or link up this code with the rest of jedi. So, there will be another PR to integrate engines with ObsSpaces, another PR will enable use within bias correction, another PR will enable Fortran, and so on. ioda-converters already uses ioda-engines, and there the change is particularly easy -  just one line of CMake code. Now that the base features are available, we can also work on porting the observation operators and qc filters, which will coincide with ioda file format changes, and so this will definitely extend into the next AOP year.

In light of the holiday break and Dan's paternity leave the main focus of the meeting was an update on what meeting participants are working on and plans for January (some team members were still away on holiday).  There was also an appeal to all team members to help out with code reviews in Dan's absence.

Steve V reported on his own work as well as broader work from the MPAS team.  He and the team have been working on implementing the unstructured grid interpolation in oops as an alternative to bump interpolation.  This work is proceeding.  Steve also mentioned that the MPAS team recently ran a several-day documentation code sprint.  There is a feature branch in jedi-docs with a PR coming soon.  Steve also mentioned that JJ has been running EDA experiments with 1-month cycling.

Maryam has been working with Mark Olah on restructuring how many tests are handled in JEDI, combining run and compare steps into a single test.  These changes are being implemented at the code (C++) level rather than in the python and shell scripts.  For now they are focusing on oops but the changes, when implemented, will impact models.  Keep an eye out for upcoming pull requests.

Mark P continues to make progress in integrating UFS into JEDI.  He is working on a PR that links fv3-jedi to UFS using stubs for now.  These stubs will be replaced with the UFS calls.

Travis reported that the Benchmark 2 test/application is now running for SOCA and this implementation uncovered a few issues that are now being addressed.  Next they will work on another Benchmark 3 application in preparation for a new planned release of SOCA.  Another major objective for the next 2 months is to fully implement ewok into SOCA.

Krissy is debugging some test failures in lfric.  Some appear to be related to a saber PR that was merged in late November (#26: vertically varying fixed radii), though that PR should have been backward compatible.  She is updating the yaml files to correctly use this feature.  She is also comparing results obtained with the bump and unstructured interpolation methods.  In January she also plans to write more lfric tests to improve code coverage.

Cory described a PR to ufo that adds a new filter to check whether an observation lies within the domain of a local area model.  Currently it is designed to be used with the FV3-LAM grid but it can be extended as needed.

Ming continues to improve the JEDI-WRF interface and to bring it up to date with the latest JEDI develop branches.  He had a few issues regarding the ingestion of regional data into JEDI.  One involved the conversion of bufr files and another involved variable transforms.  It was suggested that Steve H may be able to help with the former and Marek may be able to help with the latter.  Ming also mentioned that he will be giving an oral presentation at the AMS next week that will include an update on the effort of moving from GSI to JEDI.

Ting is working on implementing a B-matrix for FV3-LAM, with both static and ensemble components.  He would like to map the GSI B Matrix to BUMP and compare the results.  He is running into some issues, particularly with regard to the yaml files, and he was encouraged to seek help from Benjamin.

Yannick mentioned that Anna has a PR in ufo on refactoring the C++ Locations class that will be merged on Thursday.  This will require changes in the models.  See [the JEDI-Models GitHub Team page](https://github.com/orgs/JCSDA-internal/teams/jedi-models/discussions/39) for further information on the changes that are needed.

Marek has a PR that adds a configuration to GetValues that can be read in through a yaml file.  He's also working with Ollie on implementing the cubed sphere grid into atlas, building on Dan's work for FV3.   Inspired by Atlas, he's also working to implement auto-formatting in other repos with the help of clang-tidy.  There was some discussion about how to define and share parameters across models and JEDI repos in a consistent way.  Yannick agreed this would be desirable but we don't have a good solution yet.  He suggested it might be done in the the jckit repository.

C++ Locations PR will be merged today. This impacts models and most of the model work is completed. See above (JEDI3 summary) for more details.

Clementine’s PR adding Block Lanczos minimizer was merged in oops

Dan worked on FV3-JEDI background error covariances. He prepared a C384 ensemble for static B training, and Benjamin will work on setting it up for vertical regression.

Benjamin has a PR changing hybrid covariance to hold a list of covariances (instead of one static and one ensemble)

JJ raised questions about unstructured interpolation in oops

Marek is adding a Config parameter to GetValues, it would allow to configure GetValues through yaml, and use different types in GetValues in one model (via factory)

Anna worked on Coupled H(x).