2020-01-14

Maryam, in collaboration with Mark O, has a pull request under review in oops that implements a compare test in C++.  This can be used to unify the run and compare stages of tests and replaces the current compare.py script.  As such, it's used for comparing numerical values in text output so it does not apply to netcdf compare tests that use nccmp.  The current code is backward compatible so merging this PR should not break current tests.  However, after this PR is eventually merged, model teams and other developers should transition compare tests to the new method.  When all tests have been transitioned, we will deprecate compare.py.  Watch the JEDI GitHub Team page for further information.

We have also been discussing how to let developers see the results of their tests in CodeBuild.  Currently this requires AWS access but Maryam suggested that we can push the results to the public https://cdash.jcsda.org site so everyone can see it.  This is preferable to our previous idea of emailing the results to the PR author.  We will look into implementing this soon.

An even higher priority for the JEDI team is to improve the way we manage test data.  Our current method of selected team members (often Ryan and Steve for ufo and ioda) uploading branch data to S3 is inefficient and unsustainable.  One idea we have discussed is to potentially return to git-lfs for hosting the ufo and ioda branch data but to locate the data in separate repositories (e.g. ufo-data, ioda-data) and clone/fetch these as needed.  The details are still being worked out but expect changes coming soon.

Mark O is putting together some python tools to be used for profiling, leveraging the linux/gnu perf utility.  These tools will enable the user to define a matrix of simulations using different yaml files (e.g. differing resolution) and run configurations (e.g. number of MPI tasks) and will report profiling results across this matrix.  These are not ready yet - they are still under development - but, when ready, will be useful for profiling

Mark M is working on containers.  We have built new gnu and clang containers for CI and are now testing them.  The main differences are a move to the ubuntu 20.04 operating system, replacing bufrlib with bufr, and adding clang-tidy.  The clang container appears to be working and is being implemented in CI.  However, several lorenz95 tests are failing in the DualMinimizer::DoMinimize() with the default RelWithDebInfo build (e.g. test_l95_4dvar.rpcg).  These tests pass in debug mode and for other compilers and platforms so it appears to be an optimization issue.  Mark is continuing to debug.  And, he is also in the process of transitioning from intel Parallel Studio to Intel OneAPI in our CI, development, and application containers.  Stay tuned for more information on this too.

The JEDI 1 team also had a discussion with Steve H about how to handle potential conflicts between python packages and the jedi-stack.  There was some interest form EMC to use the python hdf5 package h5py in the development of new ioda-converters.  However, this comes with its own hdf5 implementation that can conflict with the hdf5 and netcdf modules in the jedi stack.  Mark O suggested two possible remedies, namely to instally h5py by linking to the jedi-stack version of hdf5 or (preferable) to do a --binary-only installation of h5py that links to its intrinsic hdf5 through an rpath and thus should not conflict with the stack.  He also recommended the use of xarray over h5py.  However, it was acknowledged that HPC users may not have the freedom to install their own python packages so a workable solution to this type of problem is still being investigated and may be platform-dependent.

We had a JEDI2 meeting yesterday [https://docs.google.com/document/d/1NObKF9eKbzyfbIm-ssBdAnPhYcAlVZ3TA586f2UentI/edit?usp=sharing], where we reviewed all of our AOP tasks to determine which issues still need to be addressed in the final AOP quarter. Since it is easy to become lost in the details, I have asked the group for further review to ensure that the issues will address the broader agency goals.

Two announcements:
        • A meeting discussing proposed expansions to QC flags will be held next week on Jan 19, 8-9 MST, 10-11 EST, 3-4 PM UTC. Invites have been sent out to everyone on my mailing list, and if you are interested in attending and have not already received a calendar invite, please email me.
        • Since last Thursday, we opened twenty-nine pull requests into UFO, which may be a record. I have merged fourteen pull requests, and I think eight more are ready to go into develop today, but I am waiting on the automated testing system. Thanks to everyone who helped review and merge, and please keep helping because the incoming rate is incredible. To speed up the process, I ask anyone opening new PRs to ensure that you have linked the PR to an issue that has epics and an estimated complexity.

From UCAR:

        • Anna reported that the C++ locations class refactor has been merged. Anna has also taken over Xin's bias correction work, and we have discussed plans to finish this. Besides her own work, Anna also reports that Travis Sluka in JEDI4 may add access to Atlas locations, and noted that Sergey Frolov has accomplished substantial work in the Halo obs distribution.

        • Steve has been working on the ObsSpace refactoring, which is a next step in the ioda-engines integration. He has also contributed to the obs team code sprint, where he has been improving ioda's obs grouping functions, and also to ioda-converters, where he is working on the generic bufr converter.

        • The obs team is finishing their second UFO code sprint. Several new generic ObsFunctions have been written, and these should be merged into develop soon. Additionally, many PRs demonstrating correct H(x),  bias correction and qc for many radiance and conventional instruments have been submitted, and more are on the way.

From the UKMO:

        • Wojciech opened a PR introducing a filter that assigns values to specified variables at locations governed by a where clause. He has also opened several PRs extending his parameter validation classes to more filters, and is working on vertically interpolating background error estimates as obs diagnostics.

        • August has continued work on the stuck check filter. Chris has been working on a profile pressure calculation PR, now merged, and is working on profile vertical averaging routines.

From EMC:

        • Nick has opened a PR to enhance the converter for ADPSFC data. Praveen presented on BUFR utilities at EMC's ObsProc / SatIngest Re-Engineering meeting.

- Benjamin: worked on instrumentation of BUMP code in saber (PR opened). In the next couple of weeks plans to work on a vertical balance in BUMP. Several technical issues to solve: dealing with large ensembles that can’t fit in memory at once (e.g. fv3-jedi training set); allowing to have different MPI distribution for applications that estimate and that use BUMP parameters.

- BJ: worked on testing multivariate B (in a cycling experiment). Tuned multivariate B in mpas-jedi is now producing results that are better or similar to univariate B. Will work on testing hybrid weight diagnostics.

- Clementine: working on adding diagnostics to the block Lanczos minimizer.

- JJ: worked on implementing and testing barycentric weights for the unstructured interpolation. JJ reported that the current weights in the unstructured interpolation may be working well, even if they are not barycentric.

- Marek: working on refactoring GetValues to allow it to take Configuration on input (would allow to configure GetValues through yaml, and to have various different GetValues that can be activated through yaml). Marek plans to work on time-interpolation GettValues next.

- Olly: working on ways to interpolate fields from one grid to the other (found a way to do that within atlas), copying the grid from one partitioning to a different partitioning.

- Rahul: working on interpolation for fv3-jedi. Bringing offline UFS program for regridding into the fv3-jedi, to be used for interpolation between outer and inner loops (without going to the unstructured grid). This code would also be applicable to mom6.

- Sergey: working on Halo Obs distribution PR. After it’s merged, plans to work on LETKF to use Halo distribution. Plans for the next month: using OpenMP in local volume solver; adding obs-space localization for land data assimilation (taking into account difference in the orography).

- Travis: tuning LETKF experiments with SOCA; 20 members covariances isn't enough. Obs-space localization with Rossby radius taken into account. Also has worked on using atlas fields for ufo::Locations.

- Yannick: plans to work on improving the code for exchanging information between background, observation error covariances and QC filters.