Yannick was occupied in another meeting today so Steve led the meeting. He began by announcing that this will be a general round-table discussion. And he reminded all JEDIS to:
We started the roundtable in Boulder with a report from Travis on the code sprint that is occurring this week. Tomorrow is the last day of the 1-week sprint that began on Monday. Prior to the code sprint, there was only a preliminary LETKF implementation in oops. So, the main objectives of the sprint were:
- to add horizontal localization
- To get the LETKF working for the toy models (qg, l95), the shallow-water model, and the MAOOAM coupled QG model.
All participants agreed that great progress has been made toward these goals. They may be close to completion by the end of the day tomorrow.
Then we proceeded with a round-table report from others in Boulder.
Mark M announced that there are new instructions posted in the jedi documentation on how to run jedi on S4. Preliminary versions of the environment modules were installed a few weeks ago but about 15% of the ufo-bundle tests were failing. This was traced to two problems, the first had to do with the use of the slurm process manager srun to run mpi jobs instead of mpirun. Solving this required a patch to eckit that has been implemented on S4 and that has been communicated to ecmwf for inclusion in future versions of eckit. The second problem had to do with the default file locking behavior of hdf5 (which is not allowed on S4). This was solved by setting the HDF5_USE_FILE_LOCKING environment variable to FALSE and re-building the code and supporting libraries. Now all ufo-bundle tests pass with the jedi/intel17-impi module. Contact Mark M if you have any questions or remaining problems.
Steve H has been working with ODC. He now has a fully functional odc-bundle with all the package ctests passing and he has a feature branch in jedi-stack that adds a build script for odc. He also has a feature branch of ioda that enables odc via a CMake flag. By default this is turned off. Mark asked if this branch includes any tests of the ODC implementation in ioda. Steve answered that prior to this change, there has been four ioda tests that addressed the odb-api functionality. In this feature branch, these have been updated to test ODC.
Steve also stressed that we have not yet made a final decision on what ioda file formats to support going forward. We continue to welcome your thoughts on this and we will plan more discussions on this as we gain familiarity with ODC.
Xin is working on variational bias correction for satellite data. He has already implemented much of the code and this week he intends to work on computing the predictors online.
Anna has been working on general cleanup and adding tests after the ufo code sprint several weeks ago. And, she has also been working on connecting NOAA GSI NKF with ufo and ioda. As part of this effort, she has implemented new fortran interfaces into ioda to access the ioda ObsSpace.
Chris H has been occupied with the sprint this week. Before that he successfully got H(x) working for the shallow water model and merged these changes in. So, he is getting closer to having 3DVar working. He has also been working on implementing other interfaces, including the GEM ensemble B and a Geometry iterator.
Chris H also reported on a meeting he had with members of Forrest Hobbs' team on implementing the jedi software stack on Hera. To make further progress, he believes we have to engage some individuals who are higher up in the NOAA management structure, including Forrest Hobbs himself. Chris also reported the good news that Singularity has been installed on Hera. The down side of that is that only selected groups have access to it, including management and his own group. Still, there is hope that the JEDI team might be granted access at some point in the future.
Maryam has been investigating test failures that occur in the Docker containers that do not occur in comparable environments outside the containers, such as the AWS jedi test AMI.
Mark O now has completed the application procedure for access to Discover and has successfully logged in. This will enable him to make further progress on workflow development. He has also been working with Mark M on defining a benchmark that will be used to quantify the overhead of the Charliecloud and Singularity containers on HPC systems. In order to test container overhead, this benchmark will need to be computationally intensive enough to run across nodes (requiring on the order of 100 MPI tasks) and to provide a meaningful performance measure (on the order of several hundred core hours). He has prepared an H(x) application that we will generalize to 3DVar and test first on Cheyenne.
Mark O also mentioned that the version of ecbuild we're using now is behind ECMWF's development version, both because of outstanding issues/bugs in their development version and because updating to their development version would require significant changes to all the JEDI repos and an up-to-date version of CMake.
Clementine has been working with the MPI implementation of H(x) in the shallow water model using fv3-jedi as a model.
Steve V has been working with Steve H on the ODC testing and performance assessment.
Guillaume has been focusing on running JEDI/SOCA on Hera. He said it was working before but was running into problems today. Later in the meeting Steve H asked him if there is anything we need to look at. Guillaume responded that he does not think it has anything to do with the code sprint. He is only seeing the problem for high-resolution calculations.
Dan has continued to work on the Poisson solver he discussed last week. He now has it working, giving reasonably accurate results for transforming from u, v, to streamfunction/potential and back. However, he wants to improve the accuracy by improving the interpolation.
Dan is also working with Virginie to refactor the GEOS AOD obs operator so it can live outside of GEOS. They intend to implement it as a jedi repo following the example of ropp. They have a current PR in ufo.
Hamideh is adding an emissivity model to CRTM. She is seeing good performance overall but there are some regional changes she wants to investigate more closely.
Ryan reported on several things he has been up to. First, he introduced a new scratch repo in jedi/jcsda for the purpose of experimenting with new concepts before implementation into JEDI. Second, in response to the discussion in some previous meetings, he has gone through and flagged a number of places in ufo, ioda, and saber that have been giving warnings during compilation. Some of these are in need of attention and should be fixed. He has created ZenHub issues in the appropriate repos and will work with other JEDIs to address them. Lastly, Ryan is developing a means for specifying QC filtering formulas using simple mathematical expressions written in the YAML file.
Emily has continued to work on QC based on diagnostics from the H(x) operators. She intends to implement a class similar to GeoVaLs that will encapsulate the H(x) diagnostics that can then be used for QC. She has generated new obs and geovals files and has issued a PR in ufo with these enhancements. Cory has been helping out with this.
John M and Alex (NRL) have been meeting this week with Yannick and Tom and have identified some action items to work on.
UKMO: Marek has been adding functionality to geovals. He also asked whether or not the 2D data structure for ROPP 2D is likely to change - in particular, is the ordering of fields fast enough? Anna responded that this will likely be refactored some time in the future for improved efficiency but was not sure on the timeline. So, she advised against any hard-coding of the field ordering. They agreed to discuss this further with Yannick offline. Marek has also been refactoring some of the LFRIc DA code to work on the C++ level rather than the fortran level.
Nancy announced that NRL has hired two new contractors and she will introduce them in an upcoming meeting. One will be working with her and Sarah on interfacing Neptune with JEDI and the other will be working on interfacing the obs layer with jedi.
Then the meeting adjourned with one more request to let us know what focused topic(s) you would like to see discussed at one of our weekly meetings.