Yannick opened by announcing that there were no general announcements. We proceeded to hearing updates from everyone.
Boulder
Steve H is working on the ODC reader in IODA. Several PRs have been merged in that comprise steps to the ODC reader functionality.
Steve H also announced that he wants to get all netcdf obs files that are checked into the IODA repository to meet the standards for a IODA file. The standards are outlined on this JEDI Wiki page (see the "File Variable Conventions" section). There are currently 12 files checked into the IODA repository that do not meet the standards. The plan is to submit a separate ZenHub issue for each file. Steve also created a milestone in ZenHub, called "Netcdf obs files meet ioda standards", for tracking this work. For now, this milestone is due on November 15, 2019
Mark M has an Intel 19 container working. He has run into an issue with the container size getting too large, primarily due to faulty behavior of an Intel installation process that is supposed to allow one to trim out optional compiler packages. No matter how it is configured, the installation process will install all compiler packages, which include packages we are not using in JEDI (and consume a significant amount of space). (Update: it turns out that this is a known bug in intel 19.0.5 that is expected to be addressed in a future release).
Mark M reported that the Debug build mode using Intel 19 is slow, but it did uncover a number of floating point exceptions. These appear to be related to BUMP (bump and QG bump test failures) and Mark with work with Benjamin to get these resolved.
Mark O has a real time H(x) work flow operating successfully, and is working on a website to publish the results. His example runs H(x) on GNSSRO and COSMIC II data, and he is waiting for permission from the COSMIC team to publish COSMIC II data and results. He has an observation database constructed on AWS S3, and will be adding GOES16 and GOES17 data next. We will need further development on the ioda converters outside of the GSI ncdiag converters in order to get more obs types into this work flow. Mark will create a new JCSDA github repository soon called jedi-rapids that contains the work flow code. "JEDI Rapids" is the name for the automated JEDI work flow.
A question about the converter speed came up during Mark O's update, with a note that Python might be too slow and compiled executables will therefore be necessary for operations. In response it was noted that we are currently focused on adding functionality (more converters) into the system. Once that is accomplished, performance tuning can be addressed. The comment about moving to compiled executables to satisfy operational requirements is a very good point and will absolutely be considered as we bring more converters on line.
Xin has PRs under review for variational bias correction, and expects these to be merged soon. Once merged, there will be one more stage to getting the bias coefficients calculated on the fly (instead of loaded from a file), which will complete the variational bias correction functionality.
Steve V has been readying MPAS for the real time H(x) processing. He is currently using a collection of scripts with CYLC for his work flow and will consult with Mark O about getting his flow integrated into JEDI Rapids.
Clementine is getting the first results from a block lanczos flow using FV3 (two PRs under review in OOPS and FV3-JEDI). These results are being generated with a low resolution grid, and the next step is to try the flow with a high resolution grid.
Maryam is installing multi-tier test capability in the SABER repo. She also recently installed Travis-CI and CodeBuild testing for the IODA-CONVERTERS and SHALLOW-WATER repos. Maryam discovered that the Intel docker container for CodeBuild is missing some necessary Python packages, and work to resolve this is under way.
Nan is developing IODA converters for the real time H(x) flow.
Ryan reported Intel 19 compiler issues related to MPI. When he builds (ufo-bundle?) using icc, the build completes successfully, but when using mpiicc the build fails. Mark M will help get this resolved.
Ryan announced that he will be out of the office for the first two weeks of November and, with Anna on leave at the same time, asked about who will handle any issues that come up in UFO. For now, send issues to Yannick and he will help get them resolved.
Ryan plans to finish his JCSDA newsletter article by the end of this week.
Ryan confirmed that he has seen issues with the Lustre file system on Hera. The symptoms are excessively long test runtimes, and HDF5 test failures. These issues appear to be caused by components outside the JEDI system.
Chris H has a series of PRs to the shallow-water repo under review which introduce: linear model (TLAD) interface, 3DVar and 4DVar. He believes that the 3DVar is working properly, but the 4DVar is not working properly. He asked for help with verifying/debugging the 4DVar from the group. After these PRs get merged, he will work on fixing LETKF.
Chris H reported that on Hera the environment module for Intel 19.0.5 was mistakenly loading 19.0.4 instead. This has just recently been repaired, and may or may not be related to other issues reported on Hera.
Mark M added that he had to insert pragma statements in strategic locations of the JEDI code that shut off optimization in order to get Intel 19.0.5 to work correctly. The pragma statements were placed around regions that the compiler optimization was mistakenly changing the functionality of the code. This has to do with the model propagation for the Lorenz model so without this change, many l95 tests fail. These cases have been reported to Intel, and hopefully Intel will be able to provide fixes soon.
EMC/GMAO
Dan is making progress with Static-B work, and has also improved performance of the Poisson solver.
Hamideh is debugging GMI H(x) issues related to emissivity.
Jong is making progress with CICE 3DVar, and is also working on reviving the WAVE JEDI interface.
Sergey has made progress with SOCA(?) LETKF.
NRL
Sarah announced that 3DVar increment is working with NEPTUNE. They are currently working on code cleanup, etc.
Sarah reported that Intel 17 with a Debug build results with 57 OOPS test failures. In a Release build, all tests pass. This behavior was seen on a local server, and is not exhibited on other systems.
Yannick asked about the status of a Navy machine of which Mark M was to have access for debug purposes. Sarah responded that Nancy is the person to contact about this.
Met Office
Marek has a PR under review that contains an all C++ implementation of the time interpolation operator.