Yannick opened the meeting with the announcement that there were no special topics to present. He reported that we are finishing up code changes for next week's Academy which include multiple variable assimilation and a QC filter examples.
Due to next week's academy and Thanksgiving the following week, the next time we will hold this meeting will be on November 29th.
Then we went around and gave updates.
Marek reported that he is getting most of the tests working with 4Dvar (LFRic). The 4D EnVar run is not minimizing properly. Yannick mentioned that there is a fix, related to an issue of how the edges between sub-windows are being handled, that is currently being worked on. Marek may need the fix for this issue before continuing. Marek added that he is getting much appreciated help from Dan. It was noted that BJ may also be able to assist since he has made a lot of progress with MPASS 4Dvar.
Steve S mentioned that he is making progress on his effort to update LFRic.
Guillaume said that he is cleaning up the SOCA model interface, and that the 4Dvar work is under way. He mentioned that there are some tests that are missing such as one for 4D obs. He will work with Dan to fill in the missing tests.
Anna reported that she and Jeff Whitaker are looking at a way to distribute obs that are spatially local to model grid points. This would be an interface that allows JEDI to identify observations that are located within a specified distance of each grid point.
Chris H mentioned that he has an updated flow to build JEDI on Theia. In this new flow he is using the version of ecbuild that we have adopted for JEDI (2.9.0) and that he had to back off on the version of the Intel compiler (from 18 to 17?) to get the flow working. Chris H also reported that he had issues with the ecbuild_find_mpi macro in OOPS. He had to comment this instruction out in order to get the Theia build to work. Others chimed in that they had also experienced this problem. It is not clear what the conditions are that lead to the problem. We decided that we need to formally debug and fix this issue, but that effort will have to wait until after next week's Academy.
A discussion also ensued concerning the need to organize and post the various build flows that have been developed for the different HPC systems. Rahul will get his build instructions posted on the Wiki. Mark volunteered to organize this effort and will start this after Thanksgiving week.
Dan reported that for both FV3 and MPAS, 4Dvar is converging. He is not sure why LFRic is not converging (as reported by Marek above). Dan mentioned that he put in a bug fix (to enable 4D EnVar) that has a "workaround" nature. He is interested in suggestions on how to improve this workaround. The workaround is related to getting GeoVaLs allocated properly. Contact Dan for more details if you are interested in helping with this. Dan has also made more progress with the FSOI work. He has added some new C++ classes as a result.
Rahul has successfully built everything that is needed for next week's Academy on Theia. Since we have very recently found a location (Hotel Cambria in College Park) that is outside the NOAA firewall, we should be good to go with the AWS setup. We will use Rahul's work as a backup. Rahul made progress with the pseudo model. He has the read of model state files working with QG, and is ready to move to the next steps. He requested help from Yannick, and they will attempt to find some time for this during next week's Academy. Rahul also reported that he is making progress on getting the MOM6 model into a JEDI repository (JCSDA/MOM6?). The current develop branch is in good shape. Not all of the test pass at this point, but the code is quite usable if anyone wants to give it a try.
Mark reported that, in addition to the ecbuild_find_mpi issue, he is experiencing build failures of ANTLR on Amazon that may arise if you build the JEDI environment using the cmake scripts in the JCSDA/docker repository. ANTLR is not needed for JEDI so it can be disabled. This has now been done in the JCSDA/docker repository. If you have any questions about this please contact Mark or Xin.
Anna was having build issues with ufo-bundle but this appears to have been solved. The all reduce MPI aspect of the dot product routine is related to the build failures. Mark mentioned that it sometimes helps if you comment out the fckit bundle in the top-level CMakeLists.txt file (ie, use the installed version of fckit). In any case, fckit needs to be the JSCDA fork version. Anna mentioned that she is working in the Docker image, which may not have been updated with the JCSDA fork. Mark will check into this.
Xin has finished the TL/AD code for multiple variables using conventional obs (Aircraft and Radiosonde). He is anticipating being able to merge these changes in soon.
BJ reported progress with MPAS. 3Dvar is working, and there remain a couple issues with 4D EnVar. The Cheyenne issues he was experiencing related to fckit have been resolved.
Ming has made progress with the WRF interface. Forecast (model stepping) is working, and the next step is getting the model field values (GetValues) for the hofx operator.
Francois mentioned that they are waiting on the IODA update before fixing some issues in the GNSSRO obs operators. Once fixed, they will be ready to do a pull request.
Steve V reported that he has successfully converted the Radiosonde netcdf file into an equivalent ODB2 file (April 15th data). This new ODB2 file works with MPAS.
Steve H is finishing up modifications to IODA to enable support for multiple variables assimilation. He is anticipating a pull request later today with this code.