Yannick and others from the leadership team were not present today due to their retreat. There were no special topics so we collected updates from everyone.
Marek reported that a UFO pull request is under review that provides an interface to Dave Rundel's RTTOV tool. Anyone using RTTOV needs to obtain a license through a simple online process. Marek is working with his management to devise a way for restricting access to the RTTOV repository to those who have the license. One solution would be to place the RTTOV repository on the JCSDA github site and restrict read/write access to only those with the license. Mark pointed out that there are 2 or 3 people who have administrator privileges on JCSDA github and they would need to get the license since they automatically have read/write access to all repositories. Mark will send the list of names of the administrators on the JCSDA github site to Marek so he can check with his management about using the JCSDA github site.
Marek also mentioned that he was able to get 4D EnVar to complete using Yannick's 4D EnVar branch. Verifying the results of this run is work in progress. Marek asked about the status of an OOPS pull request (https://github.com/JCSDA/oops/pull/145) containing BUMP bug fixes. There are two approvals by Dan and Marek, and one review with comments from BJ. Mark will follow up with this pull request and see if it can be merged in soon (hopefully, later today).
Dan added a feature to UFO that will provide the channel selection information (from the YAML configuration) to CRTM. This functionality has been merged into the develop branch. At this point, the channel selection is used to construct the H(x) vector, using only the selected channels, that will be consistent with the obs (y) vector. CRTM is still simulating all channels. Ben is working on the interface to ingest the channel selection and have CRTM only simulate the selected channels. This feature will be introduced in an upcoming pull request.
Dan is also working on a means for posting results (eg., O-A) from a variety of models for comparison purposes. He started with Slack as the place to post results. He may end up using the Wiki, but for now, when there are many posts and updates, he wanted to avoid the Wiki so that everyone will not get hit with many email notifications. Dan solicited help from the group on the best way to post results that will facilitate comparing models.
Dan noted that fckit does not currently support splitting of MPI communicators. We created an issue on the ECMWF site and were told that fixing this will first require some modifications to eckit. When eckit is ready, we offered to make the appropriate modifications on our fork of fckit and then push them to ECMWF. Dan mentioned that FV3 is good to go with the OOPS BUMP pull request that Marek is waiting for.
Hamideh is making progress on the hofx operator for GMI. She saw results with large differences between H(x) and y, and is currently debugging. Hamideh mentioned seeing negative values in model field values (that should not be there). She is working with Dan to resolve this. A tactic they are using is to dump out a set of geovals and obs that Hamideh can use to continue development in ufo-bundle.
Mark and others (Dave H, Jim, Dan, Francois, ...) have made progress with getting full resolution JEDI running on Amazon. He reported that full resolution FV3-GFS (GNU compiler) runs successfully across nodes outside of JEDI. He now has access to the Intel compiler and will try compiling JEDI using the Intel compiler instead of GNU. More generally, there are eight permutations that they have been working on, representing FV3-GFS/JEDI, GNU/Intel compilers, and single-node/multi-node AWS functionality. six of these permutations are now either working or to the stage that they can be testing. The two remaining (JEDI with intel compilers on 1 node and across multiple nodes) will be addressing in the coming days.
Mark reported the status of using split MPI communicators. On one Amazon configuration (GNU compilers + openmpi), the ECMWF develop branch gets a compile time error related to a data mismatch in the MPI send calls. This compiler error does not appear in other environments outside of Amazon. Mark is currently debugging this issue. Mark warned the group that the latest develop branch of the ECMWF fckit repository might have issues. If this happens, one can use the JCSDA fork of fckit as a workaround.
Mark announced that he has updated the default JCSDA Vagrantfile. This version fixes some issues that were reported by Clementine and others, contains both Singularity and CharlieCloud tools (the images have to be downloaded) and is based on Ubuntu. After installing the new Vagrantfile, you should either start in a new directory or clear out your existing vagrant VM. Instructions for deleting a vagrant VM can be found on ReadTheDocs: https://jointcenterforsatellitedataassimilation-jedi-docs.readthedocs-hosted.com/en/latest/developer/jedi_environment/vagrant.html. If you decide to delete your vagrant VM, be sure to save any work that is inside the VM beforehand.
Steve H is working on IODA documentation to go on the JCSDA ReadTheDocs website. This will include an overview, class structure and the interfaces associated with IODA. Steve hopes to be done this week before the IODA Workshop so this can be used as a resource.
Steve H reported that he tested IODA with a large obs data set. Using the GOES-16 channel 1 data from April 15, 2018 00Z, which consisted of 117 million locations, he was able to read those data into IODA, write out them into an output netcdf file, and verify that the data from the output netcdf file matched up with the original GOES-16 data file.
Maryam is working to replace the Boost based tests in JEDI with eckit/fckit based tests. She is currently debugging a segmentation fault that occurs in a new eckit/fckit based tests. Maryam is taking the tactic of using eckit/fckit based tests in Atlas as an example.
Steve V is preparing for the IODA workshop. He will be giving a talk on the file formats (eg., Met Office ODB) that IODA is able to ingest, along with the associated conversion processes.
JJ finished refactoring the split Increment and State for MPAS. This work was merged in last week, and included a fix for a memory leak that enabled much higher capacity model runs to complete. JJ reported a runtime performance issue with GetValues. It is related to horizontal interpolation in BUMP where the process slows way down if you interpolate one level at a time versus all levels simultaneously. JJ created a band-aid type fix that got the one-level-at-a-time process (which is what MPAS requires) to run an order of magnitude faster. He will work with Benjamin to get a generalized fix into BUMP.
Junmei has submitted a pull request into MPAS (feature/graphics) that includes Python script for plotting O-A and O-B. Dan asked if these scripts would be of general use (not just MPAS) and perhaps they could be merged into the jedi-tools repository instead. Discussions after the meeting with Yannick and Guillame brought up the possibility that we might want to devote an entire JEDI repo in the future to common analysis and visualization tools. But, for now we can merge this PR into MPAS. Junmei will add some core team members to the reviewers list, and the core team can help determine the answer to Dan's question.
Sarah (NRL) reported that their work to integrate their interpolator into the H(x) calculation is progressing nicely.
Stylianos had a bad connection (he could he everyone, but no one could hear him). Sorry about that. He reported via email after the meeting that they are working on the interface between the EMC marine obs and IODA.