Yannick opened by announcing that we will have no meeting next week, in lieu of the JEDI Academy that is taking place in Boulder.
Travis opened the discussion with some questions about how Travis-CI works. He gave the example of a pull request in ufo that is dependent on other corresponding branches in other repositories, such as oops and ioda. He was surprised that a PR passed all tests and he was speculating that Travis-CI might be smart enough to check out corresponding feature branches in other GitHub repos if they exist. Maryam responded that the default behavior for Travis-CI for ufo-bundle repos such as oops, ufo, and ioda, is to use the source of the PR (e.g. a feature branch) for the repo in question and the develop branches for all other repos. To change this behavior, you would have to edit the Travis-CI yaml file. Maryam also added that the default behavior is to only run the bundle tests for the repo in question. For example, if the PR is done for ufo, the entire ufo-bundle will be build but only the ufo tests (and their dependencies) will be run.
Travis also said that he has been looking at AirFlow workflows to facilitate cycling runs. He's working with Airflow for now but he will work with Mark O to adopt whatever workflow strategy we end up adopting for JEDI as a whole.
Maryam is continuing to enhance the automated Travis-CI testing to add the capability to run tests on AWS nodes. She acquired and set up a license for Travis-CI enterprise and is now working to configure Travis-CI to work with AWS,
Clementine continues to work on enhancing the ensemble DA functionality in oops, using the qg model as a testbed. The first outer loop is working but subsequent outer loops are diverging due to changes in the H operator. She is working on correcting this.
Steve H is using the jedi-stack build scripts to build JEDI dependencies on his Mac using clang and gfortran. The libraries are working and ufo-bundle ctests are passing but he's seeing issues with the C/C++ python interfaces. He suspects that it may be because his python installation is 32-bit and he asked if anyone knew how to determine that. Mark O suggested running readelf on the executatble to see if it is 32 bit. (Update after the meeting - readelf is a linux command but on Mac OS Steve found that the file command was able to tell him that his python installation is indeed 64 bit - so the problem must lie elsewhere).
Mark M is getting the AWS nodes and GitHub accounts ready for next week's academy.
Ming continues to work on the JEDI WRF interface. He is now working on GetValues() for the TL model.
Mark O announced that he and Rahul have created a JCSDA fork of ECMWF's ecbuild repo. They are currently working on adding improved Find cmake modules for NetCDF, LAPACK, and MKL. The develop and master versions of the JCSDA fork trace the develop and master versions at ECMWF and, as such, they cannot currently be used to build JEDI. The latest versions of ecbuild at ECMWF include changes that change interfaces (for example, requiring version numbers to be specified for each project) and exploit new cmake functionality (for example, bundles within bundles) so getting these branches to work with JEDI will require some changes to JEDI. For this reason, they have created a release/stable branch of the JCSDA fork that is based on ecbuild release 2.9.4 and, as such, can be used to build JEDI (ecmwf releases are now beyond 3.0).
Chris H mentioned that ecbuild also has a problem locating boost if the boost version is later than 1.70. This is due to changes in the boost file structure. He has a fix for that and Mark O encouraged him to add this to the JCSDA/ecbuild develop and release/stable branches.
EMC
Dan has PR in oops and fv3-jedi that builds on recent changes made by Benjamin to use compare scripts to test multi-processor MPI jobs. The intention is to do "softer' comparisons between log files that can be used, for example, to specify tolerances. But, it's not quite ready to be merged - Yannick tried it and found it registering test failures in instances where the numbers apparently matched.
Dan has also been testing the new SABER repo, which is the new home for BUMP. He's finding that it took over 10 min to run the tests on Discover and asked others to try on laptops to see if it also takes that long there. If so, we may consider speeding up the test suite.
Ben reported that Jim is continuing his work to optimize CRTM. They have found it beneficial to introduce architecture-specific compiler flags to enhance performance particularly with the vectorization of inner loops. However, they found that aggressive optimization flags can produce results that are not bitwise reproducible. However, differences are still small; results agree to within 12-13 significant digits. Ben also mentioned that they are looking into the possibility of adding an option to run CRTM in single precision to improve computational and memory efficiency. He will be in Boulder next week for the JEDI academy and plans to work closely with Jim and the JEDI team on further performance tuning.
Cory is working on the ioda converters for GSI. He's working with Anna to further automate the process of generating obs and geovals files
Hamideh is giving a talk next week at GMAO and she is preparing for it.
Emily is using Cory's packages to get obs and geovals files and is now testing a cloud fraction operator in UFO.
Chris H has a pull request in for forecast capability in the shallow-water (SW) model. It's close but Mark (M) noticed a few test failures with intel 17 compilers when -O2 or -O3 is used. This was seen on both AWS and Cheyenne but Chris was not able to reproduce it on Theia - there all tests pass with all optimization levels. Chris plans to try Cheyenne next.
Chris has made progress with interpolation for GetValues() with the SW model, but this is proving to be challenging due to simplified nature of the SW model. He may need to solicit help with this task from the others working on model interfaces.
NRL
John and Sarah reported that they have made good progress on refactoring Neptune to make it more easily integrated with JEDI and on developing the JEDI interface.
UKMO
Marek has make good progress replacing the Boost scoped pointers with std:unique_ptrs. But, he has run into a few issues. In some constructors such as DateTime, the scoped pointers had been initialized to integer values, such as 0. This did not work with the unique_ptrs so Marek initialized them to NULL instead. This seems to work and the tests pass but he wanted someone else to check and make sure that this did not introduce any unexpected consequences. Yannick said he would take a look when Marek issues a pull request. Marek also mentioned that he was seeing warnings from the object counter in a number of tests - namely the number of objects destroyed was less than the number created and Marek was concerned about memory leaks. Yannick responded that this is an artifact of the tests we have set up. The test suite uses static variables but there is nothing in the C++ standard to ensure that the order in which static objects are created and destroyed is always the same. So, these warning messages arise because sometime the object counter print statement is executed before the static objects are destructed. This is just the way the tests are set up and should not cause problems in real-world applications. We could replace the static variables with dynamic variables but this would adversely effect the computational efficiency - tests would be slower. Yannick suspects that this isn't a problem and we should not spend too much time debugging it but Marek might check with valgrind just to make sure there are no memory leaks.
Marek also confirmed that the new SABER repo works with LFRic.
Then Yannick opened the floor for further questions and comments.
Mark M made plans to consult with Jim R on the best compiler options to use for intel in debug mode - our current options make the ctest suites for the various bundles take a long time to run.
Jim then suggested that we consider adding PGI compilers to the list of compilers we test with JEDI. He gave two reasons why this might be a good idea:
- PGI compilers are free - for the community version
- PGI compilers have good support for GPUS, particularly with regard to the integration of openacc directives
However, Yannick and others mentioned that the PGI compilers tend to be behind when it comes to implementing new Fortran and C++ language features (e.g. Fortran 08 and C++11). Perhaps they have improved in recent years but Chris H is skeptical that they would be able to handle all the language features currently exploited by JEDI.