Yannick initiated the meeting by announcing that there is no special topic and the agenda was to take turns with updates.
Yannick started by announcing that he will be removing all of the develop-* variant branches in OOPS this afternoon unless there are any objections. Dan noted that UKMO might object since they are still using the develop-next branch. BJ added that they still needed develop-next as well. It was decided to keep develop-next (and of course develop) and remove all the other develop-* variants.
Yannick mentioned that he will be gone over the next two weeks (Sep 24 - Oct 5) on vacation. He will be back the week of Oct 8 working remotely for one week, then back in Boulder.
Dan announced that he is working on getting 4DVar working with radiance data and finishing FSOI. For the 4DVar, he will get rid of staggered winds and use cell-centered winds instead, and he should have 4DVar H(x) working soon. Dan asked about how to do obs and geovals binning for the incremental time steps in 4DVar. Currently, UFO and IODA can only return all geovals and obs, thus it will take changes in IODA and UFO to accommodate the binning. After some discussion, Anna and Dan volunteered to look into getting a quick fix into place. They will first look into creating an array that holds the indices of the current time step's locations. This index array would then be shared between UFO and IODA for selecting the correct lists of geovals and obs.
Guillaume said that they have model interpolation and BUMP integration completed for SOCA(?). They used the OOPS QG implementations as guides for their work. They are waiting for the memory leak issue to be fixed. Guillaume reported problems when using more than 12 cores and they are looking into why.
Hui mentioned that they are working on obs operators for GNSSRO. They are nearing the submission of pull requests to get the work merged into the develop branches. They are using pressure for their vertical coordinate during interpolation steps.
Yannick reminded everyone to follow the methodology of merging in many small increments versus one big increment. This helps avoid changes in other parts of the system inadvertently breaking your work.
Marek said that next Monday (Sep 24) they will be hosting a mini JEDI Academy to train new people. The change of variable and Model Factory implementation is completed. 4DVar is work in progress. Then Marek proceeded to ask a list of clarifying questions:
- Can we work outside bundles?
- Yes, you are not required to use bundles.
- But - be sure that you do not break the bundles for the sake of other users
- Why are eckit and fckit commented out in various bundles?
- The default is for running inside Singularity and the proper versions of eckit and fckit are pre-installed in Singularity
- Yannick reminded everyone that the purpose of the bundles (and Singularity) are for easy use by those who do not want to mess with the technical details of building software.
- Can bundles be made to automatically detect whether eckit, fckit should be compiled versus loading pre-installed libraries?
- This could be done, but currently the scripting isn't smart enough to sort out which versions to use if there exist multiple versions to choose from. Someone could work on this as an improvement. Keep in mind that it's only a quick edit to switch back and forth between compiling and using pre-installed libraries.
- What is the CPPIODA repository?
- This held a prototype for the C++ implementation for IODA work that is under way (see below). This prototype has been merged into the IDOA repository, and the CPPIODA repository has been deleted.
- Can we get "calling tree" style tracing information during job execution (the OOPS_TRACE mechanism doesn't cover enough ground)?
- There exist compiler options for both GNU and Intel that will provide detailed tracing. Give these a try first to see if they are adequate for your needs.
- If Marek and/or others wish to add explicit markers to trace entry, exit on all methods then the JEDi team has no objections
Marek mentioned an LFRic issue where it is difficult to import orography and other land information into the State object. Other models can do this, but LFRic isn't far enough developed yet to accommodate this action. At this point they are doing aqua-planet simulations. Marek solicited ideas for workarounds until LFRic gets the feature it needs to enable the import to the State object.
Yannick mentioned that we now have "small" obs files that are for the purpose of providing fast design loops while working on a problem such as this. It was also suggested that modelers focus on resolution configurations that can run on a desktop for now - we'll move to larger HPC configurations in the future.
There was no update from NOAA
Rahul has made progress with the generic model. The purpose of the generic model is to add in the capability to read in forecast information (into a Forecast object) from disk so that time consuming model runs can be skipped thus speeding up design loops. He is doing this by overloading the Forecast object, and he asked a question about at what level in the code hierarchy to do this overloading. There was some discussion about this which resulted in some direction for Rahul. Rahul also mentioned that the generic model could be used for reading in trajectory data, including non-linear trajectories.
Chris Snyder introduced Steven Vahl who is a new software engineer in NCAR. Steven will be working on developing an ODB2 reader/writer to go into the new C++ IODA implementation. He also mentioned that he has some planning information that the team working on the C++ IODA should find useful.
BJ noted that they have updated to the new obs data. They tested with the develop branch, but it uses too much memory on a Mac. It run okay on Cheyenne. There is a suspected memory leak in BUMP which could be the cause of this memory issue. With MPAS they have 3DVar, 3DVar with BUMP, and 3DEnVar working.
JJ mentioned that he is working on a repository to hold the MPAS model code (which will replace the tarball that is currently used in the mpas-bundle build process). This will better synch the main MPAS repo with the version used in JEDI.
Steve H gave a brief overview of the C++ IODA project. For the long term we want a database for storing obs data, but for the short term we want to improve the existing memory store. We decided to go with a total C++ implementation because the underlying Fortran layer is not necessary, this makes the implementation easier, and we can take advantage of canned C++ data structures. Xin found an interesting data structure called MultiIndex in the boost library that can be used for the memory store of obs data. Three of us are working on this upgrade: Steve H on the netcdf reader/writer, Xin on the memory store, and Steven Vahl on the ODB2 reader/writer. We will do a special talk about this in an upcoming meeting (likely mid-October) to present more details about this work.
Please note that the C++ IODA is being done on the develop-newmem branch in the IODA repository. Everyone needs to remain on the develop branch until the develop-newmem work is ready. We will continue to support and make necessary changes on the develop branch up to the point when we are ready to merge in the develop-newmem branch.