We started with an update from the JEDI teams
JEDI 1 (Mark)
We started our meeting yesterday by clarifying the scope of this biweekly meeting, in lieu of organizational changes with the JEDI teams that do not necessarily align with the current AOP. The JEDI1 team will continue to cover tasks JEDI 1.10-1.15 (testing, HPC modules, containers, jedi-stack, optimization) and some aspects of JEDI 1.6-1.9 (NRT system maintenance, OBS data handling). However, the development of cycling DA workflows (JEDI 1.1-1.5) through the new ewok system and its eventual integration into the NRT applications (JEDI 1.6-1.9) will be addressed by a separate series of biweekly meetings led by Yannick. Furthermore, with the addition of Steve to the team, we will also begin to address some aspects of IODA infrastructure, though the principle responsibility for JEDI 2 tasks will remain with the JEDI 2 team. So, if you haven't already, please let Mark or Yannick know which series of meetings you wish to participate in or if you would like to participate in both.
We then had an update on release-related items. A webhook for synchronization of JCSDA-internal and JCSDA has now been set up and has been deployed for jedi-stack. After we verify that it is working, it will be extended to all repos on JCSDA. Every time a pull request is merged into the develop branch of a repo on JCSDA-internal the webhook will automatically push these changes to the develop branch of the corresponding repo on JCSDA.
Maryam then gave an update on CI testing in the new, post-release set up. Since Travis-CI and CodeCov are free for public repos, this is what will be implemented for PRs to our public repos. Internal repos such as oops, saber, ioda, ufo, fv3-jedi, and ioda-converters will use AWS Codebuild. Maryam has also now added Codebuild CI to MPAS-JEDI. Some internal repos such as SOCA will still use Travis-CI but those repos that have CodeBuild do not need Travis-CI. CodeCov can be expensive for private repos, on the order of $10-$12 per user per month. So, we will implement this but only a limited number of users (JEDI core team) will have access to the detailed results.
We brought Mark O up to speed on the new data management scheme after the release, with most of the data now being served from UCAR DASH and AWS S3 as opposed to git-lfs. We briefly discussed the need for a more flexible and robust data management plan. Ryan suggested Data Version Control (https://dvc.org/doc) as something to consider.
We then had an extended discussion about the jedi-stack, including which sofware packages and versions are required, which are optional, which should be deprecated, and who is responsible for maintaining the stacks on each supported HPC system (Hera, Orion, Discover, S4, Cheyenne, AWS, and Gaffney). This is summarized in a new document on Google Docs. If you'd like access to this document, please ask Mark. We will also update the information on GitHub and ZenHub for each platform. We also briefly discussed the need for a procedure to assess the benefits and drawbacks whenever there is a request to add a new dependency to the stack. Details of this procedure will be discussed at future meetings.
We then briefly reviewed the October and November milestones. And, with regard to JEDI 5, academy preparation is ongoing this week, with focus on the development and testing of practicals and on the Zoom infrastructure to hold the academy virtually.
After this update, Chris H asked about egress costs for the test data. Mark responded that it depends on the situation. Users who download the default public branch of fv3-bundle and proceed to build it will get the release test data from UCAR DASH, at no cost to JCSDA. But, internal users and public users who build from the develop branches will get their test data from AWS S3 at a cost of about $0.1 per GB. Since a fresh clone of fv3-bundle is 2-3 GB, this corresponds to about 20-30 cents per clone. This egress cost is about the same as git LFS. However, the CI testing involves data transfer only within a single AWS region (us-east-1), from S3 to CodeBuild EC2 nodes. This does not incur a cost.
Jake requested access to submit pull requests to the RTTOV repository. Since this is a proprietary repo he does not have write access to do a push of a feature branch. NCAR may already have the appropriate license in place for access to the RTTOV repo. Mike Cooke volunteered to work with David Rundle to get this resolved.
JEDI 2 (Ryan)
Many members of the team were preoccupied with the release and with preparations for the quarterly review. This summary is based off of the JEDI2 meeting notes for Oct. 21 and Nov. 4. Those notes are viewable at these links:
October 21 - https://docs.google.com/document/d/19ezONfjM8WCIdTRRHikrWrodbia33rHiZdZzNjiEJL4
November 4 - https://docs.google.com/document/d/1hGD5Tcpe-roPQjUADWZ6fluQBt5cE7M9Ao_-FvQhN_o
A message for ioda-engines users:
We want to merge ioda-engines into the ioda repository sooner rather than later, and generally before the end of November. We want to avoid blocking progress on other tasks, and this will allow us to apply Xin's final pull requests for VarBC. Very little should change from the end-user standpoint. If you're not using ioda-engines directly, you will notice nothing. Anyone using ioda-engines, such as in ioda-converters can simply just call find_package(ioda), and the ioda-engines targets will be available to ecbuild. After the migration is complete, we will then issue subsequent PRs to link ioda with ioda-engines.
From UCAR:
• Steve and Ryan are still working on linking the ioda ObsSpace with ioda-engines. Ryan will start a pilot project to restructure the IASI data files to have multidimensional TB variables. We want to benchmark the impact of a change in the file structure on the operations of the JEDI code.
• Anna and Ryan will be working over the coming weeks to rewrite the UFO Locations class in C++.
• Anna is also working on the Halo observation distribution across MPI tasks. This builds on work started by Sergey.
Many contributions have come from the UK:
• Mike has submitted PRs to perform 1DVAR with RTTOV.
• Neill continued porting the 1DVAR GNSSRO code to JEDI. Also created an obs. function to calculate RO impact height from impact parameter and the earth’s local radius of curvature. This will be discussed at the OBS team meeting.
• Wojciech worked on updating the script that generates the boilerplate code used when developing new observation operators and filters. He added a test to the build system to ensure that the script does not go out of date. He also is refactoring the thinning filters and track checks to fully support MPI-distributed data.
• August added comparison tests (between OPS and UFO) to JEDI for TrackCheckShip and corrected found bugs in bugfix/ship-track-check-flag-mismatch. TrackCheckShip is essentially complete, and an adapted version of OPS HistoryCheck is in the works.
After this update, Hui requested that the OBS team be kept in the loop, particularly for the GNSSRO work that Neill is doing.
Mark M asked if the ufo Locations refactoring would represent the locations as an Atlas PointCloud. Anna responded not initially, but this will be added in the future.
JEDI 3 (Dan)
FV3-JEDI
Working on regional FV3-SAR applications. Adding a Static B model based on fixed length scales for meteorology and chemistry. Will add 3DVar ctest for each mode of running. Chemistry will use the new averaging kernel obs operator.
Close to having a unified FV3-JEDI - UFS develop unified build system. Just resolving some issues with updates to the FMS build system.
MPAS:
- Resolving some residual issues with their new ecbuild version of MPAS
- Working on dual resolution assimilation runs
SOCA
- Working with University of Maryland on a new C++ only interface to replace the NOAA OISST product.
Met Office
- Implementing capability to handle staggered grids through Atlas.
- Implemented code for reading the background error statistics from the OPS system.
- Added interface to OPS interpolation routines.
- Looking at stretched grids in Atlas for the UKV model.
- Adding interface to OOPS unstructured interpolation method.
Shallow water
- Some work to use Atlas interface to BUMP.
Generic GetValues
- Planning to get back to this work over the next month or so. Will deprecate type(obsop) from BUMP.
JEDI 4 (Anna)
Travis has been running experiments with LETKF in soca. He sees improvement in speed after Sergey’s PR (splitting observer and solver in two steps) and improvement in performance after Travis’s PR making LETKF work with QC.
Sergey is working on adding Halo distribution for observations distribution.
Yannick plans to continue work on cleaning up cost functions.
Benjamin is adding:
- capability to set different localization lengtshcales for different levels
- atlas meshing capability (in addition to stripack)
Clementine is working on bringing block Lanczos branch up-to-date to issue PR later.
Dan plans to work on static B and variable change cleanups.
BJ is working on BUMP correlation functions.
JJ is running experiments with EDA with low resolution model, working on tuning ensemble.
Ricardo and Fabio brought back FSOI that they worked on last fall. They are experimenting with GEOS to get meaningful estimates comparable to GSI. Fabio has added an option to save increment after each outer loop iteration.
Olly is looking for alternatives to using fftw (fft library with better license to use instead of fftw)
Marek plans to work on time interpolation, and atlas partitioner invariant to staggering.
Anna is working on a coupled H(x) application.
JEDI 5 (Yannick)
JEDI 5 work over the past few weeks has been dominated by preparation for the 5th JEDI Academy, which will be held virtually next week.
Yannick also spoke about the ewok team meeting that Mark M mentioned. He said the ewok work is coming along well and the next steps are to implement H(x) for fv3-jedi and soca. By next month we hope to have an ewok prototype ready for people to use. More updates will come; we will soon have a Thursday focused topic discussion dedicated to ewok.
After the team updates, Yannick asked if there were any general comments or problems to discuss.
Chris S referred to a GitHub discussion on using MPAS-JEDI H(x) as a front end to NCAR's DART system. He added that there are similarities here to the LETKF implementation in JEDI. Those interested should contact Chris for further information.
Since there were no other updates, Yannick invited Carwyn Pelley of the UKMO to give a presentation on the handling of Obs errors in JEDI.
The strategy put forth by Carwyn was generally well received. Hui commented that the ability to control the obs error function from the yaml file would be very useful for them to implement their GNSSRO operators in a more generic way. Greg also approved of the flexibility the PerformAction approach offers.
However, Yannick and Chris S expressed some concern that the approach as stated might combine the two distinct components of Obs filtering, namely
- Select obs based on some filter (e.g. DomainCheck, Blacklist, BoundsCheck)
- Decide what to do with them (e.g. reject, accept, etc)
There is some benefit to keeping these two actions separate. Carwyn agreed that performing an action did not fit into the prior ObsFilter concepts of BlackList, BoundsCheck, etc. And, he wasn't sure if a filter should be responsible for the AssignError operation.
Steve H asked if Carwyn was proposing to include the (non-diagonal in general) R matrix information in the ioda file itself. Carwyn wasn't sure yet if this was the right approach but Yannick said this is a step in that direction. This is something we should look in to in the coming year.
Then the discussion returned to where non-diagonal R components should be assigned. Yannick mentioned that the R matrix is currently handled by the ObsErrrorCovariance class in oops. Anna suggested that another way to achieve what Carwyn was discussing is through another ObsErrorCovariance subclass in ioda that might add off-diagonal components. This might be preferable to assigning these in the filters.
With this we were close to our time allotment for the meeting, so Yannick brought the meeting to a close, reminding people to please suggest focused topic discussions that they would like to see next week and in three weeks (there will be no meeting in two weeks in lieu of the Thanksgiving holiday in the US). To suggest meeting topics, visit the jedi-discussions ZenHub board.