Yannick opened the meeting by announcing that we will do a roundtable discussion today so we can re-synchronize with the bi-weekly schedule.
Next week we will have a focused topic discussion on C++, in particular the use of smart pointers, intended mainly as a training session for those of us who are not expert C++ programmers. We will have these C++ training sessions from time to time, in order to progressively and collectively improve our usage of C++ in JEDI.
We then started the round-table with Guillaume. He mentioned that there are several pull requests in oops that the soca/EMC groups would like to see merged soon. In particular, he mentioned one on an alternative 2D interpolation that speeds up the interpolation relative to bump (Yannick later merged this).
Hamideh has been working on pull requests in ioda and ufo that add sea surface salinity. The ioda PR has already been merged. Anna mentioned that there is an active PR in jedi-docs that updates that adds documentation on CRTM. She asked Hamideh to add a description of the salinity obs op when she is done implementing it.
Dan has issued a number of pull request recently, mainly to fv3-jedi, that cover work he had done leading up to the AMS meeting.
Guillame then brought up a question he had about how to handle the linear obs operator implementation for surface observations in CRTM. Currently this is missing the Jacobian for surface observations and he wanted to discuss with someone how best to implement it. He asked if anyone had thought about it, particularly with regard to the code organization. Anna suggested asking JJ. Yannick mentioned some recent discussions we had last week when Dan, Ryan, and Emily were visiting Boulder regarding the re-structuring of the ObsSpace data structure. He wondered if this could help with this.
Sarah commented that a similar restructuring might be needed for aerosols, particularly AOD. Yannick wondered if we should have a single interface for CRTM or possibly multiple. Anna mentioned that Bo Huang has been looking into refactoring the AOD operator in CRTM but this is with regard to radiance observations, not surface, so she wasn't sure if it would apply to the issue that Guillaume raised.
The round-table update then proceed with Ryan who said he is now rebuilding the jedi stack on Hera. Anna mentioned that ecbuild in particular is out of date and Ryan agreed that he plans to update this.
Rahul then reported that he has built the jedi-stack on Orion (for gnu/openmpi and intel/impi), building off of the native modules. He also added that the machine is fast, and relatively underutilized. He encouraged others to run on it - if they have access! Rahul has also been working with the fv3 and soca bundles and on the obs processing in EMC. He'd like to modify this workflow to use ioda data files and, when ready, he wants to work with Steve H on implementing generic filters.
Then there was some discussion about pre-processing of the data. Rahul described two levels of pre-processing: that which requires the model backgrounds and that which does not. Yannick mentioned that the pre-processing that requires the backgrounds can be done within the DA but Rahul said there may be some instances where users might want to process the data outside of the DA workflow. Stan mentioned that this would be useful for their work with regional models, particularly with regard to validation. He added that it would be beneficial to have O-B from HRRR in ioda format. Stan and Rahul agreed to discuss this further offline.
Anna has been working on generalizing the locations class and on refactoring GetValues to enable different obs operators to use different location implementations. She suggests that a promising way to achieve this is to encapsulate GetValues into two classes (nonlinear and linear) instead of having them as methods in the State and Increment classes (where they are now). She has a draft pull request in oops and invites those interested to take a look. To illustrate how these new classes would be used, she has implemented them in the lorentz model as an example (they are not yet implemented in qg).
Travis has the real-time SOCA cycling web application working and he plans to add a link to the JCSDA web page to make it more visible.
Clementine has been working with Francois to get an EDA application running on S4 that includes a full set of GSI obs and a higher-resolution (c384) fv3 model.
Andy has been working on ioda-converters for land DA.
Nan is developing a visualization tool for H(x) that he intends to make available in the DAdiagnostics repo.
Cheng is getting up to speed on the aerosol implementation in JEDI.
Maryam reported that the S3 pull requests that have been in development for several weeks have now been merged into develop. So, ioda and other repos now store their test data on AWS S3 instead of git-lfs. In order for this to work, users must use an up-to-date version of ecbuild (jcsda fork, branch 3.1.0.jcsda3). She is now working on implementing Ryan's jedi-build package to speed up the CI testing.
Mark O is cleaning up some bugs that were identified when he was demonstrating jedi-rapids last week, when Dan, Ryan, and Emily were visiting Boulder. He promises a release and a broader demonstration soon. One issue that came up that he has been working to optimize is with regard to the concatenation of netcdf files. Mark implemented two ways to do this in jedi-rapids: 1) through NCO tools, and 2) through the xarray package in python. Mark finds that xarray can be much faster, by a factor of 10-500. So, Mark now has a stand-alone executable in jedi-rapids that can be used to concatenate netcdf files. Travis asked if it will work for records and Mark answered yes. Mark also mentioned that the way we are handling radiance channels as separate variables slows things down a bit and we could likely make that more efficient. Mark also considered two different back ends, namely netcdf4 and h5netcdf, the latter of which provides direct access to the hdf5 api. This allows the user to select between different compression algorithms, a capability that has now been added to jedi-rapids. While cleaning up jedi-rapids, Mark also identified several issues with ioda-converters that could be cleaned up, such as the (deprecated) use of the nrecs dimension.
Stan asked for a use case of why we need to concatenate netcdf files. Mark O and Yannick responded that it has to do with the output of DA or H(x) applications in which each MPI task writes a different file. These then need to be post-processed to combine them into a single file.
Ricardo asked if this was an inefficiency in the diag netcdf files and Mark clarified by saying that it is not a deficiency in GSI but rather the conversion. Ricardo mentioned that GSI has "concatentators" and asked if this could be used. Mark O asked if these were implemented in Fortran (Ricardo was not sure) and if so, suspected they would use NCO. Since xarray is much faster, Mark's new implementation may still be preferred. Mark also said that python xarray is a more general tool that could be used for things other than obs files - for example, fv3 applications often write multiple shards for each tile.
Rahul asked why each task is writing a different file and wondered if we could implement a more efficient way to write. Yannick and Steve H assured Rahul that this is just the way it is done now - there is no expectation that we will continue with this scheme in the future. It is on the agenda to optimize the IO to exploit MPI parallelism but this probably won't happen until a decision is made on whether to adopt necdf4 or odb.
Steve H is working on consolidating different ioda container classes into a common class than can hold N-dimensional data. He said that, in a recent pull request to ufo, Wojciech added an aircraft filter check with methods that can query the ObsSpace to determine and process data characteristics such as how it was grouped and the type of variables that were used, and how to sort the data. Steve said this might be useful for others who are developing obs filters.
Xin has Variational bias correction working with 3DVar and he is looking into the logic of how GSI does pre-conditioning so he can implement this in JEDI.
Steve Vahl has been trying to get mpas-bundle working with the intel compilers on Cheyenne, but has not yet gotten it to work. He has also been looking into using the new interpolation implementation with mpas to improve efficiency.
Mark M has installed new modules on Discover and has updated the documentation for using the modules on Discover, S4, and Hera (this update was later merged so it is now published in the JEDI Documentation). Mark has also been going through the process of gaining access to Hera and NRL systems.
We then moved to an update from NRL where Nancy said that much of what they have been working on parallels what Rahul is working on with the obs processing. They have a prototype for ray observations and are also working on conventional observations. Nancy asked Steve H if the ioda restructuring he mentioned will allow for slant paths. Steve, Yannick, and Anna replied that this has more to do with the work Anna is doing on generalizing the Locations class. This will pave the way for the implementation of ray paths and also satellite footprints. Nancy asked if this would also handle radiosonde drift. However, Anna replied that the radiosonde data is not treated as a profile. Rather, it is treated as a set of 3D points in time. So, this can already account for radiosonde drift.
Nancy also asked others what variable they are using for humidity. The MPAS group responded that they are using specific humidity. Nancy said they are currently using relative humidity at NRL. Dan said that FV3 uses a mixture - specific humidity is used for the increments but relative humidity is used in the background error. He said this is not desirable and should be addressed at some point. Marek responded that at the UKMO they use a hybrid variable that mimics specific humidity in some contexts and relative humidity in others.
We then had a report from the Met Office. Marek has the analytic initialization working for the UM-JEDI model state. Next he plans to work on the change of variable for UM-JEDI.
Wojciech implemented an aircraft id check. He also asked if the cross error had been implemented yet. Yannick said no - currently the only ObsError that is implemented corresponds to the diagonal of the R matrix.
Yannick then closed the meeting by again announcing that next's week focused topic will be a lesson in C++. We'll see you then.