Yannick opened the meeting by announcing that JEDI team members from the Maryland/DC area have been working in Boulder this week.
No special topics this week, so we went around and collected updates from everyone.
Boulder
Mark O has been demonstrating the new workflow, jedi-rapids, to people in Boulder this week. jedi-rapids is successfully running on S4 and Discover using roughly 1/2 of the obs types that we've extracted from GSI ncdiag data. Rahul asked when a demo will be available for EMC/GMAO. Chris S expressed the same for NCAR. Yannick responded that the software is just now coming on-line and we can start planning when to do more demos. Currently, jedi-rapids can do H(x) runs and work is underway to add 3DVar, as well as to get jedi-rapids running on Cheyenne and Hera.
Steve V is working on mpas-jedi. GNU/OpenMPI is working on Cheyenne, but Steve is running into difficulties with other compiler/mpi combinations. Steve is running performance profiling and noted that a particular saber routine is taking a lot of time. Yannick acknowledged this routine and noted that it hasn't been parallelized yet. Steve V mentioned a research paper that presents a parallelization technique that could potentially be used for the saber routine, and will send a link to that paper.
Steve H announced that he has a PR under review that contains a fix for an issue that caused an abort when the number of observations being assimilated was fewer than the number of process elements. This fix allows for the assimilation of one observation which is a useful test case.
Steve H also reported that work is underway to refactor several classes in UFO/IODA which hold geovals and observation data. It was noted that several class structures (ObsVector, ObsDataVector, GeoVaLs, ObsData, ObsSpace) hold these data and the data container parts of these can be replaced with a common container class. Multi-dimensioned data will also be introduced which will allow for the storage of a single brightness temperature variable with 2D data (locations X channels).
Ryan is helping Steve H with the refactoring effort. Ryan also announced that he (along with several others) have access to Mississippi State's (MSU) Orion HPC system. Anyone with current Hera access should now have an account on Orion. Here are slides from a presentation Orion IT gave on how to get started.
Xin is attempting to run GSI on Cheyenne to help his development of the pre-conditioning step in the variational bias feature, but is running into issues with program crashes. After some discussion, it was decided that it would be difficult to exactly duplicate GSI functionality, and it would be sufficient to use data/results presented in several research papers for checking JEDI results.
Maryam is finishing up work to store test data on AWS S3 (as opposed to git-lfs). PRs (ioda, ufo, fv3-jedi, mpas-jedi) are under review for placing ioda obs files on S3. The obs files will be automatically downloaded (if not already present) when running ctest. ioda, ufo and fv3-jedi are all working correctly, and mpas-jedi is working (on Cheyenne) in interactive mode, but not working in batch mode. A feature is included that allows one to pre-download the test data and simply point the build system to that download area which is anticipated to be needed for most HPC systems. Once the mpas-jedi batch mode issue is resolved we will be merging these PRs with instructions on how to work with the S3 test data.
Clementine is learning jedi-rapids this week, and is currently adding in GEOS and GFS into the EDA scheme.
Sarah L is working with EMC, NESDIS to add in the assimilation of aerosol observations such as GOES-16, -17 and MODIS. Sarah noted that some observations become available in 5-minute cycles (e.g., GOES) and asked if we can assimilate at the 5-minute frequency. Yannick mentioned that continuous DA is a future goal, but for now we will work with larger fixed cycles such as the established 6-hour cycle.
Mark M announced that we received an additional 40,000 credits with AWS which starts in February.
Mark has PRs that introduce self-paced JEDI tutorials installed in Singularity containers. You simply pull the container and go through the instructions for each lesson. The container includes compiled code and pre-loaded data. The first example is FV3-GFS in JEDI. For now, the tutorial containers will be private (contact Mark for access), and once we have the initial JEDI release the tutorials will be made public.
Andy will be creating two new JCSDA repos in github for the land DA code. The initial work will be targeted for Cheyenne. Yannick added that JCSDA now has 100 repos on github, with 200 users!
Yannick noted that JCSDA does not have allocation (ie, a project) on Cheyenne. Chris S mentioned that it's easy to get an account on Cheyenne, but more difficult to get a project. You either need NCAR sponsorship, or work collaboratively with a university (NCAR and universities pull from two separate allocation partitions on Cheyenne), where the university collaboration is typically easy to obtain.
Emily is in Boulder this week working on incorporating the satellite radiance footprint (which can cover more than one model grid point) into the H(x) calculation. She noted that the refactoring that Steve H and Ryan are working on, in regard to a radiance variable being 2D (locations X channels), will be very helpful. Emily has also been learning jedi-rapids this week.
Hailing has a new branch in GSI which will add needed data (e.g., GSI results from RO obs ops for comparison with JEDI) to the obs/geovals data generated through the ncdiag path. Hailing announced that new real time RO assimilation results have be uploaded to S3, which can be used for testing. Also, new plots from the real time assimilation have been uploaded to the JCSDA website.
Travis reported the the real time SOCA assimilation is operating on a 1-degree ocean grid using 3DVar with a daily cycle. Ensemble DA will be the next step. Also, every night a SOCA build with production and develop branches occurs, which tests if the latest develop branch is compatible with SOCA. A new web page will be constructed soon which will alert us if any merges into develop have broken SOCA.
BJ is working on covariance functionality in mpas-jedi. Currently there are two variable changes (u,v ↔ psi, chi). He is successfully generating error statistics in 3DVar runs, and is evaluating the results. BJ asked if anyone else has been evaluating covariance, and Dan mentioned that this is working in FV3 and he can help if need be.
Ming is working on WRF in JEDI and has run into issues on Cheyenne with the Intel compiler and MPI (impi) package. Some discussion ensued about recent eckit bug fixes related to MPI and Mark M will work with Ming to get the WRF interface synced up with the proper eckit version.
Francois is validating results from EDA (GFS, c384) assimilation of July 2019 radiosonde and amsu-a observations.
Dan is helping Emily with the radiance footprint work. He is first working on a routine to find the model grid points within a given ellipse (footprint). Yannick mentioned that the Met Office has a PR containing code to find points within a given ellipse, and Dan said he will look into this code.
Anna is working on a generalized locations class where different types of locations (other than lat,lon) can be used for different obs operators.
Sergey is adding LETKF to the FV3 and SOCA DA suites.
Met Office
Wojciech has a QC related PR in ufo for an aircraft track check.
Marek is working with the Geometry class and asked if the new generalized locations class will be accessible from C++. The answer is yes. The refactoring Anna is working on will provide a C++ interface.
Marek has been building a prototype UM-JEDI interface using atlas for the data model. He built geometry, state and increment classes and is in the process of getting the analytic_init test working with a linear interpolation scheme that takes advantage of the structured grid layout. Marek has been keeping to C++ wherever possible. The next step is to understand the stencils better. The geometry class takes account of the vertical and horizontal stagger of the EndGame grid.
Oliver added a "difference states" test (for an increment) to the toy models in oops.
Kristen is ramping up on JEDI code, methodology, etc.
EMC/GMAO
Rahul has a prototype of an ATLAS based grid interface for JEDI, which covers the Geometry and State classes and provides a unified interface. He will be adding a Gaussian grid next to the toy models.
Rahul started exploring obs processing last week. He built the odb interface (ODC) on Hera and is evaluating odb as a central format within the NOAA obs processing flow.
Guillaume reported that SOCA/CICE 3DVar is successfully assimilating marine obs for a one month cycle, and that Stelios is working on an LETKF implementation in a hybrid 3DVar.
Guillaume is developing CRTM simulation for marine obs within a simple 3DVar run.
NRL
Sarah is working to resolve issues arising with saber and the NEPTUNE grid.