There was no specific topic today, so we went around and collected updates.
Steve H announced he is working on the conversion paths to get the various obs file formats into a common format that IODA can read. He is working with Anna on the GSI ncdiag to IODA path, and with Steve V on the ODB to IODA path. There is a target IODA netcdf format that Steve H and Steve V designed, so the end goal is to get all obs data files into the target netcdf format and modify the IODA reader to use the target format.
Steve V reported that he is investigating various data store mechanisms to be used by IODA. Currently we are using a netcdf file for this. Steve V ran a performance comparison between netcdf and ODB API and found that netcdf read operations are much faster than those for ODB API. Yannick added that ECMWF will be making available soon a new version of ODB (called "ODC") that will be open to us and will combine the speed of ODB1 with the archiving features of ODB API. This should be available to use in a few months from now.
Yannick announced that the core team has a new hire, Maryam, who will be working on the JEDI testing framework.
Anna is working on adding an interface to OOPS to find local obs. She has this working with the L95 model in OOPS.
Hailing is making progress with the ROPP obs operator. She is expecting to have results for this next week.
Xin has finished up the work on the C++ ObsSpace class, and is switching to working with Ming on the WRF integration into JEDI. He is investigating the TLAD aspect of this integration.
Yannick has updated the QC filter capability in JEDI. The intent of the new system is for the code to contain the generic aspects of filtering, while the specifics for the different obs operators is contained within the YAML files. Most, if not all, filtering should be able to be accomplished by writing YAML files. Yannick will provide documentation and default YAML files that can be used by the developers to create their specific YAML file. The new code is all C++ (no Fortran). Complex expressions (A or B and not C for example) are not available in the YAML yet. Hailing has started using the new QC filter scheme in her ROPP obs operator, and so far has had success.
Mark has built a random number generator package that will generate the same random number sequence, given the same seed, independent of compiler and platform. He added a new test to the system that verifies the random number sequence, and plans to add some more tests. There are C++ and Fortran interfaces available, and Mark plans to submit the pull request for this package later today. As a side result of this work, Mark added some additional testing infrastructure to OOPS.
Clementine is working on the block Lanczos minimizer. She has made a lot of progress and is expecting the minimizer to be converging soon.
JJ is working on an MPAS run at 120 km resolution. He is running into memory issues with the DRP__(?) minimizer and is currently debugging them. Yannick suggested switching to the DRP Lanczos minimizer since this is the most heavily tested option. JJ will complete his work on separating increments and states first, and then give the DRP Lanczos minimizer a try.
Yannick asked the group if anyone has been testing an assimilation of both radiosonde and aircraft data. FV3 has seen an order of magnitude difference between aircraft and radiosonde increments, and Yannick wanted to see if other models have experienced this. BJ reported the large difference when using MPAS, even with univariate 3DVar. Dan reported that with FV3, the temperature from aircraft and radiosonde were different, but not as severe a difference as seen with the winds. Anna responded that anyone doing these tests needs to use the new obs files that she has generated from GSI. These files are currently not in the IODA repository, so they need to be obtained from Anna.
BJ is working on assimilating the moisture variable. They have converted relative humidity (RH) in the obs files to specific humidity before assimilating. This was done to accommodate an algorithm somewhere. Yannick warned that it is not good practice to modify obs data. The conversion of RH to specific humidity is okay for now, but we will want to eliminate that in the long term.
Junmei has developed scripts to process results from assimilating multiple variables (Q, T, U, V).
Chris S commented that, for MPAS, they are targeting bit-for-bit match between running forecasts with the stand-alone MPAS model and the JEDI MPAS model. They are getting close but are still debugging some mis-matches.
Francois is working on a GPS Constellation QC filter. The plan is to create a separate YAML file for each satellite constellation. This will involve a lot of up-front work, but once finished there shouldn't be much subsequent change.
Tom added that there is a conceptual change in the QC filtering task. The generic aspects are in the code and the specifics are in the YAML. This should allow QC filters to be developed simply by writing a YAML file (no code changes) which fits in well with the Agile methodology.
Dan reported that AMSU-A assimilation is working again in 3D Var. The convergence isn't as good as he would like, but the process is working. The next step is to try adding in more obs types. Dan is also working on a high-resolution (50 km) 4DVar run. This is working, but he is running into issues with program crashes. Dan is making some code cleanup that includes reducing the "noise" that comes from all MPI processes writing the same messages. This should help make debugging easier. Benjamin's fixes to reduce the number of files from BUMP have been helpful. The code cleanup also includes eliminating duplicated code that resulted when the feature to split fields into state and increment was added. Dan will create a new branches in the repositories with the code cleanup changes.
Sean (UKMO) announced that he is working on the satellite DA plans for next year, and is coming up to speed on JEDI. He mentioned that David Rundell has a version of RTTOV working with UFO, and will be creating a repository for RTTOV (github, davidrundell/RTTOV). Tom asked that David make sure the RTTOV repository is private in order to avoid licensing issues. Sean asked if we are supporting HDF5. The answer is yes. The HDF5 libraries are installed since they are required for netcdf4. Steve H volunteered to be the contact point for HDF5 support. Chris S asked if RTTOV is ready to try out. Sean responded that it should be since it works in UFO, and David can be contacted for more details.
Steve S is working on the LFRic build infrastructure for GNU and Intel compilers. He is having some issues with BUMP when using the Intel compiler.
Marek is developing an identity obs operator in UFO. It is passing tests now. He asked it the SST data in the marine obs is valid. Guillaume responded that some of the marine obs data is incomplete for now, and will be fixed later on. Marek asked if we have surface pressure data. Anna responded yes, and she will work with Mark to make it available on Amazon. Marek also asked about the next steps once the identity operator work is completed. Yannick indicated that further discussion is necessary, and he will set that up.
There were some questions about file naming conventions in ioda. Steve H mentioned that there are currently two conventions supported, including a newer "@"+<suffix> format that we are transitioning toward.
Guillaume is working on a 3DVar FGAT run, and a domain check QC filter. He will try switching his domain check to the new QC scheme (ie, write the YAML file). Guillaume will be adding more filters for the ocean model next. He asked about the H(x) for 4dEnsVar and Yannick responded that it still needs work.
Nancy (NRL) reported that they are proceeding with the plans for the IODA workshop in February. 17 people are signed up now, and we are waiting for the furlough to end before we have the final attendance numbers. A discussion followed about when we should make the decision to postpone the workshop. Government employees can't book travel, UCAR employees' travel is highly restricted, and we are rapidly approaching the point where it will be too late to get travel arrangements made.
Sarah (NRL) announced that work is in progress for the NEPTUNE model integration into JEDI.
Hui is working on GNSSRO 2D bending angle obs operator, and this work needs the 2D slices from the model. Yannick responded that this is not in the system yet. Hui will start building more QC filters once Hailing's ROPP operator is working.