Yannick announced that the project leads will be in a retreat during the next week (Feb 4-8) and unavailable for next week's meeting (Feb 7), but we could still have the meeting if need be.
There were no special topics, so we went around and gave updates.
Marek reported that he saw issues related to BUMP while running with multiple process elements after a recent OOPS update. Benjamin is helping and should have a fix ready in a couple days. Yannick asked if they had tested the 4D EnVar branch yet, and Marek said that they will do that next.
Steve S reported work in progress on the model (LFRic) in regard to updates with the State and Model classes.
Yannick asked the group in general if anyone has tested the 4D EnVar branch. BJ responded that they have tried it with MPAS, have successfully produced 4D increments, but ran into a couple issues. These could be related to the MPAS-JEDI interface. Some discussion ensued, and it was noted that the default mode for the 4D EnVar branch includes two actions: no localization in time, and running BUMP for model covariances.
BJ reported that they found an issue with QC related to differences in the results of the background checks coming from a 3D Var run and a 3 EnVar run with the same backgrounds. Yannick responded that debugging this issue is already on the to-do list.
Chris S asked if there was any localization in time for 4DVar. Yannick recalled that there was no localization in time by default but you can use bump to implement it if you wish.
Marek then asked the group if we could create a place to collect scientific results where everyone can have access. Yannick proposed creating a Wiki page where we agree on a common set of background, observations, etc. where we post results of an increment so that everyone can see what is expected. Even in the case of an aqua-planet simulation, this could be useful for verifying that it is in the right order of magnitude. Dan recently created a tool to plot H(x), which could be enhanced to plot other useful quantities such as O-A and O-B. Yannick suggested starting with running a 3D Var with all models using the April 15, 2018 00Z obs and initializations, and posting the resulting H(x), O-A, O-B. Then we can compare results. Mark added that analytic initial conditions from the DCMIP are already included in oops (found in oops/src/oops/generic) and could potentially be used to provide a common initial condition across models. The core team volunteered to create the Wiki page and provide Dan's H(x) plotting script. We still need someone to volunteer to lead this effort.
JJ is working toward a pull request for separating the Increment and State for MPAS. This work has resulted in a more than 50% memory reduction for 3D Var. JJ expects to get additional memory reduction by switching to the DRP Lanczos minimizer which is his next step in this work. JJ reported memory usage differences when running different numbers of iterations of the outer loop that would suggest a memory leak. Yannick responded that debugging this is already on the to-do list.
Steve V has been investigating using MongoDB (a non-SQL database) for the IODA file store. He was able to ingest a sample (radiosonde) Met Office ODB file into MongoDB and discovered that read access is a little bit slower than netcdf. This investigation is getting tabled until some licensing concerns can be addressed.
Anna reported that she has a branch in OOPS that provides a mechanism that can be used to distribute observations according to locality. She has created a new class (ObsSpaceView) that provides a means to select a subset of the observations loaded into an ObsSpace object. She is expecting this work to be completed in the next one to two weeks.
Clèmentine has resolved issues with vagrant and is back on track with her work on the block minimizer. Mark added that the VagrantFile that is currently available on the web is based on one distributed by Charliecloud. Some users have reported inconsistent behavior with this Vagrantfile, which may be due to memory issues. During the creation of the virtual machine, the root filesystem is expanded using the disksize plugin to vagrant. We can likely solve this by using a simplified Vagrantfile that just uses the default memory. Mark plans to provide this as a second option but the priority of this depends on how many users need it right away. So, please let Mark know if you are running into problems with this vagrant file, and he can help get these resolved.
Hailing is now working on a 2D ROPP obs operator. The focus of this work is to develop a 2D geovals representation.
Ming and Xin are working on the goal of being able to point ecbuild_bundle to the main WRF repository (as opposed to creating a custom JEDI version of WRF). They have successfully compiled the WRF code, but are running into issues with some of the configuration/initialization files. They are also seeing issues with real*8, which is being compiled in WRF as little_endian. This is causing problems with the input files that are real*4.
Mark is working with Jim Rosinki and David Hahn to install intel compilers on the Amazon cloud. One of the motivations behind this is to run FV3-GFS in the cloud but another is to provide an ehanced testbed for JEDI. This will allow the core team to test JEDI with intel compilers without the need to access HPC systems like Cheyenne, Discover, or Theia.
Steve H is working on converting the GOES-16 ABI obs data to a file that IODA can ingest. He has an incomplete file (missing some meta data) containing the channel 10 obs, error and QC data that he has successfully read into IODA. Steve is working with Ben and others to fill in the missing meta data. This file will be handed off to Dan for his high resolution run (see below).
Dan is making progress on resolving an issue with excessive IO he is seeing during a high resolution run using FV3-GEOS. Dan reported that he has a high resolution FV3-GFS run successfully assimilating AMSU-A obs, and will try GOES-16 obs next. As part of this work, he has made a couple modifications to OOPS that facilitate plotting H(x).
Ben is sorting out how to convert x, y for the ABI (GOES-16 obs data) into scan angle. Discussion ensued in regard to the extraction of data from the GOES-16 file, and it was determined that the ABI x value is the scan angle, and channel 13 is likely the easiest channel to get by with hard-wired solar angles. Ben is also debugging some large brightness temperature differences that cropped up recently, possibly due to the recent CRTM version update. Anna will help debug by rolling back the recent series of updates so that the particular update that caused the brightness temperature changes can be identified.
Hamideh is working on H(x) for GMI. She is currently debugging issues that came up while trying to save geovals.
Guillaume is running multiple cycles in ocean DA. He can now successfully simulate NESDIS observations.
Nancy (NRL) announced that the IODA workshop is on for Feb 11-13. Yannick and Nancy are putting the program together, and they would like a few more volunteers for talks. The workshop is planned to start around noon on Monday (Feb 11) and end around 3:00 or 4:00 pm on Wednesday (Feb 13).
Maryam is rewriting tests in OOPS with the goal of replacing Boost based tests with Eckit/Fckit based tests. She is currently debugging an issue with the ObjectCounter class.
Yannick closed the meeting with an announcement that there are currently 4 open positions on the UCAR job board for the JEDI core team. These should be open for another week or so. If anyone knows of good candidates, please direct them to these openings.