No special topics or announcements today. We went around and gave updates.
Travis is reviving the real-time H(x) using SOCA
Sergey's PR with the grid iterator for SOCA has been merged into develop. Next is the grid iterator for FV3.
Sergey is working on testing LETKF with SOCA, and is anticipating attaining scientific validation by February.
Xin is developing the predictor part of the variational bias correction functionality, and Emily is assisting.
Maryam has a PR under review in IODA that moves the test data to S3. This scheme requires adding a test to all consumers of the IODA test data that downloads the test data to your local clone (but only if the test data is not present). The PR is almost ready to be merged, and needs to be synchronized with PRs in FV3 and UFO (which add in the test that does the test data download). Travis mentioned that SOCA is not a consumer of the IODA test data and thus does not need the new download test.
Maryam added that in the case of Hera, the downloading of data from S3 is not allowed. A provision has been added to the IODA configuration that will use a local copy of the test data, where that local copy can be maintained manually. Ryan asked what version of ecbuild is required for the automatic downloading of the test data. The JCSDA ecbuild develop branch should be used since it contains an upgrade that allows for downloading from S3.
Steve H has a PR in IODA that contains an ODB reader based on ECMWF's new ODC API library. Steve also developed a script (based on a script Steve V wrote) to convert IODA netcdf obs files to ODB, and then converted all of the IODA obs files to ODB. This script, called nc2odb.py, is checked into ioda-converters repository on the feature/nc2odb branch (PR coming soon). Steve then ran FV3 3DVar and 4DVar tests using ODB files and these tests passed. Steve is working with Maryam to add the ODB files to the S3 repository so they become available when the IODA test data is downloaded. The next step for Steve will be to collect performance data comparing netcdf with ODB.
Steve V is adding an ODB writer (based on ODC API library) to IODA. He is also working on MPAS in JEDI.
Yannick has fixed issues with H(x), 3DVar, 4DVar, filter operators that were causing program crashes. OOPS and UFO PRs have been merged in with these fixes.
Hailing is working on real-time H(x) for GNSSRO using COSMIC 2 data, and on creating a GNSSRO obs error inflation filter.
Nan reported an issue with environment modules on S4 (U of Wisc HPC system). The S4 IT team is expected to upgrade the lmod package today which will hopefully resolve this issue.
Francois is running EDA using FV3-GFS and reported an issue where JEDI aborts when at least one of the process elements (PE) has zero observations allocated to it. Steve H responded that this is a known situation that needs to get fixed (Yannick also mentioned that this needs to get fixed), and for now more observations can be added until all PEs get at least one obs to workaround this issue. Francois will add a ZenHub issue to IODA about this and assign it to Steve H.
Chris S reported that they have gained more understanding of the issue where MPAS would crash on Cheyenne when using more than one node. It turns out that if MPAS places all of its IO onto a single node, the running with multiple nodes will not crash. Still not quite at the root cause, but this is helpful information for debugging efforts.
Chris S asked for a list of which compiler/mpi combinations work and don't work on Cheyenne. Here's what was mentioned in response:
| Compiler | MPI | Works for MPAS? | Works for other? |
|---|---|---|---|
| Intel | impi | N | Y |
| Intel | mpt | N | N |
| Gnu | openMpi | Y | Y |
Mark M is actively working on resolving the Intel/mpt issues.
Guillaume reported that Hera is having issues with the file system, IO system and MPI which of course is rendering it unusable. Ming confirmed that ESRL is also experiencing these issues. Ming reported that Hera IT has identified the causes of these issues, and is now working to get them resolved.
Guillaume also asked for the status of a request to place Travis-CI into the soca-cice6 repository, and Maryam responded that she will be working on that next after the IODA S3 test data work is completed.
Dan is working on system level testing in FV3, doing cycling and high resolution experiments.
Emily has completed the July 2019 GSI runs, and has generated the associated geovals and obs files. These are deliverables for the "H(x) validation tool" release in JEDI. She is working on the QC diagnostics task now (for the same release). She has GPSRO and IASI filtering working now. The small IASI test case with 100 obs is getting a match with GSI results, and the IASI full obs set (6 million obs) is getting a mismatch on 26 obs. The mis-matches are related to surface checks, and are due to numerical rounding differences. It was decided that this was good enough since it was rounding differences, and only 26 out of 6 million obs had mis-matched. Emily will upload the YAML configuration she used for this testing so that Mark O can use that configuration in the JEDI work flow. Emily added that work on the microwave filtering is about half-way completed.
Ryan has done prep work for the upcoming IODA test file changes. He has also been assisting with the Hera debugging efforts.
Ryan has sped up the UFO build process using techniques based on what Travis did to speed up SOCA builds. Ryan reported a significant speed-up, and anticipates submitting a PR for this around the new year.
Hamideh cleaned up the SMAP obs file (ioda compliance, plus other work) which has been merged into develop. She also has PRs under review in FV3 and UFO for SSS CRTM functionality (which utilizes the new SMAP obs file).
Wojciech has a number of PRs under review in OOPS and UFO related to Parameters and temporal thinning.
Marek and Oliver are getting started on the UM model interface in JEDI.
Marek reported an issue with ensemble H(x) (enshofx) in LFRic where parallel execution sometimes crashes. Yannick responded that this is a known issue, and is affecting other models. Clementine is working now to resolve this.
Clementine added that a change to the YAML configuration is required for enshofx, and Marek confirmed that this has already been done for LFRic. Further discussion revealed that in enshofx, each model uses its own MPI communicator, and this needs to be set up by passing the Geometry communicator to the Ensemble. Marek confirmed that this has not been done for LFRic and will work on that next. Steve S asked for instructions and/or examples of how to do this and it was suggested to look at how FV3 does the communicator passing as a guide. The EDA and enshofx tests in fv3-jedi are a good place to start.
Sarah reported that they have 3DVar cycling working with Navy obs. Steve H asked if assistance is needed for the IODA issue with the obs files reported in the general communication meeting two weeks ago, and Sarah confirmed that this has been resolved.
That ended the updates and Yannick asked if there were any more comments, questions or concerns.
Clementine mentioned that if anyone is having trouble with EDA or enshofx, to please contact her for assistance.
Yannick announced that we will not have meetings for the next two weeks, due to the holidays, and will reconvene on Jan 9, 2020. Happy Holidays!