This meeting was a general roundtable update.
From: Anna Shlyaeva
JEDI2 and JEDI3 update:
Variable change: Steve V’s pull request that removes VariableChange factories from oops, and changes the VariableChange interfaces will be merged tomorrow. Let Steve know if you have any questions.
GetValues: Progress with GetValues refactoring. Yannick reported speed-up 7-8 times in the current experiments.
CRTM land classification issue: work in progress.
Variational bias correction: Jo Waller issued a PR introducing preconditioning changes to the spectral limited memory preconditioner (similar to the previously changed quasi-Newton limited memory preconditioner): https://github.com/JCSDA-internal/oops/pull/1580. Junmei Ban is testing the quasi-Newton preconditioner in cycled experiments with maps-jedi. Haixia Liu is working on generating data for the preconditioner unit test.
Coupled DA: Merged a PR introducing coupled Geometry, coupled State, coupled ModelAuxControl and a simplistic non-coupled coupled Model (the two model components step sequentially without interaction, with the same time resolution). Guillaume tested the development with fv3 and soca. Clementine is working on adding an option to split the MPI communicator and have each of the model components use their own part of the communicator. Other plans: gradually introduce N-coupled classes (all of the development is 2-coupled now); show an example of interfacing a particular coupled model using the existing coupled Geometry, coupled State, etc classes; add coupled GetValues, taking into account the GetValues refactoring that is in progress (this would allow for CoupledHofX).
EnKF development: Sergey has a PR that we plan to merge soon that adds a capability to sequentially apply several obs localizations in LETKF (similar to the list of SABER blocks for B): https://github.com/JCSDA-internal/oops/pull/1559. This changes yamls for LETKF-type applications. He also issued a bugfix for GETKF, based on a bug report from June Park (OU). He is also working on writing the local volume solver in JEDI paper.
Generic time interpolation: Steve Sandbach is working on the generic time interpolation (https://github.com/JCSDA-internal/ufo/issues/1751), the development depends on the GetValues refactoring (time interpolation will occur in GeoVaLs class). Yannick commented that he expects the optimized get values be merged first, quickly followed by time interpolation.
Updates to BUMP: Benjamin has issued several bug fixes, some related to using um-jedi with saber.
Spectral covariance in SABER: Steve V will be working on porting um-jedi spectral covariances to SABER blocks; the refactoring in SABER and um-jedi have made this task easier. Olly commented that the hard part would be in computing statistics for the spectral B. For applying spectral B, interpolation from the Gaussian grid is needed, which is implemented and thoroughly tested in atlas. For computing statistics, one would need interpolation to the Gaussian grid; the generic interpolations in atlas may help with this.
From Hui Shao:
We had the monthly meeting for UFO this week so the work reported here include in-kinds' UFO progress reports from the UFO meeting:
UKMet:
- radiance: ATMS is now ready for trialling. now working on other data types and RTTOV interface update for MW scattering
- GNSS/AMV: main focus is on the feeding JEDI with ODB files and validation is in progress.
- Conventional: Building end-to-end yaml files for both sonde and surface processing and using these to check JEDI is replicating the current processing system. Near completion for Sondes.
EMC:
- Computed bump static B for aerosols and analysis increments using hybrid B. PRs for MODIS and IRRS converters are in review.
- Work in progress for aircraft prepbufr converter and comparison of the IODA data vs GSI data set.
- Added ABI yaml files for QC and BC based on GDAS applications
GSL:
IODA V2 converters for AEROMENT AOD and absorption AOD are merged to develop
JCSDA:
- Working on NRT demo data decoding and collection (Ben R reported)
- Converters for some observation types in the WMO bufr format are in Develop now.
- Continue to work on QC steps for conventional data.
- Two PRs in review for adding GDAS COSMIC-2 obs errors and the three corner hat (3CH) into UFO. Initial tests are being performed to test the analysis and forecast impacts for implementing different obs error models.
From: Ryan Honeyager
OBS3
-We held a code review sprint for the IODA converters last Friday. Thanks to everyone who attended. I believe we have three or four PRs related to the IODA-v2 conversion that are still outstanding, and several new ones related to ingest of ATMS and GNSSRO data.
-OBS3 met last week and Eric Lingerfelt gave a general talk on R2D2's current state and planned features. We had questions about syncing local and JCSDA-wide databases, overwriting and versioning of files, and some discussion about how R2D2 should interface with IODA. We then looked at the metadata that R2D2 needs. The general consensus is that R2D2 can scan the global attributes in the IODA file headers to get most of its metadata. We discussed some of the metadata that we want, and can add these to the JEDI Data Conventions tables.
-The JEDI data conventions were posted on our ReadTheDocs site on Monday! Many thanks to everyone who contributed. You can find them under the "Inside JEDI/JEDI Data Conventions" section.
-The next step for the convention rollout is the development of a IODA file validation program that checks that every variable has the proper name, type, units, and is stored in the correct groups in the ObsSpace structure. Development of this program is in progress and will be done by the end of the year. At the same time, we will upgrade the "ioda-upgrade.x" application to switch out variable names in all of our testing files. We talked timelines at last week's OBS3 meeting, and would like to host a sprint for the conversion in early February, though dates are to be determined. It will be a week of find and replace throughout our YAML files, test data, IODA, UFO, and model interface codes. Details will be sent out in January.
-Links:
• Eric's slides: https://docs.google.com/presentation/d/1zDXNdtz-Kndx76OfkGo4OZY4NMtc2OTi/edit?usp=sharing&ouid=107877012698598856934&rtpof=true&sd=true
• JEDI Data Conventions: https://jointcenterforsatellitedataassimilation-jedi-docs.readthedocs-hosted.com/en/latest/inside/conventions/index.html
From: Jerome Barre
OBS:
- AERONET IODA converter merged to develop
- PR still open for MODIS and VIIRS
- PR on the averaging kernel operator includes 3 different PR on IODA-converters, UFO and FV3-JEDI
SABER:
- Computation of bump static b matrix for aerosols is working at C96 resolution.
From: Benjaman Ruston
BenR - working on converters for various data types to IODA2.
Have noticed that some of the existing data types likely are passing the ctest however the data may not be usable in UFO. Will begin to examine this and reach out to developers.
Further, there are no existing naming conventions for the output filenames and they vary greatly. It would be good to try to coordinate the data types for a couple specific dates to make testing the test files in UFO easier.
Regarding new datatypes, there are two microwave sensors TEMPEST and COWVR being launched 20Dec and will be placed on the International Space Station (ISS).
Lastly, there is GNSS-RO data being generated now from the PlanetiQ receivers, we are having a very preliminary look at these and have been in contact with various GNSS-RO partners.
From: Andrew Fox
Land updates:
After a couple of false starts, Zhichang has been making some great progress with UCLDAS, which is the interface to the offline land model driver under development at EMC, with pretty much everything except GetValues/VariableChange implemented.
We’re having ongoing discussions about how to best integrate this project with other land DA efforts.
On the WRF-Hydro side, we’ve had some PRs related to the speed up work by Soren, how we generate our output file names so we can work with ensembles, and dealing with some additional model variables describing bulk snowpack properties.
In review, we also have a PR from Benjamin to get us started with ensemble-based static B estimation, and I have a draft PR dealing with the variable change refactoring (https://github.com/JCSDA-internal/oops/pull/1333)
From: Francois Vandenberghe
A mock-up NRT website was deployed. It uses the PADME/BESPIN repos for binning and generating graphics on the backend, and the Voila dasboarding utility for the widgets and display on the frontend.Travis will setup an AWS CodeBuild CI pipeline to build and deploy the PADME/BESPIN repos to a backend docker image, and Rachel will work on containerizing the Voila based frontend so the resulting docker container can be used for autoscaling.
Ben and Francois ran the Unified Model HoFX application with GALWEM forecasts provided by the Air Force. The test was using RAOB and they will expand to satellite data.
Ben and Francois are working on a GNSSRO Python BUFR to IODA2 converter that will eventually supersede the current Fortran BUFR to IODA1 converter.
Fabio's PR #1407 (Adding the LinearForecast application) was temporarily closed while some other code updates are made.
From Dom Heinzeller:
PR for new epoch data rep can start switch representations
Fortran API progress targeting early January for PR
PR on Jedi stack with EC codes
Draft PR to update python module targeted for January
Progress for nightly runs on Cheyenne
From Yannick Tremolet:
Jedi 4
-Got h(x) to run with new structure in fv3.
- Include the 3dvar will provide updates on that in January.
Discussions:
Neil Bowler: Question ton including an option to include observations at the beginning of the DA windows instead of at the end/
Yannick: Easy to put in but might be down stream consequences.
Neil Bowler: Surface obs time is on the hour where there might be a significant number.
Yannick: We can look into it but may be alot of work for not much gain.
Chris Snyder: Asking about forecast impact or for testing.
Neil: Really for testing
Yanick: You can put non-strict inequality but may produce obs going twice.
We can change inequalities and run tests and see if things break.