Yannick - We decided to move the round table discussions from next week to this week. Next week's JEDI meeting is cancelled due the upcoming JEDI academy.
JEDI1 update (Mark):
The first JEDI1-related announcement is that the MPAS-JEDI release occurred smoothly last Friday. Congratulations to the MPAS and JEDI teams for making this happen. For those who haven't seen it already, please refer to the https://jcsda.org web site for further information.
A few build issues came up this week that are worth mentioning. A PR was merged to IODA that effectively and unintentionally made ODC a requirement. This was causing builds to fail for CI and for platforms like Cheyenne where ODC was not installed. This prompted two courses of action. Steve H issued a PR that again made ODC optional, meaning IODA can now build and run without it. Rick and Maryam made sure that ODC is now installed on Cheyenne and Caspar and in all the containers. So, this problem should be fixed. As always, if you run into other build issues, let us know.
Another change was with the containers. Accompanying the MPAS-JEDI release, we created new containers with an upgrade to gcc/gfortran 10.3. But we were finding that tests and applications were suddenly taking more memory, causing intermittent test failures and problems with the Academy practicals. This seemed to be not a problem in the clang containers, which use gfortran. So, we attribute it to gcc/g++ 10.3. So, we reverted the gnu-openmpi-dev container to gcc 9.3, while adding a few other enhancements (properly setting up locales to remove ecbuild warnings, AWS Codebuild upgrade). So, there were a few days of transition for the containers but they should be stable again.
In yesterday's JEDI1 meeting we had a presentation by Steve Lawrence, who is working with Henry Winterbottom, Sergey Frolov, and Jeff Whitaker on their cloud infrastructure for running cycling DA applications. Currently they are focusing on cycling 3DVar with GSI and SOCA using a CYLC workflow. They have their workflow running on both AWS and Azure, using both NOAA's ParallelWorks cloud platform and independent cloud accounts. They have done performance and cost comparisons between different cloud configurations (processor types, memory, networking, storage) and similar runs done on Hera. Performance is generally comparable to Hera and the scientific result match. In particular, they find consistent and predictable throughput in the cloud that is comparable or faster than On-Prem. For further information, contact Steve Lawrence.
On the JEDI4 side, Rick has begun to implement EWOK on an AWS cluster. And, we are just starting to get set up on NOAA's ParallelWorks platform.
Steve H continues to work on the new DateTime implementation in IODA. We are considering several different ways to represent the DateTime as integers in IODA files. Steve finds that human-readable integer representations are about a factor of 2 slower than OOPS native integer representations but still more than an order of magnitude faster than strings. We have not yet made a decision on which to go with.
Mark Potts has also created a docker container equipped to run UFS, which will be used for CI on the fv3-jedi repo to check the UFS coupling. It builds on the gnu-openmpi-dev container and adds in libufs. There is a PR in the docker repo issued today for those interested.
Progress was also made this week on CI for ops-um-jedi. A plan was implemented to host the test data on S3 and we should now be ready to implement the CI.
And, finally, we have been actively preparing for the 7th JEDI Academy, which occurs next week.
JEDI2 update (Anna):
Jedi 2.2 - Generalized locations - Francois Hebert is working on the field of view calculations (computing averages of surface quantities)
Jedi 2.3 - Use of NUOPC driver with FV3-JEDI - Mark Potts put together a singularity container (docker image) for testing fv3-jedi with a version of UFS that can be linked with the JCSDA FMS version. He is working on adding an H(x) C48 resolution test to CI. Remaining question: how do we keep that container periodically updated with UFS?
Jedi 2.6 - Ensemble DA validation for fv3-jedi - Sergey Frolov and team have GSI GETKF and JEDI GETKF running with using the same obs. Focus is on radiosondes t, u, v, q, with the latest UFO yaml files.They see some differences when assimilating all network and are working on identifying the issues.
Jedi 2.7 - Background error model validation for fv3-jedi - Benjamin Menetrier is focusing on the humidity variable: looking at changes of variables to use normalized rh or pseudo rh (GSI options); or using specific humidity and having semi-analytic change of variables between temperature and humidity (used in IFS). There was a discussion on which humidity variables different groups use in B in JEDI. BJ Jung (mpas-jedi): have an option to use pseudo rh; change of variable is inside MPAS linear variable change. BJ compared pseudo rh and specific humidity, and statistics look similar. Structure of the vertical length scale is quite different between the two, but a month-long cycling experiment shows similar results. Marek Wlasak (um-jedi, lfric): use pseudo rh. It’s one bit of covariances that doesn’t give the same results as VAR; working on identifying the differences. Operational system also has Met Office nonlinear moisture control variable (but this is not in JEDI yet). Liz Satterfield (neptune): use mixing ratio. Would like to get to pseudo rh. Cathy Thomas plans to do EnVar tests (pure EnVar) in addition to Var with static covariances.
Jedi 2.8 - Regional DA with fv3-jedi - Ting Lei has tentative results for cycling with conventional data. Main concern is the big difference H(x) for wind. Ming Hu is working with Greg Thompson on testing H(x) and analyzing the differences.
Jedi 2.9 - MPAS general updates - JJ Guerrette is working on EDA experiments.
Jedi 2.12 - Neptune general updates - Liz Satterfield reported that Neptune is cycling with winds and temperatures stably and they are working on moisture updates.
Jedi 2.14 - Cubed sphere grid into Atlas - Olly Lomax: mesh generator is in atlas develop. FunctionSpaces for NodeColumns and CellColumns for cubed sphere are under review; probably merged very soon. Next 2 steps: Make a general redistribution method and add an operation to convert cubed sphere mesh to dual mesh (with center cells). After that is done, off-the-shelf atlas interpolation should work for a cubed sphere.
Jedi 2.15 - VADER - Steve Vahl is finalizing a big OOPS PR to prepare oops for VADER. Models will need to adapt before PR goes in. Steve would work on um-jedi, and Dan Holdaway on fv3-jedi to put some examples in place.
JEDI 3 update (Anna):
Fabio Diniz issued bug fixes for saving bias correction in the HofX3D application and ObsOperator test.
Anna Shlyaeva has a draft PR that removes GetValues from traits, and creates a factory: https://github.com/JCSDA-internal/oops/pull/1397. This would allow for generic implementations of GetValues; e.g. GetValues using Interpolator classes in OOPS, or generic time interpolation GetValues. All models would need to adapt, we will announce the changes required soon.
Travis Sluka refactored ufo ObsLocalization classes (used in LETKF), they now allow for specifying length scale that is different from the one in the yaml file: https://github.com/JCSDA-internal/ufo/pull/1517.
We are discussing a code sprint on the parameters.
Wojiciech - Code sprint description. In my view this refactoring will benefit easier to make it easier to find configuration options. The second benefit would be made to catch mistakes in YAML files with automatic validation. We invite participants contribute. For validation while editing these files in schema editors. It would be great if we could have one person from each group create model interfaces in a code sprint in November. If you are interested, please send email to Anna or Wojiciech.
Yannick - We will give an update on dates when that is decided.
Chris Snyder - Could these schema that come from parameter class be used to generate documentation as a list of what an application needs in its YAML file?
Wojiciech - There are tools that convert schema to documentation, but I have not tested them. They may not tackle our schemas wince they are complex. The list is very long. To test a schema generated for an option for a single operator would probably more easy for people to use. Recently a capability has been added that makes it possible for a schema to contain descriptions for individual options.
David - For the model GeoVal orientation, do we have a conclusion with what you want to do like fix the implementation in UFO?
Yannick - I think we said yes but we have not decided what direction to go.
David - Some models will have to do nothing and some will not. Maybe some forward operator will require to be changed. They may assum an orientation.
Yannick - We will take the orientation that CRTM wants and go ahead and implement that.
Greg - I think that result of CRTM is top down. In any case, there are a couple of UFO operators that are intended to be bottom up. I think that the identity operator for SST might need it to be bottom up, not top down.
Yannick - We came to conclusion that the models are about 50/50. We should start going in that direction.
David - All of that will be done in model interface?
Yannick - Then UFO will all be written so that the input comes in that direction and we don't need to test.
Hui - Let me know when those changes will be coming in.
Yannick - I would expect the OBS team to take the lead with that.
Greg - That should not be an existing problem in UFO filters.
Yannick - We need to make a plan and make a decision and document it somewhere for clarity.
Anna - I have a draft PR in UFO that adds a check to make sure GeoVals are in the correct order.
OBS1 update (Hui):
JEDI_GDAS conventional yaml updates are in review. Communicating with EMC on speeding up the validations of the updates as well as identifying and developing the missing capabilities. Working on the R2D2 updates: smoothed out issues related to radiance data. Now adding new field (effective QC from GSI) to observations for reference validation. Added ABI to JEDI_GDAS testbed using observations from NOAA Data Lake project. A demo case for using data from different data resources. ABI yaml is based on NOAA AHI configurations.
JEDI_RRFS: the first set of yaml files is in review. Working with GSL to review and test these configurations.
Extension of current AOP UFO for absorption AOD
Obsbias outputs are added to the UFO outputs to help diagnose the bias correction
Solar zenith angle correction in develop
ROPP-UFO fix for compilation
PRs are currently in review: cross channel correlated obs error, profile modification, variable transform, satwind obs pressure, ...
OBS2 update (Francois): No update
OBS3 update (Ryan):
OBS3 met yesterday. We discussed WIGOS and how it can be used in JEDI. WIGOS identifiers are the new WMO standard for identifying stations. David Simonin gave a presentation detailing how these identifiers work. For slides, feel free to ask him or me.
Following the presentation, there was open discussion about how we could implement WIGOS. The use of these identifiers in QC is going to be quite complicated. There are old-format and new-format IDs, and IDs can also change in certain situations, so we need to retain ID mapping tables inside of UFO.
We also discussed how this would mesh with other station ID specifications. Formerly, we had a loose concept of a single stationIdentifier variable, but this is problematic because not everyone uses the same methodology for assigning station IDs. Some stations use WIGOS, but we also have aircraft tail numbers, special codes coming from METARs or TAMDAR data, and so on. Some stations may be identified under multiple schemes, while others lack any such assignment. Some of these identifiers are numbers, others are fixed-length strings, and a few are longer and have a variable length. In general, we may need to create separate variables for each type of identifier, and we would like to reserve the old "stationIdentifier" name for local use. This should be discussed more and once we agree on a strategy we can update the IODA conventions.
Speaking of the IODA conventions, we have finalized version 1.0 of the conventions document and tables, and will post them on ReadTheDocs soon. Weekly Friday meetings will continue to discuss changes to the conventions and how to connect these to other parts of JEDI. The ioda-converters will need a round of updates later this year to switch variable names, UFO YAML and source code files will need to be updated, and some of the variables used in model interfaces will also need matching changes. We are also working on expanding IODA somewhat to support all of the described data types, such as the new enumerated data type. This crosses over into Steve's DateTime improvements, and several PRs have already been accepted.
Finally, Steve H, Cory, and Ryan are engaged in a mini code sprint this week. Following the JEDI1.1 release, the converters still wrote IODAv1 files, and then v1 outputs needed to be processed through an upgrader program. We want to fix this. In this sprint, we are taking a few Python converters and are upgrading them to use the new Python IODA-v2 interface. The goal of this mini-sprint is make examples to show converter developers how to update their converter codes. We updated one of the marine converters yesterday, and this will be a great example for the other marine converters. The process is very parallelizable, so we want to have a sprint in October to port all of the Python converters. If you are a converter developer and are interested in participating in this broader sprint, please email me. The general sprint will be after next week's JEDI academy, possibly the week of the 11th or the 18th, depending on everyone's availability and so we do not conflict with the JEDI sprint.
Yannick - What about non-python converters?
Ryan - GNNSSRO and NCAR are waiting on FORTRAN converters to be converted into python.
CRTM update (Patrick):
Yesterday we had first tutorial for generating microwave instrument coefficient. We will provide documentation. There is an issue with Intel FORTRAN compiler on S4 and likely on Discover. We are looking for solution and waiting for the contributions from NOAA NESDIS for CRTM.
Cheng - Working on reflectance factor for CRTM and python code for CRTM solver. So far CRTM can only handle upward radiance.
Mark - CRTM build problem with intel . Is only in debug mode?
Patrick - This is the debug mode issue mentioned by Ben. Ideally, we would like to identify the location on the code triggering these issues. It is difficult to get verbose output about what’s going wrong. It is just saying that an internal compile error is occurring.
Ryan - I'm wondering, do you still have the weekly telecon with the Intel engineers?
Mark - Yes. It seems like a hard one to deduce. We meet tomorrow at 10am. Could be useful.
Ryan - Can we have an Intel engineer take a look? This issue has propped up several times in last three years.
Mark - Provide what compiler flags are being used and I will bring this up at meeting.
SOCA update (Guillaume): No update
LAND update (Andy):
Youlong and Clara have ongoing work with preprocessing and IODA converters for IMS snow cover fraction observations
Also working through QC associated with these obs, that included using obs that aren’t assimilated and model states that aren’t updated. The former is easy to deal with, the latter not so much with the LETKF, so that’s going to require some development. Thanks to Anna for inputs on that one.
Had a meeting with Clara, Tseganeh and Mike Baralage at EMC specifically about short term and medium term plans for offline land model capabilities. We have an existing “Frankenstein” version that will (shortly) enable cycling, but at very specific resolutions and converting between grids. All in agreement this is sub-optimal, but medium-term plans will really depend on model driver developments
I’ve been working on IODA-converters for the WRF-Hydro SWE obs for the first time in a long time - updating based on Cory’s IODA engines script - hope that’s the right thing to be doing.
Steve H - Parameters code sprint questions. We need to get rid of the dependency on OOPS in IODA and parameters is one of them. We are creating a new repo called JCKIT. Do we plan to moving parameters to JCKIT? If it remains in OOPS, we will keep OOPS as dependency. Is that something that should be done during or before the code sprint?
Wojiciech - What is status of JCKIT? This could become the first component of JCKIT. Just a question to Steve (H) are you also removing dependency on ECKIT?
Steve H - It is something to consider, but it would be more difficult.
Wojiciech - I am asking because parameter classes use the configuration classes from ECKIT.
Ryan - We should not have to remove dependency on ECKIT, but OOPS is hard to compile.
Yannick - Other things will be moved to JCKIT that will depend on ECKIT.
1 Comment
Eric Lingerfelt
Yannick - We decided to move the round table discussions from next week to this week. Next week's JEDI meeting is cancelled due the upcoming JEDI academy.
JEDI1 update (Mark):
JEDI2 update (Anna):
Next 2 steps: Make a general redistribution method and add an operation to convert cubed sphere mesh to dual mesh (with center cells). After that is done, off-the-shelf atlas interpolation should work for a cubed sphere.
JEDI 3 update (Anna):
JEDI4 update (Yannick):
Wojiciech - Code sprint description. In my view this refactoring will benefit easier to make it easier to find configuration options. The second benefit would be made to catch mistakes in YAML files with automatic validation. We invite participants contribute. For validation while editing these files in schema editors. It would be great if we could have one person from each group create model interfaces in a code sprint in November. If you are interested, please send email to Anna or Wojiciech.
Yannick - We will give an update on dates when that is decided.
Chris Snyder - Could these schema that come from parameter class be used to generate documentation as a list of what an application needs in its YAML file?
Wojiciech - There are tools that convert schema to documentation, but I have not tested them. They may not tackle our schemas wince they are complex. The list is very long. To test a schema generated for an option for a single operator would probably more easy for people to use. Recently a capability has been added that makes it possible for a schema to contain descriptions for individual options.
David - For the model GeoVal orientation, do we have a conclusion with what you want to do like fix the implementation in UFO?
Yannick - I think we said yes but we have not decided what direction to go.
David - Some models will have to do nothing and some will not. Maybe some forward operator will require to be changed. They may assum an orientation.
Yannick - We will take the orientation that CRTM wants and go ahead and implement that.
Greg - I think that result of CRTM is top down. In any case, there are a couple of UFO operators that are intended to be bottom up. I think that the identity operator for SST might need it to be bottom up, not top down.
Yannick - We came to conclusion that the models are about 50/50. We should start going in that direction.
David - All of that will be done in model interface?
Yannick - Then UFO will all be written so that the input comes in that direction and we don't need to test.
Hui - Let me know when those changes will be coming in.
Yannick - I would expect the OBS team to take the lead with that.
Greg - That should not be an existing problem in UFO filters.
Yannick - We need to make a plan and make a decision and document it somewhere for clarity.
Anna - I have a draft PR in UFO that adds a check to make sure GeoVals are in the correct order.
OBS1 update (Hui):
JEDI_GDAS conventional yaml updates are in review. Communicating with EMC on speeding up the validations of the updates as well as identifying and developing the missing capabilities. Working on the R2D2 updates: smoothed out issues related to radiance data. Now adding new field (effective QC from GSI) to observations for reference validation. Added ABI to JEDI_GDAS testbed using observations from NOAA Data Lake project. A demo case for using data from different data resources. ABI yaml is based on NOAA AHI configurations.
OBS2 update (Francois): No update
OBS3 update (Ryan):
Yannick - What about non-python converters?
Ryan - GNNSSRO and NCAR are waiting on FORTRAN converters to be converted into python.
CRTM update (Patrick):
Cheng - Working on reflectance factor for CRTM and python code for CRTM solver. So far CRTM can only handle upward radiance.
Mark - CRTM build problem with intel . Is only in debug mode?
Patrick - This is the debug mode issue mentioned by Ben. Ideally, we would like to identify the location on the code triggering these issues. It is difficult to get verbose output about what’s going wrong. It is just saying that an internal compile error is occurring.
Ryan - I'm wondering, do you still have the weekly telecon with the Intel engineers?
Mark - Yes. It seems like a hard one to deduce. We meet tomorrow at 10am. Could be useful.
Ryan - Can we have an Intel engineer take a look? This issue has propped up several times in last three years.
Mark - Provide what compiler flags are being used and I will bring this up at meeting.
SOCA update (Guillaume): No update
LAND update (Andy):
Youlong and Clara have ongoing work with preprocessing and IODA converters for IMS snow cover fraction observations
Also working through QC associated with these obs, that included using obs that aren’t assimilated and model states that aren’t updated. The former is easy to deal with, the latter not so much with the LETKF, so that’s going to require some development. Thanks to Anna for inputs on that one.
Had a meeting with Clara, Tseganeh and Mike Baralage at EMC specifically about short term and medium term plans for offline land model capabilities. We have an existing “Frankenstein” version that will (shortly) enable cycling, but at very specific resolutions and converting between grids. All in agreement this is sub-optimal, but medium-term plans will really depend on model driver developments
I’ve been working on IODA-converters for the WRF-Hydro SWE obs for the first time in a long time - updating based on Cory’s IODA engines script - hope that’s the right thing to be doing.
Steve H - Parameters code sprint questions. We need to get rid of the dependency on OOPS in IODA and parameters is one of them. We are creating a new repo called JCKIT. Do we plan to moving parameters to JCKIT? If it remains in OOPS, we will keep OOPS as dependency. Is that something that should be done during or before the code sprint?
Wojiciech - What is status of JCKIT? This could become the first component of JCKIT. Just a question to Steve (H) are you also removing dependency on ECKIT?
Steve H - It is something to consider, but it would be more difficult.
Wojiciech - I am asking because parameter classes use the configuration classes from ECKIT.
Ryan - We should not have to remove dependency on ECKIT, but OOPS is hard to compile.
Yannick - Other things will be moved to JCKIT that will depend on ECKIT.