2021-10-14

JEDI1 and JEDI4 update from Mark, Yannick, Steve, and Rick

All members of the JEDI 1 core team participated in the 7th JEDI Academy, which was held virtually last week. Another successful Academy.

One issue that we have been discussing and working on recently is how to get JEDI to work with miniconda installations of python. The problem is that miniconda installs its own versions of libraries like openssl, hdf5, netcdf, szip and lapack. And, it prefers it's own libraries to those installed by the jedi-stack. The introduction of the ioda python bindings effectively imported those preferences into jedi bundle builds. This leads to build failures, build warnings, and/or test failures. And, by extension, application failures.

The only container where this is a problem is the jedi-gnu-opempi-dev singularity container because this is the only one that uses conda. The reason it uses conda is to facilitate installation of python packages like cartopy that are used for the tutorials. This week, Steve H made changes to ioda and Mark M made changes to the container and together they got it to work; all ioda and ioda-converters tests pass in the container. But the price was high; to get it to work Mark had to change the RPATH in the conda python executable and disable RPATHs in the bundle builds.

At our JEDI1 meeting yesterday we opted for another approach. We decided to stop using conda and we recommend that other JEDI user/developers do the same where possible. For the container and HPC platforms, we can install python packages using the system-provided python or alternatives like brew or intelpython. Mark will rebuild the jedi-gnu-openmpi-dev container this week and we will provide tips and/or scripts for installing tricky python utilities like cartopy.

Travis and Ben R clarified that conda workflows are not likely to go away.  They are useful for diagnostics, for example.  Fortunately, conda environments are readily configured to be activated and deactivated.  So conda can be deactivated for building and running JEDI applications but activated for other parts of the workflow.  

In our JEDI 1 meeting yesterday (also in the OBS 3 meeting) we also discussed the representation of datetimes in ioda files and in the oops DateTime class. We decided to take an integer epoch+offset approach for the files. The conversion from this to oops native format can be done quickly as a first step. In the longer term, we may wish to change the oops native representation to match the file representation.  Steve further clarified the difference.  The epoch+offset approach is based on netcdf-CF conventions and only requires the storage of a single integer, the offset in seconds from the epoch, the latter of which can be stored as an attribute.  So, it is very efficient to read and write.  The oops native stores two integers, including a Julian date and an offset, which need not be relative to a standard epoch.  Yannick warned to be careful about how we handle operations like file concatenation to make sure they are consistent.  For further details see Ryan's briefing below.

There was then a discussion among Travis, Greg, and Steve about precision.  Is a "seconds-since-epic" approach accurate enough or should it be milliseconds.  It was agreed that it might make sense to express the offset in milliseconds to accommodate more precise timestamps but higher precision than that is unlikely to make a difference.   And, it was agreed that the integers need to be 64-bit to accommodate the time intervals of interest.

Rick and Mark are working on getting JEDI running on NOAA's AWS ParallelWorks platform. The (intel) jedi-stack is installed on the AWS plaform but Rick reported a serious roadblock for ewok and R2D2. The available python installations do not include a develop version so they do not support configuring with CMake and they do not support virtual environments. Futhermore, on the ParallelWorks Azure platform there are problems with their intel modules with regard to the C++-14 (and even C++-11) support that is needed for JEDI. Rick and Mark are working with NOAA sys admins to resolve these issues.

Yongang has identified another performance bottleneck in the unstructured interpolation due to excessive memory usage in the application of the adjoint. He is now looking into addressing this.

We created a new administrative account called jcsda-bot and this is now being used for syncing the internal and public repos. If anyone notices a public repo that is not being properly synced, please let us know. jcsda-bot will also be used for other admin purposes including HPC environment module maintenance. Its benefit over the previous windu-bot is that it is not tied to a particular user; it's tied to the jcsda-it team of Rick, Maryam, and Mark.

Mark M is working with Joel Daves (NCAR), Francois V, Rachel, and Rick to share the R2D2 S3 bucket between our NOAA and USAF AWS accounts. Nearly there.

ops-um-jedi CI is nearly ready to go. We set up the containers and an S3 bucket to host the test data and it's ready for implementation when Maryam returns from PTO next week. Steve S, Francois V Steve V, and Marek are making sure the test data in the bucket is complete and up to date. Steve S will also create a bundle for ops-um-jedi that will facilitate CI implementation.

JEDI 2-3 update from Anna

JEDI2:
No JEDI2 meetings in the last two weeks; no update.

JEDI3:
No JEDI3 meetings in the last two weeks; update based on zenhub:
- Clementine Gas has added a capability to oops to save perturbed observations (in EDA): oops#1400
- Yannick Tremolet issued a bugfix for distance-based interpolation in the unstructured interpolation in oops: oops#1406.

OBS 1 update from Hui

• OBS provided JEDI academy support last week, especially for the first three days
• Now collecting information from in-kinds for Q2 report and also updating instrument table to reflect UFO development and validations for JEDI applications
• Internally, we are summarizing our work for UFO development and identify gaps for JEDI_GDAS. We are having active discussions with EMC and working in progress for those areas
• Will have a working session next week with NRL and UKMet on ZTD operators for NRL, UKMet and GDAS applicationsA
• Adding a new approach to estimate obs error using the three Corner Hat (3CH) method. Currently, it is for RO and can be extended to other observations.
• UKMet developed a QI (Quality Index) processing filter has been added, mapping the “percent confidence” and “generating application” numbers from data providers. ODB processing in ioda signed off for all radiance obs types. RTTOV-SCATT interface nearing completion and RTTOV interface documentation including SCATT has been approved. Ongoing work for other instruments
• NRL started testing UFO using NEPTUNE. Ongoing discussion with OBS team regarding the UFO H(x) results.
• NOAA developed a new AirNow PM2.5 observation operator (upcoming PR). NASA is validating JEDI configurations against GEOSadas/GSI. GSL continues to run NRT aerosol DA using JEDI UFO, including the latest AERONET and absorption AOP development

OBS2 update from Francois V

Work to enable access to R2D2 data from the USAF SSO account (Mark, JEDI)
-Work with JEDI and Met-Office to organize the um-jedi bundle ctest data (Francois)
-Binning tool is almost ready (Travis, SOCA)
-Re-organizing the plotting module code effectively using object oriented design patterns (Rachel)
-Met with NRL to work on setting up yaml files for the Neptune-bundle (Hailing)

OBS 3 update from Ryan (read by Mark)

One announcement:

Two weeks ago, Cory, Steve, and Ryan held an internal mini-sprint to prepare to upgrade our Python ioda-converters to the new ioda-v2 format. Three converters were ported, and the sprint was an overall success. Now, we want to involve a broader group to switch over the remaining Python converters. We have scheduled a code sprint for half-days on Monday to Wednesday, October 25 - 27. Expect a general invite soon. Please consider attending if you are interested in Python converter development, want to keep abreast of changes to converters, or if you generally want to learn how observation data enters JEDI.

General updates from yesterday's OBS3 meeting [notes]:

Wojciech has reported great progress on adding diagnostics flag support to OOPS, IODA, and UFO. There are open pull requests in UFO and IODA that could use review. He also has opened an issue in UFO to discuss the best YAML syntax to use when multiple actions need to be performed on observations flagged by a filter. This is UFO issue #1560, and comments are appreciated.

Praveen and Nick are preparing unit tests for different parts of the nceplibs-bufr converter. Praveen has provided some sample plots validating GSI obs vs bufr2ioda obs for sonde data, and results are beginning to look good.

Steve brought up the issue of how we want to store timestamps in both IODA files and throughout JEDI. We have long desired to move away from storing dates and times as variable-length strings since performance is terrible. We considered several possible solutions, and the general consensus is that we will follow NetCDF's suggestion. In their approach, we use a single variable to store an offset from an epoch time, and the reference time is encoded in the variable as an attribute. The reference remains consistent for all data within a particular ObsSpace, but different ObsSpaces can use different references. All timestamps will remain in UTC time. This format is immediately parse-able by NetCDF utilities like ncdump and other utilities that understand CF-conventions. We also would like to align OOPS to using this storage method so we can avoid error-prone time conversions when reading and writing observation data. This approach can be rapidly adopted in IODA, and our code can remain backward-compatible with the old storage method. Minimal impact is expected for downstream diagnostics users; it should be easy to switch.

The remaining time was spent in a discussion of how groups are used in UFO, and the general flow of observation data in JEDI. The current system is undocumented and somewhat ambiguous. Ryan presented a schematic that led to many comments, and we ran out of time. The discussion will continue at the next OBS3 meeting on October 27, and we hope to involve VarBC and model developers. Please email Ryan if you would like to attend. We hope to see you then.

LAND update from Andy

Water model

The WRF-Hydro team has begun work with Benjamin to develop a B matrix for the snow work, although the first step is implementing the ATLAS interfaces. He’s also helped tidy up our tests so thanks for that.

I’ve converted the converter we use on that project to IODAv2, and James, who wrote the original code, is tidying it up right now, so hopefully there will be a PR related to that shortly

Soren has been doing some profiling on our var executable, and is looking into way to optimize our nearest neighbor routine

UFS

I’ve been in contact with Fabio about bringing in snow depth observations into the GDAS testbed, and have been sorting these out on Orion and testing our very simple H(x) yaml with those observations and backgrounds that are currently being used by GDAS. Looks good, but do need to check how our once daily observations will work in the system.

(Out of date, haven’t met for nearly 2 weeks) Folks at EMC have been working developing a processing pipeline, converter, and QC procedures for GTS snow observations, but coming up with a new site blocked list is proving to be pretty challenging.

Also identified a priority list for existing converters that need updating to v2

SOCA update from Guillaume

Kriti is back from maternity leave 

Guillaume has set up a workflow with Dan in which they can transfer data from the soca obs database on Orion to S3 and then to Discover through R2D2.  This will be useful for future collaborations.

The SOCA team is actively working on the first public release of SOCA, scheduled for Oct 29

Hamideh is working on bringing in infrared data into SOCA

CRTM update from Patrick (Ben was not able to attend the meeting)

CRTM Updates, Oct. 14 2021:
- Updates by Patrick:
o The first tutorial on the series on CRTM instrument coefficient generation took place recently and was focused on MW instruments. Preparations for the second tutorial have begun.
o Further investigation of the new TROPICS coefficients revealed a significant sensitivity towards the spectral resolution of the instrument SRF. So, a version 2 of the coefficients with a higher spectral resolution will likely be released soon.
o A testing dataset for the machine learning version of the CRTM has been computed. The prototype is currently implemented using TensorFlow, but the plan for December is to implement a C++ version. A separate issue is the computation of the operator Jacobians. This is straightforward for a simple neural network, but more general architectures may require an automatic differentiation framework and recommendations/guidelines on this topic are welcome.
o The delivery of the non-LTE code from NOAA NESDIS STAR is overdue by two weeks and work on mitigating the project risk for the associated AOP task has begun.
- Updates from Cheng:
o Cheng has made progress in implementing the reflectance output for the CRTM.
o Cheng is working with researchers from Texas A&M, who have identified an issue with the number of streams for the scattering solver in the CRTM as well as two-stream calculations. Hui Shao would like to be included in the Zenhub issue about this problem.
- Bryan Karpowicz has implemented a channel subset selection feature for pyCRTM and the corresponding PR might clear this week.