• Maryam is updating the CI test containers with the latest version of the NOAA-EMC NCEPLIBS-bufr library. This is being done to support the EMC BUFR converter, and should be complete soon.
• Maryam gave a presentation at this week's UCAR Software Engineering Assembly (SEA) titled "Continuous Integration in JEDI and WRF". It was well received.
• Work is underway to migrate to the latest releases of the ECMWF's ecbuild, eckit, fckit and atlas libraries.
• A JEDI software stack was successfully built on Summit.
• A PR will be submitted soon that will implement a generalized compare methodology. This PR is a replacement for a prior PR (which was reverted) using a regex package that inadvertently broke the build on Hera.

JEDI2 Meeting — March 24, 2021

Agenda:

1. JEDI2 in AOP21
2. Updates and questions
3. ZenHub boards and roadmap

Updates:

Ryan:

• Remaining AOP20 areas of work:
  • Files converted, tests pass in ioda and ufo
  • Converters need to be updated
  • Performance assessment and fixes (pre-release and AOP21)
  • Python API updates
  • Re-introduce error system
  • Add YAML links to factory constructors (this week)
• PR reviews and fixes
• Generic Filters
• Bias correction
• ioda v2 (in progress; April)
• odb, bufr engines (now - end of Q2 AOP21)
• AOP21:
  • Multiple QC variables to supplement pass-fail flag, multiple data types, better derived variable storage, native use of multidimensional variables in filters, migrate ObsDataVectors to ObsGroups.
• Variable name conventions, validation, and website (OBS3 and JEDI2)
• IODA converter development. (OBS3)
• BUFR, ODB backends. (OBS3)
• R2D2 integration with IODA and diagnostics. (OBS3, OBS2, JEDI4)
• IODA development (JEDI1, OBS3)
• Many UFO filter enhancements. (OBS3)
• Continued improvements to generic filters. (OBS3)
• GeoVaLs in C++ (JEDI2)
• Locations with field of view (JEDI2)
• OBS instrument code sprints. (OBS1, OBS3)
• VarBC (JEDI3)

Steve:

• Multiple PR’s merged into feature/ioda-v2 that fix issues in the file upgrader.
• All ufo and ioda tests are passing on feature/ioda-v2 branches with one exception
• There is an intermittent failure of test_ufo_function_linear_combination where it gets a floating point exception. Turning off trapping of floating point exceptions results in the test passing. This needs to be tracked down and fixed, but we are probably ready to try performance evaluation.
• PR merged into jedi-stack that builds new version of NCEPLIBS-bufr which is needed by the EMC bufr converter.
• Maryam is helping get the CI test containers updated with this new NCEPLIBS-bufr version and is close to having this completed.
• git lfs versioning of test data into ioda-data and ufo-data.

JEDI3 meeting - March 22, 2021

• Steve Vahl has joined the JEDI 3 team and will have the role of liaison with the Met Office
  • Also will be helping with variable change code development

• Yannick is developing performance enhancements for H(x) operation
  • Working in the context of the H(x) testbed (JEDI GDAS experiments)
  • Has 3D H(x) improvements merged into develop
  • Next step is to work on 4D H(x) which will impact the model interfaces
• Static B development
  • Full ensemble, GFS (same as GSI ensemble)
  • Full ensemble, GEOS
  • Static B is trained from latitude points
• EMC development
  • NUOPC interface for UFS is work in progress
  • FV3 LAM system
    • Doing science cycling
    • UFO validation
• Met Office development
  • B Matrix deliverable is complete
  • Replication of OPS
    • H(x) calculations
    • Atlas based implementation
• NRL development
  • Refactoring on Neptune is complete
  • Integrating Yannick's code for faster H(x)
• Oak Ridge
  • FV3 JEDI is running on Summit
• AOP 20 is ending, please help clean up the ZenHub tickets

JEDI4 meeting - March 22, 2021

New team member: Steve Vahl (UKMO liaison in the JEDI - Model interfaces team).

Updates:

Anna: Working on refactoring observation-space Localization (used in LETKF-type algorithms) to allow for features required by ocean and snow DA. Also planning to use the updated ObsLocalization interface for local observation search (moving that functionality out of ObsSpace second constructor).

Benjamin: The fv3-jedi static B matrix training is now running on an AWS parallel cluster, using Dan's ensemble on S3. Also helped Guillaume to setup a similar B matrix training for SOCA.

Clementine: merged diagnostics for Lanczos; working on fv3-jedi diagnostics. 

Dan: Working on using lagged pairs; PR in oops and saber. Reading increments from disc instead of states. 

Eric: NRL converters (innovations data -> nc4). Preparing for cycling by end of April (3DVar)

Nancy: Experiencing a problem with 3DVar (4DVar obs data set has problems, too dense obs for 3DVar). Unclear why 3DVar crashes yet.

JJ: working on variable change for model->geovals.

Junmei: working on mpas-jedi cycling with bias correction & QC filters.

Wei: working with Sergey on Halo distribution. Found a bug when there are zero obs on an MPI task; issued bugfix PR in ioda. Plan to run separate observer and solver for the experiments

Yannick: Working on how we’re passing ObsErrors between Filters and everything else; PR coming soon. Same with QC filters: working on an obvious way the QC flags are passed to the ObsOperator.

Olly: created a PR in ECMWF Atlas that implemented a repartitioner from one PE distribution to other. Currently it works for structured meshes.

Marek: have been continuing on the spectral transform work for a spectral B. I have managed to get the code to do a low-pass filter.  Now I am taking a two pronged approach to getting a functioning adjoint of an inverse transform.  It should be possible to rescale the direct transform (grid-point to spectral) to simulate the adjoint of the inverse transform. That is one approach.  The other is to allow atlas to use the adjoint of the inverse transform in atlas.  That requires changes to transi and well as atlas.  That is the other approach that I am implementing.

The PR in um-jedi that reads the spectral coefficients from an operational covariance file has been implemented in um-jedi.

• Work flow development
  • Archived bias correction files in R2D2
    • Will be able to use these for this weekend's run
  • Adding UFS forecast to EWOK
  • Decoupled diagnostics process from main flow
    • This speeds up the flow
• NOAA Data Lake
  • Generic method for creating queue templates (ie, the templates needed for extracting data)
  • Not quite fully automated yet, getting close
• S4
  • Able to build the flow stack
  • Have generic user account which can access the flow stack

• GNSSRO Zenith Total Delay (ZTD) UFO Code Sprint with UK Met-Office and NRL
• HofX improved performance testing with Ewok.
• Netcdf HofX files concatenation becomes a bottleneck.

• New obs function was developed for SSMIS CLW predictor, which leads to the full representation of GDAS-SSMIC Bias correction. The JEDI runs for SSMIS BC now match GDAS's.  Working with ESRL on validating the conventional UFO for regional applications
• IODA converter update to allow adding additional global attributes
• New ground GNSS Zenith Total Delay operator added from NRL (based on ROPP refractivity operator). UKMet ZTD operator was adapted and tested for GDAS. Reasonable results were generated for both. 
• UFO team has been performing diagnostics for weekly JEDI GDAS testbed, identifying the cause of the discrepancy with GDAS references and providing feedback to diagnostic tool development. 

• Working with EMC for new reference data sets. This will help to limit the differences due to background temporal interpolation. 

• Regional ocean DA: Static B and obs filtering (Kriti, Guillaume, Travis)
• JEDI-GODAS/NG-GODAS: To be tagged by the end of the month. Candidate configuration currently in review (JCSDA+EMC)
• Better B for soca (Benjamin, Guillaume, Travis)
• R2D2-ificaton of the soca obs "database" (Rahul)
• Nightly stable branch for soca-science (Travis)
• LETKF with sea ice obs (MinJeong)
• Investigation of MW Radiance DA from soca using the LETKF (Hamideh)

• I’ve been keeping up with 3d H(x) changes, and cleaning up tickets at the end of the AOP cycle.

• Had meeting with our various partners beyond NOAA updating them about the new AOP and making some final tweaks

• Would like to thank Anna for finding the time to work on the observation space localization with Sergey, this is really critical to our plans at the moment

• Would like to ask though who would be a good person to consult about optimizing one of our ioda-converters - IMS snow cover to be precise - which is running more slowly, and producing much larger files than we expect given past experience.

  • Steve H volunteered to help