INFRA

• CI testing upgrade (Evan)
• In progress
• Provides testing speedup and improves testing reliability
• Utilizing AWS lambda service
• Runs unit and integration (jedi-bundle for now) test suites
• Development is being done in oops and ioda repositories
• oops and ioda testing appear to be working
• Once testing is working reliably in oops and ioda, the other repos will be incrementally upgraded

• Ingest suites (Ashley)
• Completed initial obs ingest suite
• 4 obs types: AMSU-A, ATMS, GPSRO and Radiosonde (prepBUFR)
• More to come in the future
• In progress developing background ingest suite with assistance from Clementine 
• Conversion of backgrounds to a different resolution
• Not a straightforward process
• We know how to convert resolution on cold start backgrounds
• But, JEDI uses warm start backgrounds
• We are working with Rahul from EMC and Clementine to convert warmstart data to different resolutions and use with JEDI skylab experiments
• Designing a flexible system to handle various DA time windowing schemes
• Assisting OBS team (Greg) with storing feedback files in r2d2
• R2D2 (Eric / Tariq / Evan)
• Argument value validation modules (Tariq)
• Created custom Python value filtering classes
• Descriptive diagnostic output of detected value errors
• Leverages MySQL server validation capabilities
• Integrated cleanly into existing validation methods
• Extensible design makes adding more validation rules a breeze
• Continued optimization of R2D2 MySQL API (Eric and Evan)
• Updated MySQL server query and connection management to optimize resources and balance load on MySQL server on AWS
• Some Skylab and OBS INGEST tasks were timing out due to long-running, sustained open connections
• These sleeping connections were occupying a significant number of available, concurrent MySQL server processes
• Optimization of queries against the R2D2 Data Registry
• The R2D2 Data Registry tracks the physical storage location all ~7M data files 
• Time for calls to store and fetch decreased by 2-3 orders of magnitude
• Actively under development:
• Data store curation updates
• Adding “lifetime” for experiments (Eric and Yannick)
• debug - 2 weeks
• science - 6 months
• publication - 5 years
• release - indefinite
• Delete experiment function (Eric)
• Safely erase all data files and associated metadata for selected experiments 
• Filter by experiment identifier or by “lifetime” value
• Accessible to R2D2 admins and regular end users
• Auditor tool (Tariq)
• Flag inconsistencies in between R2D2’s database and data store contents
• Adding “static forecast” data type for quasi-static MPAS backgrounds (SteveV and Eric)
• Adding “static B” data type (Christian and Eric)
• spack-stack (Steve, covering for Dom)
• spack-stack-1.5.0 release plans
• Target release date is Fri Sep 8, 2023
• Items to be included are being tracked in a github project board as well as a ZenHub sub epic
• https://github.com/orgs/JCSDA/projects/1
• INFRA-53.2: https://github.com/JCSDA/spack-stack/issues/676
• Assisted with testing and debugging of 1.5.0 features (jedi-bundle and skylab on Mac, Orion and S4)
• Assisted with Mac spack-stack build issues
• IODA (Steve)
• Reader improvements - two steps left for completion of the planned io pool based reader
• MPI transfer of data from io pool tasks to non io pool tasks
• Provides a functional reader
• Can select between the current reader and the new reader with YAML control
• Idea is to use this control for upcoming benchmarking and profiling efforts
• PR today
• Pre-process the input ioda file for streamlining the downstream MPI distribution
• Provides an incremental performance enhancement
• Re-arrange obs that go to each MPI task into contiguous blocks that simply get pulled out and sent on to their target MPI task
• PR next week
• AWS Pcluster support (Ashley, Evan)
• Issue with compute nodes quitting after ~½ hour
• Resolved: The R&D cluster is up and running. We have added a couple new JCSDA core users, and eventually all of JCSDA-core will have access
• Discovered that “git lfs install” needs to be run periodically
• Can’t just run it once and forget about it (like on other platforms)
• PR (Evan) is under review containing and automated solution to handle this

ALGO

• Document for atlas <-> jedi model interfaces and plan for the implementation. Presentation by Francois H.

• Work on control-pert EDA, creating workflow for that and preparing for comparison with EIDA. (Clementine, Kate Huxtable (visiting from MetOffice), Yannick).

• Updated generic repos to compile with C++17 (Francois H)

• Merged PR that introduces the reduced format for extended GeoVaLs which unblocks the field-of-view development. (PR by Wojciech Śmigaj)

• Sequential option for 4DEnVar: different time slots now can be either distributed across different MPI ranks, or be on the same MPI rank (Yannick), merge soon (no changes needed in the model interfaces).

• Exploring different communication options in 4DEnVar (for parallel) and ways to optimize serialization (Francois H, MMM).

• Work on 4D covariances in SABER: refactoring to distinguish between different block chains (for static and for ensemble covariances) - work in progress, introduction of FieldSet3D (thin wrapper around atlas::FieldSet that also stores time information) and FieldSet4D - PRs in oops (Anna)

• B matrix training suite development (Christian).

• Continued work on MPAS-Skylab (Steve V, Clementine, Yannick, MMM)

• TODO: ask Francois to create issues for eckit (especially)/atlas updates for spack-stack1.N+1 

COMPO

• A new ioda-converter for TEMPO synthetic data was added
• Work on adding post processing task in EWOK has started
• GEOS-CF forecast PR is under review
• Work on adding tracers to fv3 linear model is in progress

MARINE

• ctests for 4denvar and hbrid4denvar are now added  to soca [ soca PR#936]
• OASIM (Ocean Atmosphere Spectral Irradiance Model ) is now built as part of soca-bundle [soca PR#937 ]
• We added regional and global range test filters for ocean_profile observations in skylab [skylab PR#68 ].
• Working to add more filters as specified in OBSPROC-95

OBS

• Ben and Francois V. had a meeting with GMAO to move NASA’s GNSSRO capability forward into JEDI. The GMAO’s (GSI) HofX is now reproduced. The remaining piece is to unify and consolidate the error inflation method. 

• Lindsey and Sam are testing NOAA’s’ ADP-SFC (surface reports), ADP-UPA (radiosondes and aircraft measurements) BUFR to IODA decoders. Upper air releaseTime is missing in the ADP upper air.

• Sam is reviewing Praveen’s PR (#1321) on the RTMA ADPSFC ioda converters

• Sam is also working on NOAA’s IASI decoder with the help from EMC. She is waiting on input from EMC.

• Lindsey fixed minor bugs in SIMOBS (longitude from 0 to 360, instead of -180,+180)

• Greg has been making good progress on the water Vapor channels with the ABI. 

• Francois V. is working with NRL to finalize their GNSS-RO pathway and the ROPP 1D operator implementation into the NEPTUNE-JEDI bundle.

CRTM

• PR in changes enables polarization angle support like NOAA 21 ATMS. In review. Removes all ecbuild from CRTM. Cmake only with testing harness as well.

• PR In progress for V3 to be a pure cmake build system.

• Notice bug in process with channel subsetting and openmp. OpenMP has seg faults. Anything else works fine. Works fine in debug mode.

• The reason for going to cmake to support all partners and remove legacy build system with autotools. Lots of support options. Will not support legacy build system.

EMC

• No updates.

Met Office

• Global obs processing jedi based all of the must have obs for target date have been imported
• Multi obs jobs going well. ODB database working for all variables.
• DA side model interface work.
• Covariance and EDA PR has been merged.
• SABER doc updated.
• Localization merged into saber develop.
• HTLM had end of last week a small HTLM restructure with PR merged.
• 3dvar cycling successfully with a limited set of obs.
• Obs usage increasing

GMAO

• Going over the changes worked out for observer in HOFX since more recent updates. Almost done for hofx. Maybe more PRs
• Trying to use cycle mode hofx. Problem with YAMLs. Have problems.

Y – We do not have ensemble bkgs for every hour

• Ricardo can get this for us.

ROMS

• 4dvar issues. ROMs with OOPS not the same solution as standalone. Kernel is so complicated.

NRL

• OBS ozone ssmis

• Solver dual resolution implementation. Testing with that

• In house JEDI training

• Mixed success

• Different platforms. JEDI documentation is hard to follow. Video links are broken from academy. We were provided access to jedi edu. Could not use. Could not install docker on any nrl computers.

• Working with modelling folks for fherbert changes

MMM

• no update