View Source

JEDI Weekly Notes from 6/8/2023

I am still new so expect some acronyms and lingo to be wrong :)

Spack-stack: A new python script has been merged into the develop branch of JCSDA/spack-stack which will report to the user duplicate packages after running the concretize step. This is a great addition to the debugging utilities. Thanks to Alexander Richert of NOAA-EMC for this addition!
IODA:

Work continues on the ODB/hdf5 comparison experiment. Thanks to David Davies and Michael Cooke of the Met Office for their help with this effort!
Work continues on the IODA reader improvements - The plan now is to move the completed reader improvements to the skylab-v6 release, but to still have a basic functional reader (containing at least a subset of the planned improvements) in the develop branch by the end of this quarter

CI: Updated unittest build system to work with ecbuild. Tests run fully and pass on first repo, working on pull request code-status updating.
R2D2: optimizations going in for store and fetch, administrative functions added to sync data stores
Near Real Time: An observation ingest workflow was added to EWOK and successfully fetches, converts, and stores gdas gpsro data from ncep ftp into R2D2. More ingest points will be added
Research and development: An AWS cluster was spun up for the observations team to start ROMEX work. Work is being done to use Globus to transfer observation files from NCAR's RDA directly to JCSDA's AWS S3 bucket.

Many ALGO members out, but there are PRs going in
FGAT cost function added recently and can be used for testing with different models
4denvar issues and working with Ricardo to resolve (more below)
VADER PR introduces a new model Traits class for ModelData - discussions to come
Reminder there is an interface change announcements in github JCSDA-internal for discussions and notifications at https://github.com/orgs/JCSDA-internal/discussions

Pyioda converters have a PR to reorganize using a single python path and collaboration with Corey. Ben will reach out to Nancy with change information
Obs-proc meeting highlights:

CRTM v3 testing ongoing and going well, only subtle differences found so far
On track for CRTM v3 for skylab v5
Skylab v5 sensors will include cobrus(?) and temper sms(?)
Will look into testing CRTM with aerosols
Looking to bufr dumps to move away from pre-processing
Snow assimilation updates to processing of those files
Working with NASA and NRL on the GMI and AMSER2(?) files and getting a feed
MetOffice work on profiling code and searching for bottle necks, watch for PRs to address these and help reduce obs space

CRTM 2.4 and CRTM 3 will be available in the spack-stack and can be referenced via “module load”. More discussion to come on how to handle different version of CRTM with the slight differences between the two

See notes from Ben R on CRTM v3 work and testing, differences in output are expected but nothing major showing so far. Note microwave surface channels have the largest difference showing cooler in some locations
AOD issue will hopefully be fixed quickly and a priority

Progress made in GEOS and cycling, a couple small issues to resolve before going into testing with JEDI
Stretch grid progress with HofX
Testing with stretch grid of GHC(?) and goal of cycling
Different state/model benchmarks:

Work on the generic interface for ATLAS is moving forward and in the testing phase. Jerome has suggestions to make it more flexible
Started the discussion with MetPlus with how to incorporate UFO/JEDI

Using VADER with SOCA and all marine related variable transformations (0 right now) but goals of incorporating temperature variables and more
Waiting for Dom to assist with spack-stack integrations and compatibility
Adopting generic oops interpolation from Francios, more UFO PRs will be coming to handle missing values
Goal of next skylab using the generic oops interpolation to assist with post processing
Working on hybrid envar, oasim hofx work,
Discussion with Ricardo on 4denvar issues - does not appear in BUMP but does when including GSI operator

Observing system configuration complete with validation of all obs types using GSI geovals (hofx, counts, and final errors all match). Cleaning up and preparing final PRs to close this out.
Background error model work on 4denvar cost function still being tracked down, getting a hybrid 3dvar cycle going
Coupled DA development is looking at using VADER and refactoring
Up next: transition validated configuration to using cycle system from Ricardo, correlated observation error, varbc for radiances,
Question (Dan): quarterly release of spack-stack prior to skylab release to allow for users to transition and ideally people update when it rolls out
Question (Ricardo): CRTM moved locations or copy or point? If the filename is the same the coefficients will be, but site moved to FTP site for better consistency and older version are also included on that site
Question (Dan): Need to diverge with CRTM, will UFO built with crtm 2.4.0 cause ctest failures? Yes, likely to get tolerance failures and Ben J is happy to assist with issues

Discovered the new spack-stack had issues with hdf5 and was reinstalled by Dom before he left
Working on getting native IODA converts up and running
Science experiment work with 3DVAR FGAT with unusual behavior is being looked into
Bias correction is in for contestants and scan angle but trouble with thickness

Needed to change compiler version to get to work
Ecbuild with debug slows down the tests by a lot! Looks like this is a default setting (look up). The way IODA is compiled causes a big slow down (from 500s to 7000s) and might be left over from testing
Follow up from Yannick: Switched to c++17 in new eckit so compilers will need to be up to date