Francois H led a discussion about upcoming model interface changes related to Atlas. Francois started with a summary of the changes and impacts on the model interfaces in this presentation:
Highlights from the presentation include:
- Atlas helps by presenting model grids and data in a generic fashion
- Updates impact the Geometry, State and Increment classes in the model interfaces.
- For models that use a grid that does not have a corresponding built-in grid in Atlas, a new application called MeshBuilder has been added that will help with the conversion.
- The primary effort when using MeshBuilder is the construction of the connectivity data structure.
- See slides for details and examples
- A coordinated effort will be required to prepare and merge the PRs for this effort (likely to take place in October, 2023)
- Please see the branches
feature/atlas
in oops/saber/fv3-jedi for the changes in OOPS generic code and model interfaces. These branches are in a mature state but not entirely complete yet.
- Please see the branches
Moving forward we can hold discussions in this GitHub discussion: https://github.com/orgs/JCSDA-internal/discussions/62. Note that there is a link to a design document included in the discussion page.
A great discussion ensued and the meeting was recorded. Contact Phillip Gibbs for gaining access to the recording. Here are highlights of the discussion:
- spack-stack versions of atlas, eckit, fckit, fiat and ectrans will need be upgraded to newer versions (this is being tracked in ZenHub)
- There may be some cases where the new Atlas interface will enable reducing current memory usage (eg, reduction of fieldsets)
- In cases where it is necessary to provide a mapping between x,y space and lat,lon space, the requirement for this mapping may be eliminated.
- This mapping will be covered by the specification of the connectivity
- If you utilize MeshBuilder, for now the only way to pass those results to a subsequent process is to dump the information into a file and have the subsequent process read it out of the file.
- Francois demonstrated using the MeshBuilder with a walk through of a couple cases
- For now, supported polygons include quadrilaterals and triangles.
- Pentagons exist in Atlas but are not utilized in the MeshBuilder
- For now hexagons are not supported in Atlas
- There exists an option to have Atlas compute the connectivity which utilizes Delauney Triangulation
- However, it's not generally recommended to use this option
- Creates a lot of triangles
- Creates rough boundary edges and halo regions
- Not repeatable when the number of MPI tasks changes
- For now, supported polygons include quadrilaterals and triangles.
- Atlas has recently been updated with a fix that follows conservation rules when computing areas
- A fair amount of time was spent on the topic of staggered grids
- For the goal of keeping the algorithms generic in JEDI, the winds need to be at the grid centers.
- Note that the models do not need to change their grids, the moving of the winds to/from the grid centers can be done in the model interface.
- The group thought this was reasonable and do-able, but some concerns were raised about having to do too many interpolations to meet the winds at the grid center requirement
- The updates in the new Atlas interface should be do-able for regional models
- Be careful to treat the lateral boundary conditions differently than the "ghost" points. Ghost points are related to MPI partitioning.
- Francois did an interface for fv3 regional model, and he ended up creating a halo outside of Atlas containing the boundary conditions and then passed in that halo to Atlas.
- Be careful to treat the lateral boundary conditions differently than the "ghost" points. Ghost points are related to MPI partitioning.
- Once the new Atlas interface is in place, there are plans to do optimization work in JEDI which should benefit everyone.
- It was noted that the grid going from Atlas to Saber is currently a point cloud so this will need to be addressed.
- A handy technique for verifying the correctness of a new model Atlas interface is to plot the grid
- Francois used a technique of inserting a call to plot the grid in the JEDI code which can be visually inspected after the run.
- Used a library called gmsh
- See slides for details
- Dan added that we may want to add a scatter plot to the verification methods. This could be useful to see if the right MPI task is picking up the right points.
- It might not be possible to use a more quantitative approach (than plotting with visual inspection) to verify correctness. It's quite difficult to know what correct means.
- Francois used a technique of inserting a call to plot the grid in the JEDI code which can be visually inspected after the run.