Francois H led a discussion about upcoming model interface changes related to Atlas. Francois started with a summary of the changes and impacts on the model interfaces in this presentation:
Highlights from the presentation include:
- Atlas helps by presenting model grids and data in a generic fashion
- Updates impact the Geometry, State and Increment classes in the model interfaces.
- For models that use a grid that does not have a corresponding built-in grid in Atlas, a new application called MeshBuilder has been added that will help with the conversion.
- The primary effort when using MeshBuilder is the construction of the connectivity data structure.
- See slides for details and examples
- A coordinated effort will be required to prepare and merge the PRs for this effort (likely to take place in October, 2023)
Moving forward we can hold discussions in this GitHub discussion: https://github.com/orgs/JCSDA-internal/discussions/62. Note that there is a link to a design document included in the discussion page.
A great discussion ensued and the meeting was recorded. Contact Phillip Gibbs for gaining access to the recording. Here are highlights of the discussion:
- spack-stack versions of atlas, eckit, fckit, fiat and ectrans will need be upgraded to newer versions (this is being tracked in ZenHub)
- There may be some cases where the new Atlas interface will enable reducing current memory usage (eg, reduction of fieldsets)
- In cases where it is necessary to provide a mapping between x,y space and lat,lon space, the requirement for this mapping may be eliminated.
- This mapping will be covered by the specification of the connectivity
- If you utilize MeshBuilder, for now the only way to pass those results to a subsequent process is to dump the information into a file and have the subsequent process read it out of the file.
- Francois demonstrated using the MeshBuilder with a walk through of a couple cases
- For now, supported polygons include quadrilaterals and triangles.
- Pentagons exist in Atlas but are not utilized in the MeshBuilder
- For now hexagons are not supported in Atlas
- There exists an option to have Atlas compute the connectivity which utilizes Delauney Triangulation
- However, it's not generally recommended to use this option
- Creates a lot of triangles
- Creates rough boundary edges and halo regions
- Not repeatable when the number of MPI tasks changes
- For now, supported polygons include quadrilaterals and triangles.
- Atlas has recently been updated with a fix that follows conservation rules when computing areas
- A fair amount of time was spent on the topic of staggered grids
- For the goal of keeping the algorithms generic in JEDI, the winds need to be at the grid centers.
- Note that the models do not need to change their grids, the moving of the winds to/from the grid centers can be done in the model interface.
- The group thought this was reasonable and do-able, but some concerns were raised about having to do too many interpolations to meet the winds at the grid center requirement
- The updates in the new Atlas interface should be do-able for regional models
- Be careful to treat the lateral boundary conditions differently than the "ghost" points. Ghost points are related to MPI partitioning.
- Francois did an interface for fv3 regional model, and he ended up creating a halo outside of Atlas containing the boundary conditions and then passed in that halo to Atlas.
- Be careful to treat the lateral boundary conditions differently than the "ghost" points. Ghost points are related to MPI partitioning.
- Once the new Atlas interface is in place, there are plans to do optimization work in JEDI which should benefit everyone.
- It was noted that the grid going from Atlas to Saber is currently a point cloud so this will need to be addressed.
- A handy technique for verifying the correctness of a new model Atlas interface is to plot the grid
- Francois used a technique of inserting a call to plot the grid in the JEDI code which can be visually inspected after the run.
- Used a library called gmsh
- See slides for details
- Dan added that we may want to add a scatter plot to the verification methods. This could be useful to see if the right MPI task is picking up the right points.
- It might not be possible to use a more quantitative approach (than plotting with visual inspection) to verify correctness. It's quite difficult to know what correct means.
- Francois used a technique of inserting a call to plot the grid in the JEDI code which can be visually inspected after the run.