These instructions are specifically for NCAR computers derecho and casper.
The following steps describe creating a new grid and using it in a beta version of CESM3. They make use of existing executables on derecho or casper. More general instructions for CESM2.2 are available in the step-by-step guide.
For instructions on connecting to derecho or casper see: https://ncar.github.io/CESM-Tutorial/notebooks/prereqs/prereqs_overview.html
Naming Conventions
$(grid_name) = ne0np4.NAME.ne30xR where NAME is a unique name for your grid, ne30 is for the base resolution, R is for the refinement factor of the highest resolution region. All variable resolution grids with 4 GLL points must begin with ne0np4.
$(grid_label) = NAME_ne30xR
In the examples below, a grid for a small refined region at ne30x8 (ne240, 1/8 deg) centered over Nanjing will be created, using NAME=Nanjing, R=8, so:
- $(grid_name) = ne0np4.Nanjing.ne30x8
- $(grid_label) = Nanjing_ne30x8
Set up a repository for your grid
Create a directory in your /glade/work/$USER/ space for a repository of your created files ($REPO). For example:
- $REPO = /glade/work/emmons/tutorial_Nanjing/ne0np4.Nanjing.ne30x8/
Create a new variable resolution mesh (VRM)
In your repository directory, create a 'grids' subdirectory.
On casper start the VRM Editor from your $REPO/grids directory:
> cd $REPO/grids
> /glade/work/emmons/tutorial_Nanjing/VRM_tools/VRM_Editor/src/VRM_Editor
If you are working on a different computer, download the VRM_tools software by running:
> git clone https://github.com/ESMCI/Community_Mesh_Generation_Toolkit.git
and follow instructions in the documentation.
Set up the base grid
On the ‘VRM’ tab, select:
- Grid Type: CubeSquared
- Base Resolution: 30
- Click Generate VarMesh. See image below for what it should look like.
Adjust Longitude Shift, Rotate-X and Rotate-Y so that your region of interest is centered within a cube face. You can zoom in/out by using the scroll button of your mouse. The right and left arrow keys move the map sideways; up/down to shift the map N/S.
For example, set Longitude Shift to 30 and Rotate-X to 25, then click Generate VarMesh, to produce grid shown below.
When satisfied with the face location, record the Longitude Shift, Rotate-X and Rotate-Y values (you will need them later).
Create draft refinement region
The default resolution of the Editor map is a bit coarse, so read in a higher resolution map. In the "Actions" menu (upper left), select Read Refinement Map.
Open /glade/work/emmons/Community_Mesh_Generation_Toolkit_Nov23/VRM_tools/VRM_Editor/src/REFMAP_1440x720.nc. For easier access, copy this file to your working directory.
On the ‘Edit’ tab, click ‘Edit refine Map’.
Select ‘Polygon Editor’ - a green box appears over the Pacific. Drag and adjust the 8 points to create a refinement region. Use the scroll button of your mouse to zoom in and out, and the arrow keys to shift the map. Try to have the polygon borders be parallel to the grid lines.
Click ‘Apply’ (with Value =1.00)
To save this, click 'Exit Edit Mode' and 'Yes' to save.
Go to 'Display' tab, check Refinement Map to see the refined region as a red box.
Go back to ‘VRM’ tab. For best results (in many cases), the following settings are recommended:
- Refine Type = LOWCONN
- Select Refine Level (1 = ~0.5 deg, 2=0.25 deg, 3=⅛ deg=14km, …). Resolution in the refined region is the base grid resolution divided by 2^(refinement level)
- Smoothing Options: Spring, with Iterations = 3, Length = 3
Click 'Generate VarMesh'. This case has Refine Lev = 3, to give a refined mesh with 1/8 degree resolution at center:
The LOWCONN setting uses templates that span 2x2 base elements to transition between resolutions, and the SPRING smoothing rounds out the element shapes to reduce sharp angles. There may be situations when other settings give better results.
Uneven edges might be removed with small adjustments (0.1) to Longitude Shift or Rotate-X, -Y values.
It is best to have a couple of rows at each intermediate resolution. This can be done by making a 'halo' in the Editor: make the polygon a bit larger than the refined region, and set the value at the appropriate fraction. For an ne30 base grid with Refine Level=3 (ne240), the halo region surrounding the finest refinement (at ne120 resolution) should have a Value=0.66.
Then a second halo, at ne60 resolution, can be created with Value=0.33.
If making a refined region with the finest resolution ne120, then make one halo with Value=0.5.
When satisfied with the halos, Exit Edit Mode, Yes to save, and go back to VRM tab. With the same settings as above (LOWCONN, Refine lev =3, etc.), click Generate VarMesh:
The halos can be adjusted repeatedly in the Edit menu until satisfied with the grid. Once the grid is at least close to what you want proceed to steps below.
Save the Refinement Map - under the Actions menu. This writes a netcdf file of the map of refinement values (0 for no refinement, 1 for maximum refinement). Save the files as something like: REFMAP_Nanjing_ne30x8.nc.
Save frequently as the VRM Editor on the Mac tends to crash. You can then Read the refinement later and start adjusting the grid from that point.
Also, make note of any Longitude Shift and Rotate-X, -Y values as those are not saved in the Refinement map file.
If you are happy with your grid, under Actions: Write Exodus File - give it a name like Nanjing_ne30x8_EXODUS.nc. If you want to make further manual edits to your grid, do not save the EXODUS file.
Write Refinement Grid - for manual editing of refinement region and halos
If you have trouble making a grid with smooth borders and transitions around the refinement region, you can save the grid as a text file and manually edit it.
Once you have a rough version of your grid, save a Refinement Grid - in the Actions menu, select 'Write Refinement Grid'.
This is a text file similar to the Refinement Map, so name it something like: RefGrid_Nanjing_ne30x8.dat
Record the Longitude Shift, Rotate-X and Rotate-Y values.
Then edit the RefGrid file to have straight edges for each refine level. For a refinement factor of 8 (refine level=3) the file has 0 for no refinement and 1, 2, 3, etc. for each refine level.
This example shows irregular borders, and different width halos:
After editing, a clean grid template looks like this:
Use Create_VRMgrid to create EXODUS file from Refinement Grid
The command line ‘Create_VRMgrid’ will create an EXODUS file from the updated Refinement Grid file.
> /glade/work/emmons/tutorial_Nanjing/VRM_tools/VRM_Editor/src/Create_VRMgrid
--refine_type "LOWCONN" {or "CUBIT"}
--grid_type "CubeSquared"
--resolution 30
--refine_level {refinement [1,2,3 …]}
--smooth_type "SPRING" --smooth_dist 3 --smooth_iter 3
--x_rotate {xrot_value} --y_rotate {yrot_value} --lon_shift {lonshift_value}
--refine_file REFMAP_{yourLabel}_{resolution}.nc
--refine_cube RefGrid_{yourLabel}_{resolution}.dat
--output {yourLabel}_{resolution}_EXODUS.nc
(be sure this is all on one line)
For this example: /glade/derecho/scratch/emmons/nanjing_musica_tutorial> /glade/work/emmons/tutorial_Nanjing/VRM_tools/VRM_Editor/src/Create_VRMgrid --refine_type "LOWCONN" --grid_type "CubeSquared" --resolution 30 --refine_level 3 --smooth_type "SPRING" --smooth_dist 3 --smooth_iter 3 --x_rotate 25 --y_rotate 0 --lon_shift 30 --refine_file REFMAP_Nanjing_ne30x8.nc --refine_cube RefGrid_Nanjing_ne30x8.dat --output Nanjing_ne30x8_EXODUS.nc
Plot EXODUS file with /glade/u/home/emmons/tutorial_Nanjing_notebooks/Plot_exodus_grid.ipynb
Create grid files
Several types of grid files need to be created. These grids are described on the overview page.
Create SCRIP and LATLON grid files from EXODUS file
On casper:
> cd $REPO/grids
> cp /glade/work/emmons/tutorial_Nanjing/VRM_tools/VRM_ControlVolumes/src/input.nl input-Nanjing.nl
Edit input-Nanjing_ne30x8.nl to have your path to $REPO/grids and the grid name. Then run Gen_ControlVolumes:
> /glade/work/emmons/tutorial_Nanjing/VRM_tools/VRM_ControlVolumes/src/Gen_ControlVolumes.exe input-Nanjing.nl > LOG_Nanjing
This produces Nanjing_ne30x8_np4_SCRIP.nc and Nanjing_ne30x8_np4_LATLON.nc
Examine the LATLON file to get the number of columns in your new grid:
> ncdump -h Nanjing_ne30x8_np4_LATLON.nc
This prints (among other things):
ncol = 60482 ;
Create ESMF mesh file from SCRIP file
On casper, load needed modules:
> module load mpi-serial/2.3.0
> module load esmf/8.5.0
To find the path to ESMF_Scrip2Unstruct, run:
> module show esmf/8.5.0
Find the listing for the "PATH" directory:
"PATH","/glade/u/apps/casper/23.10/spack/opt/spack/esmf/8.5.0/mpi-serial/2.3.0/oneapi/2023.2.1/dfkx/bin"
In the directory with your SCRIP file, run:
> /glade/u/apps/casper/23.10/spack/opt/spack/esmf/8.5.0/mpi-serial/2.3.0/oneapi/2023.2.1/dfkx/bin/ESMF_Scrip2Unstruct Nanjing_ne30x8_np4_SCRIP.nc Nanjing_ne30x8_np4_MESH.nc 0
Note the 0 at the end.
This creates the ESMF mesh file: Nanjing_ne30x8_np4_MESH.nc
Generate CESM input files on new grid
To keep with previous conventions for the structure of your new grid repository create directories inic, atmsrf and topo in your repository:
> cd $REPO
> mkdir inic
> mkdir maps
> mkdir atmsrf
> mkdir topo
The files created below are generally labeled with today's date. Even if you make files on different days, all the files in one repository should have the same date (YYMMDD).
Regrid CAM IC file
The resulting file is assigned to ncdata in user_nl_cam.
> cd $REPO/inic
> cp /glade/work/emmons/tutorial_Nanjing/VRM_tools/gen_CAMncdata/TEMPLATES/interpic_script_TEMPLATE.sh interpic_script_Nanjing.sh
> vi interpic_script_Nanjing.sh
Edit the script to point to your grid files.
# USER CHANGES
VRdate="YYMMDD"
VRgridName="ne0np4.NAME.ne30xR"
VRgridLabel="NAME_RESOL"
VRrepoPath="your_repo_path"
# end of USER CHANGES
On casper:
> module load nco
> module load ncl
> qcmd -- 'sh interpic_script_Nanjing.sh > log_Nanjing'
Check there are no errors in the log file. This should have created an ic file with the dimensions of your new grid: cami-mam4_0000-01-01_ne0np4.Nanjing.ne30x8_L32_c240809.nc
Regrid 'atmsrf' file
The resulting file is assigned to drydep_srf_file in user_nl_cam.
Copy the template script to your working directory and edit for your grid. Be sure directory $REPO/maps exists.
> cd $REPO/atmsrf
> cp /glade/work/emmons/tutorial_Nanjing/VRM_tools/gen_atmsrf/TEMPLATES/gen_atmsrf_TEMPLATE.ncl gen_atmsrf_Nanjing.ncl
> vi gen_atmsrf_Nanjing.ncl
> module load ncl
> module load esmf
> qcmd -- 'ncl gen_atmsrf_Nanjing.ncl > log_Nanjing'
Check there are no errors in the log file. This writes atmsrf_ne0np4.Nanjing.ne30x8_240809.nc.
Create Topography file
The resulting file is assigned to bnd_topo in user_nl_cam.
> qcmd -l walltime=12:00:00 -l select=1:ncpus=1 -- /glade/work/emmons/Topo_derecho/cube_to_target/cube_to_target --rrfac_manipulation --grid_descriptor_file='/glade/work/emmons/tutorial_Nanjing/ne0np4.Nanjing.ne30x8/grids/Nanjing_ne30x8_np4_SCRIP.nc' --intermediate_cs_name='/glade/campaign/cgd/amp/pel/topo/cubedata/gmted2010_modis_bedmachine-ncube3000-220518.nc' --output_grid='Nanjing_ne30x8' --rrfac_max=8 --smoothing_scale=100.0 -u 'Louisa Emmons, emmons@ucar.edu' > out_Nanjing