Versions Compared

Old Version 29

changes.mady.by.user Louisa Emmons

Saved on

compared with

New Version Current

changes.mady.by.user Louisa Emmons

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: SST file through 2024

These instructions are specifically for NCAR computers derecho and casper.

The following steps describe creating a new grid and using it in a beta version of CESM3.  Instructions for CESM2.2 are available in the step-by-step guide.

...

> cd ${VRM_tools}/VRM_Editor/src
> module load gcc/12.2.0  
> module load ncarenv/23.10
> qmake VRM_Editor.pro
> make

Running "make" produces a lot of warning messages; that’s ok.  For Linux users, with the Qt and netCDF packages installed, building the editor should be as simple as for Casper.

Correct the file Create_VRMgrid.cpp:

Change:

       CommandLineDouble (MyGridYRotate       ,"x_rotate"      ,0.0);

To: 

       CommandLineDouble (MyGridYRotate       ,"y_rotate"      ,0.0);

Build Create_VRMgrid on Casper: 

...

The resulting file is assigned to ncdata in user_nl_cam.

> cd $REPO/inic
> cp /glade/work/emmons/tutorial_Nanjing/VRM_tools/gen_CAMncdata/TEMPLATES/interpic_script_TEMPLATE.sh interpic_script_Nanjing.sh
> vi interpic_script_Nanjing.sh

Edit the script to point to your grid files.  This template includes the path to an existing executable of interpic, which is available in the CESM source code.

# USER CHANGES
VRdate="YYMMDD"
VRgridName="ne0np4.NAME.ne30xR"
VRgridLabel="NAME_RESOL"
VRrepoPath="your_repo_path"
# end of USER CHANGES

On casper:

This script uses the interpic program provided in CESM source code, but must first be compiled.  If you do not already have CESM code, see "Set up CESM3" below.

On casper, in CESM source code, ./components/cam/tools/interpic_new:

Edit Makefile:
l.15 from: LIB_NETCDF := /usr/local/lib
     to: LIB_NETCDF := $(NETCDF)/lib
l.18 from: INC_NETCDF := /usr/local/include
     to: INC_NETCDF := $(NETCDF)/include
l.99 from: LDFLAGS = -L$(LIB_NETCDF) -lnetcdf
     to: LDFLAGS = -L$(LIB_NETCDF) -lnetcdff -lnetcdf  

Currently Loaded Modules:
  1) ncarenv/23.10 (S)   2) intel/2023.2.1   3) ncarcompilers/1.0.0   4) hdf5/1.12.2   5) netcdf/4.9.2

> gmake

This should have created the executable ‘interpic’.  Alternatively, the instructions below use an existing executable available on casper.

> cd $REPO/inic
> cp /glade/work/emmons/tutorial_Nanjing/VRM_tools/gen_CAMncdata/TEMPLATES/interpic_script_TEMPLATE.> module load nco
> module load ncl
> qcmd -- 'sh interpic_script_Nanjing.sh
> >vi loginterpic_script_Nanjing'

Check there are no errors in the log file. This should have created an ic file with the dimensions of your new grid: cami-mam4_0000-01-01_ne0np4.Nanjing.ne30x8_L32_c240809.nc

Regrid 'atmsrf' file

The resulting file is assigned to drydep_srf_file in user_nl_cam.

Copy the template script to your working directory and edit for your grid. Be sure directory $REPO/maps exists.

> cd $REPO/atmsrf
> cp /glade/work/emmons/tutorial_Nanjing/VRM_tools/gen_atmsrf/TEMPLATES/gen_atmsrf_TEMPLATE.ncl gen_atmsrf_Nanjing.ncl
> vi gen_atmsrf_Nanjing.ncl
> module load ncl
> module load esmf
.sh


Edit the script to point to your grid files.  This template includes the path to an existing executable of interpic, which is available in the CESM source code. 

# USER CHANGES
VRdate="YYMMDD"
VRgridName="ne0np4.NAME.ne30xR"
VRgridLabel="NAME_RESOL"
VRrepoPath="your_repo_path"
# end of USER CHANGES

On casper:

> module load nco
> module load ncl
>> qcmd -- 'nclsh geninterpic_atmsrfscript_Nanjing.nclsh > log_Nanjing'

Check there are no errors in the log file. This writes atmsrf_ne0np4.should have created an ic file with the dimensions of your new grid: cami-mam4_0000-01-01_ne0np4.Nanjing.ne30x8_240809L32_c240809.nc.

Create Topography file

The resulting file is assigned to bnd_topo in user_nl_cam.

The following steps get the Topo software (https://github.com/NCAR/Topo), build the cube_to_target executable and run it. Documentation for Topo is available on its wiki page.

On casper:

On your own computer, you can find ${VRM_tools}/gen_CAMncdata/TEMPLATES/interpic_script_TEMPLATE.sh, and use the interpic executable that you created.  You can start with any *.cam.i.* file that you have available.

Regrid 'atmsrf' file

The resulting file is assigned to drydep_srf_file in user_nl_cam.

On casper: copy the template script to your working directory and edit for your grid. Be sure directory $REPO/maps exists.

> cd $REPO/atmsrf
> cp /glade/work/emmons/tutorial_Nanjing/VRM_tools/gen_atmsrf/TEMPLATES/gen_atmsrf_TEMPLATE.ncl gen_atmsrf_Nanjing.ncl
> vi gen_atmsrf_Nanjing.ncl
> module load ncl> cd $REPO
> git clone https://github.com/NCAR/Topo.git Topo
> cd Topo
> cd cube_to_target
> module load gccesmf
> module list

Currently Loaded Modules:

  1) ncarenv/23.10 (S)   2) ncl/6.6.2   3) gcc/12.2.0   4) ncarcompilers/1.0.0   5) hdf5/1.12.2   6) netcdf/4.9.2   7) cuda/12.2.1   8) ucx/1.14.1   9) openmpi/4.1.6  10) esmf/8.5.0

> gmake -f Makefile clean
> gmake -f Makefile

Put following in a script or be sure entire command is on one line:

> qcmd -l walltime=12:00:00 -l select=1:ncpus=1 -- ./cube_to_target --rrfac_manipulation
--grid_descriptor_file='/glade/work/emmons/tutorial_Nanjing/ne0np4.Nanjing.ne30x8/grids/Nanjing_ne30x8_np4_SCRIP.nc'
--intermediate_cs_name='/glade/campaign/cgd/amp/pel/topo/cubedata/gmted2010_modis_bedmachine-ncube3000-220518.nc'
--output_grid='Nanjing_ne30x8' --rrfac_max=8 --smoothing_scale=100.0 -u 'Louisa Emmons, emmons@ucar.edu' > out_Nanjing

Modify the bold text to point to your SCRIP file, grid name, etc.

Can add option --output_data_directory to specify location of final topography file (be sure it already exists).

This can take several hours.  

Generate CTSM (CLM) surface datasets

On derecho, use CTSM5.2 source code to create the needed surfdata and landuse_timeseries files.

 qcmd -- 'ncl gen_atmsrf_Nanjing.ncl > log_Nanjing'

Check there are no errors in the log file. This writes atmsrf_ne0np4.Nanjing.ne30x8_240809.nc. 

On your own computer: find ${VRM_tools}/gen_atmsrf/TEMPLATES/gen_atmsrf_TEMPLATE.ncl

Create Topography file

The resulting file is assigned to bnd_topo in user_nl_cam.

The following steps get the Topo software (https://github.com/NCAR/Topo), build the cube_to_target executable and run it. Documentation for Topo is available on its wiki page.

On casper:

> cd $REPO
> git clone https://github.com/NCAR/Topo.git Topo
> cd Topo
> cd cube_to_target
> module load gcc
> module list

Currently Loaded Modules:

  1) ncarenv/23.10 (S)   2) ncl/6.6.2   3) gcc/12.2.0   4) ncarcompilers/1.0.0   5) hdf5/1.12.2   6) netcdf/4.9.2   7) cuda/12.2.1   8) ucx/1.14.1   9) openmpi/4.1.6  10) esmf/8.5.0

> gmake -f Makefile clean
> gmake -f Makefile

Put following in a script or be sure entire command is on one line:

> qcmd -l walltime=12:00:00 -l select=1:ncpus=1 -- ./cube_to_target --rrfac_manipulation
--grid_descriptor_file='/glade/work/emmons/tutorial_Nanjing/ne0np4.Nanjing.ne30x8/grids/Nanjing_ne30x8_np4_SCRIP.nc'
--intermediate_cs_name='/glade/campaign/cgd/amp/pel/topo/cubedata/gmted2010_modis_bedmachine-ncube3000-220518.nc'
--output_grid='Nanjing_ne30x8' --rrfac_max=8 --smoothing_scale=100.0 -u 'Louisa Emmons, emmons@ucar.edu' > out_Nanjing

Modify the bold text to point to your SCRIP file, grid name, etc.

Can add option --output_data_directory to specify location of final topography file (be sure the directory already exists).

This can take several hours.  

Generate CTSM (CLM) surface datasets

On derecho, use CTSM5.3 source code to create the needed surfdata and landuse_timeseries files. (Latest tag is good to use, as of 1/25/25 it is ctsm5.3.020).

> git > git clone https://github.com/ESCOMP/CTSM ctsm5.2.0073
> cd ctsm5.2.0073
> git checkout ctsm5.23.0070
> bin/git-fleximod update

...

Run the make namelist script (modify the bold text for your resolution, MESH file and your ncol for model-mesh-nx)

(npl) > /glade/work/emmons/cesm_src_derecho/ctsm5.2.007/tools/mksurfdata_esmf/gen_mksurfdata_namelist --res Nanjing_ne30x8 --start-year 1979 --end-year 2026 --ssp-rcp SSP3-7.0 --model-mesh /glade/work/emmons/tutorial_Nanjing/ne0np4.Nanjing.ne30x8/grids/Nanjing_ne30x8_np4_MESH.nc --model-mesh-nx 60482 --model-mesh-ny 1

...

This creates the netcdf files that will be used in your simulation, specified in user_nl_clm: fsurdat = '/repo_path/land/surfdata_ ... .nc' and flanduse_timeseries = '/repo_path/land/landuse.timeseries_ ... .nc'.  In your $REPO, create a directory 'land' (mkdir $REPO/land) and copy these 2 nc files to it.

> conda deactivate

Anchor
Add grid to CESM
Add grid to CESM
Set up CESM3 with new grid

On Updated June 2025: On derecho, checkout a CESM3 beta tag (that uses ctsm5.2).  It is recommended the source code is cloned into your work (or scratch, if necessary; you might run out of space in home) directory where sufficient storage is available. CESM3 has replaced manage_externals with git-fleximod, which requires the following commands:

> git clone https://github.com/ESCOMP/CESM.git clone https://github.com/ESCOMP/CESM.git cesm3_0_beta01
> cd cesm3_0_beta01
> git checkout cesm3_0_beta01
> ./bin/git-fleximod update

...

 cesm3_0_beta01
> cd cesm3_0_beta01
> git checkout cesm3_0_beta01
> ./bin/git-fleximod update

Below, $CESMCODE refers to yourcesm3_0_beta01 directory.

CCS_CONFIG

1. Add an entry for your new ESMF mesh file in <cesm>/ccs_config/component_grids_nuopc.xml

For this grid:

  <domain name="ne0np4.Nanjing.ne30x8">
    <nx>60482</nx> <ny>1</ny>
    <mesh>/glade/work/emmons/tutorial_Nanjing/ne0np4.Nanjing.ne30x8/grids/Nanjing_ne30x8_np4_MESH.nc</mesh>
    <desc>ne0np4.Nanjing.ne30x8 is a Spectral Elem 1-deg grid with a 1/8 deg refined region over east China:</desc>
    <support>Test support only</support>
  </domain>

2. Add a grid alias entry in <cesm>/ccs_config/modelgrid_aliases_nuopc.xml

  <model_grid alias="ne0Nanjingne30x8_ne0Nanjingne30x8_mt12" not_compset="_POP">
    <grid name="atm">ne0np4.Nanjing.ne30x8</grid>
    <grid name="lnd">ne0np4.Nanjing.ne30x8</grid>
    <grid name="ocnice">ne0np4.Nanjing.ne30x8</grid>
    <mask>tx0.1v3</mask>
  </model_grid>

Note the “grid name” must match the ”domain name”  in

CCS_CONFIG

1. Add an entry for your new ESMF mesh file in <cesm>/ccs_config/ component_grids_nuopc.xml

For this grid:

  <domain name="ne0np4.Nanjing.ne30x8">
    <nx>60482</nx> <ny>1</ny>
    <mesh>/glade/work/emmons/tutorial_Nanjing/ne0np4.Nanjing.ne30x8/grids/Nanjing_ne30x8_np4_MESH.nc</mesh>
    <desc>ne0np4.Nanjing.ne30x8 is a Spectral Elem 1-deg grid with a 1/8 deg refined region over east China:</desc>
    <support>Test support only</support>
  </domain>

2. Add a grid alias entry in <cesm>/ccs_config/modelgrid_aliases_nuopc.xml

  <model_grid alias="ne0Nanjingne30x8_ne0Nanjingne30x8_mt12" not_compset="_POP">
    <grid name="atm">ne0np4.Nanjing.ne30x8</grid>
    <grid name="lnd">ne0np4.Nanjing.ne30x8</grid>
    <grid name="ocnice">ne0np4.Nanjing.ne30x8</grid>
    <mask>tx0.1v3</mask>
  </model_grid>

Note the “grid name” must match the ”domain name”  in component_grids_nuopc.xml.

...

.

Anchor
Run CAM
Run CAM
Set up a CAM case for testing new grid

On derecho, go to the directory (or create one) for all your CESM cases (in your /glade/u/home/$USER or /glade/work/$USER directory). Create a new case with FHIST compset:

> /glade/work/emmons/cesm_src_derecho/cesm3_0_beta01/cime/scripts/create_newcase --case /glade/work/emmons/tutorial_Nanjing/cases/f.e3beta01.FHIST.Nanjing_ne30x8.01 --res ne0Nanjingne30x8_ne0Nanjingne30x8_mt12 --compset FHIST --run-unsupported --project AACD0004 --pecount 2048

This creates a new directory, $CASEROOT = /glade/work/emmons/tutorial_Nanjing/cases/f.e3beta01.FHIST.Nanjing_ne30x8.01.

> cd $CASEROOT
> ./case.setup

Add to user_nl_cam:

 ncdata = '/glade/work/emmons/tutorial_Nanjing/ne0np4.Nanjing.ne30x8/cami-mam4_0000-01-01_ne0np4.Nanjing.ne30x8_L32_c240809.nc'
 bnd_topo = '/glade/work/emmons/tutorial_Nanjing/ne0np4.Nanjing.ne30x8/topo/topo/Nanjing_ne30x8_gmted2010_modis_bedmachine_nc3000_Laplace0100_noleak_20240729.nc'
 drydep_srf_file = '

On derecho, go to the directory (or create one) for all your CESM cases (in your /glade/u/home/$USER or /glade/work/$USER directory). Create a new case with FHIST compset:

> /glade/work/emmons/cesm_src_derecho/cesm3_0_beta01/cime/scripts/create_newcase --case /glade/work/emmons/tutorial_Nanjing/cases/fne0np4.e3beta01.FHISTNanjing.ne30x8/atmsrf/atmsrf_ne0np4.Nanjing.ne30x8_ne30x8.01 --res ne0Nanjingne30x8_ne0Nanjingne30x8_mt12 --compset FHIST --run-unsupported --project AACD0004 --pecount 2048This creates a new directory, $CASEROOT = 240809.nc'
 se_refined_mesh = .true.
 se_mesh_file = '/glade/work/emmons/tutorial_Nanjing/cases/fne0np4.e3beta01.FHIST.Nanjing.ne30x8/grids/Nanjing_ne30x8_EXODUS.01.

> cd $CASEROOT
> ./case.setup

Add to user_nl_cam:

nc'

 inithist = 'DAILY'


Set the timestep based on recommended value in: https://github.com/ESMCI/Community_Mesh_Generation_Toolkit/blob/master/VRM_tools/Docs/CAM-tsteps-inic-for-newgrids_v0.pdf

./xmlchangeATM_NCPL=384


Add to user_nl_clm:

fsurdat  ncdata = '/glade/work/emmons/tutorial_Nanjing/ne0np4.Nanjing.ne30x8/cami-mam4_0000-01-01_ne0np4.Nanjing.ne30x8_L32_c240809.nc'
 bnd_topo = '/glade/work/emmons/tutorial_Nanjing/ne0np4.Nanjing.ne30x8/topo/topo/Nanjing_ne30x8_gmted2010_modis_bedmachine_nc3000_Laplace0100_noleak_20240729land/surfdata_Nanjing_ne30x8_SSP3-7.0_1979_78pfts_c240809.nc'
 drydepflanduse_srf_file timeseries = '/glade/work/emmons/tutorial_Nanjing/ne0np4.Nanjing.ne30x8/atmsrfland/atmsrflanduse.timeseries_ne0np4.Nanjing.ne30x8_240809.nc'
 se_refined_mesh = .true.
 se_mesh_file = '/glade/work/emmons/tutorial_Nanjing/ne0np4.Nanjing.ne30x8/grids/Nanjing_ne30x8_EXODUS.nc'

 inithist = 'DAILY'

Nanjing_ne30x8_SSP3-7.0_1979-2026_78pfts_c240809.nc'

Add the override flag in your individual case directory to ensure CTSM doesn’t error out due to an unsupported grid:

./xmlchange --append CLM_BLDNML_OPTS="-no-chk_res"


> qcmd -- ./case.build

Set up to run a few days. 

./xmlchange RUN_STARTDATE="2010-01-01"

./xmlchange STOP_N="5"


If it does not run, try adjusting parameters, as described in Set the timestep based on recommended value in: https://github.com/ESMCI/Community_Mesh_Generation_Toolkit/blob/master/VRM_tools/Docs/CAM-tsteps-inic-for-newgrids_v0.pdf

...

Add to user_nl_clm:

fsurdat = '/glade/work/emmons/tutorial_Nanjing/ne0np4.Nanjing.ne30x8/land/surfdata_Nanjing_ne30x8_SSP3-7.0_1979_78pfts_c240809.nc'
flanduse_timeseries = '/glade/work/emmons/tutorial_Nanjing/ne0np4.Nanjing.ne30x8/land/landuse.timeseries_Nanjing_ne30x8_SSP3-7.0_1979-2026_78pfts_c240809.nc'

Add the override flag in your individual case directory to ensure CTSM doesn’t error out due to an unsupported grid:

./xmlchange --append CLM_BLDNML_OPTS="-no-chk_res"

> qcmd -- ./case.build

Set up to run a few days. 

./xmlchange RUN_STARTDATE="2010-01-01"

./xmlchange STOP_N="5"

If it does not run, try adjusting parameters, as described in https://github.com/ESMCI/Community_Mesh_Generation_Toolkit/blob/master/VRM_tools/Docs/CAM-tsteps-inic-for-newgrids_v0.pdf 

Once it runs, change the frequency of writing IC files from daily to monthly (in user_nl_cam: inithist = 'MONTHLY') and  run a year.  Save the final *cam.i.* file to use for future CAM simulations (ncdata in user_nl_cam). 

Save the final CLM restart file (*.clm2.r.*.nc) to use for finidat in user_nl_clm in future runs.

...

.pdf 

Once it runs, change the frequency of writing IC files from daily to monthly (in user_nl_cam: inithist = 'MONTHLY') and  run a year.  Save the final *cam.i.* file to use for future CAM simulations (ncdata in user_nl_cam). 

Save the final CLM restart file (*.clm2.r.*.nc) to use for finidat in user_nl_clm in future runs.

Anchor
Run CAM-chem
Run CAM-chem
Set up a CAM-chem case for production

Create a case with FCnudged compset

> /glade/work/emmons/cesm_src_derecho/cesm3_0_beta01/cime/scripts/create_newcase --case /glade/work/emmons/tutorial_Nanjing/cases/f.e3beta01.FCnudged.Nanjing_ne30x8.01 --res ne0Nanjingne30x8_ne0Nanjingne30x8_mt12 --compset FCnudged  --run-unsupported --project AACD0004 --pecount 2048

Regrid IC file

You will need to regrid a previous CAM-chem IC file to your grid (to use for ncdata in user_nl_cam).  If you have an IC file on a Finite Volume grid (f09) use 'interpic' as shown above.  If starting from a file on ne30 grid, use the ncl script at: https://github.com/NCAR/IPT/tree/master/Initial_conditions

A sample script for regridding a f09 CAM-chem file to the new grid is: /glade/work/emmons/tutorial_Nanjing/ne0np4.Nanjing.ne30x8/inic/interpic_script_camchem_Nanjing.sh

Regrid emissions files to your grid

Regridding emissions

Python regridding function: https://ncar.github.io/CAM-chem/examples/functions/Regridding.html

Example Python notebooks for regridding: /glade/u/home/emmons/EMISSIONS/regrid_notebooks_samples/

Regrid meteorology (MERRA2 or GEOS-FP) files to your grid

Regridding meteorological data

Update user_nl_clm

Use the CLM restart file from your CAM spinup for finidat in user_nl_clm.

Update user_nl_cam

If simulating post-2015, update lower boundary conditions file:

 flbc_file = '/glade/p/cesmdata/cseg/inputdata/atm/waccm/lb/LBC_17500116-25001216_CMIP6_SSP585_0p5degLat_c20200824.nc'

Add list of emissions files on new grid (srf_emis_specifier and ext_frc_specifier)

Update sea surface temperature (SST) file

./xmlchange SSTICE_DATA_FILENAME='/glade/p/cesmdata/inputdata/atm/cam/sst/sst_HadOIBl_bc_1x1_1850_2021_c141021.nc'
./xmlchange SSTICE_YEAR_END=2021

OR

./xmlchange SSTICE_DATA_FILENAME='/glade/campaign/cgd/cas/asphilli/cam_input_data/sst_COBE-SST2_bc_0.9x1.25_1850_2024_c250627.nc'
./xmlchange SSTICE_YEAR_END=2024
./xmlchange SSTICE_MESH_FILENAME='/glade/campaign/cesm/cesmdata/inputdata/share/meshes/fv0.9x1.25_141008_polemod_ESMFmesh.nc'

NOTE: Model will regrid SST file to model grid, but the SSTICE_MESH file must match the grid of the SST file!

Create a case with FCnudged compset

> /glade/work/emmons/cesm_src_derecho/cesm3_0_beta01/cime/scripts/create_newcase --case /glade/work/emmons/tutorial_Nanjing/cases/f.e3beta01.FCnudged.Nanjing_ne30x8.01 --res ne0Nanjingne30x8_ne0Nanjingne30x8_mt12 --compset FCnudged  --run-unsupported --project AACD0004 --pecount 2048

Regrid IC file

You will need to regrid a previous CAM-chem IC file to your grid (to use for ncdata in user_nl_cam).  If you have an IC file on a Finite Volume grid (f09) use 'interpic' as shown above.  If starting from a file on ne30 grid, use the ncl script at: https://github.com/NCAR/IPT/tree/master/Initial_conditions

A sample script for regridding a f09 CAM-chem file to the new grid is: /glade/work/emmons/tutorial_Nanjing/ne0np4.Nanjing.ne30x8/inic/interpic_script_camchem_Nanjing.sh

Regrid emissions files to your grid

Regridding emissions

Python regridding function: https://ncar.github.io/CAM-chem/examples/functions/Regridding.html

Example Python notebooks for regridding: /glade/u/home/emmons/EMISSIONS/regrid_notebooks_samples/

Regrid meteorology (MERRA2 or GEOS-FP) files to your grid

Regridding meteorological data

Update user_nl_clm

Use the CLM restart file from your CAM spinup for finidat in user_nl_clm.

Update user_nl_cam

If simulating post-2015, update lower boundary conditions file:

 flbc_file = '/glade/p/cesmdata/cseg/inputdata/atm/waccm/lb/LBC_17500116-25001216_CMIP6_SSP585_0p5degLat_c20200824.nc'

Add list of emissions files on new grid (srf_emis_specifier and ext_frc_specifier)

Update sea surface temperature (SST) file

./xmlchange SSTICE_DATA_FILENAME='/glade/p/cesmdata/inputdata/atm/cam/sst/sst_HadOIBl_bc_1x1_1850_2021_c141021.nc'
./xmlchange SSTICE_YEAR_END=2021

Using existing Nanjing_ne30x8 grid files

...