Child pages
  • Run CAM-Chem on Cheyenne
Skip to end of metadata
Go to start of metadata

This example describes how to to run CAM-chem out-of-the-box with specified dynamics on the NCAR cheyenne HPC. More information on running CESM is available from the CESM Tutorials (e.g., http://www.cesm.ucar.edu/events/tutorials/2019/coursework.html). However, this page includes specific instructions for chemistry simulations. The compset required for this setup is named “FCSD”.

  • Get a Cheyenne account for the NCAR High performance Computer system if you don’t have one.

(A) Create a new case

1. Make a copy of the source code from the latest release in your own directory, following the instructions at:

http://www.cesm.ucar.edu/models/cesm2/release_download.html

It is best to put this code in your 'home' or 'work' directory (not scratch). When running on cheyenne you do not need to get any of the input data files. This directory is referred to as $CESM_ROOT below.


Tip for New Users:

Having problems? Try the Troubleshooting page.

> $CESM_ROOT/cime/scripts/create_newcase --case <your_path+$CASENAME> --mach cheyenne --res f09_f09_mg17 --compset FCSD

You may need to add --run_unsupported to the call line if you are not running with a scientifically validated compset and resolution combination.

Note 1: For additional help and options, type ./create_newcase -h 

Note 2: To find the possible compset types $CESM_ROOT/cime/scripts/query_config --compsets cam. 

A new directory $CASEROOT = <your_path+$CASENAME> is created

Tip for New Users:

Information on the compsets can be found here: http://www.cesm.ucar.edu/models/cesm2/config/compsets.html

(B) Set up your case

2. From within $CASEROOT

>./case.setup 


3. Make changes to the model configuration using the *.xml files; you can edit the files directly or use the xmlchange tool (in your case directory).

Changes to env_build.xml must be made before building, or you will need to re-build.

See advanced pages to change the chemistry source code.

(C) Build the Executable

Tip for New Users:

You can use script xmlquery to query a variable in the xml files before modifying a variable with xmlchange command. For example:

>./xmlquery CALENDAR

4. Compile and build the model in $CASEROOT

>qcmd -- ./case.build


Note: you cannot run ./case.build interactively from the cheyenne prompt because it uses too much memory: you must use 'qcmd'.

5. Make changes to the model runtime setup: Changes to env_run.xml can be made at any time:

If not starting in the default 2005, change date of RUN_START: see Changing Dates of a Run (also see namelist changes)

Biogeochemistry (bgc) option

<entry id="CLM_BLDNML_OPTS" value="-bgc bgc -crop -irrig=.true."> (Note The -irrig option can ONLY be used with clm4_0 physics)

<entry id="CLM_CONFIG_OPTS" value="-phys clm4_0">

or, use default for “specified phenology” (satellite LAI):

<entry id="CLM_BLDNML_OPTS" value="-bgc sp">

The default is a test run -- 5 days.

Change STOP_OPTION  and STOP_N to desired values.

Check the SST ice file covers the dates you require:

<entry id="SSTICE_DATA_FILENAME" value="$DIN_LOC_ROOT/atm/cam/sst/sst_HadOIBl_bc_0.9x1.25_1850_2016_c170525.nc">

<entry id="SSTICE_YEAR_END" value="2016">

6. Check namelist settings in the namelist files user_nl_cam and user_nl_clm. Most CAM-chem related namelist variables are in CaseDocs/atm_in, but MEGAN and drydep are in CaseDocs/drv_flds_in (these files are created during build).  To modify any of these, copy the appropriate lines to user_nl_cam and edit there.

If RUN_STARTDATE has been changed, you have to also change the date of the initial meteorology file in user_nl_cam to start at the corresponding date.

For other changes see namelist changes or advanced options page.

Tip for New Users:

There are many namelist variables. You can find their definitions at: http://www.cesm.ucar.edu/models/cesm2/settings/current/cam_nml.html

Optionally run:

>./preview_namelists

after changing any user_nl_* files. NOTE: most changes in user_nl_* files do not require re-building. However, during a run (CONTINUE_RUN = TRUE) no changes can be made to history output (fincl lists). If you want to change history output, create a new or branch run.

7. Check the run setup. In the env_batch.xml file make sure to have your project added correctly:  <entry id="PROJECT" value=$YOUR_PROJECT_CODE>

Change your walltime if desired: <entry id="JOB_WALLCLOCK_TIME" value="12:00:00"> (12 hours max)

8. If you make changes to env_build.xml variables or SourceMods after setting up and building, you may have to clean your setup and build again:

>./case.setup --clean
>./case.setup

OR

>./case.setup --reset

(which cleans, then does setup)

followed by

>./case.build --clean
>qcmd -- ./case.build

(D) Run the Model

9. Submit run to queue:

>./case.submit

While running, output is written to <run_dir>: /glade/scratch/<username>/$CASENAME/run

After the run completes successfully, output files are moved to the short term archive: /glade/scratch/<username>/archive/$CASENAME/atm/hist (similar directories exist for other model components: lnd, etc.)

Restart and initial conditions files are written to: /glade/scratch/<username>/archive/$CASENAME/rest

Note: long term archiving is currently not working

10. Useful commands while model is running:

check your run progress: 

>qstat –u <user>

If you find an issue and need to delete your run:

>qdel <JobID

If your model run doesn't complete, try some of the suggestions on the troubleshooting page.

11. To continue a run from restart files, for example after an initial start up, change CONTINUE_RUN to TRUE in the env_run.xml file.

Tip for New Users:

High performance computing systems often have maximum wall times (e.g. JOB_WALLCLOCK_TIME = 12 hours), meaning a long run will need to be split into several smaller runs. In this case, change the “RESUBMIT” value in env_run.xml file to greater than zero. For example, you can simulate 10 years by changing STOP_OPTION=nyears, STOP_N=1 and RESUBMIT=9. This will perform an initial run of 1 years + (9 resubmits x 1 years per job) = 10 years.





  • No labels