cam3_6_61
https://svn-ccsm-models.cgd.ucar.edu/cam1/trunk_tags/cam3_6_61
Need modified versions of models/atm/cam/bld/namelist_files: namelist_defaults_cam.xml, namelist_definitions.xml that register namelist variable nu_q; available on NCAR goldhill file system in ~mirin/namelist_register.
./configure -phys cam3_5_1 -spmd -nosmp -dyn homme -hgrid ne16np4
Must reference netcdf, p-netcdf, lapack. Instructions for jaguar are below.
Code successfully executed in aqua-planet mode on jaguar for one simulated day. (Did not create history files nor attempt to restart.)
homme_cam3_6_39
https://svn-ccsm-models.cgd.ucar.edu/cam1/branches/homme_cam3_6_39
./configure -phys cam3_5_1 -spmd -nosmp -dyn homme -hgrid ne16np4 -pnc_lib /contrib/pnetcdf/lib -pnc_include /contrib/pnetcdf/include
CAM3_6_39 (includes everything from 3_6_32 below) and:
- updated to CAM3_6_40 (for PIO bugfixes)2
- HOMME (09/05/12): total energy fixer and updates to area weights 1
- support for time-split physics (instead of process split) 1
- merge in scalable chemistry from Jim's: cs_cam3_6_34/models/atm/cam/src/chemistry/mozart
- Needed for BG/P: reduce number of global coordinate arrays allocated and removed tropopose observation based backup algorithm. 1
- BGP print_memusage support 1
- Patch from Tony Craig: fix tri-grid mapping glitch at coastlines
- Patch from Pat Worley: more scalable phys_grid_init() 1
1 scheduled for CAM trunk (maybe 3_6_51)
2 requres pio51_prod: run "cd models/utils/pio ;
svn switch https://svn-ccsm-models.cgd.ucar.edu/pio/trunk_tags/pio51_prod/pio"
homme_cam3_6_32 ( can now run CAM3_5 physics!)
https://svn-ccsm-models.cgd.ucar.edu/cam1/branches/homme_cam3_6_32
./configure -phys cam3_5 -spmd -nosmp -dyn homme -hgrid ne16np4 -pnc_lib /contrib/pnetcdf/lib -pnc_include /contrib/pnetcdf/include
CAM3_6_32 plus the following patches:
- scalable surface fields restart (ported from Jim's PIO branch)
- cubed-sphere chemistry support (ported from Jim's CS branch)
- HOMME (09/4/14)
- addition of area weights to history file
- updates to dynamics/homme: some outfld() support, physics_dt bugfixes (these should go on the trunk when Jim commits the PIO branch)
- other updates: FQps on restart. support for writing cam.i.* files
Machine dependent configure options
Additional options used on Sandia Blackrose (Intel/OpenMPI/Infiniband cluster)
-fc ifort -nc_inc /home/mataylo/netcdf/tbird-intel/include -nc_lib /home/mataylo/netcdf/tbird-intel/lib -pnc_inc /home/mataylo/pnetcdf/tbird-intel/include -pnc_lib /home/mataylo/pnetcdf/tbird-intel/lib -ldflags "-L/usr/lib -llapack -lmpi_f77" -cppdefs "-DNO_MPIINPLACE -DNO_MPI2" -mpi_inc $MPIHOME/include -mpi_lib $MPIHOME/lib
Also, edit the Makefile and add -DLLNL_PELOTON to get "use pio" fixes for Intel ifort
Additional options used on ANL BG/P
-pnc_inc /soft/apps/parallel-netcdf-1.0.2/include -pnc_lib /soft/apps/parallel-netcdf-1.0.2/lib -nc_inc /soft/apps/netcdf-3.6.2/include -nc_lib /soft/apps/netcdf-3.6.2/lib -target_os bgp
And then change "-lblas_bgp" to "-lblas" in the Makefile.
Additional options used on ORNL XT4 (Jaguar)
module load netcdf/3.6.2 (more recent version should work)
module load p-netcdf/1.0.3
module load lapack
setenv LAPACK_LIBDIR /sw/xt5/lapack/3.1.1/cnl2.1_pgi8.0.3/lib (update according to value of LAPACK_LIB)
-fc ftn -cc cc -ldflags '-L$LAPACK_LIBDIR -llapack'
When using Lustre parallel filesystem (ORNL, SNL, LLNL), edit the Makefile and add PIO_LUSTRE_HINTS here:
ifneq ($(strip $(LIB_PNETCDF)),)
CPPDEF += -D_PNETCDF -DUSEMPIIO -DPIO_LUSTRE_HINTS
USER_LDFLAGS += -L$(LIB_PNETCDF) -lpnetcdf
endif
NOTE: latest (2009) version of Lustre (Installed on LLNL's Dawn, maybe JaguarPF?) does not require this setting, and will perform much better if this setting is not used.
Start with CAM default namelist
./build-namelist -namelist "&camexp aqua_planet=.true. /" -csmdata /fs/cgd/csm/inputdata
On jaguar, csmdata directory is /tmp/proj/ccsm/inputdata.
If using build-namelist utility, must provide values for bndtvs, focndomain, fatmlndfrc, fatmgrid, fsurdat, faerdep; these are not used in aqua_planet mode so any value will suffice.
Edit the resulting atm_in as follows:
To match Dave Williamsons' CAM 3.1 parameters:
use CAM 3.4.10 physics ( currently not possible with CAM 3.5+)
CH4VMR = 1.650e-6
N2OVMR = 0.306e-6
CO2VMR = 3.480e-4
use symmetric ozone data:
bndtvo = /ptmp/jedwards/cam-tests/testdata/hommeapeozone.nc
in drv_in, timemgr_inparm orbital parameters:
orb_ECCEN = 0.
orb_OBLIQ = 0.
orb_MVELP = 0.
Physics timestep "dtime" has to be a multiple of the dynamics timestep. To run close to a 5min physics timestep (to compare with Dave Williamson's results)
dtime = 360
(and also in drv_in, atm_cpl_dt = 360)
Note: for benchmarking runs, we use dtime=1800.
Resolution specific input files (not needed with -phys cam3_5 )
bndtvaer = AerosolMass_V_ne16np4_clim_c071213.nc
ncdata = cami_0000-01-01_ne16np4_L26_ape_c071213.nc
Note1: need to add instructions on how to compute these
Note2: for now, Mark will generate 1 and 2 degree files and store on bluefire_
HOMME specific options:
phys_loadbalance = -1
HOMME Resolution Dependent Parameters
2 degree resolution:
homme_ne = 16
homme_nsplit = 2 ( dtime/180)
dtime = 360
nu = 1e16
nu_q = 1e16
1 degree resolution:
homme_ne = 30
homme_nsplit = 4 (dtime/90)
nu = 1e15
nu_q = 1e15
0.5 degree resolution:
homme_ne = 60
homme_nsplit = 40 (dtime/45)
nu = 1e15
nu_q = 1e15
Note: The high-res settings below require changing dtime so that nsplit<=45. Larger values of nsplit have trouble with our out-of-balance initial condition. They may be possible if we use a spun-up initial condition file.
0.25 degree resolution: (runs on 86,400 MPI processes)
dtime = 900
homme_ne = 120
homme_nsplit = 45 (dtime/20)
nu = 1.5e13
nu_q = 1.5e13
0.125 degree resolution: (runs on 345,600 MPI processes)
dtime = 450
homme_ne = 240
homme_nsplit = 45 (dtime/10)
nu = 2e12
nu_q = 2e12
History Output
Recommend output for high-res APE runs. Tweaked to produce 30 day means instead of monthly means. If restart frequency is a multiple of 30 days, no history restart will be needed. Precip data for 6h histograms is in the h1 file. Also added 1 hour snapshots of TMQ for quick animations (h2 file). Velocity snapshots are output every 6 days, used to compute mean KE spectra (h3 file).
NDENS = 2,2,2,2,2
MFILT = 1,120,720,5
NHTFRQ = -720,-6,-1,-144
FINCL1 = 'DUV ','DVV ',
'ZMDQ ','ZMDT ','TT ','ZZ ',
'OMEGAU ','VZ ',
'DCCLDLIQ','DCCLDICE'
FEXCL1 = 'DISED ','DLSED ','DQSED ','EVAPPREC','EVAPSNOW',
'HEVAP ','HKEIHEAT','HKFLXPRC',
'HKFLXSNW','HKNTPRPD','HKNTSNPD','HMELT ','HPROGCLD',
'HREPART ','HSED ','PCSNOW ','PRECSC ',
'PRECSED ','PRECSL ','PRODPREC','RAINSED ',
'SNOWSED ',
'TSICE ','ZMDLF ','ZMEIHEAT','ZMFLXPRC','ZMFLXSNW',
'ZMNTPRPD','ZMNTSNPD'
FINCL2 = 'PRECC ','PRECL '
FINCL3 = 'TMQ:I '
FINCL4 = 'U:I ','V:I ','PS:I '