CCSM ESMF Stage-1 Design Overview

Notes about how we are using ESMF in our Stage-1 implimentation in the Community Climate System Model (CCSM).

Time synchronization – down from parent

The start and stop time is set at the top level driver. The top level driver establishes a synchronization clock that is then passed down to sub-components in a read-only manner. Sub-components, set their internal clocks based on the synchronization clock from the parent to set the start, stop, and current times. When a component runs – it runs from it's current time (it checks to make sure it's current time is the same as the sync-clock time) until it's next alarm time.
Example...

 !
 ! Initialization
 !
 call ESMF_Initialize()
 call ESMF_VMGetGlobal( vm )
 call ESMF_VMGet(       vm, localpet=petid, mpicommunicator=mpicom ) 
 masterproc = (petid == 1)
 call shr_timemgr_SetupClock(     ClockSetup, log_print=masterproc, &  
                                                   clock_out=ESyncClock )
 ! Initialize all components (two phase initialization, with coupling and merging imbetween)
 ...
 !
 ! Time loop
 !
do while( .not. ESMF_ClockIsStopTime( ESyncClock )  )
     if ( shr_inputinfo_RunModel( initinfo,  atm  ) .and. ESMF_AlarmIsRinging( atm_alarm ) )then   
        ! Couple last time period first
        call ESMF_CplCompRun ( cc_map_a2l, import=a2x_a, export=a2x_l, rc=rc )
        call ESMF_CplCompRun ( cc_map_a2i, import=a2x_a, export=a2x_i, rc=rc )
        call ESMF_CplCompRun ( cc_map_a2o, import=a2x_a, export=a2x_o, rc=rc )
        ! Run atmosphere from curr_time until curr_time + atm_alarm%interval
        call ESMF_GridCompRun( gc_atm,     import=x2a_a, export=a2x_a, ESyncClock, phase=1 )
   end if
   if ( shr_inputinfo_RunModel( initinfo,  lnd  ) .and. ESMF_AlarmIsRinging( lnd_alarm ) )then
       ! Couple last time first
       call ESMF_CplCompRun (  cc_map_l2a, import=l2x_l, export=l2x_a, rc=rc )
       ! Note: cs_l is the composite state pointing to: a2x_l
        ! Run land from curr_time until curr_time + lnd_alarm%interval
       call ESMF_GridCompRun ( gc_mrg_x2l, import=cs_l, export=x2l_l, rc=rc ) 
       call ESMF_GridCompRun(	 gc_lnd,     import=x2l_l, export=l2x_l, ESyncClock, phase=1 )
       call ESMF_GridCompRun(	 gc_lnd,	                                      phase=2 )
       call ESMF_AlarmRingerOff( lnd_alarm )
   end if
   if ( shr_inputinfo_RunModel( initinfo,  ice  ) .and. ESMF_AlarmIsRinging( ice_alarm ) )then
        ! Couple last time first
        call ESMF_CplCompRun (  cc_map_i2a, import=i2x_i, export=i2x_a, rc=rc )
        call ESMF_CplCompRun (  cc_map_i2o, import=i2o_i, export=i2o_o, rc=rc )
        ! Note: cs_i is the composite state pointing to: a2x_i, o2i_i
        ! Run ice from curr_time until curr_time + ice_alarm%interval
        call ESMF_GridCompRun ( gc_mrg_x2i, import=cs_i, export=x2i_i, rc=rc )
        call ESMF_GridCompRun(  gc_ice,     import=x2i_i, export=i2x_i, ESyncClock, phase=1 )
        call ESMF_GridCompRun(  gc_ice,	                                       phase=2 )
        call ESMF_AlarmRingerOff( ice_alarm )
   end if
   if ( shr_inputinfo_RunModel( initinfo,  ocn  ) .and. ESMF_AlarmIsRinging( ocn_alarm )  )then
        ! Couple last time first
       call ESMF_CplCompRun(   cc_map_o2a, import=o2x_o, export=o2x_a, rc=rc)
        call ESMF_CplCompRun (  cc_map_o2i, import=o2x_o, export=o2x_i, rc=rc )
        ! Note: cs_o is the composite state pointing to: i2o_o and a2x_o
        ! Run ocean from curr_time until curr_time + ocn_alarm%interval
        call ESMF_GridCompRun ( gc_mrg_x2o, import=cs_o, export=x2o_o, rc=rc ) 
        call ESMF_GridCompRun(  gc_ocn,     import=x2o_o, export=o2x_o, ESyncClock, phase=1 )
         call ESMF_AlarmRingerOff( ocn_alarm )
   end if
   if ( shr_inputinfo_RunModel( initinfo,  atm  ) .and. ESMF_AlarmIsRinging( atm_alarm ) )then
       ! Note: cs_a is the composite state pointing to: l2x_a, i2x_a, and o2x_a
       call ESMF_GridCompRun( gc_mrg_x2a, import=cs_a, export=x2a_a, rc=rc )
       call ESMF_GridCompRun(	gc_atm,     import=x2a_a,               phase=2 )
       call ESMF_AlarmRingerOff( atm_alarm )
   end if
   call ESMF_AdvanceClock( ESyncClock )
end do

Top level driver code

Alarms that CCSM components listen to:

CCSM components are expected to listen to the following alarms.

• The alarm that rings when they are called (atm, ocn, ice or lnd)
• The CCSM restart alarm

  CCSM components have a given set of expected import/export fields

  CCSM components are expected to have certain import/export fields. They also include the date and time-of-day as integer attributes added to each import/export state.

  No use of grid in gridded components

  We do NOT use the grid in the gridded component – as it is dangerous to do so. Grid information should be retrieved from the specific field within the export state from a component. In some cases sub-components will have several different grids or grid decompositions within the same state.
  All CCSM components will be heavy-weight components with the ability to establish their own grid.

  Data allocations

  Most data allocated in components at either phase 1 or phase 2 initialization.

  Functions in set-services

• Init-1
• Init-2
• Run-1
• Run-2
• Finalize

  Science objective of CCSM

  From the CCSM "about us" page...
  Development of a comprehensive CCSM that accurately represents the principal components of the climate system and their couplings requires both wide intellectual participation and computing capabilities beyond those available to most U.S. institutions. The CCSM, therefore, must include an improved framework for coupling existing and future component models developed at multiple institutions, to permit rapid exploration of alternate formulations. This framework must be amenable to components of varying complexity and at varying resolutions, in accordance with a balance of scientific needs and resource demands. In particular, the CCSM must accommodate an active program of simulations and evaluations, using an evolving model to address scientific issues and problems of national and international policy interest.
  The CCSM project will address important areas of climate system research. In particular, it is aimed at understanding and predicting the climate system. The long-term goals of the CCSM project are simple but ambitious. They are:
• to develop and to work continually to improve a comprehensive CCSM that is at the forefront of international efforts in modeling the climate system, including the best possible component models coupled together in a balanced, harmonious modeling framework;
• to make the model readily available to, and usable by, the climate research community, and to actively engage the community in the ongoing process of model development;
• to use the CCSM to address important scientific questions about the climate system, including global change and interdecadal and interannual variability; and
• to use appropriate versions of the CCSM for calculations in support of national and international policy decisions.

  Component diagram

• Driver
  • Atmosphere
    • Atmosphere-merger (takes, land, ocean and sea-ice and merges it into atmosphere import state)
    • Atmosphere to land coupler
    • Atmosphere to ocean coupler
    • Atmosphere to sea-ice coupler
  • Land
    • Land-merger (takes atmosphere data and maps into land import state)
    • Land to atmosphere coupler
  • Ocean
    • • Ocean-merger (merges sea-ice and atmosphere data into ocean import state)
      • Ocean to atmosphere coupler
      • Ocean to sea-ice coupler
  • Sea-ice
    • Ice-merger (merges ocean, and atmosphere data into ice import state)
    • Sea-ice to ocean coupler
    • Sea-ice to atmosphere coupler
      Eventually we want to have ensemble operation. We'll also have a single-column mode that works somewhat differently than the global system driver.

      Status

      All grids are available at Stage-1. At Stage-2 we need to be able to represent the POP dipole grid.

      Mode of execution – sequential

      Stage-1 development is totally seqential. Later we hope to develop a concurrent version and/or hybrid.

      ESMF utilities

• Redist
• Time-manager
• Comm
  Later, regrid, config, and possibly log-err will be added.

  Sample Output

This is from a previous CCSM3 case – NOT using ESMF.
500 mbar geo-potential height
http://www.ccsm.ucar.edu/experiments/ccsm3.0/atm/b30.009-obs_501-520/set7/set7_ANN_Z3_500_NCEP_NP_obsc.png
Ocean mixed layer depth
http://www.ccsm.ucar.edu/experiments/ccsm3.0/ocn/b30.009/571_600/HMXL.gif
Annual mean ice-thicknesss
http://www.ccsm.ucar.edu/experiments/ccsm3.0/ice/b30.009-b30.004_401-410/contour/diff_con_ann_hi_csim4_NH.gif