Requirements are arranged according to the two-stage evaluation plan. Stage 1 involved implementing the uncoupled, sequential, single executable CAM/CLM model using ESMF. Stage 2 involves implementing the coupled, concurrent CCSM3 model in ESMF. To the extent possible, the existing CCSM3.0 architecture will be preserved in Stage-2. Both stages require significant code restructuring, but the restructured CAM and CLM from Stage 1 can be used, without modification, in Stage 2.
Stage 1 is further subdivided into Stage 1.a and Stage1.b. Stage 1.a implements the code restructuring in CAM and CLM necessary to prepare them for the use of the ESMF superstructure, but results in executable code that does NOT actually use the ESMF superstructure. Stage 1.b implements any other code development necessary to use the ESMF superstructure and results in an actual ESMF application.
Requirements are categorized as Required by CCSM science , Required by ESMF , and Desired by CSEG . Required by CCSM science indicates requirements imposed on the software by CCSM scientists. Required by ESMF indicates requirements imposed by ESMF. Desired by CSEG indicates design choices deemed appropriate by CSEG.
Requirements for all steps of Stage-1 and Stage-2
These are requirements for code created both for Stage-1 and Stage-2.
- Required by CCSM science
- 1. The hub-spoke pattern of CCSM3 is adopted in this framework. Hence five components are created: atmosphere, sea-ice, land, ocean and hub. Communication between components is always done through the hub component.
- 2. List of import and export states to components will include the complete set of concurrent CCSM3 fields exchanged and this may represent a larger set of data than actually used..
- Features desired by CSEG
- 3. As much as possible only high level code in existing models should be modified.
- As a principle the minimum amount of code should be modified to accomplish the requirements.
- Some components are termed throw away components , because they are not used in Stage-2, a minimum level of effort and modification should be given to these components, since they have a short life-cycle.
- Code changes should also be limited to limit the amount of testing that is required in different configurations and run-time options.
- Code changes that improve the design of a given model and are approved by the liaison for the given model are allowed.
- 4. The code-flow of the top-level drivers should allow the developer to easily understand how components are being called and what is communicated into and out of each component.
Requirements for Stage-1 both 1.a and 1.b
- 4. The code-flow of the top-level drivers should allow the developer to easily understand how components are being called and what is communicated into and out of each component.
- Required by CCSM science
- 5. The top-level driver will be referred to as sequential CCSM and not stand-alone CAM, as CAM is merely an atmospheric component to the system. We will refer to the version of sequential CCSM that satisfies all these requirements as seq_ccsm1.0
- 6. All components have the same coupling interval and model time-step. And further the coupling interval equals the model time-step.
- Features desired by CSEG
- 7. Answers are shown to be bit-for-bit for differing numbers of tasks and threads and bit-for-bit under restart on all dycores and supported platforms.
Requirements for Stage-1.a pre-ESMF version seq_ccsm1.0
The pre-ESMF version will be incrementally checked into cam_dev. This version will not be round-off or even bit-for-bit with current stand-alone CAM, but the current climate will be maintained.
- 7. Answers are shown to be bit-for-bit for differing numbers of tasks and threads and bit-for-bit under restart on all dycores and supported platforms.
- Required by CCSM science
- 8. Sequential CCSM must provide all of functionality provided for in stand-alone CAM in some mode. Some of the modes that are required are:
- The model system must be tested to work stand-alone as sequential CCSM but also backwards compatible with: SCAM, CCSM and SOM as build-time options.
- Aqua-planet, and adiabatic modes must also be supported as run-time options.
- 9. The CLM component will also not lose any of it s required functionality.
- CLM component must also be tested to work with CCSM, and offline-CLM.
- 10. Code will be verified to work on platform-compiler combinations supported by CAM and CCSM when checked into cam_dev.
- 11. For each check-in – code will be shown to be bit-for-bit, within round-off, or have the same climate (with standard long simulations performed) of standard CAM stand-alone of same version on all supported platforms for all three dy-cores (EUL, FV, SLD) using the standard CAM test-suite at the time of the tests.
Requirements for the ESMF versions in both Stage-1.b and Stage-2
- Required by CCSM science
- 12. The atmosphere, sea-ice, land, and ocean component interfaces to the high level driver are designed such that it is easy to swap any one components out for another of the same model-type, e.g. CAM and data-atmosphere can be swapped, without changing the high level system. The high level system should not have any #ifdef or other code logic that depends on the type of underlying component (whether it s active, data, or dead).
- 13. Code will be verified to work on platform-compiler combinations supported by CAM and CCSM on the date ESMF validation is begun.
- Required by ESMF
- 14. Each CCSM component is setup as an ESMF gridded component.
- 15. ESMF components are required to have: initialization, run, and finalization methods and are allowed to have multiple phases of each.
16. Components can have their own internal clock that may or may not use ESMF time-manager, but it does need to synchronize with the ESMF clock passed down into it. - 17. With noted exceptions the only way to communicate with other components is via import and export states.
- use statements , common blocks, or direct MPI communication is disallowed between model components.
- This requirement is relaxed for the CAM-DOM and CAM-CSIM components at Stage-1, which are allowed to use use statements to CAM modules. After Stage-1 no components are allowed to relax this requirement.
- ESMF does not allow communication to other ESMF components midway in an initialization, run or finalization phases.
- 18. The build system will require that ESMF be built separately. Only one version of ESMF will be used in the code (hence the calls to the prototype time-manager for CAM and CLM will be updated to the version of ESMF used in the exercise).
- 19. Import and export states will use pointers to internal data.
- Features desired by CSEG
- 20. The design will confine the dependence on ESMF to a thin wrapper layer, allowing ESMF to be swapped for something else.
- 21. Design should allow sequential and concurrent mode to share as much code as possible still allowing the call reordering that is required to become concurrent.
Requirements for Stage-1.b seq_ccsmesmf1.0
At this point a branch is created off of cam_dev from seq_ccsm1.0 to test ESMF functionality for stage-1.
- Required by CCSM science
- 22. Code will be shown to be within round-off of seq_ccsm1.0 on all supported platforms for all three dycores (eul, fv, sld) using the standard CAM test-suite from seq_ccsm1.0.
- 23. Memory usage compared to seq_ccsm1.0 must not increase by more than 20% on the production IBM platform (bluesky).
- 24. Performance on Cray-X1 and IBM for Eulerian dycore and default physics is no worse than 5% degradation from seq_ccsm1.0 for the tests shown in Table-1 of the ESMF Evaluation Stage 1 plan.
- Required by ESMF
- 25. CAM, CLM, CAM-DOM, and CAM-CSIM must be ESMF components called with ESMF GridComp methods.
- Features desired by CSEG
- 26. Design should allow as much of the code to be used as possible without rework when moving onto Stage-2. The CAM and CLM gridded components developed in stage-1 will work in stage-2 without modification.
Requirements for Stage-2
If Stage-1 is successful and ESMF is accepted for use in CCSM by the SSC,
- 26. Design should allow as much of the code to be used as possible without rework when moving onto Stage-2. The CAM and CLM gridded components developed in stage-1 will work in stage-2 without modification.
- Required by CCSM science
- 1. All components run with an integer number of cycles per day.
- 2. Land, sea-ice, and atmosphere components all run with the same number of cycles per day. Ocean couples to the rest at a frequency that is some integer multiple of the atmosphere cycles per day.
- 3. The hub will check the consistency of grid information from each component at initialization.
- 4. The hub will be responsible for determining what fields will be passed between components. And this determination will be done at run-time rather than compile-time.
- 5. It will be possible to run CCSM in both a sequential and concurrent manner with any combination of active or data components.
- Required by ESMF
- 6. CAM, CLM, POP, CSIM, and data-7 components must be ESMF components called with ESMF GridComp methods.
- 7. Separate driver grid components will exist for sequential and concurrent CCSM.
- 8. Concurrent CCSM will work as a SPMD single-executable with CCSM components as subroutine calls for the main driver.
- 9. The number of phases for initialize, run and finalize is set by the points when communication needs to happen.
- 10. There is some mode for which answers are shown to be bit-for-bit for differing numbers of tasks and threads and bit-for-bit under restart for all components on all component configurations and supported platform-compiler combinations.
- Features desired by CSEG
- 11 Drivers for sequential and concurrent execution should be as similar as possible made up with components that allow for either type of operation. The main difference between the two should be the order of the calls.
- 12 All components will obtain grid and surface information (land-masks, surface-height and standard-deviation) from a single shared input file.
- 13 As much as possible code from stage-1 will be re-used here with as little of changes as possible.