August 14, 2012 mlevy)
(4:13p) Final changes before making tag 03:
Just a couple minor fixes to ensure CVMix still works with POP.
Date: 14 Aug 2012 Name: pop2/branches/vmix_project Summary: Updated vertical_mix.F90 to move loops out of CVMix code. Also needed to add dens and dens_lwr to vmix_put(). Files Modified: M 39453 source/vertical_mix.F90 M 39453 source/vmix/vmix_put_get.F90
(2:01p) The first draft of the standalone code is ready to be uploaded, but I want to make sure that the changes I made here are handled correctly on the POP side of things as well... so there will be one more update today (hopefully limited to vertical_mix.F90) to account for that and then I'll make a tarball with the stand-alone driver.
Date: 14 Aug 2012 Name: pop2/branches/vmix_project Summary: Removed vmix_init from vmix_kinds_and_types.F90 (for now, just use vmix_put to initialize values as needed). Also updated the coeffs routines (and vmix_init_bkgnd_BryanLewis) to use a single column as input rather than an array of columns. I also updated many of the local [loop-indexing] variables to meet standards laid out in the wiki (kw to loop over vertical interfaces vs kt to loop over tracer points / cell centers... no kt loops exist yet). Files Modified: M 39452 source/vmix/vmix_background.F90 M 39452 source/vmix/vmix_convection.F90 M 39452 source/vmix/vmix_put_get.F90 M 39452 source/vmix/vmix_kinds_and_types.F90
August 13, 2012 (mlevy)
Sorry for the late update, I committed this right before a couple of meetings yesterday and didn't have a chance to post details here.
(1:52p) I'm taking steps towards the standalone driver, but have a few more things to do. The "initialization" stage now looks like this:
call vmix_put(Vmix_params, 'max_nlev', nlev) call vmix_put(Vmix_params, 'prandtl', 0.0_vmix_r8) call vmix_put(Vmix_vars(1), 'nlev', nlev) call vmix_put(Vmix_vars(1), 'visc', 0.0_vmix_r8) call vmix_put(Vmix_vars(1), 'diff', 0.0_vmix_r8) call vmix_put(Vmix_vars(1), 'z', iface_depth)
These calls will change slightly when we move to pointers, but the first stand-alone driver I make available will still rely on memory copies. Also, unrelated to the pointers, the 'z' will become 'zw' per Steve's vertical naming convention.
Before making a tag and making the stand-alone driver available, I want to take the column-loops out of the CVMix code (so I can use Vmix_vars instead of Vmix_vars(1) all over the place). That will also be a good step towards something Todd suggested last week -- having a single CVMix routine that calls all the necessary coeffs routines so the logic to determine what type of mixing to use is done inside CVMix instead of in the models.
Date: 13 Aug 2012 Name: pop2/branches/vmix_project Summary: Working on the stand-alone driver - I added the put_get module for packing / unpacking data and I uncovered a bug in the Bryan-Lewis mixing. Specifically, I needed to store pi (vmix_PI) because I like the bl1 + (bl2/PI)*atan(bl3*(z - bl4)) formulation (note the PI in the denominator of the second term). For pack and unpack, I made a software engineering decision that may force us to meet F95 standards instead of F90 -- I initialized a component of a derived datatype in the declaration. This way you can't pack arrays without specifying nlev first. Files Modified: M 39331 source/vmix/vmix_background.F90 A 0 source/vmix/vmix_put_get.F90 M 39331 source/vmix/vmix_kinds_and_types.F90
August 8, 2012 (mlevy)
(3:38p) While putting together the Using CVMix with different ocean models page, I noticed a few things that didn't look right in how I used the modules in POP. This update doesn't affect any of the modules in the vmix/ directory, it's POP-only code.
Date: 08 Aug 2012 Name: pop2/branches/vmix_project Summary: Two quick fixes to vertical_mix.F90, one important and one not. The one that is: for constant mixing, Vmix_bkgnd_params%static_visc and static_diff are dimension (1,1), but I was trying to evaluate them at (1, 2:km+1). Also, I realized that some of the work that is being done for all background mixing only needs to be done if we are using convective mixing as well... so I added a couple of if statements. These should always evaluate to "True," though, so it doesn't actually change anything. Files Modified: M 39293 source/vertical_mix.F90
August 7, 2012 (mlevy)
(10:02p) Final changes before making tag 02:
I just cleaned up the protex documentation a little bit.
Date: 07 Aug 2012 Name: pop2/branches/vmix_project Summary: Cleaned up the documentation, and am somewhat happy with the way the protex code looks. Files Modified: M 39290 source/vmix/vmix_background.F90 M 39290 source/vmix/vmix_convection.F90 M 39290 source/vmix/vmix_kinds_and_types.F90
(5:33p) I think this is a smart way to handle the different possibilities for constant mixing... diff / visc can be
- scalars,
- 1D in the vertical,
- 1D in the horizontal (lat-lon),
- 2D (varying both vertically and horizontally),
- Specified by a specific algorithm -- so far, only Bryan-Lewis has been implemented
The static fields will be stored in a 2D array in a vmix_bkgnd_params_type variable... in case 1 it will be a 1x1 array, in case 2 it will be ncol x 1, in case three it will be 1 x nlev+1, and in cases 4 or 5 it will be ncol x nlev+1. The initialization is done using a procedure interface, so vmix_backgrounds.F90 contains vmix_init_bkgnd_scalar, vmix_init_bkgnd_1D, vmix_init_bkgnd_2D, and vmix_init_bkgnd_BL. Further, the vmix_init_bkgnd_1D routine will assume the array varies in the vertical unless you send it ncol.
Todd had asked about how CVMix will handle errors, and right now the answer is it doesn't, but when the error-handling module gets added in the first thing to check is to ensure that the dimensions match in the 1D or 2D cases... otherwise you might (for example) pass in a 1x30 array and have CVMix try to initialize a 1x60 array. Also, the arrays in vmix_bkgnd_params_type should only be allocated once - right now, if they are already allocated then the vmix_init_bkgnd_* routines return without changing anything but it would be way better to either error out or at least print a warning.
Date: 07 Aug 2012 Name: pop2/branches/vmix_project Summary: Finally set up vmix_bkgnd_init to handle the fact that the static viscosities and diffusivities can be scalar (the only current implementation in POP), 1D (varying in vertical or horizontal), or 2D (varying in both the vertical and horizontal). In vmix_coeffs_bkgnd, vmix now counts on two logical flags in the vmix_bkgnd_params_type (lvary_vertical and lvary_horizontal) to know the dimensions of static_visc and static_diff. Files Modified: M 39262 source/vertical_mix.F90 M 39262 source/vmix/vmix_background.F90 M 39262 source/vmix/vmix_kinds_and_types.F90
August 3, 3012 (mlevy)
(3:37p) I added Bryan-Lewis diffusion, but don't really have a good way to test it until we get the stand-alone driver build. Vmix now allocates space for distance from the top of the ocean at both tracer points and cell interfaces. I also added cell thicknesses and distances between tracer points; as of now, I haven't run anything where POP needs to use those values, so they are just initialized to zero and wasting memory. Once all (or at least "most") of the mixing modules are populated, I'll work on logic to figure out what variables are actually needed given the type of mixing requested.
Along a similar note, I think I want to move the vmix_init() routine into each model. Right now it's in vmix_kinds_and_types.F90, which is definitely not the right place for it, but I think it should be in the vmix package at all. I'm envisioning a POP_vmix_interaction.F90 file (as part of POP) that would handle all of the initialization as well as contain routines to populate fields. In fact, that might take care of the problem of allocating memory for variables we don't need, since that module can just allocate memory if it's trying to populate a variable that hasn't been allocated yet. That would fit well with the stand-alone vmix_driver goal, because the stand-alone driver would need a similar module... so I'll add that to the tag 03 milestone.
Still to do before making tag 02 -- I need to expand vmix_bkgnd_init_static to handle 1D and 2D inputs for the mixing coefficients as opposed to just the constants that it deals with now. The "hard" part of that is figuring out how to know if a 1D array is varying vertically or if it is varying in the lat-lon direction. I've been thinking that I'd just have two logical flags in vmix_bkgnd_params :: lvaries_vert and lvaries_horz; then the vmix_bkgnd_coeffs routine can check whether either (or both) are true and know which static values need to be sent back to the model.
Date: 03 Aug 2012
Name: pop2/branches/vmix_project
Summary: Introduced Bryan-Lewis diffusion into vmix_background. This meant
restructuring vmix_background.F90 a little bit: there are now
two initialization routines (vmix_init_bkgnd_static and _BL) and
the old routine vmix_init_bkgnd is now an interface to determine
which method to call (based on the arguments provided, since BL
mixing needs four parameters passed in and the static mixing only
needs two).
I created a new datatype, vmix_global_params_type, with the intent
of using it to store any parameters that are used by multiple
mixing techniques. Right now it contains max_nlev, the most levels
a column can have ("km" in POP), and prandtl, the Prandtl number.
Steve mentioned that different mixing methods will use different
Prandtl numbers, so I anticipate this value will get moved to the
mixing-specific parameter types. But it's better than storing it
on every column, since it doesn't vary spatially.
I also tried to improve the documentation in vmix_background, esp
with the addition of Bryan-Lewis mixing, but I think there is
still more work to be done on that front.
Files Modified:
M 39173 source/vertical_mix.F90
M 39173 source/vmix/vmix_background.F90
M 39173 source/vmix/vmix_kinds_and_types.F90
August 2, 2012 (mlevy)
(10:06p) Lots of clean-up, moving towards the tag 02 goal laid out earlier. Still need to add Bryan-Lewis and option to read from a file and update documentation, though.
Date: 02 Aug 2012 Name: pop2/branches/vmix_project Summary: No longer store POP's km (max number of levels) in Vmix_input_type. Instead, Vmix_inputs%nlev is the number of cells in that particular column, and memory for viscosity, diffusivity, etc is allocated accordingly. Also, vmix_background had some variables using the pop naming conventions that needed to be updated. For example, we are now using bkgnd_visc instead of const_vvc for the background viscosity field. Lastly, renamed vmix_convective.F90 vmix_convection.F90 (similar change to the name of the module inside the file). Files Modified: M 39132 source/vertical_mix.F90 M 39132 source/vmix/vmix_background.F90 D 39132 source/vmix/vmix_convective.F90 A + - source/vmix/vmix_convection.F90 M 39132 source/vmix/vmix_kinds_and_types.F90
July 30, 2012 (mlevy)
(5:24p) Making a list made it pretty easy to go through and pick off some issues... this update addresses points 2, 3, and 4 in the earlier post. I'll try to update the documentation tomorrow morning and then make tag 02, which will be the first tag that looks like what we've envisioned for this mixing package rather than POP.
Date: 30 July 2012 Name: pop2/branches/vmix_project Summary: Combined vmix_input_type and vmix_output_type into vmix_data_type, cleaned up initialization routines so vmix_init_bkgnd only sets up background settings (added vmix_init() to vmix_kinds_and_types.F90 which may need to be renamed now that it contains more than just kinds and types). Also fixed the indexing issue with viscosity and diffusivity, both now go from 1:km+1, with i=n corresponding to the top interface of cell k=n, and i=km+1 corresponding to the bottom interface. Still need to add protex documentation to vmix_init(). Files Modified: M 39032 source/vmix/vmix_kinds_and_types.F90 M 39032 source/vmix/vmix_background.F90 M 39032 source/vmix/vmix_convective.F90 M 39032 source/vertical_mix.F90
(3:24p) I talked with Gokhan this morning, who pointed out that POP's constant mixing is a combination of background mixing (with constant coefficient) and convective mixing. Rather than handling both in a single module, he recommended splitting out the convective part.
Date: 30 July 2012 Name: pop2/branches/vmix_project Summary: Created vmix_convective module. What POP called "constant mixing" is background mixing with a constant background followed by convective mixing to handle instabilities. Files Modified: M 39014 source/vertical_mix.F90 M 39014 source/vmix/vmix_background.F90 A 0 source/vmix/vmix_convective.F90 M 39014 source/vmix/vmix_kinds_and_types.F90
This commit compiles and runs, but there is still lots to do:
- The documentation needs to be updated -- I copied vmix_background.F90 to vmix_convective.F90 and cleaned up the code, but I didn't do much with the comments
- Some decisions need to be made about what each module contains. Right now vmix_background and vmix_convective both have an initialization routine and a routine to actually return the respective coefficients. However, the background mixing initialization routine also handles allocating memory for some of the components in Vmix_inputs and Vmix_outputs, which should probably be done in a separate Vmix_init routine (possibly in vmix_kinds_and_types.F90?)
- Speaking of Vmix_inputs and Vmix_outputs, I don't think it makes sense to carry around both of these. Vmix_outputs right now only contains diffusivity, viscosity, and nlev, while Vmix_inputs contains those as well as other variables needed by the mixing routines (such as density). I like the idea of just having Vmix_vars and using that for both input and output.
- Before progressing too much further, I'd like to nail down a naming convention for components of the soon-to-be-merged Vmix_input_type and Vmix_output_type. Gokhan raised the possibility of adding an _iface to variables on the interface, with all others being cell-centered variables:
type, public :: vmix_data_type integer :: nlev ! Number of levels integer :: deepcell_id ! Index of deepest cell real(vmix_r8) :: prandtl ! Prandtl number real(vmix_r8), allocatable :: visc_iface(:) ! nlev+1 real(vmix_r8), allocatable :: diff_iface(:,:) ! nlev+1 x 2 real(vmix_r8), allocatable :: dens(:) ! nlev real(vmix_r8), allocatable :: dens_lwr(:) ! nlev end type vmix_input_type
Talking about this raised another issue that I need to fix, and it's something Steve has been mentioning for weeks: right now viscosity is an array indexed from 1:nlev instead of 1:nlev+1, and interface i=n is the bottom interface of cell k=n. I need to go back through and initialize it properly, then make sure all the bookkeeping is taken care of.
July 28, 2012 (mlevy)
(3:43p) These are the last small changes leading to the first vmix branch tag:
I'm making the tag here because the constant mixing module compiles without any POP dependence and I'm happy-ish with the documentation / interfaces for the vmix package (I've attached two protex PDF files to this progress report). As mentioned in this update, there are still a few things to consider changing -- I don't fully understand how POP determined if a column was unstable, so I copied its checks in a rather clunky manner, and some of the parameters I thought we'd want access to in the vmix package probably belong in the models themselves.
Note about these progress reports: everything that shows up in blocks below are copied from the file BranchProgress; there is also a BranchChangeLog which is updated with each new tag that basically summarizes all the BranchProgress reports that were created between tags.
Date: 28 July 2012 Name: pop2/branches/vmix_project Summary: Cleaned up the module usage in vmix_background.F90. One thing I don't like about the current implementation: to determine if a column is unstable, POP compared RHOK (density at level k) to RHOKP1 (density at level k+1). However, the way the calculation was done, RHOKP1 at level k was not equal to RHOK at level k+1... since we're storing everything in a column, it would be nice to only compute and store RHO once per level. Also, I cleaned up the calls to vmix_coeffs_bkgnd. I was passing in the number of levels, despite that being stored in Vmix_inputs. Besides removing that, I changed the way the loop indices were passed to help with OMP blocking. One more step might be to make the cols and cole commands optional, defaulting to 1 and ncol if they are not set... but at this point that doesn't seem necessary. Files Modified: M 39013 source/vertical_mix.F90 M 39013 source/vmix/vmix_background.F90 M 39013 source/vmix/vmix_kinds_and_types.F90
(1:50p) The summary pretty much says it all... I cleaned up the kind type naming convention to avoid namespace conflicts and also worked on the protex documentation. I want to make sure everything still compiles / runs, then I'll add the "only" options to the use statements in vmix_background.F90. (I also want to switch from subroutine-level use statements to module-level use statements, at least for this case -- I believe both subroutines need the same public data from vmix_kinds_and_types.F90.)
Date: 28 July 2012 Name: pop2/branches/vmix_project Summary: Cleaned up some of the protex documentation, added svn keywords to keep track of revision history, and renamed the real kind number to avoid potential namespace conflicts (using vmix_r8 instead of real_kind). This commit is meant to move things from my laptop to bluefire, where I'll clean up the use statements, make sure the package still compiles, and then I'll tag it. Files Modified: MM 39011 source/vmix/vmix_background.F90 MM 39011 source/vmix/vmix_kinds_and_types.F90
July 26, 2012 (mlevy)
(5:33p) Last update for today, I promise!
Code now compiles, runs, and is bit-for-bit with POP2 (comparing standard history file of daily averages after 2 months of running) when using vmix_type = 'const'. Now I'll go back to make sure the formatting is consistent and the comments / documentation are sufficient. My goal is to make a tag and upload the documentation generated by protex (should just be a PDF file) early next week.
Date: 26 July 2012 Name: pop2/branches/vmix_project Summary: Found bug that was preventing bit-for-bit results: I forgot to initialize convect_visc in the parameters data structure. Now that this setup is bfb with the original POP code, I'll focus on the software engineering side of things. I also found a bug that didn't affect my results because I was running with a single block per MPI task. Files Modified: M 38974 source/vmix/vmix_background.F90 M 38974 source/vertical_mix.F90
(4:26p) Code now compiles and runs, but isn't bit-for-bit even though I'm still fairly convinced it should be. The only substantial change is in the chain of calls from baroclinic.F90 -> vertical_mix.F90 -> vmix_background.F90 (vmix_const.F90 in the original), so I'll go through all of those calls line by line and see if I can figure out what is going wrong. This is very, very close to running (which is where I wanted to be on July 20th, but that's besides the point).
Date: 26 July 2012 Name: pop2/branches/vmix_project Summary: Lots more little bugs taken care of: among others, I inverted an if statement (if A then C else B instead of if A then B else C) and forgot to initialize viscosity and diffusivity after setting the initial values in the parameters derived type. Good news: this completes a two month run without blowing up. Bad news: it's not bit-for-bit with the original code (in fact, SST is off by a max of about 20% in the history files) Files Modified: M 38964 source/vmix/vmix_kinds_and_types.F90 M 38964 source/vmix/vmix_background.F90 M 38964 source/vertical_mix.F90 M 38964 source/baroclinic.F90
(12:10p) Code now compiles, beginning tests to ensure it runs. I think it will be bit-for-bit with POP at this point, so far the only differences are in memory management (actual computation should still be the same, unless I botched something in the OMP blocks).
Date: 26 July 2012 Name: pop2/branches/vmix_project Summary: Lots of little bugs taken care of, and the code now compiles with constant mixing in the vmix/ directory. Before I make a tag, I need to compare the output to the old POP and make sure code meets the standards we drew up. Files Modified: M 38960 source/vertical_mix.F90 M 38960 source/vmix/vmix_background.F90 M 38960 source/vmix/vmix_kinds_and_types.F90 M 38960 source/baroclinic.F90
(10:18a) Hopefully this will be the first of many entries today. Last night I got really close having the constant mixing pulled out -- and came up with a plan that I think makes sense for changing how the mixing driver actually gets called. Also, I went back to giving each mixing type its own module, rather than just having one module for all the initialization work and another for calculating the mixing coefficients.
One of the downsides to copy / pasting code is that if the code being copied doesn't meet our "good coding practices" guide, it needs to be edited... and I haven't gotten that far. So my plan for today is to get the constant mixing to work and then go back with my editing hat on and fix variable names, use statements, documentation / comments, etc. Once that's done, I'll make a tag and solicit opinions from everyone.
Date: 25 July 2012 Name: pop2/branches/vmix_project Summary: Switched back to one module per mixing method, so vmix_initialize and vmix_coeffs got rolled into vmix_background. This should set up a nice template for all the other modules we create, but for now it is pretty sloppy. Once everything compiles, I'll still need to go back and make sure the module fits the "good software practices" we laid out a couple weeks ago... I know for a fact that the code as it stands does not. To get rid of the loop over k in baroclinic.F90, I created a new routine vmix_coeff2 which will eventually replace vmix_coeffs in vertical_mix (at that time, I'll go back to the old name). This routine will receive an array of entire column at once, pack into the Vmix_input_type data structure, call into the vmix/ directory, and then unpack from the Vmix_output_type data structure. As of this commit, none of those three steps are done and I still need to determine exactly what variables need to be packed for inputs. Files Modified: M 38957 source/vertical_mix.F90 A + - source/vmix/vmix_background.F90 D 38957 source/vmix/vmix_initialize.F90 D 38957 source/vmix/vmix_coeffs.F90 M 38957 source/vmix/vmix_kinds_and_types.F90 M 38957 source/baroclinic.F90
July 23, 2012 (mlevy)
(9:39p) More work on separating the "constant" mixing (at the request of Steve and Gokhan, we are now calling this "background" mixing, though still focusing on the special case where the background field is constant) from the rest of POP. There is a call to the equations of state in the case of convective diffusion that still needs to be moved outside of the vmix/ directory, and the call to the mixing routine also needs to be updated because POP currently sends one level at a time and I would rather send an entire column. I assume that's how other models are set up, but I haven't actually checked yet.
I expect to create the first real tag in the next day or two with a working background mixing module. At that point, I'd like to take a step back and discuss the implementation in more detail to make sure everyone is happy with the setup.
Date: 23 July 2012 Name: pop2/branches/vmix_project Summary: Update pop2.buildexe.csh to also include source/vmix/. Also continue work on constant coefficient mixing (now known as a subset of the "background" class of mixing). I think I want to go back to each mixing type having its own module, rather than all the initialization routines, all the coeffs routines, etc. Issue -- currently, vmix_coeffs() is called from baroclinic.F90 inside two loops: one over the number of blocks (okay from our end) and one over the number of levels (not okay for how we've drawn this up). Files Modified: M 38923 source/vertical_mix.F90 A 0 source/vmix/vmix_initialize.f90 M 38923 source/vmix/vmix_coeffs.F90 M 38923 source/vmix/vmix_kinds_and_types.F90 M 38923 bld/pop2.buildexe.csh M 38923 BranchChangeLog
July 17, 2012 (mlevy)
(1:09p) I created a new branch of the repository: https://svn-ccsm-models.cgd.ucar.edu/pop2/branches/vmix_project
I also started looking into what I need to do to separate vmix_const.F90 from POP, and ran into some expected issues right off the bat:
- Kind Numbers: Obviously I can't have the vertical mixing package depend on the POP kinds; Brian Eaton suggested creating our own list of kind numbers and then having the initialization routine in the model driver compare the model's kind numbers to those in the vmix package.
- Defining Data Structures: I think it will make sense to define vmix_parameter_types for each of the mixing routines (vmix_const_param_type, vmix_rich_param_type, vmix_kpp_param_type) and use that to pass the inputs that come (for pop, at least) from namelists. For constant mixing this will just be the constant viscosity and diffusivity; for Richardson mixing it will the background diffusivity and viscosity as well as the coefficient for the Richardson number term; for KPP it will be a whole slew of parameters. If we include these parameters in vmix_input_type then we are storing these parameters on each column, rather than once per core.
I made this initial repository commit so everyone can see the direction I'm heading... but I didn't make a tag because it seems silly to tag a revision that doesn't compile. One thing I added was BranchChangeLog to try to keep track of everything I committed, and for the sake of completeness I'll copy that to wiki here:
Date: 17 July 2012 Summary: Starting to work on separating vertical mixing from POP. Created the source/vmix/ directory, started work on the constant mixing. I think all the initialization will occur in vertical_mix.F90 and the three coeffs routines (vmix_const_coeffs, vmix_rich_coeffs, and vmix_kpp_coeffs) will be moved to vmix/vmix_coeffs.F90. It might make more sense to keep the individual modules for each mixing method, though... Currently this does not build. Three things need to be done: 1. I need to create vmix_pack_and_unpack.F90 (with the pack and unpack routines); this will be in pop2/source. 2. I need to rewrite vmix_const_coeffs() to be column-independent even in the case where diffusion is simulating convection. 3. I need to modify the build system to look for files in source/vmix Files Modified: D 38726 source/vmix_const.F90 M 38726 source/vertical_mix.F90 A 0 source/vmix A + - source/vmix/vmix_coeffs.F90 A 0 source/vmix/vmix_kinds_and_types.F90