This is the home page for the TOY08 Student Information space.
Week 3 (Computational Science Lab) Information
Course content
Lectures
Tutorials on tools and techniques
Lab work
Large numerical problems may consume years. We will only have one week!!!
Final student presentations on Friday (~10 - 15 minutes)
Problems
Please report any problems by email to toy4@ucar.edu
Logistics
Crypto cards & UCAS passwords
Please verify operation of your crypto card and UCAR password by ssh'ing to frost and storm0 ASAP using the procedures below. Report any problems by email to toy4@ucar.edu.
Frost:
ssh -l your_ncar_login_name frost.ucar.edu
Storm0:
Ssh -l your_ncar_login_name gate.ucar.edu
Storm0.scd.ucar.edu (simply type storm0.scd.ucar.edu at the prompt after ssh'ing to gate)
Note: As of 7/15 Frost accounts have not been activated. Students will be notified as soon as their accounts are active.
Venues
Main Seminar Room: Most lectures will be held in the MSR
SCD VisLab: Lab work and tutorials will be held in the SCD VisLab
Damon Room: Six workstations are located in the Damon room and may be used for remote visualization (the systems are equipped with the TurboVNC client)
VAPOR
For much of the work in week 3 you will be using the VAPOR visual data analysis tool (www.vapor.ucar.edu)
We highly recommend that you experiment with VAPOR prior to week 3 using some of the sample data sets we provide.
VAPOR Documentation
Documentation for VAPOR may be found www.vapor.ucar.edu/doc. In particular we recommend:
An Overview of VAPOR Data Collections
VAPOR Quick Start Guide
There are a couple of options for experimenting with VAPOR in advance of week 3
LOCAL: Use your own laptop
- 3D gfx hardware required
- Installation of VAPOR and test data required (documented on the VAPOR web site)
REMOTE: Use NCAR Data Analysis and Visualization resources
- Preferred method
- Use your laptop as remote TurboVNC client
- TurboVNC client software installation required
or
- Use NCAR workstations in Damon Room (email room)
Sample data sets
Sample data sets may be found on the VAPOR web site:
www.vapor.ucar.edu/download (if running VAPOR on your laptop)
and on the storm cluster in /cxfs/DASG/clyne/TOY (if running VAPOR remotely)
NCAR Data Analysis and Visualization resources for remote viz
Information on using NCAR's data analysis resources remotly may be found at http://www.cisl.ucar.edu/hss/dasghttp://www.cisl.ucar.edu/hss/dasg in the following documents:
Quick Start Guide
Remote Visualization (Follow directions for "external" users)
VAPOR
Scratch disk space is available on the storm cluster in /ptmp/your_login_name
TurboVNC PCs
NCAR is providing six PCs (three linux and three windows) that are equipped with the TurboVNC software needed to support remote visualization with VAPOR. These PCs will be located in the Damon Room for the first two weeks of summer school. During the third week some, or all, of these systems will be moved to the VisLab as needed. Login information is written on the blackboard in the Damon Room.
DNS and LES simulation code notes for running on Frost
These are notes for setting up and using the Joe Werne's TRIPLE code.
1. setting up your user account
- change your umask to 022 in your ~/.cshrc file (or equivalent file if you
are using a different shell). this will make it easier to share your results
with your team members and with your instructors.
- add the following to your path: /home/werne/PST/bin /home/werne/triple/bin
the best way to do this is to add it to your .cshrc file:
set path=(. /home/werne/PST/bin /home/werne/triple/bin $path)
- create the directory ~/triple/3d/bin
2. modifying codeit (the execution script for running the TRIPLE code)
- you must set the variable $TripleHomeUser in codeit to include your home
directory:
$TripleHomeUser = "/home/<usr>/triple";
- don't mess with the nearby variable $TripleHome.
- to run codeit (after codein is ready), do the following:
codeit >& codeit_X.out &
this will run codeit and save standard output to the specified file. the
X is intended to be incremented for continuation runs.
3. preparing your work space to run.
- you must create the run-time directory for your project runs. a subdirectory
name has been set already in the codein file for your project. to find it,
look in codein and find the variable called DED. The variable $a{WORKDIR}
is equivalent to /ptmp/<usr>/, so you can leave that as is. the tail is the subdirectory
name you need to create (e.g., mkdir /ptmp/<usr>/R5kth60a09 - where <usr>
is your user name).
4. editing codein to prepare a run
- codein holds all the input parameters for the run. all the parameters you
may want to modify are the following:
NX, NY, NZ, ----------------------------- grid size
NCPU, NCPUNOW, ----------------------------- number of processors (keep equal)
NTOTAL, NSTEP0, NSTEP1, NSTEP2, NSTEP3, ---- output timing
PREVIFAR, CURRENT0, PROG, SCRIPT, ---------- run and executable names
QUEUE, QUEUE2, JOB_T, TOT_T ---------------- run queue and total job time
- keep in mind that NZ/(2*NCPU) and NX/NCPU must be integers.
- likewise, all of the NTOTAL/NSTEP? must also be integers.
- the queues available to you are
debug: NCPU=<256; JOB_T,TOT_T=< 2 hours
reserved: NCPU=<1024; JOB_T,TOT_T=< 24 hours
keep QUEUE=QUEUE2 and JOB_T=TOT_T
- setting NSTEP1 about 50 or 60 is good
- setting NSTEP2 >= 120 or larger is good. this paramter controls writing
3D volumes. if you make it too small, we'll run out of disk space.
- for the KH DNS problems, max(NX,NY,NZ) will be (800,270,800). for GW DNS
problems, max(NX,NY,NZ) will be (720,360,720). larger than this and you
will have trouble finishing on time. you may be able to finish by leaving
these set at these values, but when fine-scale turbulence weakens, you can
choose smaller values. to find acceptable values, use the command "factors
NX NY NZ", where NX, etc. are replaced with the values you want to use.
values must be of the form 2^n*3^m*5^p, where n, m, p are integers. more
will be said about how to choose NX, NY, NZ below in the section titled
"choosing the best grid". for the LES problems, you can choose grids that
are a factor of 5 smaller than the maximum values mentioned above (so you
won't have any trouble finishing on time).
- PREVIFAR specifies the location of the previous run on the archival storage
system. the tail of PREVIFAR specifies the local run name for the restart
data. after a run, you must run a processing script to concatenate the
output files, move them to archival storage, and prepare the restart data.
you cannot continue a run until it is processes. processing the data is
described below where SCRIPT is explained.
- CURRENT0 is the name of the run you are preparing.
- PROG is the name of the executable. you should change this from run to run.
it resides in your ~/triple/3d/bin/ directory. if you reuse the name from
a previous run, it will overwrite the previous value used, which might cause
a problem if you want to save the executable for testing or reuse if
something goes wrong.
- SCRIPT is the name of the script that submits the executable to frost's
queueing system. codeit runs SCRIPT to get the job in the queue. you
must run SCRIPT after the run is finished to process the restart data,
concatenate the volume (and other) data, and move the results to archival
storage. since you must run the script by hand to process the data, it
is important that each run has a unique name for SCRIPT. to run it after
the job is finished, do the following.
~/triple/3d/bin/SCRIPT cleanup >& SCRIPT-cleanup.out &
here SCRIPT should be replaced with its specified value in codein.
collecing the output is a good idea so we can monitor problems, if they
appear.
5. choosing the best grid
- NX,NY,NZ are chosen by you. you can just set them to the largest values
i mentioned earlier and hope you are able to finish your project on time.
i chose the problem size so this is feasible, but it could be tight. you
can speed up the evolution in terms of wall-clock time by using the
minimum number of spectral modes necessary. output saved in
CURRENT0/o/CURRENT0.out.output helps you evaluate this. to take
advantage of this, do the following:
cd CURRENT0/o/
for KH problems: grep -e "_max " *.output
for GW problems: grep _max_xz *.output
the output includes the time on the left and values of the ratio of the
grid spacing over the kolmogorov scale for different methods of averaging.
for the GW problem, this number is off by a factor of 1.333 (oops!), but
that's not critical. just make sure you keep this value below about 1.34
for the GW solutions and 1.8 for the KH problems and your solutions will
be well resolved.
note however that the kolmogorov scale only makes sense to use as a guide
when the flow is fully turbulent. during the transitional flow in the
beginning and at the end, required resolutions will be greater than that
indicated by this ratio. it is always wise to augment statistical
metrics with visual inspection when monitoring the evolving resolution
requirements.
you can change NX, NY, and NZ from run to run, and the code will perform
spectral interpolation when continuing with a different grid from the
previous run.
6. examining your output
- you should look at the contents of the CURRENT0/o/CURRENT0.dat1 file
as soon as it is available. it contains the following parameters:
1.time, 2.dt, 3.theta^2 (theta is the potential temperature, 4.u.u
(u.u = u^2+v^2+w^2), 5.blah - ignore this, 6.cost, 7.sint, 8.cosv,
9.sinv (cost, sint, cosv, sinv are the coefficients for the sine and
cosine parts of the primary gravity wave as it evolves. you'll need
this to assess the wave amplitude versus time.), 10.max(theta),
11.min(theta), 12.max(u), 13.min(u), 14.max(v), 15.min(v), 16.max(w),
17.min(w), 18.max(vort_1), 19.min(vort_1), 20.max(vort_2), 21.min(vort_2),
22.max(vort_3), 23.min(vort_3), 24.theta_mid, 25.u_mid, 26.w_mid. this
file is a 32-bit (i.e., single precision), fixed record length, direct
access (i.e., it just has numbers in it - no header info).
- the volume data is only concatenated into single files after running
the post-processing script. it is then moved into the archival storage
system. you can find it by issuing a command like the following:
archive ls KH/3d/R22003d/R2200Ri05_1
archive get -C KH/3D/R2003d/R2200Ri05_1 R2200Ri05_1.txyz.03.001
these commands are based on the msread and mswrite commands. if you know
those, they work too.
the volume files are 32-bit, fixed-record-length, direct-access files in
a left-handed coordinate frame (Z,Y,X). for the KH problem, Z is vertical,
Y is in the spanwise direction across the flow direction, and X is the
streamwise direction. for the GW problem, Y points in the direction of
the wave's phase normal. Z is in the direction of the wave group
velocity, and Y is spanwiase, orthogonal to the group velocity.
7. so where are my codein and codeit files?
- codeit is in /ptmp/werne/R22003d/codeit -- take it and modify it as
necessary. The file should be copied to the directory you created in step (3)
- codein: the codein files for your project are in the following locations:
you will need to modify them to continue with the next run.
1. /ptmp/werne/R22003d/codein
2. /ptmp/werne/R24003d/codein
3. /ptmp/werne/R34003d/codein
4. see above. you must modify the following parameters in codein for
this project: ISGS=2, FSGS=0 (not used), ASGS=2, CSGS=0.1 (not used
for dynamic model)
5. /ptmp/werne/R5kth60a09/codein
6. /ptmp/werne/R5kth72a09/codein
7. /ptmp/werne/R5kth80a09/codein
8. see above. you must modify the following parameters in codein for
this project: ISGS=2, FSGS=0 (not used), ASGS=2, CSGS=0.1 (not used
for dynamic model)
8. Put a small dummy file on the mass store. It doesn't matter which file. For example go to the directory where you copied the "codein" file and run:
msrcp codein mss:/<MY_LOGIN_NAME_IN_CAPITAL_LETERS>/dummy_file
For Joe Werne the command would be:
msrcp codein mss:/WERENE/dummy_file
this will make sure your mass store directory exists. Type "msls" to make sure that this file was transfered.
9. Run codeit as described in (2) above (just type "codeit" in the directory where this script is located. This will create subdirectories for the old and new simulation.
this will automatically create a version file that we need to delete. First, delete the file called /ptmp/username/R*/R*/R*_2/R*_2.version (This assumes that you are going to use the R*_2 simulation to restart the new simulation.)
10. Copy the dat0 from the restart simulation to your /ptmp directory. For example from when you are in the /ptmp/username/R*/R* directory type:
cp -R /ptmp/werne/R*/R*/R*_2/dat0 R*_2/.
10. Now, copy the .version file from /ptmp/werne using
cp -R /ptmp/werne/R*/R*/R*_2/R*.version R*_2/.
11. Prepare a test job
Edit codein and change the name of PREVIFAR directory to the name of the simulation to use for the restart. If this is the first time editing codein, you just need to use the number following the underscore listed in CURRENT0 since this was the previous job. Then increment CURRENT0, PROG, AND SCRIPT by one. Finally, change SUBMIT to "no".
12. Run codeit again to make sure that the code compiles.
13. If successful, edit codein so that SUBMIT="yes", then type "codeit" -- you are a supercomputer!
14. How long is this run going to take? How long should I run?
The time it takes the code to complete on frost is given by:
wall time (in seconds) = C.NX.NY.NZ.NTOTAL ln(NX.NY.NZ) / NCPU
where C=1.48e-6.
For the KH DNS problems, try to get at least to t=150. If you can get as far as t=200, great! But to do that you will likely have to restart with a coarser grid when the resolution requirements have reduced.
For the GW DNS problems, try to get to t=50 or so. If you can get farther, great! Your goal is to get to a time where the max(vorticity components) are significant and fluctuating.
Data Analysis
The analysis you should do on your DNS/LES data is described in the HomeWork.pdf file located in frost:/ptmp/werne/student_info.
You will also find there an idl routine named lookdat1.pro that plots <uu+vv+ww> and Ri<tt> and the vorticity maxima versus time. Here <> refers to a volume average. Read the comments in the idl routine.
Preparing your simulation output data for analysis with VAPOR
To analyze your 3D output fields with VAPOR and IDL you will need to do the following:
- Copy the data you are interested in to the storm cluster. You only have 150GBs of space in your /ptmp directory on storm so don't copy more than you need.
- Transform the raw outputs from the simulation code into the VAPOR data format. The process for this is described by "VAPOR Documentation" section above
The basic steps for these operations are as follows:
1. Log on to one of the storm nodes (storm0, storm1... storm4) by ssh'ing through gate.ucar.edu:
- ssh my_login_name@gate.ucar.edu
at the gate.ucar.edu prompt type one of the storm node names. For example:
- storm2.scd.ucar.edu
2. From your storm window identify which files on the MSS archive you will copy to storm. The 3D volume file names contain the string "xyz" in them. The string the precedes the "xyz" indicates the variable name. For example, "uxyz" would be the U component of velocity. The files with the ".list" extension contain ascii text with other useful information about your run. The following command will list all of the files that you have:
- msls -lR
3. Change working directories to your storm /ptmp directory (all storm nodes see the same /ptmp file system):
- cd /ptmp/my_user_name
4. Copy the files from the MSS with the msrcp command. For example:
- msrcp mss:/USER_NAME_IN_CAPS/file_name file_name
5. Set up your environment to use the VAPOR commands:
- source /fs/local/bin/vapor-setup.csh
You can put the above source command in your .cshrc
6. Create a VAPOR .vdf file as described in the VAPOR documentation above with the command vdfcreate. The command:
- vdfcreate -level 2 -dimension 401x150x400 -varnames t:u:v:w -numts 20 R2200Ri05.vdf
would create a .vdf file named "R2200Ri05.vdf" that is ready to contain up to 20 time steps; 4 variables named "t", "u", "v", and "w"; for a volume with grid dimension 401x150x400; and would have two multi-resolution transformation levels. You may want more than two transform levels for larger data than 401x150x400. Note that X and Z are transposed in Joe's simulation code.
7. For each simulation output file transform it into a VAPOR Data Collection (VDC), associated with your .vdf file, using the raw2vdf command. For example, the command
- raw2vdf -swapbytes -varname t -ts 0 R2200Ri05.vdf R2200Ri05_1.txyz.00.001
would transform the temperature volume file, "R2200Ri05_1.txyz.00.001", inserting it at the first time step in the VDC. Note, the -swapbytes option is need to handle endian differences between Frost and the storm cluster.
8. Now you are ready to visualize with VAPOR!
IDL Scripts
The template IDL scripts that you will need to create derived quantities from you 3D data fields are on the storm cluster in /cxfs/DASG/clyne/TOY/idl
Copy all of these files to your /ptmp directory before using them. Note, the scripts write data to the current working directory. Do NOT try to run them from your home directory. They should be run from /ptmp or you will run out of space.
These scripts are intended to be used from data "exported" from VAPOR. If you want to read data in other formats you will have to modify them.
There are two scripts that are intended to be edited by you as needed:
vorticity_ie.pro : computes vorticity and writes the result to a vapor data set, vorticity_ie.vdf
eke_ie.pro : computes eddy kinetic energy and writes results to a vapor data set, eke_ie.vdf
To run either script you must first "export" data from VAPOR via the 'Data/Export to IDL' button. Be careful about exporting a region that is too large, or you will run out of memory, etc.
Multiple time steps
A number of you have asked how to calculate derived quantities (e.g. vorticity) for multiple time steps. There is a new script, vorticity_multi_ie.pro, that demonstrates how to accomplish this for vorticity. It's a quick hack and you'll need to edit to variables before you can use it: TSFIRST, and TSLAST. These are documented in the script. In order to use it you'll need new versions of expregions.pro and impregions.pro as well. All of these scripts may be found in
/cxfs/DASG/clyne/TOY/idl
Making animations out of jpeg files
All, you can create mpeg animations on the storm nodes with the ffmpeg command. The basic syntax is
ffmpeg -r FRAMERATE -qmax QUALITY -i FILEPREFIX%04d.jpg MOVIFILE.mp4
Where
FRAMERATE : is the movie play back rate in frames per second. I suggest 5 or 10
QUALITY: is an integer quality number. 2 will give the best results (what I recommend). Larger values will give poorer quality, but smaller files
FILEPREFIX: is the prefix for the jpeg files you generated with VAPOR. The '%04d' is used to match the index numbers of your jpegs.
MOVIEFILE: is simply the name of your output file.
For example:
ffmpeg -r 10 -qmax 2 -i otp%04d.jpg mymovie.mp4
Note, I believe the first jpeg file must be indexed 0000, for example myfile0000.jpg. Also, the jpegs must be in sequential order with no gaps
Also, you may or may not be able to play these from within powerpoint/keynote. If you can't, you should still be able to play them in a stand alone movieplayer (e.g. quicktime, windows media player)