| Docker | Singularity |
---|
Permissions | the process inside the container is running with “root” privileges | user permissions inside the Singularity container are the same as outside the container |
Filesystem | has no direct access to the host’s file system except for those directories that are explicitly cross mounted into the container. | the process that the user is running sees the user’s home directory, and the user’s environment is shared seamlessly. |
Image | Docker images come in layers, and they are assembled like layers of a cake to generate a running image | Singularity image is an actual file that you could put in entirety on your desktop, and move around as you would with any other file. |
Step-by-step guide
Installation (
You don’t need to have Docker installed)
Please visit here to install the Singularity tools on your computer. (root privilege is needed)
Check out the Singularity image
[xinzhang@localhost jedi]$ singularity pull shub://JCSDA/singularity
Progress |===================================| 100.0%
Done. Container is at: /home/xinzhang/jedi/JCSDA-singularity-master.simg
Running the Singularity image
[xinzhang@localhost jedi]$ singularity shell -e JCSDA-singularity-master.simg # -e option cleans all environment before running container
Singularity: Invoking an interactive shell within container...
Singularity JCSDA-singularity-master.simg:~/jedi> cd
Singularity JCSDA-singularity-master.simg:~> pwd
/home/xinzhang
Check out the GSI code and get case (T24) data
Singularity JCSDA-singularity-master.simg:~> mkdir -p ~/jedi/code
Singularity JCSDA-singularity-master.simg:~> cd ~/jedi/code
Singularity JCSDA-singularity-master.simg:~> git clone https://Your_UserName@bitbucket.org/jcsda/gsi-vlab.git gsi # Check out the GSI code, please replace Your_UserName with your bitbucket username
Singularity JCSDA-singularity-master.simg:~> cd gsi/
Singularity JCSDA-singularity-master.simg:~> git submodule init # Check out the external repositories
Submodule 'fix' (git@bitbucket.org:jcsda/fix) registered for path 'fix'
Submodule 'libsrc' (git@bitbucket.org:jcsda/external-libs) registered for path 'libsrc'
Singularity JCSDA-singularity-master.simg:~> git submodule update # Check out the external repositories
Cloning into 'fix'...
remote: Counting objects: 332, done.
remote: Compressing objects: 100% (235/235), done.
remote: Total 332 (delta 102), reused 320 (delta 96)
Receiving objects: 100% (332/332), 961.58 MiB | 357.00 KiB/s, done.
Resolving deltas: 100% (102/102), done.
Checking connectivity... done.
Submodule path 'fix': checked out 'd13e5a23dde538ceefedc39746b029878ee6ff60'
Cloning into 'libsrc'...
remote: Counting objects: 896, done.
remote: Compressing objects: 100% (630/630), done.
remote: Total 896 (delta 263), reused 890 (delta 260)
Receiving objects: 100% (896/896), 5.57 MiB | 1.59 MiB/s, done.
Resolving deltas: 100% (263/263), done.
Checking connectivity... done.
Submodule path 'libsrc': checked out '8fa69e1fd011b658de6cd2e9f08cf1ad77566f4f'
Singularity JCSDA-singularity-master.simg:~> cd fix/
Singularity JCSDA-singularity-master.simg:~> git checkout rev1
Singularity JCSDA-singularity-master.simg:~> cd ~/jedi
Singularity JCSDA-singularity-master.simg:~> wget ftp://ftp.ucar.edu/pub/mmm/xinzhang/caseT24.tar.gz
Singularity JCSDA-singularity-master.simg:~> tar xvf caseT24.tar.gz
Singularity JCSDA-singularity-master.simg:~>$ rm caseT24.tar.gz
Singularity JCSDA-singularity-master.simg:~> ls -la
total 1163160
drwxrwxr-x. 5 xinzhang xinzhang 64 Oct 4 12:11 .
drwxrwxr-x. 4 xinzhang xinzhang 33 Oct 4 11:30 ..
drwxrwxr-x. 3 xinzhang xinzhang 17 Oct 4 11:30 code
drwxrwxr-x. 4 xinzhang xinzhang 36 Oct 3 13:34 data
drwxrwxr-x. 2 xinzhang xinzhang 32 Oct 3 13:55 test
1) T24 background(size: 52M):
On Theia: /scratch3/BMC/wrfruc/mhu/jedi/casedata/T24_2016030406/gest24/
On Docker: ~/jedi/data/case/T24_2016030406/gest24
aircraft_t_bias.gdas.2016030400 sfcf05.gdas.2016030400 sigf04.gdas.2016030400
biascr.gdas.2016030400 sfcf06.gdas.2016030400 sigf05.gdas.2016030400
biascr_pc.gdas.2016030400 sfcf07.gdas.2016030400 sigf06.gdas.2016030400
radstat.gdas.2016030400 sfcf08.gdas.2016030400 sigf07.gdas.2016030400
sfcf03.gdas.2016030400 sfcf09.gdas.2016030400 sigf08.gdas.2016030400
sfcf04.gdas.2016030400 sigf03.gdas.2016030400 sigf09.gdas.2016030400
2) T24 ensemble forecast(size:178M):
On Theia:/scratch3/BMC/wrfruc/mhu/jedi/casedata/T24_2016030406/enst24
On Docker: ~/jedi/data/case/T24_2016030406/enst24
20 member, hourly output during 3-h to 09-h forecast initialized from 2016030400
sfg_2016030400_fhr03s_mem020
sfg_2016030400_fhr04s_mem020
sfg_2016030400_fhr05s_mem020
sfg_2016030400_fhr06s_mem020
sfg_2016030400_fhr07s_mem020
sfg_2016030400_fhr08s_mem020
sfg_2016030400_fhr09s_mem020
3) Observations:
On Theia:/scratch3/BMC/wrfruc/mhu/jedi/casedata/T24_2016030406/obs
On Docker: ~/jedi/data/case/T24_2016030406/obs
3.1) Original observation file: 1bamua.gdas.2016030406, prepbufr.gdas.2016030406
3.2) Small observation file: 1bamua.gdas_picked, prepbufr_picked
List of observations in prepbufr_listobs.txt:
header: number, level number, SID XOB YOB DHR TYP ELV SAID T29
observation:POB QOB TOB ZOB UOB VOB PWO CAT PRSS
obs quality: PQM QQM TQM ZQM WQM NUL PWQ
obs error: POE QOE TOE NUL WOE NUL PWE
List of observations in 1bamua_listobs.txt:
header: number, channel number, SAID FOVN YEAR MNTH DAYS HOUR MINU SECO CLAT CLON CLATH CLONH HOLS
header 2: SAZA SOZA BEARAZ SOLAZI
obs (channel):
Build GSI code and run GSI case
suppose you already checked out the GSI code as $HOME/jedi/code/gsi and have test case directory as $HOME/jedi/data.
Singularity JCSDA-singularity-master.simg:~> cd ~
Singularity JCSDA-singularity-master.simg:~> mkdir -p ~/jedi/build/gsi # Prepare the directory for GSI building, out-of-source building
Singularity JCSDA-singularity-master.simg:~> cd ~/jedi/build/gsi # Enter into directory for GSI building
Singularity JCSDA-singularity-master.simg:~> export FC=mpif90
Singularity JCSDA-singularity-master.simg:~> cmake -DBUILD_CORELIBS=ON -DUSE_WRF=OFF -DBUILD_GLOBAL=ON ~/jedi/code/gsi
Singularity JCSDA-singularity-master.simg:~> make -j`nproc` # parallel compilation with the availble cores
Singularity JCSDA-singularity-master.simg:~> ls -al bin/
total 44560
drwxr-xr-x. 2 root root 44 Sep 29 17:24 .
drwxr-xr-x. 9 1000 1001 251 Sep 29 17:17 ..
-rwxr-xr-x. 1 root root 8266248 Sep 29 17:24 enkf_gfs.x
-rwxr-xr-x. 1 root root 37358984 Sep 29 17:23 gsi_global.x
Singularity JCSDA-singularity-master.simg:~> cd /jedi/test
# Please modify following variables to reflect the correct PATHS in ./run_gsi_picked.csh
####################################################
# set DATA = ~/jedi/test/$PSLOT
# set APRUN = "mpirun -np 2 "
# set CASEDIR = ~/jedi/data/case/T24_2016030406
# set BASEGSI = ~/jedi/code/gsi/
# set FIXcrtm = ~/jedi/data/CRTM_2.2.3/
# set GSIEXEC = ~/jedi/build/gsi/bin/gsi_global.x
######################################################
Singularity JCSDA-singularity-master.simg:~> ./run_gsi_picked.csh
After a successful GSI run, we can check the O-B for each observation by reading diagnosis files.
1) combine subdomain drag files to one drag file:
> cd ~/jedi/test/testT24_picked
> ls dir.*
> cat dir.0000/conv_01 dir.0001/conv_01 > conv_ges
> cat dir.0000/amsua_metop-b_01 dir.0001/amsua_metop-b_01 > amsua_metop-b_ges
2) compile read_diag utilities:
> cd ~/jedi/code/gsi/util/Analysis_Utilities/read_diag
3) Edit makefile and change the variable "INCLD" to reflect the correct path of GSI code, for example :
INCLD= -I/home/xinzhang/jedi/build/gsi/include
> make
Should generate three executables: read_diag_conv.exe read_diag_conv_ens.exe read_diag_rad.exe
3) read conventional observations:
> vi namelist.conv
It should be set up like this:
&iosetup
infilename='/home/xinzhang/jedi/test/testT24_picked/conv_ges',
outfilename='./results_conv_ges',
/
> ./read_diag_conv.exe
Now, we should see a text file: results_conv_ges.
The content of each column is listed below
1-3: variable name,station ID,itype,
4-7: obs relative time,latitude,longitude,pressure,
8-14:iuse,observation,background, O-B (for wind, U and then V)
4) read radiance diag:
> vi namelist.rad
It should be set up like:
&isotope
infilename='/home/xinzhang/jedi/test/testT24_picked/amsua_metop-b_ges',
outfilename='./results_amsua_metop-b_ges',
/
> ./read_diag_rad.exe
Now, we should see a new text file: results_amsua_metop-b_ges
The content of this text file is listed below:
head information (number 1-26):
1:observation latitude (degrees)
2:observation longitude (degrees)
3:model (guess) elevation at observation location
4:observation time (hours relative to analysis time)
5:sensor scan position
6:satellite zenith angle (degrees)
7:satellite azimuth angle (degrees)
8:solar zenith angle (degrees)
9:solar azimuth angle (degrees)
10:sun glint angle (degrees) (sgagl)
11:fractional coverage by water
12:fractional coverage by land
13:fractional coverage by ice
14:fractional coverage by snow
15:surface temperature over water (K)
16:surface temperature over land (K)
17:surface temperature over ice (K)
18:surface temperature over snow (K)
19:soil temperature (K)
20:soil moisture OR graupel water path (if gmi .or. saphir)
21:surface land type
22:vegetation fraction OR scattering index from AMSU-A (if radmod%lcloud_fwd .and. sea)
23:snow depth OR integrated CLWP (kg/m**2) from background (if radmod%lcloud_fwd .and. sea)
24:surface wind speed (m/s)
25:cloud fraction (%) OR if microwave:
cloud liquid water (kg/m**2) OR
cloud liquid water (kg/m**2) if(radmod%lcloud_fwd .and. sea)
clw (kg/m**2) from retrievals if(gmi .or. amsr2)
26:cloud top pressure (hPa) OR if microwave:
total column precip. water (km/m**2) OR
retrieved CLWP (kg/m**2) from simulated BT if((radmod%lcloud_fwd .and. sea) .or. gmi .or. amsr2)
channel information (loop through each channel with column list), for each column:
1: observed brightness temperature (K)
2: observed - simulated Tb with bias correction (K)
3: observed - simulated Tb with no bias correction (K)
4: inverse observation error
5: quality control mark or event indicator
6: surface emissivity
7: stability index
8: indicator of cloudy consistency (if lcloud_fwd) OR d(Tb)/d(Ts)
Check out the OOPS code
Singularity JCSDA-singularity-master.simg:~> cd ~/jedi/code
Singularity JCSDA-singularity-master.simg:~> git clone -b feature/ufo https://github.com/UCAR/oops.git # Check out the oops code to oops directory
Build and test OOPS code
Singularity JCSDA-singularity-master.simg:~> cd ~
Singularity JCSDA-singularity-master.simg:~> mkdir -p ~/jedi/build/oops # Prepare the directory for oops building, out-of-source building
Singularity JCSDA-singularity-master.simg:~> cd ~/jedi/build/oops
Singularity JCSDA-singularity-master.simg:~> ecbuild --build=release -DLAPACK_LIBRARIES=$LAPACK_LIBRARIES ~/jedi/code/oops
Singularity JCSDA-singularity-master.simg:~> make -j`nproc`
...
...
[ 99%] Built target test_qg_4dvar_obsbias.x
[ 99%] Built target test_qg_3dfgat.x
Scanning dependencies of target test_qg_4densvar.x
[ 99%] Built target test_qg_4densvar.x
[ 99%] Linking CXX executable test_qg_increment
[ 99%] Built target test_qg_increment
[100%] Linking CXX executable test_qg_localization
[100%] Built target test_qg_localization
Singularity JCSDA-singularity-master.simg:~> ctest
...
...
108/108 Test #108: test_qg_dfi ...................... Passed 0.46 sec
100% tests passed, 0 tests failed out of 108
Label Time Summary:
boost = 1.13 sec (53 tests)
executable = 1.16 sec (56 tests)
fortran = 0.04 sec (3 tests)
oops = 23.89 sec (108 tests)
script = 22.73 sec (52 tests)
Total Test time (real) = 23.94 sec
Download WRF and WPS
Singularity JCSDA-singularity-master.simg:~> cd ~/jedi/code
Singularity JCSDA-singularity-master.simg:~> wget http://www2.mmm.ucar.edu/wrf/src/WPSV3.9.1.TAR.gz
Singularity JCSDA-singularity-master.simg:~> tar xvf WPSV3.9.1.TAR.gz
Singularity JCSDA-singularity-master.simg:~> rm -f WPSV3.9.1.TAR.gz
Singularity JCSDA-singularity-master.simg:~> wget http://www2.mmm.ucar.edu/wrf/src/WRFV3.9.1.1.TAR.gz
Singularity JCSDA-singularity-master.simg:~> tar xvf WRFV3.9.1.1.TAR.gz
Singularity JCSDA-singularity-master.simg:~> rm -f WRFV3.9.1.1.TAR.gz
Singularity JCSDA-singularity-master.simg:~> ls -la
total 12
drwxrwxr-x. 6 xinzhang xinzhang 53 Oct 5 16:01 .
drwxrwxr-x. 13 xinzhang xinzhang 196 Oct 5 14:45 ..
drwxrwxr-x. 13 xinzhang xinzhang 243 Oct 5 14:29 gsi
drwxrwxr-x. 10 xinzhang xinzhang 4096 Oct 5 11:13 oops
drwxr-xr-x. 7 xinzhang xinzhang 4096 Aug 17 11:10 WPS
drwxr-xr-x. 17 xinzhang xinzhang 4096 Aug 28 15:01 WRFV3
Build WRF and WPS code
Singularity JCSDA-singularity-master.simg:~> cd ~/jedi/code/WRFV3
Singularity JCSDA-singularity-master.simg:~> ./configure # Select 34 for dmpar gnu
Singularity JCSDA-singularity-master.simg:~> vi configure.wrf # Becasue WRF has problem to test mpi2 support, please add -DMPI2_SUPPORT after DM_CC = mpicc
Singularity JCSDA-singularity-master.simg:~> ./compile em_real # For unknown reason, module_ra_rrtmg_swf.f90 might fail to be compiled due to internal compiler error, typing "./compile em_real" again usually can solve the problem.
...
...
==========================================================================
build started: Thu Oct 5 22:12:18 UTC 2017
build completed: Thu Oct 5 22:21:58 UTC 2017
---> Executables successfully built <---
-rwxr-xr-x. 1 root root 36472488 Oct 5 22:21 main/ndown.exe
-rwxr-xr-x. 1 root root 36346432 Oct 5 22:21 main/real.exe
-rwxr-xr-x. 1 root root 35944328 Oct 5 22:21 main/tc.exe
-rwxr-xr-x. 1 root root 40051304 Oct 5 22:21 main/wrf.exe
==========================================================================
Singularity JCSDA-singularity-master.simg:~> cd ../WPS
Singularity JCSDA-singularity-master.simg:~> ./configure # Select 3 for dmpar gfortran
Singularity JCSDA-singularity-master.simg:~> vi configure.wrf # edit COMPRESSION_LIBS and COMPRESSION_INC as following lines and remove -f90=gfortran from DM_FC and remove -cc=gcc from DM_CC
...
...
...
#
# Settings for Linux x86_64, gfortran (dmpar)
#
#
COMPRESSION_LIBS = -L/usr/local/lib -ljasper -lpng -lz
COMPRESSION_INC = -I/usr/local/include
FDEFS = -DUSE_JPEG2000 -DUSE_PNG
SFC = gfortran
SCC = gcc
DM_FC = mpif90
DM_CC = mpicc
...
...
...
Singularity JCSDA-singularity-master.simg:~> ./compile
Singularity JCSDA-singularity-master.simg:~> ls -al *.exe
lrwxrwxrwx. 1 root root 23 Oct 6 16:02 geogrid.exe -> geogrid/src/geogrid.exe
lrwxrwxrwx. 1 root root 23 Oct 6 16:02 metgrid.exe -> metgrid/src/metgrid.exe
lrwxrwxrwx. 1 root root 21 Oct 6 16:02 ungrib.exe -> ungrib/src/ungrib.exe
Download MPAS code
Singularity JCSDA-singularity-master.simg:~> cd ~/jedi/code
Singularity JCSDA-singularity-master.simg:~> git clone -b v5.2 https://github.com/MPAS-Dev/MPAS-Release.git
Build MPAS code
Singularity JCSDA-singularity-master.simg:~> cd ~/jedi/code/MPAS-Release
Singularity JCSDA-singularity-master.simg:~> vi ./src/core_atmosphere/physics/checkout_data_files.sh # Replace the MPAS-Data address to https (see next line below)
Singularity JCSDA-singularity-master.simg:~> git diff
diff --git a/src/core_atmosphere/physics/checkout_data_files.sh b/src/core_atmosphere/physics/checkout_data_files.sh
index e62b466..f2c3cba 100755
--- a/src/core_atmosphere/physics/checkout_data_files.sh
+++ b/src/core_atmosphere/physics/checkout_data_files.sh
@@ -56,7 +56,7 @@ fi
which git
if [ $? -eq 0 ]; then
echo "*** trying git to obtain WRF physics tables ***"
- git clone git://github.com/MPAS-Dev/MPAS-Data.git
+ git clone https://github.com/MPAS-Dev/MPAS-Data.git
if [ $? -eq 0 ]; then
mv MPAS-Data/atmosphere/physics_wrf/files/* physics_wrf/files
rm -rf MPAS-Data
Singularity JCSDA-singularity-master.simg:~> make gfortran CORE=atmosphere
...
...
make[2]: Leaving directory '/home/xinzhang/jedi/code/MPAS-Release/src/core_atmosphere'
*******************************************************************************
MPAS was built with default double-precision reals.
Debugging is off.
Parallel version is on.
Papi libraries are off.
TAU Hooks are off.
MPAS was built without OpenMP support.
MPAS was built with .F files.
The native timer interface is being used
Using the PIO 1.x library.
*******************************************************************************
make[1]: Leaving directory '/home/xinzhang/jedi/code/MPAS-Release'
Test MPAS
Singularity xinzhang8noaa-singularity-master.simg:~/jedi/code/MPAS-Release> wget ftp://ftp.ucar.edu/pub/mmm/xinzhang/mpas_10242_case.tgz
Singularity xinzhang8noaa-singularity-master.simg:~/jedi/code/MPAS-Release> tar xvf mpas_10242_case.tgz
Singularity xinzhang8noaa-singularity-master.simg:~/jedi/code/MPAS-Release> mpirun -np 8 atmosphere_model # or mpirun -np 1 atmosphere_model
task 0 of 8 is running
task 1 of 8 is running
task 2 of 8 is running
task 3 of 8 is running
task 6 of 8 is running
task 7 of 8 is running
task 4 of 8 is running
task 5 of 8 is running
Check out the FV3 code
Singularity JCSDA-singularity-master.simg:~> cd ~/jedi/code
Singularity JCSDA-singularity-master.simg:~> git clone https://Your_UserName@bitbucket.org/jcsda/comfv3.git comfv3 # Check out the FV3 code, please replace Your_UserName with your bitbucket username
Singularity JCSDA-singularity-master.simg:~> cd comfv3
Singularity JCSDA-singularity-master.simg:~> git submodule init
Singularity JCSDA-singularity-master.simg:~> git submodule update
Singularity JCSDA-singularity-master.simg:~> git branch -a
Singularity JCSDA-singularity-master.simg:~> git checkout feature/gfortran_build
Singularity JCSDA-singularity-master.simg:~> cd FV3
Singularity JCSDA-singularity-master.simg:~> git checkout feature/gfortran_build
Singularity JCSDA-singularity-master.simg:~> cd ../NEMS
Singularity JCSDA-singularity-master.simg:~> git checkout feature/gfortran_build
Build the FV3 code
Singularity JCSDA-singularity-master.simg:~> cd ~/jedi/code/comfv3/release/v0/exp/
Singularity JCSDA-singularity-master.simg:~> ./build.sh macgnu
...
...
...
Elapsed time 66 seconds. Compiling HYDRO=Y 32BIT=N finished
+ cp /home/xinzhang/jedi/code/comfv3/release/v0/exp/../../../tests/fv3_1.exe ../NEMS/exe/fv3_gfs_hydro.prod.64bit.x
+ rm /home/xinzhang/jedi/code/comfv3/release/v0/exp/../../../tests/fv3_1.exe
+ exit 0
Check out and build the CRTM V2.2.3 code
Singularity JCSDA-singularity-master.simg:~> cd ~/jedi/code
Singularity JCSDA-singularity-master.simg::~/jedi/code> git clone https://YourUserName@bitbucket.org/jcsda/crtm-release.git crtm_v2.2.3 # please replace Your_UserName with your bitbucket username
Singularity JCSDA-singularity-master.simg::~/jedi/code> cd crtm_v2.2.3/
Singularity JCSDA-singularity-master.simg::~/jedi/code/crtm_v2.2.3> source config-setup/gfortran.setup
Singularity JCSDA-singularity-master.simg::~/jedi/code/crtm_v2.2.3>./configure --prefix=/home/`whoami`/jedi/build/crtm
Singularity JCSDA-singularity-master.simg::~/jedi/code/crtm_v2.2.3> make
Singularity JCSDA-singularity-master.simg::~/jedi/code/crtm_v2.2.3> make check
Singularity JCSDA-singularity-master.simg::~/jedi/code/crtm_v2.2.3> make install
Related articles
-
Page:
-
Page:
-
Page:
-
Page:
-
Page:
Build MPAS code