Describe when someone would need this information. For example "when connecting to wi-fi for the first time".
Docker | Singularity | |
---|---|---|
Permissions | the process inside the container is running with “root” privileges | user permissions inside the Singularity container are the same as outside the container |
Filesystem | has no direct access to the host’s file system except for those directories that are explicitly cross mounted into the container. | the process that the user is running sees the user’s home directory, and the user’s environment is shared seamlessly. |
Image | Docker images come in layers, and they are assembled like layers of a cake to generate a running image | Singularity image is an actual file that you could put in entirety on your desktop, and move around as you would with any other file. |
Add the steps involved:
Installation (
You don’t need to have Docker installed)Please visit here to install the Singularity tools on your computer. (root privilege is needed)
It is critical to install the latest official release. http://singularity.lbl.gov/release-2-4, as of Oct. 2, 2017 Click here for instructions to install Singularity on Mac OS X.
|
Check out the Singularity image
[xinzhang@localhost jedi]$ singularity pull shub://JCSDA/singularity Progress |===================================| 100.0% Done. Container is at: /home/xinzhang/jedi/JCSDA-singularity-master.simg |
Running the Singularity image
[xinzhang@localhost jedi]$ singularity shell -e JCSDA-singularity-master.simg # -e option cleans all environment before running container Singularity: Invoking an interactive shell within container... Singularity JCSDA-singularity-master.simg:~/jedi> cd Singularity JCSDA-singularity-master.simg:~> pwd /home/xinzhang |
All the steps below are optional and depend on what you intend to use. To work with the jedi-bundle, follow the instructions in the README of that repository once you have reached this point. To exit the singularity container or vagrant at any time, type exit |
Singularity JCSDA-singularity-master.simg:~> mkdir -p ~/jedi/code Singularity JCSDA-singularity-master.simg:~> cd ~/jedi/code Singularity JCSDA-singularity-master.simg:~> git clone https://Your_UserName@bitbucket.org/jcsda/gsi-vlab.git gsi # Check out the GSI code, please replace Your_UserName with your bitbucket username Singularity JCSDA-singularity-master.simg:~> cd gsi/ Singularity JCSDA-singularity-master.simg:~> git submodule init # Check out the external repositories Submodule 'fix' (git@bitbucket.org:jcsda/fix) registered for path 'fix' Submodule 'libsrc' (git@bitbucket.org:jcsda/external-libs) registered for path 'libsrc' Singularity JCSDA-singularity-master.simg:~> git submodule update # Check out the external repositories Cloning into 'fix'... remote: Counting objects: 332, done. remote: Compressing objects: 100% (235/235), done. remote: Total 332 (delta 102), reused 320 (delta 96) Receiving objects: 100% (332/332), 961.58 MiB | 357.00 KiB/s, done. Resolving deltas: 100% (102/102), done. Checking connectivity... done. Submodule path 'fix': checked out 'd13e5a23dde538ceefedc39746b029878ee6ff60' Cloning into 'libsrc'... remote: Counting objects: 896, done. remote: Compressing objects: 100% (630/630), done. remote: Total 896 (delta 263), reused 890 (delta 260) Receiving objects: 100% (896/896), 5.57 MiB | 1.59 MiB/s, done. Resolving deltas: 100% (263/263), done. Checking connectivity... done. Submodule path 'libsrc': checked out '8fa69e1fd011b658de6cd2e9f08cf1ad77566f4f' Singularity JCSDA-singularity-master.simg:~> cd fix/ Singularity JCSDA-singularity-master.simg:~> git checkout rev1 |
Singularity JCSDA-singularity-master.simg:~> cd ~/jedi Singularity JCSDA-singularity-master.simg:~> wget ftp://ftp.ucar.edu/pub/mmm/xinzhang/caseT24.tar.gz Singularity JCSDA-singularity-master.simg:~> tar xvf caseT24.tar.gz Singularity JCSDA-singularity-master.simg:~>$ rm caseT24.tar.gz Singularity JCSDA-singularity-master.simg:~> ls -la total 1163160 drwxrwxr-x. 5 xinzhang xinzhang 64 Oct 4 12:11 . drwxrwxr-x. 4 xinzhang xinzhang 33 Oct 4 11:30 .. drwxrwxr-x. 3 xinzhang xinzhang 17 Oct 4 11:30 code drwxrwxr-x. 4 xinzhang xinzhang 36 Oct 3 13:34 data drwxrwxr-x. 2 xinzhang xinzhang 32 Oct 3 13:55 test |
1) T24 background(size: 52M): On Theia: /scratch3/BMC/wrfruc/mhu/jedi/casedata/T24_2016030406/gest24/ On Docker: ~/jedi/data/case/T24_2016030406/gest24 aircraft_t_bias.gdas.2016030400 sfcf05.gdas.2016030400 sigf04.gdas.2016030400 biascr.gdas.2016030400 sfcf06.gdas.2016030400 sigf05.gdas.2016030400 biascr_pc.gdas.2016030400 sfcf07.gdas.2016030400 sigf06.gdas.2016030400 radstat.gdas.2016030400 sfcf08.gdas.2016030400 sigf07.gdas.2016030400 sfcf03.gdas.2016030400 sfcf09.gdas.2016030400 sigf08.gdas.2016030400 sfcf04.gdas.2016030400 sigf03.gdas.2016030400 sigf09.gdas.2016030400 2) T24 ensemble forecast(size:178M): On Theia:/scratch3/BMC/wrfruc/mhu/jedi/casedata/T24_2016030406/enst24 On Docker: ~/jedi/data/case/T24_2016030406/enst24 20 member, hourly output during 3-h to 09-h forecast initialized from 2016030400 sfg_2016030400_fhr03s_mem020 sfg_2016030400_fhr04s_mem020 sfg_2016030400_fhr05s_mem020 sfg_2016030400_fhr06s_mem020 sfg_2016030400_fhr07s_mem020 sfg_2016030400_fhr08s_mem020 sfg_2016030400_fhr09s_mem020 3) Observations: On Theia:/scratch3/BMC/wrfruc/mhu/jedi/casedata/T24_2016030406/obs On Docker: ~/jedi/data/case/T24_2016030406/obs 3.1) Original observation file: 1bamua.gdas.2016030406, prepbufr.gdas.2016030406 3.2) Small observation file: 1bamua.gdas_picked, prepbufr_picked List of observations in prepbufr_listobs.txt: header: number, level number, SID XOB YOB DHR TYP ELV SAID T29 observation:POB QOB TOB ZOB UOB VOB PWO CAT PRSS obs quality: PQM QQM TQM ZQM WQM NUL PWQ obs error: POE QOE TOE NUL WOE NUL PWE List of observations in 1bamua_listobs.txt: header: number, channel number, SAID FOVN YEAR MNTH DAYS HOUR MINU SECO CLAT CLON CLATH CLONH HOLS header 2: SAZA SOZA BEARAZ SOLAZI obs (channel): |
suppose you already checked out the GSI code as $HOME/jedi/code/gsi and have test case directory as $HOME/jedi/data.
Singularity JCSDA-singularity-master.simg:~> cd ~ Singularity JCSDA-singularity-master.simg:~> mkdir -p ~/jedi/build/gsi # Prepare the directory for GSI building, out-of-source building Singularity JCSDA-singularity-master.simg:~> cd ~/jedi/build/gsi # Enter into directory for GSI building Singularity JCSDA-singularity-master.simg:~> export FC=mpif90 Singularity JCSDA-singularity-master.simg:~> cmake -DBUILD_CORELIBS=ON -DUSE_WRF=OFF -DBUILD_GLOBAL=ON ~/jedi/code/gsi Singularity JCSDA-singularity-master.simg:~> make -j`nproc` # parallel compilation with the availble cores Singularity JCSDA-singularity-master.simg:~> ls -al bin/ total 44560 drwxr-xr-x. 2 root root 44 Sep 29 17:24 . drwxr-xr-x. 9 1000 1001 251 Sep 29 17:17 .. -rwxr-xr-x. 1 root root 8266248 Sep 29 17:24 enkf_gfs.x -rwxr-xr-x. 1 root root 37358984 Sep 29 17:23 gsi_global.x Singularity JCSDA-singularity-master.simg:~> cd /jedi/test # Please modify following variables to reflect the correct PATHS in ./run_gsi_picked.csh #################################################### # set DATA = ~/jedi/test/$PSLOT # set APRUN = "mpirun -np 2 " # set CASEDIR = ~/jedi/data/case/T24_2016030406 # set BASEGSI = ~/jedi/code/gsi/ # set FIXcrtm = ~/jedi/data/CRTM_2.2.3/ # set GSIEXEC = ~/jedi/build/gsi/bin/gsi_global.x ###################################################### Singularity JCSDA-singularity-master.simg:~> ./run_gsi_picked.csh |
After a successful GSI run, we can check the O-B for each observation by reading diagnosis files. 1) combine subdomain drag files to one drag file: > cd ~/jedi/test/testT24_picked > ls dir.* > cat dir.0000/conv_01 dir.0001/conv_01 > conv_ges > cat dir.0000/amsua_metop-b_01 dir.0001/amsua_metop-b_01 > amsua_metop-b_ges 2) compile read_diag utilities: > cd ~/jedi/code/gsi/util/Analysis_Utilities/read_diag 3) Edit makefile and change the variable "INCLD" to reflect the correct path of GSI code, for example : INCLD= -I/home/xinzhang/jedi/build/gsi/include > make Should generate three executables: read_diag_conv.exe read_diag_conv_ens.exe read_diag_rad.exe 3) read conventional observations: > vi namelist.conv It should be set up like this: &iosetup infilename='/home/xinzhang/jedi/test/testT24_picked/conv_ges', outfilename='./results_conv_ges', / > ./read_diag_conv.exe Now, we should see a text file: results_conv_ges. The content of each column is listed below 1-3: variable name,station ID,itype, 4-7: obs relative time,latitude,longitude,pressure, 8-14:iuse,observation,background, O-B (for wind, U and then V) 4) read radiance diag: > vi namelist.rad It should be set up like: &isotope infilename='/home/xinzhang/jedi/test/testT24_picked/amsua_metop-b_ges', outfilename='./results_amsua_metop-b_ges', / > ./read_diag_rad.exe Now, we should see a new text file: results_amsua_metop-b_ges The content of this text file is listed below: head information (number 1-26): 1:observation latitude (degrees) 2:observation longitude (degrees) 3:model (guess) elevation at observation location 4:observation time (hours relative to analysis time) 5:sensor scan position 6:satellite zenith angle (degrees) 7:satellite azimuth angle (degrees) 8:solar zenith angle (degrees) 9:solar azimuth angle (degrees) 10:sun glint angle (degrees) (sgagl) 11:fractional coverage by water 12:fractional coverage by land 13:fractional coverage by ice 14:fractional coverage by snow 15:surface temperature over water (K) 16:surface temperature over land (K) 17:surface temperature over ice (K) 18:surface temperature over snow (K) 19:soil temperature (K) 20:soil moisture OR graupel water path (if gmi .or. saphir) 21:surface land type 22:vegetation fraction OR scattering index from AMSU-A (if radmod%lcloud_fwd .and. sea) 23:snow depth OR integrated CLWP (kg/m**2) from background (if radmod%lcloud_fwd .and. sea) 24:surface wind speed (m/s) 25:cloud fraction (%) OR if microwave: cloud liquid water (kg/m**2) OR cloud liquid water (kg/m**2) if(radmod%lcloud_fwd .and. sea) clw (kg/m**2) from retrievals if(gmi .or. amsr2) 26:cloud top pressure (hPa) OR if microwave: total column precip. water (km/m**2) OR retrieved CLWP (kg/m**2) from simulated BT if((radmod%lcloud_fwd .and. sea) .or. gmi .or. amsr2) channel information (loop through each channel with column list), for each column: 1: observed brightness temperature (K) 2: observed - simulated Tb with bias correction (K) 3: observed - simulated Tb with no bias correction (K) 4: inverse observation error 5: quality control mark or event indicator 6: surface emissivity 7: stability index 8: indicator of cloudy consistency (if lcloud_fwd) OR d(Tb)/d(Ts) |
Singularity JCSDA-singularity-master.simg:~> cd ~/jedi/code Singularity JCSDA-singularity-master.simg:~> git clone -b feature/ufo https://github.com/UCAR/oops.git # Check out the oops code to oops directory |
Singularity JCSDA-singularity-master.simg:~> cd ~ Singularity JCSDA-singularity-master.simg:~> mkdir -p ~/jedi/build/oops # Prepare the directory for oops building, out-of-source building Singularity JCSDA-singularity-master.simg:~> cd ~/jedi/build/oops Singularity JCSDA-singularity-master.simg:~> ecbuild --build=release -DLAPACK_LIBRARIES=$LAPACK_LIBRARIES ~/jedi/code/oops Singularity JCSDA-singularity-master.simg:~> make -j`nproc` ... ... [ 99%] Built target test_qg_4dvar_obsbias.x [ 99%] Built target test_qg_3dfgat.x Scanning dependencies of target test_qg_4densvar.x [ 99%] Built target test_qg_4densvar.x [ 99%] Linking CXX executable test_qg_increment [ 99%] Built target test_qg_increment [100%] Linking CXX executable test_qg_localization [100%] Built target test_qg_localization Singularity JCSDA-singularity-master.simg:~> ctest ... ... 108/108 Test #108: test_qg_dfi ...................... Passed 0.46 sec 100% tests passed, 0 tests failed out of 108 Label Time Summary: boost = 1.13 sec (53 tests) executable = 1.16 sec (56 tests) fortran = 0.04 sec (3 tests) oops = 23.89 sec (108 tests) script = 22.73 sec (52 tests) Total Test time (real) = 23.94 sec |
Singularity JCSDA-singularity-master.simg:~> cd ~/jedi/code Singularity JCSDA-singularity-master.simg:~> wget http://www2.mmm.ucar.edu/wrf/src/WPSV3.9.1.TAR.gz Singularity JCSDA-singularity-master.simg:~> tar xvf WPSV3.9.1.TAR.gz Singularity JCSDA-singularity-master.simg:~> rm -f WPSV3.9.1.TAR.gz Singularity JCSDA-singularity-master.simg:~> wget http://www2.mmm.ucar.edu/wrf/src/WRFV3.9.1.1.TAR.gz Singularity JCSDA-singularity-master.simg:~> tar xvf WRFV3.9.1.1.TAR.gz Singularity JCSDA-singularity-master.simg:~> rm -f WRFV3.9.1.1.TAR.gz Singularity JCSDA-singularity-master.simg:~> ls -la total 12 drwxrwxr-x. 6 xinzhang xinzhang 53 Oct 5 16:01 . drwxrwxr-x. 13 xinzhang xinzhang 196 Oct 5 14:45 .. drwxrwxr-x. 13 xinzhang xinzhang 243 Oct 5 14:29 gsi drwxrwxr-x. 10 xinzhang xinzhang 4096 Oct 5 11:13 oops drwxr-xr-x. 7 xinzhang xinzhang 4096 Aug 17 11:10 WPS drwxr-xr-x. 17 xinzhang xinzhang 4096 Aug 28 15:01 WRFV3 |
Singularity JCSDA-singularity-master.simg:~> cd ~/jedi/code/WRFV3 Singularity JCSDA-singularity-master.simg:~> ./configure # Select 34 for dmpar gnu Singularity JCSDA-singularity-master.simg:~> vi configure.wrf # Becasue WRF has problem to test mpi2 support, please add -DMPI2_SUPPORT after DM_CC = mpicc Singularity JCSDA-singularity-master.simg:~> ./compile em_real # For unknown reason, module_ra_rrtmg_swf.f90 might fail to be compiled due to internal compiler error, typing "./compile em_real" again usually can solve the problem. ... ... ========================================================================== build started: Thu Oct 5 22:12:18 UTC 2017 build completed: Thu Oct 5 22:21:58 UTC 2017 ---> Executables successfully built <--- -rwxr-xr-x. 1 root root 36472488 Oct 5 22:21 main/ndown.exe -rwxr-xr-x. 1 root root 36346432 Oct 5 22:21 main/real.exe -rwxr-xr-x. 1 root root 35944328 Oct 5 22:21 main/tc.exe -rwxr-xr-x. 1 root root 40051304 Oct 5 22:21 main/wrf.exe ========================================================================== Singularity JCSDA-singularity-master.simg:~> cd ../WPS Singularity JCSDA-singularity-master.simg:~> ./configure # Select 3 for dmpar gfortran Singularity JCSDA-singularity-master.simg:~> vi configure.wrf # edit COMPRESSION_LIBS and COMPRESSION_INC as following lines and remove -f90=gfortran from DM_FC and remove -cc=gcc from DM_CC ... ... ... # # Settings for Linux x86_64, gfortran (dmpar) # # COMPRESSION_LIBS = -L/usr/local/lib -ljasper -lpng -lz COMPRESSION_INC = -I/usr/local/include FDEFS = -DUSE_JPEG2000 -DUSE_PNG SFC = gfortran SCC = gcc DM_FC = mpif90 DM_CC = mpicc ... ... ... Singularity JCSDA-singularity-master.simg:~> ./compile Singularity JCSDA-singularity-master.simg:~> ls -al *.exe lrwxrwxrwx. 1 root root 23 Oct 6 16:02 geogrid.exe -> geogrid/src/geogrid.exe lrwxrwxrwx. 1 root root 23 Oct 6 16:02 metgrid.exe -> metgrid/src/metgrid.exe lrwxrwxrwx. 1 root root 21 Oct 6 16:02 ungrib.exe -> ungrib/src/ungrib.exe |
Singularity JCSDA-singularity-master.simg:~> cd ~/jedi/code Singularity JCSDA-singularity-master.simg:~> git clone -b v5.2 https://github.com/MPAS-Dev/MPAS-Release.git |
Singularity JCSDA-singularity-master.simg:~> cd ~/jedi/code/MPAS-Release Singularity JCSDA-singularity-master.simg:~> vi ./src/core_atmosphere/physics/checkout_data_files.sh # Replace the MPAS-Data address to https (see next line below) Singularity JCSDA-singularity-master.simg:~> git diff diff --git a/src/core_atmosphere/physics/checkout_data_files.sh b/src/core_atmosphere/physics/checkout_data_files.sh index e62b466..f2c3cba 100755 --- a/src/core_atmosphere/physics/checkout_data_files.sh +++ b/src/core_atmosphere/physics/checkout_data_files.sh @@ -56,7 +56,7 @@ fi which git if [ $? -eq 0 ]; then echo "*** trying git to obtain WRF physics tables ***" - git clone git://github.com/MPAS-Dev/MPAS-Data.git + git clone https://github.com/MPAS-Dev/MPAS-Data.git if [ $? -eq 0 ]; then mv MPAS-Data/atmosphere/physics_wrf/files/* physics_wrf/files rm -rf MPAS-Data Singularity JCSDA-singularity-master.simg:~> make gfortran CORE=atmosphere ... ... make[2]: Leaving directory '/home/xinzhang/jedi/code/MPAS-Release/src/core_atmosphere' ******************************************************************************* MPAS was built with default double-precision reals. Debugging is off. Parallel version is on. Papi libraries are off. TAU Hooks are off. MPAS was built without OpenMP support. MPAS was built with .F files. The native timer interface is being used Using the PIO 1.x library. ******************************************************************************* make[1]: Leaving directory '/home/xinzhang/jedi/code/MPAS-Release' |
Singularity xinzhang8noaa-singularity-master.simg:~/jedi/code/MPAS-Release> wget ftp://ftp.ucar.edu/pub/mmm/xinzhang/mpas_10242_case.tgz Singularity xinzhang8noaa-singularity-master.simg:~/jedi/code/MPAS-Release> tar xvf mpas_10242_case.tgz Singularity xinzhang8noaa-singularity-master.simg:~/jedi/code/MPAS-Release> mpirun -np 8 atmosphere_model # or mpirun -np 1 atmosphere_model task 0 of 8 is running task 1 of 8 is running task 2 of 8 is running task 3 of 8 is running task 6 of 8 is running task 7 of 8 is running task 4 of 8 is running task 5 of 8 is running |
Singularity JCSDA-singularity-master.simg:~> cd ~/jedi/code Singularity JCSDA-singularity-master.simg:~> git clone https://Your_UserName@bitbucket.org/jcsda/comfv3.git comfv3 # Check out the FV3 code, please replace Your_UserName with your bitbucket username Singularity JCSDA-singularity-master.simg:~> cd comfv3 Singularity JCSDA-singularity-master.simg:~> git submodule init Singularity JCSDA-singularity-master.simg:~> git submodule update Singularity JCSDA-singularity-master.simg:~> git branch -a Singularity JCSDA-singularity-master.simg:~> git checkout feature/gfortran_build Singularity JCSDA-singularity-master.simg:~> cd FV3 Singularity JCSDA-singularity-master.simg:~> git checkout feature/gfortran_build Singularity JCSDA-singularity-master.simg:~> cd ../NEMS Singularity JCSDA-singularity-master.simg:~> git checkout feature/gfortran_build |
Singularity JCSDA-singularity-master.simg:~> cd ~/jedi/code/comfv3/release/v0/exp/ Singularity JCSDA-singularity-master.simg:~> ./build.sh macgnu ... ... ... Elapsed time 66 seconds. Compiling HYDRO=Y 32BIT=N finished + cp /home/xinzhang/jedi/code/comfv3/release/v0/exp/../../../tests/fv3_1.exe ../NEMS/exe/fv3_gfs_hydro.prod.64bit.x + rm /home/xinzhang/jedi/code/comfv3/release/v0/exp/../../../tests/fv3_1.exe + exit 0 |
Singularity JCSDA-singularity-master.simg:~> cd ~/jedi/code Singularity JCSDA-singularity-master.simg::~/jedi/code> git clone https://YourUserName@bitbucket.org/jcsda/crtm-release.git crtm_v2.2.3 # please replace Your_UserName with your bitbucket username Singularity JCSDA-singularity-master.simg::~/jedi/code> cd crtm_v2.2.3/ Singularity JCSDA-singularity-master.simg::~/jedi/code/crtm_v2.2.3> source config-setup/gfortran.setup Singularity JCSDA-singularity-master.simg::~/jedi/code/crtm_v2.2.3>./configure --prefix=/home/`whoami`/jedi/build/crtm Singularity JCSDA-singularity-master.simg::~/jedi/code/crtm_v2.2.3> make Singularity JCSDA-singularity-master.simg::~/jedi/code/crtm_v2.2.3> make check Singularity JCSDA-singularity-master.simg::~/jedi/code/crtm_v2.2.3> make install |
When you start a Singularity container instance from the xinzhang8noaa-singularity-master.simg, we already prepapre the gnu version 7.2 compilers and most of the necessary libraiestools for GSI, OOPS, WRF etc., all libraries are installed under /usr/local, which include
The major NCEP libraries are also installed at :
|
You may also want to use visual panels to communicate related information, tips or things users need to be aware of. |
Related articles appear here based on the labels you select. Click to edit the macro and add or change labels.
|