Building ufo-bundle with ODB support

This document describes how to build ufo-bundle and include ODB support in the IODA subsystem. We are focusing on ODB_API (ODB2) support for now. Note that netcdf support will be present regardless of the presence of ODB support. The obs data interface that will be used is triggered by the filename ("obsfile" spec) that is entered in the JSON configuration file in the "Observations/ObsTypes/ObsType" section. According to the specification in the obsfile configuration keyword:

• filename with a .nc4 or .nc suffix →  netcdf is used
  
  

  Example Expand source

  "ObsType": "Radiosonde",
          "ObsData": {
            "ObsDataIn": {
              "obsfile": "Data/diag_t_01_wprofiles.nc4"
            }
          }

• filename with a .odb suffix → ODB_API (ODB2) is used
  
  

  Example Expand source

  "ObsType": "Radiosonde",
          "ObsData": {
            "ObsDataIn": {
              "obsfile": "Data/diag_t_01_wprofiles.odb"
            }
          }

By default, running the ufo-build process will include only netcdf support. By adding cmake controls (see below) to ecbuild commands, you can ODB2. In the initial "ODB support" code, only the Radiosonde obs type is enabled to use the ODB2 interface.

Steps for building ufo-bundle with ODB

Build the ODB packages

Do this step, only once at the beginning, to build, test and install the ODB packages. These include metkit, and odb_api (ODB2). This process takes a while so it is separated from the ufo-bundle build to not impose the long build time during iterations of building and testing ufo-bundle. The plan is to install these packages soon in the JCSDA Singularity image (also could be done for other systems outside Singularity) so that this step will be eliminated for users.

Two helper scripts (setup_odb_build.sh and run_odb_build.sh) have been included in the jedi-tools repository that will automate the build process for you. You can look in these scripts if you want to see what is being done to get the ODB packages built and installed.

Here is a brief description of the steps, and an example will be given below.

1. cd to a directory where you want the build/test/install process to take place
2. execute setup_odb_build.sh
3. execute run_odb_build.sh

This creates a subdirectory structure in the directory you chose to do the build in step 1, where the resulting installed packages exist in ./odb/install.

Build the ufo-bundle

Ecbuild options are used to distinguish between building without ODB with  ODB2 as shown here:

• Without ODB2
     No need to add any options to ecbuild
• With ODB2
     Add the following options to ecbuild:
  • -DENABLE_ODB_API=1
  • -DODB_API_PATH=<path_to_odb_install_directory>
    • It is important to specify <path_to_odb_install_directory> using a full path (ie, not a relative path)\
      
      

  Using mpif90

  You may need to use mpif90 as the fortran compiler (as opposed to gfortran). This can be accomplished by setting the environment variable FC to "mpif90" before running ecbuild. In bash this is done by executing:

  export FC="mpif90"

The build process for ODB uses the pre-installed versions of ecbuild and eckit in the Singularity image, so it is important to comment out the corresponding lines in the CMakeLists.txt file in ufo-bundle:

...
#ecbuild_bundle( PROJECT eckit GIT "https://github.com/ECMWF/eckit.git" TAG 0.18.5 )
#ecbuild_bundle( PROJECT fckit GIT "https://github.com/ECMWF/fckit.git" TAG 0.5.0 )

ecbuild_bundle( PROJECT oops GIT "https://github.com/JCSDA/oops.git"        BRANCH develop  UPDATE )
ecbuild_bundle( PROJECT gsw  GIT "https://github.com/JCSDA/GSW-Fortran.git" BRANCH develop  UPDATE )
ecbuild_bundle( PROJECT crtm GIT "https://github.com/JCSDA/crtm.git"        BRANCH develop  UPDATE )
ecbuild_bundle( PROJECT ioda GIT "https://github.com/JCSDA/ioda.git"        BRANCH develop  UPDATE )
ecbuild_bundle( PROJECT ufo  GIT "https://github.com/JCSDA/ufo.git"         BRANCH develop  UPDATE )
...

Other than the extra ecbuild options, building ufo-bundle proceeds as in the same manner as before.

Let's take a look at an example:

Example

Say that you are starting from scratch and have decided to build with ODB2 support and to do the work in your $HOME/projects directory. After starting up Singularity, type in the following commands to build the ODB packages and ufo-bundle:

# First, clone ufo-bundle and make the edits to the top level CMakeLists.txt
# With the latest ufo-bundle, the eckit and fckit ecbuild_bundle commands should be commented out
cd $HOME/projects
git clone https://github.com/JCSDA/ufo-bundle.git
vi ufo-bundle/CMakeLists.txt  # comment out the eckit and fckit ecbuild_bundle lines

# The jedi-tools repo needs to be cloned so that you have access to the helper scripts.
#
# As of 9/18/18, the odb build scripts are in a branch called "feature/odb-build-script", which has been
# pull requested into the develop branch. It might take a couple days for these scripts to appear in
# the default master branch.
#
# Depending on when you try this, you may need to checkout the "feature/odb-build-script" or "develop"
# branch in order to have access to the build scripts, using one of the following commands:
#     git checkout feature/odb-build-script
#     git checkout develop
#
cd $HOME/projects
git clone https://github.com/JCSDA/jedi-tools.git

# Make sure you are in $HOME/projects to build ODB. This build
# only needs to be done once. Note that this process takes a long
# time (over 1/2 hour).
#
cd $HOME/projects
jedi-tools/buildscripts/ODB/setup_odb_build.sh
jedi-tools/buildscripts/ODB/run_odb_build.sh

# Check the log files and make sure things went okay before proceeding.
#
# The odb libraries and include files will be installed in:
#
#   $HOME/projects/odb/install
#
# Use this path for the subsequent ufo-bundle build.
#
# Build ufo-bundle. Let's build ufo in the directory:
#
#  $HOME/projects/ufo-bundle/build
#
cd ufo-bundle
mkdir build
cd build
ecbuild -DODB_API_PATH=$HOME/projects/odb/install -DENABLE_ODB_API=1 ..
make -j4
ctest