NASA uses HDF-EOS (also called HDF-EOS2) and HDF-EOS5 for their satellite data. HDF-EOS uses HDF4 and HDF-EOS5 uses HDF5. HDF4 and HDF5 are incompatible with each other. The RTTOV libraries we use to support radiance assimilation require HDF5. Consequently, any HDF4 source file for radiances cannot be directly converted to a DART radiance observation sequence file because of the innate reliance on the HDF5 support for the RTTOV metadata routines. Technically, this is a DART problem because the radiance metadata code could be separated from the forward operator code. The metadata code has no HDF-related dependencies.

http://hdfeos.org/software/h4cflib.php  is a package that converts HDF-EOS files to CF-compliant netCDF files. That website provides binaries as well as source. For the AIRS AMSU-A converter, I convert the HDF-EOS files to netCDF and then to DART observation sequence file format. I don't like the 2step process, but it does work around the HDF4/HDF5 incompatiblity issue. Building your own will require building the HDF4 and HDF-EOS packages.

https://support.hdfgroup.org/products/hdf5_tools/h4toh5  is a conversion library and tool. I have had a difficult time trying to install it, but the following build script for the required elements may provide the correct environment. This would be nice because the h4toh5 command line utility can convert an HDF-EOS2 file to an HDF5 file that can be accessed by netCDF-4.


Here is my build script for Cheyenne. Note it mentions the szip package but does not build it ... and probably should. 


#!/bin/sh

# ------------------------------------------------------------------------------
##
## The NASA Earthdata Data Access Services portal serves as the download site:
## https://wiki.earthdata.nasa.gov/display/DAS/Toolkit+Downloads
##
## The following packages were downloaded:
##
##    zlib-1.2.11.tar.gz
##    jpegsrc.v9b.tar.gz
##    hdf-4.2.13.tar.gz
##    HDF-EOS2.20v1.00.tar.Z
##    HDF-EOS2.20v1.00_TestDriver.tar.Z
##    szip-2.1.1.tar.gz
##    hdf5-1.8.19.tar.gz
##    HDF-EOS5-5.1.16.tar.Z
##    HDF-EOS5-5.1.16_TESTDRIVERS.tar.Z
##
## The documentation files were downloaded:
##
##    HDF-EOS_REF.pdf
##    HDF-EOS_UG.pdf
##    HDF-EOS5_REF.pdf
##    HDF-EOS5_UG.pdf
##
## Some other useful websites for HDF and HDF-related products are:
## https://portal.hdfgroup.org/display/support/Downloads
## https://hdfeos.org/software/library.php#HDF-EOS2
## https://opensource.gsfc.nasa.gov/projects/HDF-EOS2/index.php

# these are the modules I used
# module purge
# module add ncarenv/1.3
# module add intel/19.0.5
# module add ncarcompilers/0.5.0
# module add mpt/2.22
# module add netcdf/4.7.3
# module add mkl/2019.0.5

# If you are trying to build hdf, hdf5 - make sure you do not have those modules loaded.

if ( `false` ); then
  for i in zlib-1.2.11.tar.gz \
           jpegsrc.v9b.tar.gz \
           hdf-4.2.13.tar.gz \
           HDF-EOS2.20v1.00.tar.Z \
           HDF-EOS2.20v1.00_TestDriver.tar.Z \
           szip-2.1.1.tar.gz \
           hdf5-1.8.19.tar.gz \
           HDF-EOS5-5.1.16.tar.Z \
           HDF-EOS5-5.1.16_TESTDRIVERS.tar.Z
  do
    tar -zxvf $i
  done
fi

# ------------------------------------------------------------------------------
# start with smaller libs, work up to HDF-EOS.
# ------------------------------------------------------------------------------

# set the installation location of the final libraries
H4_PREFIX=/glade/work/${USER}/local/hdf-eos
H5_PREFIX=/glade/work/${USER}/local/hdf-eos5

# make the target install dirs
mkdir -p ${H4_PREFIX}/{lib,bin,include,man,man/man1,share}
mkdir -p ${H5_PREFIX}/{lib,bin,include,man,man/man1,share}

# record the build script and environment
echo                         >  ${H4_PREFIX}/build_environment_log.txt
echo 'the build script'      >> ${H4_PREFIX}/build_environment_log.txt
cat    $0                    >> ${H4_PREFIX}/build_environment_log.txt
echo                         >> ${H4_PREFIX}/build_environment_log.txt
echo '=====================' >> ${H4_PREFIX}/build_environment_log.txt
echo 'the build environment' >> ${H4_PREFIX}/build_environment_log.txt
echo                         >> ${H4_PREFIX}/build_environment_log.txt
env | sort                   >> ${H4_PREFIX}/build_environment_log.txt

# start with smaller libs, work up to HDF-EOS.

echo ''
echo '======================================================================'
if [ -f ${H4_PREFIX}/lib/libz.a ]; then
   echo 'zlib already exists - no need to build.'
else

   export CFLAGS='-fPIC'
   export FFLAGS='-fPIC'

   echo 'building zlib at '`date`
   cd zlib-1.2.11 || exit 1
   ./configure --prefix=${H4_PREFIX} || exit 1
   make clean     || exit 1
   make           || exit 1
   make test      || exit 1
   make install   || exit 1
   cd ..
fi

echo ''
echo '======================================================================'
# This is peculiar - on Cheyenne:
# If I build with --libdir=H4_PREFIX, subsequent linking works.
# If I build with --libdir=H4_PREFIX/lib, subsequent linking FAILS with an 
# undefined reference to 'rpl_malloc'.
if [ -f ${H4_PREFIX}/libjpeg.a ]; then
   echo 'jpeg already exists - no need to build.'
else
   echo 'buiding jpeg at '`date`
   cd jpeg-9b        || exit 2
   ./configure CC='icc -Df2cFortran' CFLAGS='-fPIC' FFLAGS='-fPIC' \
               --prefix=${H4_PREFIX} --libdir=${H4_PREFIX} || exit 2
   make clean        || exit 2
   make              || exit 2
   make test         || exit 2
   make install      || exit 2
   cd ..
fi

echo ''
echo '======================================================================'
if [ -f ${H4_PREFIX}/lib/libmfhdf.a ]; then
   echo 'hdf4 already exists - no need to build.'
else
   echo 'building hdf4 at '`date`
   # (apparently there is no 'make test')

   cd hdf-4.2.13 || exit 3
   ./configure CC='icc -Df2cFortran' CFLAGS='-fPIC' FFLAGS='-fPIC' \
               --prefix=${H4_PREFIX} \
               --disable-netcdf \
               --with-zlib=${H4_PREFIX} \
               --with-jpeg=${H4_PREFIX} || exit 3
   make clean    || exit 3
   make          || exit 3
   make install  || exit 3
   cd ..
fi

echo ''
echo '======================================================================'
if [ -f ${H4_PREFIX}/lib/libhdfeos.a ]; then
   echo 'hdf-eos already exists - no need to build.'
else
   echo 'building HDF-EOS2.20v1.00 at '`date`
   echo 'after expanding the .tar.gz file, the source is in "hdfeos"'
   cd hdfeos    || exit 4
   # (the CC options are crucial to provide Fortran interoperability)
   ./configure CC='icc -Df2cFortran' CFLAGS='-fPIC' FFLAGS='-fPIC' \
               --prefix=${H4_PREFIX} \
               --enable-install-include \
               --with-zlib=${H4_PREFIX} \
               --with-jpeg=${H4_PREFIX} \
               --with-hdf=${H4_PREFIX} || exit 4
   make clean   || exit 4
   make         || exit 4
   make install || exit 4
   cd ..
fi


#-------------------------------------------------------------------------------
# HDF-EOS5 record the build script and environment
#-------------------------------------------------------------------------------

echo                         >  ${H5_PREFIX}/build_environment_log.txt
echo 'the build script'      >> ${H5_PREFIX}/build_environment_log.txt
cat    $0                    >> ${H5_PREFIX}/build_environment_log.txt
echo                         >> ${H5_PREFIX}/build_environment_log.txt
echo '=====================' >> ${H5_PREFIX}/build_environment_log.txt
echo 'the build environment' >> ${H5_PREFIX}/build_environment_log.txt
echo                         >> ${H5_PREFIX}/build_environment_log.txt
env | sort                   >> ${H5_PREFIX}/build_environment_log.txt

echo '======================================================================'
if [ -f ${H5_PREFIX}/lib/libhdf5.a ]; then
   echo 'hdf5 already exists - no need to build.'
else
   echo 'building hdf5 at '`date`
   # (there is apparently no 'make test')

   cd hdf5-1.8.19 || exit 3
   ./configure CC='icc -Df2cFortran' CFLAGS='-fPIC' FFLAGS='-fPIC' \
               --prefix=${H5_PREFIX} \
               --enable-fortran \
               --enable-fortran2003 \
               --enable-production \
               --with-zlib=${H4_PREFIX} || exit 3
   make clean          || exit 3
   make                || exit 3
   make check          || exit 3
   make install        || exit 3
   make check-install  || exit 3
   cd ..
fi

echo ''
echo '======================================================================'
if [ -f ${H5_PREFIX}/lib/libhe5_hdfeos.a ]; then
   echo 'hdf-eos5 already exists - no need to build.'
else
   echo 'building hdf-eos5 at '`date`
   cd hdf-eos5-5.1.16    || exit 4
   # (the CC options are crucial to provide Fortran interoperability)
   ./configure CC='icc -Df2cFortran' CFLAGS='-fPIC' FFLAGS='-fPIC' \
               --prefix=${H5_PREFIX} \
               --enable-install-include \
               --with-zlib=${H4_PREFIX} \
               --with-hdf5=${H5_PREFIX} || exit 4
   make clean   || exit 4
   make         || exit 4
   make check   || exit 4
   make install || exit 4
   cd ..
fi

exit 0


  • No labels