Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Meta data can also correspond to the variables in the file. Using our satellite instrument example, there would be a total of 11 variables (nvars = 11) in the file. Meta data such and channel number and channel frequency would then be store in vectors that are nvars (11) in length. MetaData related to the variables use the variable name suffix "@VarMetaData". For example the channel frequencies would be stored in a netcdf variable named "channel_frequency@VarMetaData".

File Variable Conventions

The variables inside a IODA observation data file need to adhere to the following conventions:

NETCDF files

  • Variable names must contain a group name
    • The group name is the "@<group_name>" suffix tagged onto the netcdf variable name.
    • Each variable in the file is specified as "<variable_name>@<group_name>"
  • Variables must use the netcdf fill value for missing data marks
    • The netcdf variable attribute "_FillValue" contains the fill value
  • Variables must use the correct data types
    • Allowed types are: int, float, char (no double precision)
    • All numeric data are float except for QC marks which are int
      • The checker script (see below) only verifies that variables with group "PreQC" are integer type
  • Variable data cannot contain invalid numerical values (nan, inf, -inf)

The plan is to issue warnings now about variables that violate these conventions so that these violations can be located and fixed. Once all files are fixed, then the convention violation warnings will be changed to errors.

A checker script (check_ioda_nc.py) has been written for netcdf files, and will be made available in a pull request that is due to be merged around November 1, 2019. Once all files pass the checker, a ctest will be added to the ioda repository that runs the checker script on all of the netcdf files in the ioda repository so new files that come in will be immediately verified.

Variable names by observation types

In the following tables, the Data Group column denotes if the variable is observation data, meta data related to locations, or meta data related to variables. The entries in this column are the variable group name suffixes without the "@". An entry of ObsValue implies that you also have the ObsError and PreQC variables in the file.

...