Skip to end of metadata
Go to start of metadata

LDM

Otho is receiving the data and putting it to:

/var/autofs/mnt/rapdmg1/data/goesr/

 

Details for receiving data

 I see the data on LDM @ idd.unidata.ucar.edu:

GOES notifyme on IDD  Expand source

 

Grab it like this:

Where chomp.py, is a simple script to drop the WMO header (36 bytes worth) so that the rest is a valid netcdf file (thanks daryl herzmann akrherz@iastate.edu via unidata.ucar.edu )

import sys
fn = sys.argv[1]
data = sys.stdin.read()
o = open(fn, 'wb')
o.write(data[36:])
o.close()

 

New configuration looks like this:

prepare_goes-r.py is a modified version of chomp that optionally can create a dated log file to report which files it has processed and the WMO header contents that it has removed.

 

User Notes from LDM conversation on ldm-users

TIRC is CONUS, TIRU is hemispheric and TIRP is perhaps Puerto Rico.

PAA, PAB, etc. are the tile IDs. The tiles need to be merged to form the entire image.

C and U are center position CONUS and Full Disk, respectively.  GOES-16 is currently at the PLT position at 89.5 West.  Once it has been relocated to its final position (East or West), the headers will be changed to reflect that.

 

Displaying the GOES16 Data

 

Mosaicing the data

From unidata via the ldm support email list:

 Click here to expand...
quoted from a post by Ryan May at Unidata:
We currently have re-assembled the tiles on our test server:
http://thredds-test.unidata.ucar.edu/thredds/catalog/satellite/goes16/catalog.html
I wrote a Python script to do the assembly and stripping of header AND footer directly piped from the LDM. That code is freely available here: 
https://github.com/Unidata/ldm-alchemy (look at goes-restitch.py) This runs only on Python 3.6.
The pqact being used for the data (which runs one instance of the script for all 16 channel

LDM

Otho is receiving the data and putting it to:

/var/autofs/mnt/rapdmg1/data/goesr/

 

Details for receiving data

 I see the data on LDM @ idd.unidata.ucar.edu:

GOES notifyme on IDD  Expand source

 

Grab it like this:

Where chomp.py, is a simple script to drop the WMO header (36 bytes worth) so that the rest is a valid netcdf file (thanks daryl herzmann akrherz@iastate.edu via unidata.ucar.edu )

import sys
fn = sys.argv[1]
data = sys.stdin.read()
o = open(fn, 'wb')
o.write(data[36:])
o.close()

 

New configuration looks like this:

prepare_goes-r.py is a modified version of chomp that optionally can create a dated log file to report which files it has processed and the WMO header contents that it has removed.

 

User Notes from LDM conversation on ldm-users

TIRC is CONUS, TIRU is hemispheric and TIRP is perhaps Puerto Rico.

PAA, PAB, etc. are the tile IDs. The tiles need to be merged to form the entire image.

C and U are center position CONUS and Full Disk, respectively.  GOES-16 is currently at the PLT position at 89.5 West.  Once it has been relocated to its final position (East or West), the headers will be changed to reflect that.

 

Displaying the GOES16 Data

 

Mosaicing the data

From unidata via the ldm support email list:

 Click here to expand...
quoted from a post by Ryan May at Unidata:
We currently have re-assembled the tiles on our test server:
http://thredds-test.unidata.ucar.edu/thredds/catalog/satellite/goes16/catalog.html
I wrote a Python script to do the assembly and stripping of header AND footer directly piped from the LDM. That code is freely available here: 
https://github.com/Unidata/ldm-alchemy (look at goes-restitch.py) This runs only on Python 3.6.
The pqact being used for the data (which runs one instance of the script for all 16 channels of a particular sector/time) is here:
https://github.com/Unidata/TdsConfig/blob/master/threddsDev/pqacts/pqact.satellite
I don't have any cycles to really support that script, but I haven't noticed any problems with it on our server.
For the "true color" RGB imagery, you can see a sample of the simple approach here:
http://nbviewer.jupyter.org/github/Unidata/unidata-python-workshop/blob/master/notebooks/GOES_RGB_Demo/GOES_RGB_Image.ipynb
With the veggie channel, things are too green on land, but otherwise not too bad. I just wish they would fix the problem where the channel maxing out ends up set to 0 instead of 255.

 

 

Todo

  • LDM - need to cleanup the directory placement of data.
    • move to YYYY/MMDD/  - COMPLETED
    • don't delete old data - move away - COMPLETED (moved data to goesr-old to be archived)
    • update this page with pqact & chome changes. - COMPLETED
    • check chomp into cvs - rename from chomp.py -COMPLETED (checked into CVS under projects/ldm/goes/scripts/prepare_goes-r.py)
    • add logging & write WMO header to log (debug) - COMPLETED
    • commit changes
    • Tile the data.  - use netcdf operators - http://nco.sourceforge.net/
    • Add archiving & scrubbing
      • archive old directory structure as one big tarball
  • Documenting GOES-16 capabilities on a RAL Wiki page.
  • Look at netcdf data more - fields?  


Task list from Bob

 Click here to expand...


The GOES-16 task will entail -
1) Acquiring via LDM as much GOES-16 data as is possible
at this time;
2) Setting up data storage directory structures and
scrubbing mechanisms;
3) Setting up processing to stitch together tiled data
into a single netCDF file;
4) Perhaps putting an archiving mechanism in place;
5) Perhaps developing a web mechanism for viewing the data;
6) Documenting GOES-16 capabilities on a RAL Wiki page.

The GeoServer tasks are a little less defined and will be
done as time is available with priority given to the GOES-16
tasks. GeoServer tasks may entail -
1) Installing GeoServer on a RAL infrastructure machine;
2) Making some of the GOES-16 data available via GeoServer;
3) Perhaps making some other gridded data sets available
on GeoServer (eg. MRMS radar composite);
4) Perhaps making some feature product(s) (like METARS)
available via GeoServer;
5) Perhaps making some of the data displayable via the
GeoServer WMS and OpenLayers.
6) Documenting GeoServer capabilities on a RAL Wiki page.


From Matthias:

 It would be good if Paul could team up with Cathy for a brown bag talk to discuss GOES-16, products, data volumes, etc. at some point maybe early April?


s of a particular sector/time) is here:
https://github.com/Unidata/TdsConfig/blob/master/threddsDev/pqacts/pqact.satellite
I don't have any cycles to really support that script, but I haven't noticed any problems with it on our server.
For the "true color" RGB imagery, you can see a sample of the simple approach here:
http://nbviewer.jupyter.org/github/Unidata/unidata-python-workshop/blob/master/notebooks/GOES_RGB_Demo/GOES_RGB_Image.ipynb
With the veggie channel, things are too green on land, but otherwise not too bad. I just wish they would fix the problem where the channel maxing out ends up set to 0 instead of 255.

 

 

Todo

  • LDM - need to cleanup the directory placement of data.
    • move to YYYY/MMDD/  - COMPLETED
    • don't delete old data - move away - COMPLETED (moved data to goesr-old to be archived)
    • update this page with pqact & chome changes. - COMPLETED
    • check chomp into cvs - rename from chomp.py -COMPLETED (checked into CVS under projects/ldm/goes/scripts/prepare_goes-r.py)
    • add logging & write WMO header to log (debug) - COMPLETED
    • commit changes
    • Tile the data.  - use netcdf operators - http://nco.sourceforge.net/
    • Add archiving & scrubbing
      • archive old directory structure as one big tarball
  • Documenting GOES-16 capabilities on a RAL Wiki page.
  • Look at netcdf data more - fields?  
 "Final" status update from Aaron
 An update on where we stand on the GOES 16 work.  I was able to get GeoServer running on the host but I have been waiting for the following two items since Tuesday of last week:
  • get PostGres/PostGIS installed on satops3 (required for GeoServer and gridded data)
  • open the firewall rules to allow other hosts to contact satops3 on port 8080.  I worked around this by using ssh port forwarding but obviously if we want a shared resource it needs to be fixed

Once these tasks are accomplished by SNAT the next steps would be:
  • determine whether LDM should be running on otho or satops3 (as I recall Paul had thought it should be on satops3?)
  • rename files so they include a date format recognized by GeoServer (such as 20170331T1650Z_TISJ_S01.nc)
  • post-process the GOES files so they have a time dimension.  They currently only have x and y
  • determine which GOES 16 products are of interest in GeoServer (fields - S01..., domains such as CONUS)
  • (optional) figure out how to configure tiled datasets in GeoServer (I started an email thread on geoserver-users here) but ran out of time to track down the issues with the GOES 16 original tiles (not the mosaics)
  • configure the appropriate datasets in GeoServer pointing to the PostGIS instance on satops3
  • configure the color scales/color maps as custom SLDs in GeoServer

Since I could not work directly on satops3 due to the issues up top I worked on getting the GOES 16 data running on my development machine with an older version of GeoServer.  What I did last week was:
  • Fixed an issue with GeoServer access to the native netCDF library
  • Pointed GeoServer to a "full" data directory.  This allowed it to start up normally
  • Changed the admin password (since this may be an exposed host).  The credentials for accessing the web UI are admin:goes16
  • Created an 'NCAR' workspace where we will add our data later
  • Configured some sensible global GeoServer default settings
  • Created https://github.com/NCAR/geoserver-configs - a RAL repository for keeping existing GeoServer config starting points and instructions for setting up GeoServer.  As I went through this process on satops3 I documented each of the steps, which was a surprisingly lengthy process
  • Created https://github.com/NCAR/geoserver-config-generator - a RAL repository for a utility I created for auto-generating gridded data configurations for GeoServer.  This utility was created to create the GeoServer ImageMosaic indexer configurations as well as the CRS/SRS configuration needed in GeoServer.  Creating the correct projections/SRS configurations by hand is particularly tricky, so the hope is that this will allow us to quickly get a starting point for putting gridded data into GeoServer

I was able to get the mosaiced version of two different GOES 16 datasets running in my GeoServer instance (TIRC and TISJ).  The config generator utility is the only reason I got the latter GOES 16 dataset running, there was a good amount of guesswork about how to create the proper SRS information for GeoServer to use.  As mentioned above I still do not have a working tiled configuration.
I anticipate these GitHub repos being shared resources for getting GeoServer configured going forward. 


Task list from Bob

 Click here to expand...


The GOES-16 task will entail -
1) Acquiring via LDM as much GOES-16 data as is possible
at this time;
2) Setting up data storage directory structures and
scrubbing mechanisms;
3) Setting up processing to stitch together tiled data
into a single netCDF file;
4) Perhaps putting an archiving mechanism in place;
5) Perhaps developing a web mechanism for viewing the data;
6) Documenting GOES-16 capabilities on a RAL Wiki page.

The GeoServer tasks are a little less defined and will be
done as time is available with priority given to the GOES-16
tasks. GeoServer tasks may entail -
1) Installing GeoServer on a RAL infrastructure machine;
2) Making some of the GOES-16 data available via GeoServer;
3) Perhaps making some other gridded data sets available
on GeoServer (eg. MRMS radar composite);
4) Perhaps making some feature product(s) (like METARS)
available via GeoServer;
5) Perhaps making some of the data displayable via the
GeoServer WMS and OpenLayers.
6) Documenting GeoServer capabilities on a RAL Wiki page.


From Matthias:

 It would be good if Paul could team up with Cathy for a brown bag talk to discuss GOES-16, products, data volumes, etc. at some point maybe early April?


  • No labels