Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Expand
quoted from a post by Ryan May at Unidata:
We currently have re-assembled the tiles on our test server:
http://thredds-test.unidata.ucar.edu/thredds/catalog/satellite/goes16/catalog.html
I wrote a Python script to do the assembly and stripping of header AND footer directly piped from the LDM. That code is freely available here: 
https://github.com/Unidata/ldm-alchemy (look at goes-restitch.py) This runs only on Python 3.6.
The pqact being used for the data (which runs one instance of the script for all 16 channel

LDM

Otho is receiving the data and putting it to:

/var/autofs/mnt/rapdmg1/data/goesr/

 

Details for receiving data

 I see the data on LDM @ idd.unidata.ucar.edu:

Code Block
titleGOES notifyme on IDD
collapsetrue
julius:~% notifyme -vxl- -h idd.unidata.ucar.edu -p "TI[RS].[01][0-9] KNES"
Mar 08 19:39:36 notifyme[25015] NOTE: Starting Up: idd.unidata.ucar.edu: 20170308193936.528 TS_ENDT {{ANY, "TI[RS].[01][0-9] KNES"}}
Mar 08 19:39:36 notifyme[25015] NOTE: LDM-5 desired product-class: 20170308193936.528 TS_ENDT {{ANY, "TI[RS].[01][0-9] KNES"}}
Mar 08 19:39:36 notifyme[25015] INFO: Resolving idd.unidata.ucar.edu to 128.117.140.3 took 0.120516 seconds
Mar 08 19:39:36 notifyme[25015] DEBUG: NOTIFYME(idd.unidata.ucar.edu) returns OK
Mar 08 19:39:36 notifyme[25015] NOTE: NOTIFYME(idd.unidata.ucar.edu): OK
Mar 08 19:39:37 notifyme[25015] INFO: 601ac6600cd6d1b56bd41db706d67306    1230889 20170308193936.541  NOTHER 1454964  TIRC15 KNES 081937 PAI
Mar 08 19:39:37 notifyme[25015] INFO: 064c903805608cc9cc0ae66c6701b230    1129585 20170308193937.241  NOTHER 1454965  TIRC14 KNES 081937 PAG
Mar 08 19:39:37 notifyme[25015] INFO: 8663636dcd5bed49344af1e86dc11045    1567772 20170308193937.807  NOTHER 1454966  TIRC05 KNES 081937 PAH
Mar 08 19:39:38 notifyme[25015] INFO: 92c823b5cb379f1cbeb9ead4707e43d1    1043516 20170308193938.302  NOTHER 1454967  TIRC16 KNES 081937 PAG
Mar 08 19:39:38 notifyme[25015] INFO: 3f57488693d857fdd68d3decbec58406     876359 20170308193938.787  NOTHER 1454968  TIRC10 KNES 081937 PAI
Mar 08 19:39:39 notifyme[25015] INFO: fddcd3cca962e591e947ca10c0e2d202    1274017 20170308193939.284  NOTHER 1454969  TIRC07 KNES 081937 PAG
Mar 08 19:39:39 notifyme[25015] INFO: 626a47c007cd0524bb8d30067aa6e199    1046518 20170308193939.771  NOTHER 1454970  TIRC09 KNES 081937 PAH
Mar 08 19:39:40 notifyme[25015] INFO: eb41b080cc3fb11f4784aefbd54c0e87    1554225 20170308193940.507  NOTHER 1454971  TIRC03 KNES 081937 PAH
Mar 08 19:39:41 notifyme[25015] INFO: 964c40b93872fb6d9c1989b4c4f75cc0    1225624 20170308193941.211  NOTHER 1454972  TIRC11 KNES 081937 PAI
Mar 08 19:39:41 notifyme[25015] INFO: 819fdc134e96609993f6a033ebef4ba9    1445062 20170308193941.916  NOTHER 1454973  TIRC14 KNES 081937 PAH
Mar 08 19:39:42 notifyme[25015] INFO: a79c7e77b22ae259a0c11e8f00a04677     804807 20170308193942.216  NOTHER 1454974  TIRC09 KNES 081937 PAI
Mar 08 19:39:43 notifyme[25015] INFO: 09549a969daa019639ba472589d6cbab    1431102 20170308193943.137  NOTHER 1454975  TIRC13 KNES 081937 PAH
Mar 08 19:39:43 notifyme[25015] INFO: a2d276fd76b7b769d4164e9208a0e7a0     790246 20170308193943.581  NOTHER 1454976  TIRC10 KNES 081937 PAJ
Mar 08 19:39:44 notifyme[25015] INFO: d64c15f6adacae956c44059fb61d9930    1008289 20170308193944.056  NOTHER 1454977  TIRC08 KNES 081937 PAK
Mar 08 19:39:44 notifyme[25015] INFO: 5d8b3d2a46934ca54330ce2eadbac80c    1074094 20170308193944.563  NOTHER 1454978  TIRC12 KNES 081937 PAI
Mar 08 19:39:45 notifyme[25015] INFO: 18fb317f3fb027f52341eb0d5ad6c76f    1097439 20170308193945.227  NOTHER 1454979  TIRC16 KNES 081937 PAI
Mar 08 19:39:45 notifyme[25015] INFO: ee9d8bce191ed741ee4640883f0dca87      20218 20170308193945.230  NOTHER 1454980  TISZ02 KNES 081938 PAC
Mar 08 19:39:47 notifyme[25015] INFO: f782b94464a71307710cd57d8a3a5475    1374805 20170308193945.773  NOTHER 1454981  TIRC06 KNES 081937 PAI

 

Grab it like this:

Code Block
NOTHER  (TI....) (....) (......) (...)
        PIPE    -close  python /home/ldm/chomp.py /home/ldm/goesr/\1_\2_\3_\4.nc

Where chomp.py, is a simple script to drop the WMO header (36 bytes worth) so that the rest is a valid netcdf file (thanks daryl herzmann akrherz@iastate.edu via unidata.ucar.edu )

import sys
fn = sys.argv[1]
data = sys.stdin.read()
o = open(fn, 'wb')
o.write(data[36:])
o.close()

 

New configuration looks like this:

Code Block
NOTHER  (TI..)(..) .... (..)(....) (...)        PIPE    -close  /home/ldm/bin/prepare_goes-r.py /var/autofs/mnt/rapdmg1/data/goesr/%Y/%m\3/S\2/%Y%m\3_\4_\1_S\2_\5.nc /home/ldm/logs

prepare_goes-r.py is a modified version of chomp that optionally can create a dated log file to report which files it has processed and the WMO header contents that it has removed.

 

User Notes from LDM conversation on ldm-users

TIRC is CONUS, TIRU is hemispheric and TIRP is perhaps Puerto Rico.

PAA, PAB, etc. are the tile IDs. The tiles need to be merged to form the entire image.

C and U are center position CONUS and Full Disk, respectively.  GOES-16 is currently at the PLT position at 89.5 West.  Once it has been relocated to its final position (East or West), the headers will be changed to reflect that.

 

Displaying the GOES16 Data

 

Mosaicing the data

From unidata via the ldm support email list:

Expand
quoted from a post by Ryan May at Unidata:
We currently have re-assembled the tiles on our test server:
http://thredds-test.unidata.ucar.edu/thredds/catalog/satellite/goes16/catalog.html
I wrote a Python script to do the assembly and stripping of header AND footer directly piped from the LDM. That code is freely available here: 
https://github.com/Unidata/ldm-alchemy (look at goes-restitch.py) This runs only on Python 3.6.
The pqact being used for the data (which runs one instance of the script for all 16 channels of a particular sector/time) is here:
https://github.com/Unidata/TdsConfig/blob/master/threddsDev/pqacts/pqact.satellite
I don't have any cycles to really support that script, but I haven't noticed any problems with it on our server.
For the "true color" RGB imagery, you can see a sample of the simple approach here:
http://nbviewer.jupyter.org/github/Unidata/unidata-python-workshop/blob/master/notebooks/GOES_RGB_Demo/GOES_RGB_Image.ipynb
With the veggie channel, things are too green on land, but otherwise not too bad. I just wish they would fix the problem where the channel maxing out ends up set to 0 instead of 255.

 

 

Todo

  • LDM - need to cleanup the directory placement of data.
    • move to YYYY/MMDD/  - COMPLETED
    • don't delete old data - move away - COMPLETED (moved data to goesr-old to be archived)
    • update this page with pqact & chome changes. - COMPLETED
    • check chomp into cvs - rename from chomp.py -COMPLETED (checked into CVS under projects/ldm/goes/scripts/prepare_goes-r.py)
    • add logging & write WMO header to log (debug) - COMPLETED
    • commit changes
    • Tile the data.  - use netcdf operators - http://nco.sourceforge.net/
    • Add archiving & scrubbing
      • archive old directory structure as one big tarball
  • Documenting GOES-16 capabilities on a RAL Wiki page.
  • Look at netcdf data more - fields?  


Task list from Bob

Expand


The GOES-16 task will entail -
1) Acquiring via LDM as much GOES-16 data as is possible
at this time;
2) Setting up data storage directory structures and
scrubbing mechanisms;
3) Setting up processing to stitch together tiled data
into a single netCDF file;
4) Perhaps putting an archiving mechanism in place;
5) Perhaps developing a web mechanism for viewing the data;
6) Documenting GOES-16 capabilities on a RAL Wiki page.

The GeoServer tasks are a little less defined and will be
done as time is available with priority given to the GOES-16
tasks. GeoServer tasks may entail -
1) Installing GeoServer on a RAL infrastructure machine;
2) Making some of the GOES-16 data available via GeoServer;
3) Perhaps making some other gridded data sets available
on GeoServer (eg. MRMS radar composite);
4) Perhaps making some feature product(s) (like METARS)
available via GeoServer;
5) Perhaps making some of the data displayable via the
GeoServer WMS and OpenLayers.
6) Documenting GeoServer capabilities on a RAL Wiki page.


From Matthias:

 It would be good if Paul could team up with Cathy for a brown bag talk to discuss GOES-16, products, data volumes, etc. at some point maybe early April?


s of a particular sector/time) is here:
https://github.com/Unidata/TdsConfig/blob/master/threddsDev/pqacts/pqact.satellite
I don't have any cycles to really support that script, but I haven't noticed any problems with it on our server.
For the "true color" RGB imagery, you can see a sample of the simple approach here:
http://nbviewer.jupyter.org/github/Unidata/unidata-python-workshop/blob/master/notebooks/GOES_RGB_Demo/GOES_RGB_Image.ipynb
With the veggie channel, things are too green on land, but otherwise not too bad. I just wish they would fix the problem where the channel maxing out ends up set to 0 instead of 255.

 

 

Todo

  • LDM - need to cleanup the directory placement of data.
    • move to YYYY/MMDD/  - COMPLETED
    • don't delete old data - move away - COMPLETED (moved data to goesr-old to be archived)
    • update this page with pqact & chome changes. - COMPLETED
    • check chomp into cvs - rename from chomp.py -COMPLETED (checked into CVS under projects/ldm/goes/scripts/prepare_goes-r.py)
    • add logging & write WMO header to log (debug) - COMPLETED
    • commit changes
    • Tile the data.  - use netcdf operators - http://nco.sourceforge.net/
    • Add archiving & scrubbing
      • archive old directory structure as one big tarball
  • Documenting GOES-16 capabilities on a RAL Wiki page.
  • Look at netcdf data more - fields?  
Expand
title"Final" status update from Aaron
 An update on where we stand on the GOES 16 work.  I was able to get GeoServer running on the host but I have been waiting for the following two items since Tuesday of last week:
  • get PostGres/PostGIS installed on satops3 (required for GeoServer and gridded data)
  • open the firewall rules to allow other hosts to contact satops3 on port 8080.  I worked around this by using ssh port forwarding but obviously if we want a shared resource it needs to be fixed

Once these tasks are accomplished by SNAT the next steps would be:
  • determine whether LDM should be running on otho or satops3 (as I recall Paul had thought it should be on satops3?)
  • rename files so they include a date format recognized by GeoServer (such as 20170331T1650Z_TISJ_S01.nc)
  • post-process the GOES files so they have a time dimension.  They currently only have x and y
  • determine which GOES 16 products are of interest in GeoServer (fields - S01..., domains such as CONUS)
  • (optional) figure out how to configure tiled datasets in GeoServer (I started an email thread on geoserver-users here) but ran out of time to track down the issues with the GOES 16 original tiles (not the mosaics)
  • configure the appropriate datasets in GeoServer pointing to the PostGIS instance on satops3
  • configure the color scales/color maps as custom SLDs in GeoServer

Since I could not work directly on satops3 due to the issues up top I worked on getting the GOES 16 data running on my development machine with an older version of GeoServer.  What I did last week was:
  • Fixed an issue with GeoServer access to the native netCDF library
  • Pointed GeoServer to a "full" data directory.  This allowed it to start up normally
  • Changed the admin password (since this may be an exposed host).  The credentials for accessing the web UI are admin:goes16
  • Created an 'NCAR' workspace where we will add our data later
  • Configured some sensible global GeoServer default settings
  • Created https://github.com/NCAR/geoserver-configs - a RAL repository for keeping existing GeoServer config starting points and instructions for setting up GeoServer.  As I went through this process on satops3 I documented each of the steps, which was a surprisingly lengthy process
  • Created https://github.com/NCAR/geoserver-config-generator - a RAL repository for a utility I created for auto-generating gridded data configurations for GeoServer.  This utility was created to create the GeoServer ImageMosaic indexer configurations as well as the CRS/SRS configuration needed in GeoServer.  Creating the correct projections/SRS configurations by hand is particularly tricky, so the hope is that this will allow us to quickly get a starting point for putting gridded data into GeoServer

I was able to get the mosaiced version of two different GOES 16 datasets running in my GeoServer instance (TIRC and TISJ).  The config generator utility is the only reason I got the latter GOES 16 dataset running, there was a good amount of guesswork about how to create the proper SRS information for GeoServer to use.  As mentioned above I still do not have a working tiled configuration.
I anticipate these GitHub repos being shared resources for getting GeoServer configured going forward. 


Task list from Bob

Expand


The GOES-16 task will entail -
1) Acquiring via LDM as much GOES-16 data as is possible
at this time;
2) Setting up data storage directory structures and
scrubbing mechanisms;
3) Setting up processing to stitch together tiled data
into a single netCDF file;
4) Perhaps putting an archiving mechanism in place;
5) Perhaps developing a web mechanism for viewing the data;
6) Documenting GOES-16 capabilities on a RAL Wiki page.

The GeoServer tasks are a little less defined and will be
done as time is available with priority given to the GOES-16
tasks. GeoServer tasks may entail -
1) Installing GeoServer on a RAL infrastructure machine;
2) Making some of the GOES-16 data available via GeoServer;
3) Perhaps making some other gridded data sets available
on GeoServer (eg. MRMS radar composite);
4) Perhaps making some feature product(s) (like METARS)
available via GeoServer;
5) Perhaps making some of the data displayable via the
GeoServer WMS and OpenLayers.
6) Documenting GeoServer capabilities on a RAL Wiki page.


From Matthias:

 It would be good if Paul could team up with Cathy for a brown bag talk to discuss GOES-16, products, data volumes, etc. at some point maybe early April?