Otho is receiving the data and putting it to:
Details for receiving data
I see the data on LDM @ idd.unidata.ucar.edu:
Grab it like this:
import sys fn = sys.argv data = sys.stdin.read() o = open(fn, 'wb') o.write(data[36:]) o.close()
New configuration looks like this:
prepare_goes-r.py is a modified version of chomp that optionally can create a dated log file to report which files it has processed and the WMO header contents that it has removed.
User Notes from LDM conversation on ldm-users
TIRC is CONUS, TIRU is hemispheric and TIRP is perhaps Puerto Rico.
PAA, PAB, etc. are the tile IDs. The tiles need to be merged to form the entire image.
C and U are center position CONUS and Full Disk, respectively. GOES-16 is currently at the PLT position at 89.5 West. Once it has been relocated to its final position (East or West), the headers will be changed to reflect that.
Displaying the GOES16 Data
- One Option - plotting NetCDF data with Python
Mosaicing the data
From unidata via the ldm support email list:
quoted from a post by Ryan May at Unidata:
We currently have re-assembled the tiles on our test server:
I wrote a Python script to do the assembly and stripping of header AND footer directly piped from the LDM. That code is freely available here:
https://github.com/Unidata/ldm-alchemy (look at goes-restitch.py) This runs only on Python 3.6.
The pqact being used for the data (which runs one instance of the script for all 16 channels of a particular sector/time) is here: https://github.com/Unidata/TdsConfig/blob/master/threddsDev/pqacts/pqact.satellite
I don't have any cycles to really support that script, but I haven't noticed any problems with it on our server.
For the "true color" RGB imagery, you can see a sample of the simple approach here:
With the veggie channel, things are too green on land, but otherwise not too bad. I just wish they would fix the problem where the channel maxing out ends up set to 0 instead of 255.
- LDM - need to cleanup the directory placement of data.
- move to YYYY/MMDD/ - COMPLETED
- don't delete old data - move away - COMPLETED (moved data to goesr-old to be archived)
- update this page with pqact & chome changes. - COMPLETED
- check chomp into cvs - rename from chomp.py -COMPLETED (checked into CVS under projects/ldm/goes/scripts/prepare_goes-r.py)
- add logging & write WMO header to log (debug) - COMPLETED
- commit changes
- Tile the data. - use netcdf operators - http://nco.sourceforge.net/
- Add archiving & scrubbing
- archive old directory structure as one big tarball
- Documenting GOES-16 capabilities on a RAL Wiki page.
- Look at netcdf data more - fields?
- get PostGres/PostGIS installed on satops3 (required for GeoServer and gridded data)
- open the firewall rules to allow other hosts to contact satops3 on port 8080. I worked around this by using ssh port forwarding but obviously if we want a shared resource it needs to be fixed
- determine whether LDM should be running on otho or satops3 (as I recall Paul had thought it should be on satops3?)
- rename files so they include a date format recognized by GeoServer (such as 20170331T1650Z_TISJ_S01.nc)
- post-process the GOES files so they have a time dimension. They currently only have x and y
- determine which GOES 16 products are of interest in GeoServer (fields - S01..., domains such as CONUS)
- (optional) figure out how to configure tiled datasets in GeoServer (I started an email thread on geoserver-users here) but ran out of time to track down the issues with the GOES 16 original tiles (not the mosaics)
- configure the appropriate datasets in GeoServer pointing to the PostGIS instance on satops3
- configure the color scales/color maps as custom SLDs in GeoServer
- Fixed an issue with GeoServer access to the native netCDF library
- Pointed GeoServer to a "full" data directory. This allowed it to start up normally
- Changed the admin password (since this may be an exposed host). The credentials for accessing the web UI are admin:goes16
- Created an 'NCAR' workspace where we will add our data later
- Configured some sensible global GeoServer default settings
- Created https://github.com/NCAR/geoserver-configs - a RAL repository for keeping existing GeoServer config starting points and instructions for setting up GeoServer. As I went through this process on satops3 I documented each of the steps, which was a surprisingly lengthy process
- Created https://github.com/NCAR/geoserver-config-generator - a RAL repository for a utility I created for auto-generating gridded data configurations for GeoServer. This utility was created to create the GeoServer ImageMosaic indexer configurations as well as the CRS/SRS configuration needed in GeoServer. Creating the correct projections/SRS configurations by hand is particularly tricky, so the hope is that this will allow us to quickly get a starting point for putting gridded data into GeoServer
Task list from Bob
The GOES-16 task will entail -
1) Acquiring via LDM as much GOES-16 data as is possible
at this time;
2) Setting up data storage directory structures and
3) Setting up processing to stitch together tiled data
into a single netCDF file;
4) Perhaps putting an archiving mechanism in place;
5) Perhaps developing a web mechanism for viewing the data;
6) Documenting GOES-16 capabilities on a RAL Wiki page.
The GeoServer tasks are a little less defined and will be
done as time is available with priority given to the GOES-16
tasks. GeoServer tasks may entail -
1) Installing GeoServer on a RAL infrastructure machine;
2) Making some of the GOES-16 data available via GeoServer;
3) Perhaps making some other gridded data sets available
on GeoServer (eg. MRMS radar composite);
4) Perhaps making some feature product(s) (like METARS)
available via GeoServer;
5) Perhaps making some of the data displayable via the
GeoServer WMS and OpenLayers.
6) Documenting GeoServer capabilities on a RAL Wiki page.
It would be good if Paul could team up with Cathy for a brown bag talk to discuss GOES-16, products, data volumes, etc. at some point maybe early April?