Skip to end of metadata
Go to start of metadata

Sources of Archive data

AWS

Other

Receiving GOES 16 data.

You can get the untiled netcdf files from Amazon, Google or OCC:



Direct Readout

GOES Rebroadcast (GRB) signal.  The GRB system is the primary space relay of data products from the GOES-R ground system processing. GRB will replace the GOES VARiable (GVAR) service. This service allows users in the Western Hemisphere to receive full-resolution imagery and weather products in near real time. The delivery of this information is essential to weather forecasting across the AmericasNorth, South, and Central— and the Caribbean.

GRB is a 31 megabytes per second direct readout broadcast that replaces the 2.1 megabytes per second GVAR service. The GRB signal is relayed by the GOES-R satellite to compatible ground receive terminals in the satellite’s field of view.

(31 MB/second = 2.67 TB/day)


LDM

Otho is receiving the data and putting it to:

/var/autofs/mnt/rapdmg1/data/goesr/


Level-2 products

The level-2 (L2) products are derived satellite products produced by NESDIS. They are based on the level-1 ABI data. These data will not be in the GRB stream. Currently access is only available through NOAAPORT. Eventually, the software package that creates the L2 products will be publicly available. The software package could be run alongside the CSSP-Geo package. As such we are left with NOAAPORT.



NOAA Service Change Notice 17-73


Service Change Notice 17-73

National Weather Service Headquarters Silver Spring, MD

905 AM EDT Thu Jun 15 2017

 

To:      Subscribers:

         -NOAA Weather Wire Service

         -Emergency Managers Weather Information Network

         -NOAAPORT

         Other NWS Partners and NWS Employees

 

From:    Joseph Pica

         Director, NWS Office of Observations

 

Subject: Add Geostationary Operational Environmental

         Satellite–16 (GOES-16) Level 2 Derived Products

         to the Satellite Broadcast Network (SBN or

         NOAAPORT), effective June 14, 2017

 

Effective on or after 15 UTC Wednesday, June 14, 2017,

additional GOES-16 Level 2 Derived Products will be added to the

SBN (also known as NOAAPORT). These products are based on the

GOES-16 Advanced Baseline Imager (ABI). The SBN feed of these

GOES-16 products will be in a pre-operational validation and

testing phase through at least November 2017. 

 

NOAA's GOES-16 satellite has not been declared operational and

its data are preliminary and undergoing testing. Users receiving

these data through any dissemination means assume all risk

related to their use of GOES-16 data and NOAA disclaims any and

all warranties, whether express or implied, including (without

limitation) any implied warranties of merchantability or fitness

for a particular purpose. These products are being added to

enable the testing of networks and systems with actual GOES-16

data, and so that users will be able to evaluate the products

during this validation/test period. 

 

It is not recommended that these preliminary products be used in

operational forecast processes, especially during the initial

months of this validation/test period. Any downstream postings

of these data should clearly convey their preliminary, non-

operational and in-testing nature. Furthermore, this data stream

may be subject to temporary unannounced data outages and

modifications during this validation and testing period.  One or

more follow-on notices will provide information on the progress

and conclusion of the validation period.

 

Refer to Service Change Notice 17-61 which introduced a

different set of GOES-16 derived products to the SBN:  

 

http://www.nws.noaa.gov/os/notification/scn17-61goes1_3.htm

 

Those derived products will continue to flow as the new Level 2

derived products described herein are added to the network.    

 

The new GOES-16 level 2 derived products, the first 11

characters of their WMO headers and their geographical regions

of coverage are as follows:

 

 GOES-16 Product                   WMO Header    Regions

 ---------------                   ----------    -------

 Cloud Optical Depth               IXTY01 KNES   F,C

 Cloud Particle Size Distribution  IXTW01 KNES   F,C,M

 Derived Motion Winds              IXTU99 KNES   F,C,M

 

Notes on table, above:

1. Region codes refer to GOES-16 geographical sectors, where F

refers to ABI Full Disk, C refers to ABI CONUS, and M refers to

ABI mesoscale.

 

2. KNES signifies that NOAA/NESDIS originated the products. 

 

Approximate hourly product counts and hourly volumes for the new

GOES-16 level 2 derived products are as follows:

 

 GOES-16 Product            Hourly Count    Hourly Volume

 ---------------            -------------   -------------

 Cloud Optical Depth        24/hour         144 MBytes/hour

 Cloud Particle Size Dist.  136/hour        360 MBytes/hour

 Derived Motion Winds       168/hour        400 Mbytes/hour

 

Product counts and volumes will vary due to ABI mode changes,

diurnal variations, file-compression variability, and other

factors.

 

For the foreseeable future, the SBN’s Experimental channel 

(PID 106) will be used to disseminate these GOES-16 Level 2

derived products; however, NWS anticipates the SBN suite of

GOES-16 products described here will evolve. Several of these

products are likely to be discontinued after an evaluation

period. Other products eventually will be moved to other SBN

channels and possibly modified.  Additional GOES-16 products are

planned for addition to the SBN during July 2017. Future service

change notices will describe these changes in advance.

 

The file format for these products is netCDF4.

 

GOES-16 remains stationed over its 89.5 degrees west longitude

station. From this location, it is undergoing checkout and

validation. GOES-16 will be moved to the GOES-East position at

75 degrees west longitude, once it is declared operational in

November 2017. 

 

Additional information about the flow of GOES-16 products on the

SBN has been posted at the following web site and will be

periodically updated:

 

http://www.nws.noaa.gov/noaaport/html/GOES_R.shtml

 

Critical weather or other factors could delay the activation of

these products on the SBN. 

 

For questions pertaining to this change or upcoming plans for

the addition of GOES-16 products onto NOAAPORT, please contact:

 

     Brian Gockel

     NOAA/NWS Office of Observations

     Surface and Upper Air Division, Services Branch

     Silver Spring, MD 20910

     Email: Brian.Gockel@noaa.gov

   and

     AWIPS Network Control Facility (NCF) Help Desk

     NOAA/NWS Office of Central Processing

     Silver Spring, MD 20910

     Email: nws.ncf.supervisors@noaa.gov

 

For questions regarding the scientific or technical content of

the NOAAPORT-disseminated GOES-16 products please contact:

 

     Environmental Satellite Processing Center (ESPC) Help Desk

     Suitland, Maryland 20746

     Phone: 301-817-3880

     Email: ESPCOperations@noaa.gov

 

National Service Change Notices are online at:

 

http://www.weather.gov/os/notif.htm

 



NOAA Service Change Notice 17-61 


Service Change Notice 17-61

National Weather Service Headquarters Silver Spring, MD

350 PM EDT Wed May 31 2017

 

To:      Subscribers:

         -NOAA Weather Wire Service

         -Emergency Managers Weather Information Network

         -NOAAPORT

         Other NWS Partners and NWS Employees

 

From:    Joseph Pica

         Director, NWS Office of Observations

 

Subject: Add Geostationary Operational Environmental

         Satellite - 16 (GOES-16) Level 2 Derived Products

         to the Satellite Broadcast Network (SBN or

         NOAAPORT), effective May 31, 2017

 

Effective on or after 15UTC Wednesday, May 31, 2017, GOES-16

Level 2 Derived Products will be added to the SBN (also known as

NOAAPORT). These products are based on the GOES-16 Advanced

Baseline Imager (ABI). The SBN feed of these GOES-16 products

will be in a pre-operational validation and testing phase

through at least November 2017. 

 

NOAA's GOES-16 satellite has not been declared operational and

its data are preliminary and undergoing testing. Users receiving

these data through any dissemination means assume all risk

related to their use of GOES-16 data and NOAA disclaims any and

all warranties, whether express or implied, including (without

limitation) any implied warranties of merchantability or fitness

for a particular purpose.  These products are being added to

enable the testing of networks and systems with actual GOES-16

data and so that users will be able to evaluate the products

during this validation/test period. 

 

It is not recommended that these preliminary products be used in

operational forecast processes, especially during the initial

months of this validation/test period. Any downstream postings

of these data should clearly convey their preliminary, non-

operational and in-testing nature. Furthermore, this data stream

may be subject to temporary unannounced data outages and

modifications during this validation and testing period. One or

more follow-on notices will provide information on the progress

and conclusion of the validation period.

 

Refer to prior Service Change Notice 17-24 which introduced

GOES-R imagery to the SBN: 

 

http://www.nws.noaa.gov/os/notification/scn17-24goes16.htm

That imagery will continue to flow as the new Level 2 derived

products described herein are added to the network.    

 

The new GOES-16 level 2 derived products, the first 

11 characters of their WMO headers and their geographical

regions of coverage are as follows:

 

  GOES-16 Product                 WMO Header    Regions

  ---------------                 ----------    -------

  Aerosol Detection (Smoke+Dust)  IXTA99 KNES   F,C,M

  Aerosol Optical Depth           IXTB99 KNES   F,C

  Cloud Top Phase                 IXTD99 KNES   F,C,M

  Cloud Top Height                IXTG99 KNES   F,C,M

  Cloud Top Pressure              IXTX01 KNES   F,C

  Cloud Top Temperature           IXTI99 KNES   F,M

  Derived Stability Indices       IXTN99 KNES   F,C,M

  Total Precipitable Water        IXTO99 KNES   F,C,M

  Clear Sky Mask                  IXTH99 KNES   F,C,M

  Fire/Hot Spot Characterization  IXTJ99 KNES   F,C

  Land Surface (Skin) Temperature IXTK99 KNES   F,C,M

 

Notes on table above:

1. Region codes refer to GOES-16 ABI sectors, where F refers to

Full Disk, C refers to CONUS and M refers to mesoscale.

 

2. Derived Stability Indices include Convective Available

Potential Energy (CAPE), Lifted Index (LI), K Index (KI),

Showalter Index and Total Totals Index (TT).

 

3. KNES signifies that NOAA/NESDIS is the creating data center. 

 

For the foreseeable future, the SBN’s Experimental channel (PID

106) will be used to disseminate these GOES-16 Level 2 derived

products; however, NOAA anticipates the SBN suite of GOES-16

products described here will evolve. Several of these products

are likely to be discontinued after an evaluation period. Other

products eventually will be moved to other SBN channels and

possibly modified. Additional GOES-16 products are planned for

addition to the SBN during June 2017. Future service change

notices will describe these changes in advance.

 

The file format for these products is netCDF4.

 

GOES-16 remains stationed over its 89.5 degrees west longitude

station. From this location, it is undergoing checkout and

validation. GOES-16 will be moved to the GOES-East position at

75 degrees west longitude, once it is declared operational in

November 2017. 

 

Additional information about the flow of GOES-16 products on the

SBN has been posted at the following web site and will be

periodically updated:

 

http://www.nws.noaa.gov/noaaport/html/GOES_R.shtml

Critical weather or other factors could delay the activation of

these products on the SBN. 

 

For questions pertaining to this change or upcoming plans for

the addition of GOES-16 products onto NOAAPORT, please contact:

 

     Brian Gockel

     NOAA/NWS Office of Observations

     Surface and Upper Air Division, Services Branch

     Silver Spring, MD 20910

     Email: Brian.Gockel@noaa.gov

 

For questions regarding the scientific or technical content of

the NOAAPORT-disseminated GOES-16 products please contact:

 

     Environmental Satellite Processing Center (ESPC) Help Desk

     Suitland, Maryland 20746

     Phone: 301-817-3880

     Email: ESPCOperations@noaa.gov

 

National Service Change Notices are online at:

 

http://www.weather.gov/os/notif.htm


 

Details for receiving data

 I see the data on LDM @ idd.unidata.ucar.edu:

GOES notifyme on IDD
julius:~% notifyme -vxl- -h idd.unidata.ucar.edu -p "TI[RS].[01][0-9] KNES"
Mar 08 19:39:36 notifyme[25015] NOTE: Starting Up: idd.unidata.ucar.edu: 20170308193936.528 TS_ENDT {{ANY, "TI[RS].[01][0-9] KNES"}}
Mar 08 19:39:36 notifyme[25015] NOTE: LDM-5 desired product-class: 20170308193936.528 TS_ENDT {{ANY, "TI[RS].[01][0-9] KNES"}}
Mar 08 19:39:36 notifyme[25015] INFO: Resolving idd.unidata.ucar.edu to 128.117.140.3 took 0.120516 seconds
Mar 08 19:39:36 notifyme[25015] DEBUG: NOTIFYME(idd.unidata.ucar.edu) returns OK
Mar 08 19:39:36 notifyme[25015] NOTE: NOTIFYME(idd.unidata.ucar.edu): OK
Mar 08 19:39:37 notifyme[25015] INFO: 601ac6600cd6d1b56bd41db706d67306    1230889 20170308193936.541  NOTHER 1454964  TIRC15 KNES 081937 PAI
Mar 08 19:39:37 notifyme[25015] INFO: 064c903805608cc9cc0ae66c6701b230    1129585 20170308193937.241  NOTHER 1454965  TIRC14 KNES 081937 PAG
Mar 08 19:39:37 notifyme[25015] INFO: 8663636dcd5bed49344af1e86dc11045    1567772 20170308193937.807  NOTHER 1454966  TIRC05 KNES 081937 PAH
Mar 08 19:39:38 notifyme[25015] INFO: 92c823b5cb379f1cbeb9ead4707e43d1    1043516 20170308193938.302  NOTHER 1454967  TIRC16 KNES 081937 PAG
Mar 08 19:39:38 notifyme[25015] INFO: 3f57488693d857fdd68d3decbec58406     876359 20170308193938.787  NOTHER 1454968  TIRC10 KNES 081937 PAI
Mar 08 19:39:39 notifyme[25015] INFO: fddcd3cca962e591e947ca10c0e2d202    1274017 20170308193939.284  NOTHER 1454969  TIRC07 KNES 081937 PAG
Mar 08 19:39:39 notifyme[25015] INFO: 626a47c007cd0524bb8d30067aa6e199    1046518 20170308193939.771  NOTHER 1454970  TIRC09 KNES 081937 PAH
Mar 08 19:39:40 notifyme[25015] INFO: eb41b080cc3fb11f4784aefbd54c0e87    1554225 20170308193940.507  NOTHER 1454971  TIRC03 KNES 081937 PAH
Mar 08 19:39:41 notifyme[25015] INFO: 964c40b93872fb6d9c1989b4c4f75cc0    1225624 20170308193941.211  NOTHER 1454972  TIRC11 KNES 081937 PAI
Mar 08 19:39:41 notifyme[25015] INFO: 819fdc134e96609993f6a033ebef4ba9    1445062 20170308193941.916  NOTHER 1454973  TIRC14 KNES 081937 PAH
Mar 08 19:39:42 notifyme[25015] INFO: a79c7e77b22ae259a0c11e8f00a04677     804807 20170308193942.216  NOTHER 1454974  TIRC09 KNES 081937 PAI
Mar 08 19:39:43 notifyme[25015] INFO: 09549a969daa019639ba472589d6cbab    1431102 20170308193943.137  NOTHER 1454975  TIRC13 KNES 081937 PAH
Mar 08 19:39:43 notifyme[25015] INFO: a2d276fd76b7b769d4164e9208a0e7a0     790246 20170308193943.581  NOTHER 1454976  TIRC10 KNES 081937 PAJ
Mar 08 19:39:44 notifyme[25015] INFO: d64c15f6adacae956c44059fb61d9930    1008289 20170308193944.056  NOTHER 1454977  TIRC08 KNES 081937 PAK
Mar 08 19:39:44 notifyme[25015] INFO: 5d8b3d2a46934ca54330ce2eadbac80c    1074094 20170308193944.563  NOTHER 1454978  TIRC12 KNES 081937 PAI
Mar 08 19:39:45 notifyme[25015] INFO: 18fb317f3fb027f52341eb0d5ad6c76f    1097439 20170308193945.227  NOTHER 1454979  TIRC16 KNES 081937 PAI
Mar 08 19:39:45 notifyme[25015] INFO: ee9d8bce191ed741ee4640883f0dca87      20218 20170308193945.230  NOTHER 1454980  TISZ02 KNES 081938 PAC
Mar 08 19:39:47 notifyme[25015] INFO: f782b94464a71307710cd57d8a3a5475    1374805 20170308193945.773  NOTHER 1454981  TIRC06 KNES 081937 PAI

 

Grab it like this:

NOTHER  (TI....) (....) (......) (...)
        PIPE    -close  python /home/ldm/chomp.py /home/ldm/goesr/\1_\2_\3_\4.nc

Where chomp.py, is a simple script to drop the WMO header (36 bytes worth) so that the rest is a valid netcdf file (thanks daryl herzmann akrherz@iastate.edu via unidata.ucar.edu )


The header offset is 32 bytes. There is also a four-byte footer.


import sys
fn = sys.argv[1]
data = sys.stdin.read()
o = open(fn, 'wb')
o.write(data[32:-4])
o.close()

 

New configuration looks like this:

NOTHER  (TI..)(..) .... (..)(....) (...)        PIPE    -close  /home/ldm/bin/prepare_goes-r.py /var/autofs/mnt/rapdmg1/data/goesr/%Y/%m\3/S\2/%Y%m\3_\4_\1_S\2_\5.nc /home/ldm/logs

prepare_goes-r.py is a modified version of chomp that optionally can create a dated log file to report which files it has processed and the WMO header contents that it has removed.

 

User Notes from LDM conversation on ldm-users

TIRC is CONUS, TIRU is hemispheric and TIRP is perhaps Puerto Rico.

PAA, PAB, etc. are the tile IDs. The tiles need to be merged to form the entire image.

C and U are center position CONUS and Full Disk, respectively.  GOES-16 is currently at the PLT position at 89.5 West.  Once it has been relocated to its final position (East or West), the headers will be changed to reflect that.

 

Displaying the GOES16 Data

 

Mosaicing the data

From unidata via the ldm support email list:

 Click here to expand...
quoted from a post by Ryan May at Unidata:
We currently have re-assembled the tiles on our test server:
http://thredds-test.unidata.ucar.edu/thredds/catalog/satellite/goes16/catalog.html
I wrote a Python script to do the assembly and stripping of header AND footer directly piped from the LDM. That code is freely available here: 
https://github.com/Unidata/ldm-alchemy (look at goes-restitch.py) This runs only on Python 3.6.
The pqact being used for the data (which runs one instance of the script for all 16 channels of a particular sector/time) is here: https://github.com/Unidata/TdsConfig/blob/master/threddsDev/pqacts/pqact.satellite
I don't have any cycles to really support that script, but I haven't noticed any problems with it on our server.
For the "true color" RGB imagery, you can see a sample of the simple approach here:
http://nbviewer.jupyter.org/github/Unidata/unidata-python-workshop/blob/master/notebooks/GOES_RGB_Demo/GOES_RGB_Image.ipynb
With the veggie channel, things are too green on land, but otherwise not too bad. I just wish they would fix the problem where the channel maxing out ends up set to 0 instead of 255.

 

 

Todo

  • LDM - need to cleanup the directory placement of data.
    • move to YYYY/MMDD/  - COMPLETED
    • don't delete old data - move away - COMPLETED (moved data to goesr-old to be archived)
    • update this page with pqact & chome changes. - COMPLETED
    • check chomp into cvs - rename from chomp.py -COMPLETED (checked into CVS under projects/ldm/goes/scripts/prepare_goes-r.py)
    • add logging & write WMO header to log (debug) - COMPLETED
    • commit changes
    • Tile the data.  - use netcdf operators - http://nco.sourceforge.net/
    • Add archiving & scrubbing
      • archive old directory structure as one big tarball
  • Documenting GOES-16 capabilities on a RAL Wiki page.
  • Look at netcdf data more - fields?  
 "Final" status update from Aaron
 An update on where we stand on the GOES 16 work.  I was able to get GeoServer running on the host but I have been waiting for the following two items since Tuesday of last week:
  • get PostGres/PostGIS installed on satops3 (required for GeoServer and gridded data)
  • open the firewall rules to allow other hosts to contact satops3 on port 8080.  I worked around this by using ssh port forwarding but obviously if we want a shared resource it needs to be fixed

Once these tasks are accomplished by SNAT the next steps would be:
  • determine whether LDM should be running on otho or satops3 (as I recall Paul had thought it should be on satops3?)
  • rename files so they include a date format recognized by GeoServer (such as 20170331T1650Z_TISJ_S01.nc)
  • post-process the GOES files so they have a time dimension.  They currently only have x and y
  • determine which GOES 16 products are of interest in GeoServer (fields - S01..., domains such as CONUS)
  • (optional) figure out how to configure tiled datasets in GeoServer (I started an email thread on geoserver-users here) but ran out of time to track down the issues with the GOES 16 original tiles (not the mosaics)
  • configure the appropriate datasets in GeoServer pointing to the PostGIS instance on satops3
  • configure the color scales/color maps as custom SLDs in GeoServer

Since I could not work directly on satops3 due to the issues up top I worked on getting the GOES 16 data running on my development machine with an older version of GeoServer.  What I did last week was:
  • Fixed an issue with GeoServer access to the native netCDF library
  • Pointed GeoServer to a "full" data directory.  This allowed it to start up normally
  • Changed the admin password (since this may be an exposed host).  The credentials for accessing the web UI are admin:goes16
  • Created an 'NCAR' workspace where we will add our data later
  • Configured some sensible global GeoServer default settings
  • Created https://github.com/NCAR/geoserver-configs - a RAL repository for keeping existing GeoServer config starting points and instructions for setting up GeoServer.  As I went through this process on satops3 I documented each of the steps, which was a surprisingly lengthy process
  • Created https://github.com/NCAR/geoserver-config-generator - a RAL repository for a utility I created for auto-generating gridded data configurations for GeoServer.  This utility was created to create the GeoServer ImageMosaic indexer configurations as well as the CRS/SRS configuration needed in GeoServer.  Creating the correct projections/SRS configurations by hand is particularly tricky, so the hope is that this will allow us to quickly get a starting point for putting gridded data into GeoServer

I was able to get the mosaiced version of two different GOES 16 datasets running in my GeoServer instance (TIRC and TISJ).  The config generator utility is the only reason I got the latter GOES 16 dataset running, there was a good amount of guesswork about how to create the proper SRS information for GeoServer to use.  As mentioned above I still do not have a working tiled configuration.
I anticipate these GitHub repos being shared resources for getting GeoServer configured going forward. 


Task list from Bob

 Click here to expand...


The GOES-16 task will entail -
1) Acquiring via LDM as much GOES-16 data as is possible
at this time;
2) Setting up data storage directory structures and
scrubbing mechanisms;
3) Setting up processing to stitch together tiled data
into a single netCDF file;
4) Perhaps putting an archiving mechanism in place;
5) Perhaps developing a web mechanism for viewing the data;
6) Documenting GOES-16 capabilities on a RAL Wiki page.

The GeoServer tasks are a little less defined and will be
done as time is available with priority given to the GOES-16
tasks. GeoServer tasks may entail -
1) Installing GeoServer on a RAL infrastructure machine;
2) Making some of the GOES-16 data available via GeoServer;
3) Perhaps making some other gridded data sets available
on GeoServer (eg. MRMS radar composite);
4) Perhaps making some feature product(s) (like METARS)
available via GeoServer;
5) Perhaps making some of the data displayable via the
GeoServer WMS and OpenLayers.
6) Documenting GeoServer capabilities on a RAL Wiki page.


From Matthias:

 It would be good if Paul could team up with Cathy for a brown bag talk to discuss GOES-16, products, data volumes, etc. at some point maybe early April?












  • No labels