Skip to end of metadata
Go to start of metadata

Initial Notes

  • The data comes in from the LDM in the following format:
  • The current configuration (as of 3/16/2017) of the LDM is

This produces files with the following format:

/var/autofs/mnt/rapdmg1/data/goesr/DDHH/MM/ZZZZSS_KKKK_DDHHMM_TTT.nc where

D = day

H = hour

M = minute

Z = region (TIRC is CONUS, TIRU is hemispheric and TIRP is perhaps Puerto Rico, TISL = ?, TISM = ?)

S = sensor

K  = center ID, i.e. KNES (Center ID List)

T = tile ID

For example, /var/autofs/mnt/rapdmg1/data/goesr/1602/15/TISM09_KNES_160215_PAA.nc

 

  • The new format follows:

/var/autofs/mnt/rapdmg1/data/goesr/YYYY/MMDD/HH/tiles/SSS/YYYYMMDD_HHmm_ZZZZ_SSS_TTT.nc where

Y = year

M = month

D = day

S = sensor (preceded with S)

H = hour

m = minute

Z = region

T = tile ID 

For example, /var/autofs/mnt/rapdmg1/data/goesr/2017/0316/02/tiles/S09/20170316_0215_TISM_S09_PAA.nc

  • The new configuration in etc/pqact.conf would be:

This was determined using this website as reference: pqact.conf

 

Archive Old Data

The old data was moved to /var/autofs/mnt/rapdmg1/data/goesr-old to be archived to the HPSS.

The data will be moved to the HPSS in a tarball: /RAPDMG/GOESR/2017/goes_r_2017_03_09-16.tar.gz

 

 

Tiling

Resources

ncview /var/autofs/mnt/rapdmg1/data/goesr/2017/0317/S10/0727_TIRC_PAE.nc

http://netcdf-group.1586084.n2.nabble.com/concatenate-netcdf-files-td3209242.html

https://sourceforge.net/p/nco/discussion/9830/thread/2782cde1/

http://www.unidata.ucar.edu/mailing_lists/archives/thredds/2011/msg00270.html


  • A user from a forum (link above) wrote an NCL script to combine data. It would have to be modified to work with our data. The script is here:

     Click here to expand...

 

0727_TIRC_PA[A-F].nc are 1024 x 1024

0727_TIRC_PA[G-I].nc are 1024 x 512

  • We will have to figure out how to combine these. We could merge 2 of them to get 1024 x 1024 but then we have 1 left over!
  • The assumption of the last bullet is incorrect. See section "Grid Layouts" below for how tiles are arranged.

The following script was used to combine the tiles by x, but the result was a tall set of data instead of the correct shape (as expected). We will need to figure out how the tiles fit together then add logic to combine them the correct way (by x then by y)

 Click here to expand...

The following script was written to combine TIRC sensor 10 data:

 Click here to expand...

Can be improved to be more generic to tile other sensor data


Timing

  • I ran a script to combine tiles based on filename. I passed the script 

    ~/rapdmg1/data/goesr/2017/0320/S01/20170320_1542_TIRC_S01_PAY.nc and it correctly combined all 1542_TIRC_S01 tiles (30 of them) into a single NetCDF file

  • I will run the script on the 5 combinations of tiles on otho and icculus to compare the time it takes to process.
tilesRun on othoRun on icculus
9  
301 hour 40 min1 hour 18 min
1083 hour 23 min 
16  
4  

The conclusion is this method will take too long for our purposes. I will investigate using NCL instead.

NetCDF File Cleanup

  • We want to remove unused global attributes that are related to tiles after tiling since they are not relevant.

https://sourceforge.net/p/nco/discussion/9830/thread/62e97cb1/

https://sourceforge.net/p/nco/discussion/9830/thread/e6a01e3e/

This command works.The above mentions that the tool should be able to handle regex to remove multiple global attributes, but this fix was added for nco 4.5.1. otho and icculus both currently run nco 4.4.8. 

We want to remove the following attributes:

  • tile_center_longitude
  • tile_center_latitude
  • tile_row_offset
  • tile_column_offset
  • product_tile_width
  • product_tile_height
  • history (lists every tile filename)

Also, the chunking info looks incorrect in the final output file

 

Grid Layouts

TIRC

Every 30 minutes on 12 and 42

Sensors 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16

3x3 = 9 tiles

AAABAC
ADAEAF
AGAHAI


Where A-F are 1024x1024 (xy) and G-I are 1024x512

TIRC - Sensor 1, 3, 5

6x5 = 30 tiles

AAABACADAEAF
AGAHAIAJAKAL
AMANAOAPAQAR
ASATAUAVAWAX
AYAZBABBBCBD

Where all tiles are 1024x1024

TIRC - Sensor 2

12x9 = 108 tiles

AAABACADAEAFAGAHAIAJAKALAM
ANAOAPAQARASATAUAVAWAXAYAZ
BABBBCBDBEBFBGBHBIBJBKBLBM
BNBOBPBQBRBSBTBUBVBWBXBYBZ
CACBCCCDCECFCGCHCICJCKCLCM
CNCOCPCQCRCSCTCUCVCWCXCYCZ
DADBDCDDDEDFDGDHDIDJDKDLDM
DNDODPDQDRDSDTDUDVDWDXDYDZ
EAEBECEDEEEFEGEHEIEJEKELEM

Where all tiles are 1024x1024

TIRP

Every 30 minutes on 06 and 36

S01  S02  S03  S04  S05  S06  S07  S08  S09  S10  S11  S12  S13  S14  S15  S16  

4        16     4      1       4      1       1      1      1       1      1       1      1      1       1      1

 

TIRP - Sensor 2

AAABACAD
AEAFAGAH
AIAJAKAL
AMANAOAP

Where all tiles are 1024x1024 except AD, AH, AL, and AP are 768x1024

TIRP - Sensors 1, 3, 5

AAAB
ACAD

Where

AA is 1024x1024

AB is 896x1024

AC is 1024x896

AD is 896x896

TIRU

Every 30 minutes on 06 and 36

4 tiles for each sensor

AAAB
ACAD

where

AA is 1024x1024

AB is 784x1024

AC is 1024x784

AD is 784x784

TISI and TISJ

TISI =  30°N ≤ Lat. < 45°N and 75°W < Long. ≤ 90°W

TISJ = 30°N ≤ Lat. < 45°N and 60°W < Long. ≤ 75°W

every 6 minutes from 0000

S01  S02  S03  S04  S05  S06  S07  S08  S09  S10  S11  S12  S13  S14  S15  S16  

4         9      4      1      4      1       1       1      1       1      1      1       1      1       1      1    

S05 occasionally out (based on 3/19/2017 data feed)


TISI - Sensors 1, 3, 5

AAAB
ACAD

where 

AA is 1024x1024

AB is 39x1024

AC is 1024x392

AD is 39x392


TISI - Sensor 2

AAABAC
ADAEAF
AGAHAI

where

All tiles are 1024x1024 except

AC,AF are 78x1024

AG, AH are 1024x786

AI is 78x786 

Script to generate the number of tiles per sensor statistics
 Click here to expand...
Script to generate tile dimension information

These charts were created using the following code that can be modified to read other times and regions

 Click here to expand...


 

Other Notes

  • If the chomp.py script is run given a file that doesn't exist, the exception causes an error instead of handling it properly

 

  • The metadata describing how to recompose original SCMI products with the tiles are described in section 4.2. (4.0 is only 4 section in the PDF document)

 

GeoServer 

  • The NetCDF plugin for GeoServer supports gridded NetCDF files having dimensions following the COARDS convention (custom, Time, Elevation, Lat, Lon).
    • We will need to modify these files to include these dimensions, as they currently only have x,y
    • Two dimensional non-independent latitude-longitude coordinate variables aren’t currently supported, i.e. lat(x,y) and lon(x,y)

 

 

 

  • No labels