Initial Notes

TIRC15 KNES 081937 PAI
NOTHER  (TI....) (....) (....)(..) (...)        PIPE    -close  /home/ldm/bin/chomp.py /var/autofs/mnt/rapdmg1/data/goesr/\3/\4/\1_\2_\3\4_\5.nc

This produces files with the following format:

/var/autofs/mnt/rapdmg1/data/goesr/DDHH/MM/ZZZZSS_KKKK_DDHHMM_TTT.nc where

D = day

H = hour

M = minute

Z = region (TIRC is CONUS, TIRU is hemispheric and TIRP is perhaps Puerto Rico, TISL = ?, TISM = ?)

S = sensor

K  = center ID, i.e. KNES (Center ID List)

T = tile ID

For example, /var/autofs/mnt/rapdmg1/data/goesr/1602/15/TISM09_KNES_160215_PAA.nc

 

/var/autofs/mnt/rapdmg1/data/goesr/YYYY/MMDD/HH/tiles/SSS/YYYYMMDD_HHmm_ZZZZ_SSS_TTT.nc where

Y = year

M = month

D = day

S = sensor (preceded with S)

H = hour

m = minute

Z = region

T = tile ID 

For example, /var/autofs/mnt/rapdmg1/data/goesr/2017/0316/02/tiles/S09/20170316_0215_TISM_S09_PAA.nc

NOTHER    (TI..)(..) .... (..)(..)(..) (...)    PIPE    -close    /home/ldm/bin/prepare_goes-r.py /var/autofs/mnt/rapdmg1/data/goesr/%Y/%m\3/\4/tiles/S\2/%Y%m\3_\4\5_\1_S\2_\6.nc /home/ldm/logs

This was determined using this website as reference: pqact.conf

 

Archive Old Data

The old data was moved to /var/autofs/mnt/rapdmg1/data/goesr-old to be archived to the HPSS.

The data will be moved to the HPSS in a tarball: /RAPDMG/GOESR/2017/goes_r_2017_03_09-16.tar.gz

 

 

Tiling

Resources

ncview /var/autofs/mnt/rapdmg1/data/goesr/2017/0317/S10/0727_TIRC_PAE.nc

http://netcdf-group.1586084.n2.nabble.com/concatenate-netcdf-files-td3209242.html

https://sourceforge.net/p/nco/discussion/9830/thread/2782cde1/

http://www.unidata.ucar.edu/mailing_lists/archives/thredds/2011/msg00270.html


 

0727_TIRC_PA[A-F].nc are 1024 x 1024

0727_TIRC_PA[G-I].nc are 1024 x 512

The following script was used to combine the tiles by x, but the result was a tall set of data instead of the correct shape (as expected). We will need to figure out how the tiles fit together then add logic to combine them the correct way (by x then by y)

#! /bin/bash

ts=( A B C D E F ) #G H I )

for t in ${ts[@]}; do
  echo ncecat -O /home/ldm/rapdmg1/data/goesr/2017/0317/S10/0727_TIRC_PA$t.nc 0727_TIRC_PA$t.nc
  echo ncpdq -O -a x,record 0727_TIRC_PA$t.nc 0727_TIRC_PA$t.nc
  echo ncwa -O -a record 0727_TIRC_PA$t.nc 0727_TIRC_PA$t.nc
done

cmd="ncrcat "

for t in ${ts[@]}; do
  cmd=$cmd"0727_TIRC_PA"$t".nc "
done

cmd=$cmd" out.nc"
echo $cmd

The following script was written to combine TIRC sensor 10 data:

Can be improved to be more generic to tile other sensor data

#! /usr/bin/env python

import os

filename="/home/ldm/rapdmg1/data/goesr/2017/0320/S10/20170320_1527_TIRC_S10_PAG.nc"
filename = filename[0: -5]

grid = []
grid.append(["AA", "AB", "AC"])
grid.append(["AD", "AE", "AF"])
grid.append(["AG", "AH", "AI"])

fcmd = "ncrcat "
for y in grid:
         rcmd = "ncrcat "
         rout = "1527_TIRC_S10_P"
         for x in y:
                  path = "{:s}{:s}.nc".format(filename, x)
                  basename = os.path.basename(path)
                  dirname = os.path.dirname(path)
                  tmpdir = "{:s}/tmp".format(dirname)
                  if not os.path.exists(tmpdir):
                           os.mkdir(tmpdir)
                  tmppath = "{:s}/{:s}".format(tmpdir,basename)
                  cmd = "ncecat -O {:s} {:s}".format(path, tmppath)
                  print cmd
                  cmd = "ncpdq -O -a x,record {:s} {:s}".format(tmppath, tmppath)
                  print cmd
                  cmd = "ncwa -O -a record {:s} {:s}".format(tmppath, tmppath)
                  print cmd
                  rcmd += tmppath + " "
                  rout += x+"_"
         merged_file = "{:s}/{:s}.nc".format(tmpdir,rout[0: -1])
         rcmd += merged_file + " "
         print rcmd
         print "\n" 
         cmd = "ncecat -O {:s} {:s}".format(merged_file, merged_file)
         print cmd
         cmd = "ncpdq -O -a y,record {:s} {:s}".format(merged_file, merged_file)
         print cmd
         cmd = "ncwa -O -a record {:s} {:s}".format(merged_file, merged_file)
         print cmd
         print "\n"
         fcmd += merged_file + " "

fcmd += " {:s}/1527_TIRC_S10_FULL.nc".format(tmpdir)
print fcmd


Timing

tilesRun on othoRun on icculus
9  
301 hour 40 min1 hour 18 min
1083 hour 23 min 
16  
4  

The conclusion is this method will take too long for our purposes. I will investigate using NCL instead.

NetCDF File Cleanup

https://sourceforge.net/p/nco/discussion/9830/thread/62e97cb1/

https://sourceforge.net/p/nco/discussion/9830/thread/e6a01e3e/

ncatted -a tile_center_longitude,global,d,, outfile

This command works.The above mentions that the tool should be able to handle regex to remove multiple global attributes, but this fix was added for nco 4.5.1. otho and icculus both currently run nco 4.4.8. 

We want to remove the following attributes:

Also, the chunking info looks incorrect in the final output file

chunk dimensions:
    y   = 1024 // unlimited
    x   = 1024
...
Chunking Info:      [ 1024 <y | unlimited> x 1024 <x> ]

 

Grid Layouts

TIRC

Every 30 minutes on 12 and 42

Sensors 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16

3x3 = 9 tiles

AAABAC
ADAEAF
AGAHAI


Where A-F are 1024x1024 (xy) and G-I are 1024x512

TIRC - Sensor 1, 3, 5

6x5 = 30 tiles

AAABACADAEAF
AGAHAIAJAKAL
AMANAOAPAQAR
ASATAUAVAWAX
AYAZBABBBCBD

Where all tiles are 1024x1024

TIRC - Sensor 2

12x9 = 108 tiles

AAABACADAEAFAGAHAIAJAKALAM
ANAOAPAQARASATAUAVAWAXAYAZ
BABBBCBDBEBFBGBHBIBJBKBLBM
BNBOBPBQBRBSBTBUBVBWBXBYBZ
CACBCCCDCECFCGCHCICJCKCLCM
CNCOCPCQCRCSCTCUCVCWCXCYCZ
DADBDCDDDEDFDGDHDIDJDKDLDM
DNDODPDQDRDSDTDUDVDWDXDYDZ
EAEBECEDEEEFEGEHEIEJEKELEM

Where all tiles are 1024x1024

TIRP

Every 30 minutes on 06 and 36

S01  S02  S03  S04  S05  S06  S07  S08  S09  S10  S11  S12  S13  S14  S15  S16  

4        16     4      1       4      1       1      1      1       1      1       1      1      1       1      1

 

TIRP - Sensor 2

AAABACAD
AEAFAGAH
AIAJAKAL
AMANAOAP

Where all tiles are 1024x1024 except AD, AH, AL, and AP are 768x1024

TIRP - Sensors 1, 3, 5

AAAB
ACAD

Where

AA is 1024x1024

AB is 896x1024

AC is 1024x896

AD is 896x896

TIRU

Every 30 minutes on 06 and 36

4 tiles for each sensor

AAAB
ACAD

where

AA is 1024x1024

AB is 784x1024

AC is 1024x784

AD is 784x784

TISI and TISJ

TISI =  30°N ≤ Lat. < 45°N and 75°W < Long. ≤ 90°W

TISJ = 30°N ≤ Lat. < 45°N and 60°W < Long. ≤ 75°W

every 6 minutes from 0000

S01  S02  S03  S04  S05  S06  S07  S08  S09  S10  S11  S12  S13  S14  S15  S16  

4         9      4      1      4      1       1       1      1       1      1      1       1      1       1      1    

S05 occasionally out (based on 3/19/2017 data feed)


TISI - Sensors 1, 3, 5

AAAB
ACAD

where 

AA is 1024x1024

AB is 39x1024

AC is 1024x392

AD is 39x392


TISI - Sensor 2

AAABAC
ADAEAF
AGAHAI

where

All tiles are 1024x1024 except

AC,AF are 78x1024

AG, AH are 1024x786

AI is 78x786 

Script to generate the number of tiles per sensor statistics
#! /usr/bin/env python
import os
import glob
regions = ("TISI", "TISJ", "TIRC", "TIRP", "TIRU")
top_dir = "/home/ldm/rapdmg1/data/goesr/2017/0319"
for r in regions:
  print r
  line = ""
  for sensor in range(1,17):
    line += "S{:02d}  ".format(sensor)
  print line
  for h in range(0,24):
    for m in range(0,60,6):
      items=[]
      for sensor in range(1,17):
        sdir="{:s}/S{:02d}".format(top_dir,sensor)
        files="{:s}/*{:02d}{:02d}_{:s}*".format(sdir,h,m,r)
        items.append(len(glob.glob(files)))
      all_zero = True
      for i in items:
          if i != 0:
              all_zero = False
      if not all_zero:
        print "{:02d}{:02d}".format(h,m)
        line = ""
        for i in items:
            line +=str(i) + "    "
        print line
  print "\n"
 
Script to generate tile dimension information

These charts were created using the following code that can be modified to read other times and regions

#! /bin/bash

for f in /home/ldm/rapdmg1/data/goesr/2017/0320/S02/20170320_1527_TIRC_S02_P*; do
  echo $f
  ncl_filedump $f | grep -A 3 "^dimensions"
  ncl_filedump $f | grep tile | grep offset
done


 

Other Notes

Traceback (most recent call last):

  File "/home/ldm/bin/chomp.py", line 11, in <module>

    if exc.errno != errno.EEXIST:

NameError: name 'errno' is not defined

 

 

Projections

TISI - lambert_projection

TIRU - fixedgrid_projection

TIRP - mercator_projection

TIRC - lambert_projection

TISJ - lambert_projection

 

Ncview

https://www.myroms.org/forum/viewtopic.php?t=1930

From: http://meteora.ucsd.edu/~pierce/docs/ncview.README

3) Ncview puts copies of the colormap files (files with extension
   ".ncmap") in a system-wide directory, whose location defaults to
   "/usr/local/lib/ncview".  If you want to change this location,
   edit the Imakefile and change the value of "NCVIEW_LIB_DIR"
   accordingly.  If you don't want any system-wide directory at
   all, comment the line out; you can then put the .ncmap files
   in one of 3 places: 1) In a directory named by the environmental 
   variable "NCVIEWBASE"; 2) In your home directory, if you don't
   define NCVIEWBASE; 3) In the directory which you are running
   ncview from.

 

 

GeoServer 

 

./configure --with-zlib=/data/goes16/GeoServer/NetCDF-libs --prefix=/data/goes16/GeoServer/NetCDF-libs --enable-threadsafe --with-pthread=/usr
configure: error: Can't find or link to the z library. Turn off netCDF-4 and      opendap with --disable-netcdf-4 --disable-dap, or see config.log for errors.
CPPFLAGS=-I/data/goes16/GeoServer/NetCDF-libs/include LDFLAGS=-L/data/goes16/GeoServer/NetCDF-libs/lib ./configure --prefix=/data/goes16/GeoServer/NetCDF-libs
bash-4.1$ ./startup.sh 
GEOSERVER DATA DIR is /data/goes16/GeoServer/Data
WARNING: Module not found [ssl]
2017-03-31 18:37:11.733:INFO::main: Logging initialized @179ms
2017-03-31 18:37:11.876:INFO:oejs.Server:main: jetty-9.2.13.v20150730
2017-03-31 18:37:11.890:INFO:oejdp.ScanningAppProvider:main: Deployment monitor [file:/data/goes16/GeoServer/geoserver-2.11.0/webapps/] at interval 1
2017-03-31 18:37:12.700:INFO:oejw.StandardDescriptorProcessor:main: NO JSP Support for /geoserver, did not find org.eclipse.jetty.jsp.JettyJspServlet
2017-03-31 18:37:12.705:WARN:oejw.WebAppContext:main: Failed startup of context o.e.j.w.WebAppContext@567a4593{/geoserver,file:/data/goes16/GeoServer/geoserver-2.11.0/webapps/geoserver/,STARTING}{/geoserver}
java.lang.reflect.InvocationTargetException

HERE MARKS WHERE AARON STARTED WORK