Child pages
  • 2017 Data Processing Steps
Skip to end of metadata
Go to start of metadata

PIREP

  • /d1/TAIWIN_VX/scripts/pull_PIREP.ksh (calls pull_pireps_for_icing.pl)
  • Output written to: /d1/TAIWIN_VX/data/PIREP/txt/YYYYMMDD/YYYYMMDDHH_pireps.txt
    • Pulls one hour worth of reports centered at the top of the hour

METAR

Original METAR processing steps: Used metar_sql2csv_th.pl script

John's METAR processing steps

  • /d1/TAIWIN_VX/scripts/pull_METAR.ksh (calls /d1/TAIWIN_VX/scripts/metar_sql2met.pl)
    • metar_sql2met.pl was edited from the original to output desired METAR obs in a MET-friendly format
    • pull_METAR.ksh runs metar_sql2met.pl which outputs a MET-friendly txt file which is then converted to nc using ascii2nc
    • 2017 enhancements: additions to the usage of metar_sql2met.pl
      • -o Write to specified output file name - if not specified, uses the default
      • -s Now can read in a file list rather than specifying station ID's individually
      • -n Skips empty weather strings
      • -y Only uses empty weather strings

mPING

  • Tuvok crontab entry:
    # Pull mPING data for 3 days prior 
    00 06 * * * /d1/TAIWIN_VX/scripts/pull_mping.py `date -d '-3 day' +\%Y\%m\%d` > /d1/TAIWIN_VX/data/mPING/pull_mping_cron.log 2>&1
  • Ran ascii2nc on the files:
    /usr/local/met-5.1/bin/ascii2nc $i proc_nc/mping_$f.nc

MRMS

  • Data located: /d1/TAIWIN_VX/data/MRMS/ (we can use the data under native and regrid on the fly to our fcst domain with MET)
    • gauge corrected (missing 20170321)
    • precipitation type (missing 20170321)

HRRR-NCEP-wrfnat (HRRRop) model data

  • Input data sitting here: /d1/TAIWIN_VX/data/HRRR-NCEP-wrfnat/native/grb2/YYYYMMDDHH
  • /d1/TAIWIN_VX/scripts/run_extractFields_2nc_update.ksh (calls extractField_2nc_update.ncl)
    • This code was updated to Greg's latest on 20170206 (hence the "_update") 
  • Output written to: /d1/TAIWIN_VX/data/HRRR-NCEP-wrfnat/native/nc/YYYYMMDDHH
  • Gen_vx_mask steps
    • /d1/TAIWIN_VX/scripts/run_gen_vx_mask.ksh - currently reads in native grib2 files
      • Run on RWMR to separate the rain and freezing rain mixing ratios
      • Thresholded on temperature <=273.15 for freezing rain and temperatures >273.15 for rain
      • Produces 2 nc files, one each for RWMR and FRMR
        • The files have the regions that are masked out set to -9999 (e.g. for RWMR, the regions with temps below freezing were set to -9999)

HRRR-wrfnat (HRRRx) model data

  • Input data sitting here: /d1/TAIWIN_VX/data/HRRR-wrfnat/native/grb2/YYYYMMDDHH
  • /d1/TAIWIN_VX/scripts/run_extractFields_2nc.ksh (calls extractField_2nc.ncl)
    • This code was updated to Greg's latest on 20170206
  • Output written to: /d1/TAIWIN_VX/data/HRRR-wrfnat/native/nc/YYYYMMDDHH
  • Gen_vx_mask steps
    • /d1/TAIWIN_VX/scripts/run_gen_vx_mask.ksh - currently reads in native grib2 files
      • Run on RWMR to separate the rain and freezing rain mixing ratios
      • Thresholded on temperature <=273.15 for freezing rain and temperatures >273.15 for rain
      • Produces 2 nc files, one each for RWMR and FRMR
        • The files have the regions that are masked out set to -9999 (e.g. for RWMR, the regions with temps below freezing were set to -9999)

Aloft (PIREP) verification processing

  • /d1/TAIWIN_VX/scripts/run_wrf_extract_HRRR_update_pirep.ksh (calls wrf_extract_update)
    • Need to change lines 47-55 for HRRRop or HRRRx
  • Input includes the two files:
    • /d1/TAIWIN_VX/data/PIREP/txt/YYYYMMDD/YYYYMMDDHH_pireps.txt 
    • /d1/TAIWIN_VX/data/HRRR-NCEP-wrfnat/native/nc/YYYYMMDDHH/*wrfnat.nc
  • Output written to: /d1/TAIWIN_VX/data/HRRR-NCEP-wrfnat/comp2pirep/YYYYMMDDHH/HH 
    • Includes to following information:
      • PIREP information: PIREP, pirepId, input_lat, input_lon, xi, yj, pirep_icgSev, pirep_icgTyp, pirep_icgBas, pirep_icgTop, input_time, wrfTimes(index_time)
      • Point information: n, ii(n), jj(n), xlat(n), xlon(n), topo(n)   Note: where n ranges from 1-36 (6x6 box)
      • Model output at each point for the entire column (50 hybrid levels): pres(k,n)*0.01, hght(k,n), temp(k,n)-273.16, uwnd(k,n), vwnd(k,n), wwnd(k,n), qv(k,n)*1000., qc(k,n)*dens(k,n)*1000., qr(k,n)*dens(k,n)*1000., qi(k,n)*dens(k,n)*1000., qs(k,n)*dens(k,n)*1000., qg(k,n)*dens(k,n)*1000., ni(k,n)*dens(k,n)*0.001, nr(k,n)*dens(k,n)*0.001, nc(k,n)*dens(k,n)*1.E-6, dens(k,n), NINT(mvdc(k,n)*1.E6), ', ', mvdr(k,n)*1.E3, NINT(mvdj(k,n)*1.E6), acc_c(k,n), acc_r(k,n), vertFlag(k)
  • Greg looks in concentric rings (the first point is the NW corner and it moves clockwise from there)
    • 1 ring around (2x2; 4 WRF columns)
    • 2 rings around (4x4; 16 WRF columns)
    • 3 rings around (6x6; 36 WRF columns)
    • 4 rings around (8x8; 64 WRF columns)
    • 5 rings around (10x10; 100 WRF columns)
    • 6 rings around (12x12; 144 WRF columns)
    • Explicit negative may only be closest ring
      • Greg specifically concerned with upslope conditions - could penalize the model when there is a distinct clear edge to the cloud for a reason
  • /d1/TAIWIN_VX/scripts/run_wrfcol2mpr.ksh (calls wrfcol2mpr.R Rscript)
    • Input read from: /d1/TAIWIN_VX/data/HRRR*wrfnat/comp2pirep/YYYYMMDDHH
    • Output written to: /d1/TAIWIN_VX/data/HRRR_PIREP_mpr/HRRRop and /d1/TAIWIN_VX/data/HRRR_PIREP_mpr/HRRRex
    • Loop over the input date directories and, for each, call the wrfcol2mpr.R Rscript.
    • The wrfcol2mpr.R script reads and processes all of the *_wrfColumsn.csv ascii files for the current date.  There is one input csv file for each PIREP.  It writes a single MPR output file containing data for all PIREPs for that day.
    • Each input csv file, contains information for the 64 points (8x8) surrounding the PIREP location.
    • For each PIREP, the script writes 16 output MPR lines, 4 each for the 8x8, 6x6, 4x4, and 2x2 rings.
      • 2 MPR lines for ice accretion (ACC) and 2 MPR lines for mixing ratio (MIXR).
      • For each variable, both the maximum value (MAX) in the ring is written and the ratio of points greater than 0 (NBRHD).

Surface (METAR/mPING/MRMS) verification processing

  • Grid-to-grid verification
    • gen_pcp_combine_opHRRR.ksh

      • generates pcp_combine commands to "add" opHRRR data into 1-hr accumulations

      • 'name="APCP"; level="R735";' -name "APCP_01" → This needs to be checked for 2017 processing

      • need to pipe out to run script, e.g.: run_pcp_combine_opHRRR.ksh

    • run_grid_stat_01h_gcorr.ksh

      • runs grid-stat on 1-h forecast (model data run through pcp-combine) and obs (MRMS gcorr)

      • MET config: GridStatConfig_01h_GCorr

    • run_grid_stat_01h_PType_RWMR.ksh

      • runs grid-stat on 1-h forecast (model data run through gen_vx_mask to threshold T/RWMR) and obs (MRMS PrecipFlag)

      • MET config: Currently pointing to GridStatConfig_01h_PType_RWMR_MRthresh (Is this what we want??)

    • run_grid_stat_01h_PType_SNMR.ksh

      • runs grid-stat on 1-h forecast (model data) and obs (MRMS PrecipFlag)

      • MET config: Currently pointing to GridStatConfig_01h_PType_SWMR_MRthresh (Is this what we want??)

  • Grid-to-point verification
    • Pre-processing steps

      • utilities/run_gen_vx_mask.ksh

        • runs gen_vx_mask to separate rain and freezing rain mixing ratios using RWMR and temp variables in model output
        • for freezing rain, RWMR is thresholded on T<=273.15K
        • for rain, RWMR is thresholded on T>273.15K
        • values fitting within threshold criteria will be retained and "bad" values are set to -9999
      • utilities/run_ascii2nc
        • runs ascii2nc on MPING data to get in MET-friendly format
      • pull_METAR.ksh (see more detailed noted above)

        • runs metar_sql2met.pl which outputs a MET-friendly txt file which is then converted to nc using ascii2nc

    • run_point_stat_mping_metar.ksh
      • runs point-stat on forecast and obs
      • forecast: model data run through gen_vx_mask to threshold T/RWMR and non-threshold grib data
      • obs: METAR and MPING data (both run through MET pre-processing steps)
      • MET configs:

PointStatConfig_PTypeCat

PointStatConfig_PTypeSNMR

PointStatConfig_PTypeRWMR_Tgt0

PointStatConfig_PTypeRWMR_Tle0

PointStatConfig_PTypeGRLE

    • run_stat_analysis_byinit.ksh
      • runs stat-analysis on output from point-stat for each init and fhr
      • outputs CTC line type
    • run_stat_analysis_byinit_byvxmask.ksh
      • runs stat-analysis on output from point-stat for each init and fhr on the following masks: APL GMC LMV MDW NEC SEC SPL
      • outputs CTC line type
    • run_stat_analysis_byinit_bystid.ksh (hold off on processing for now)
      • runs stat-analysis on output from point-stat for each init and fhr on the following SID: KATL KBOS KBWI KCLT KDEN KDFW KDTW KLGA KMEM KMSP KORD KPHL KSEA KSLC
      • outputs CTC line type
  • No labels