Performance Metrics from Proposal:

• The product improvement will be evaluated by using specific probabilistic forecast measures such as, rank histograms, CRPSS and reliability for selected variables and thresholds. Current version of the product will serve as a baseline.
• MODE will be used to assess the probabilistic guidance for both common events to extreme events by forming objects at several different user-relevant thresholds.  Decreased centroid distance, area ratios closer to 1 and an examination of the distributions of intensities within the MODE objects will be used to measure success.  Additionally, a value closer to 1 for a performance measure called the Median of Maximum Interest (MMI) will also be used. The first year will be used as the benchmark.  
• The measure of success of the social science research is providing research-based guidance that aids the development by NOAA/ESRL and NCAR/RAL of effective ensemble-based probabilistic precipitation forecasts, where "effective" means that forecasters can and do use the forecasts to provide desired forecast information to their users. Success will be illustrated using analysis of data from the interviews and observations with forecasters.

SOW provided to GSD during proposal phase

Year 1

• —Month 1-2:
  • —Identify metrics and variables needed.
• —Months 3-9:
  • —Develop verification system on NOAA supercomputer (e.g. Zeus or Theia) for 1-2 ensembles and 1-2 deterministic baselines.
  • —Develop initial operating capability for Ensemble-MODE evaluation of rain and snow bands.
• —Months 9-12:
  • —Attend HMT-WPC.
  • —Explore use of MODE-TD on variables relevant to rain and snowband prediction.
  • —Demonstrate initial operating capability.
• —Months 12:
• • —Identify enhancements needed to verification system per user feedback.