Issue 3: Reproducibility of refractivity results
The comparison of results from four different centers was based on large vertical layers, large latitudinal zones, and some time-smoothing of the data. Yet there remain small but non-zero differences in the four refractivity time series, and it seems important to note that these are not retrieved meteorological data, but raw refractivity. What are the sources of these differences and can they be eliminated? If not, why not?
Response from Kevin Trenberth:
See also answer above. Issues of how retrievals are made especially the homogeneity of the ionosphere seem to be important. Given that the technique is a limb sounding to produce a vertical profile, the rays do not just sample the vertical column; a full 4D var system might account for the structure. This issue can be better quantified and maybe reduced, but will remain and leave some residual effects, as I understand it. However, many of these effects should be random and thus average out for climate purposes.
5 Comments
Tom Yunck
The differences would disappear if all groups processed the data in exactly the same way. The value of these comparisons is that they give one measure, however imperfect, of the mechanization error in the retrieval methods. I would tend to use the quality of the agreement of the top 3 or 4 groups as a proxy for this error. If and when they converge to a "benchmark" level, presumably the mature techniques can then be reapplied to historical data for benchmarking. What matters most is the stability of these agreements over the long term.
Tony Mannucci
The comparison of results from different centers has just begun. We call it the "RO Trends" study. At this time, the reasons for differences are not fully understood. Ben Ho has shown that differences are occurring in part because each center uses a different set of profiles in computing the monthly-mean averages. Each center determines which profiles to use in the averaging. Quality control criteria differ among the centers.
The differences among the centers can be eliminated once they are understood. This is an important step, but not a sufficient step to establish a climate benchmark. Even if all the centers agree, all centers could be making similar errors or similar erroneous assumptions. The benchmark must be established via careful analysis of error sources. See issue 5.
I note here that quality control is an important issue that must be addressed when formulating a benchmark. It is possible that none of the centers now approach quality control in an optimal way for climate benchmarking.
Ben Ho
Response from Ben Ho
My response and several more figures are linked below.
As I presented in AMS breakfast meeting, the non-zero fractional
refractivity difference among different centers is most likely due to
sampling issue. This is re-confirmed by pixel to pixel comparison between
GFZ and UCAR refractivity (see my response to issue 4).
Figs. of Detail Response
Detail Response
Seth Gutman
Consolidated comments from Gutman, Yoe and Reale
Seth Gutman's Response
Small differences between the centers can easily be explained by differences in the algorithms, protocols and hardware used to estimate refractivity or bending angle and retrieve geophysical parameters from the GPS observations. The question is, are the differences between the centers significant from a climatological perspective? If so, then further work needs to be done to understand the differences. If not, then the issue is moot.
Jim Yoe's Response
Agreed.
Tony Reale's Response
If relevant, the impact of such differences on "retrievals" can be respectively quantified using the collocations of COSMIC, radiosondes, dropsondes, polar satellites, etc through NPIVS. If relevant, and then found to be insignificant then ditto their projected impact on climate (regardless of whether refractivity or product are assimilated). If relevant, NPIVS could address this. My understanding from Sethis that it is "not" relevant.
Jens Wickert
Response from Jens Wickert
I think the agreement of 0.001% between UCAR and GFZ in refractivity is already absolutely great, we should note this. This number shows, that it is possible to reach very consistent results, even after the quite large number of processing steps, starting from the raw data level. We should continue this kind of investigations to reach consistent results. I'm sure we can reach this.
I would expect larger deviations between the centers in the lower troposphere (implementation of wave optics) and above 25 km (ionospheric correction), as we saw in previous studies. But also here I think it is possible to reach consistency. But it requires a bit more (joint) work.
Another more general point is, that we will see (at least I would expect this) smaller differences e.g. in the retrieval from different centers, for future mission, when we can use more advanced receivers (e.g. smaller noise at higher altitudes, already observed by Metop). Also the new navigation signals (GPS-M, GALILEO) will be the base for the application of more advanced retrieval. E.g. the application of a third carrier frequency will very probably increase the accuracy of the ionosphere calibration at higher altitudes. Also the availability of L2C will enable the application of the FSI on two frequencies separately (should be advantageous for the LT profiling). Also additional information (e.g. on the ionosphere itself) can help to improve the retrieval. There are still some interesting scientific issue in the future, GPS RO is still not at the end of its potential.