The Relative Merit of Multiple Climate Models
Reifen, C. and Toumi, R. 2009. Climate projections: Past performance no guarantee of future skill? Geophysical Research Letters 36: 10.1029/2009GL038082.
Reifen and Toumi examined this question "in an observational context" for what they describe as "the first time." Working with the 17 climate models employed by the IPCC (2007) in its Fourth Assessment Report, they determined how accurately individual models, as well as various subsets of the 17 models, simulated the temperature history of Europe, Siberia and the entire globe over a selection period (such as 1900-1919) and a subsequent test period (such as 1920-1939), asking the question: are the results of the test period as good as those of the selection period? ... and following this procedure while working their way through the entire 20th century at one-year time-steps for not only 20-year selection and test intervals, but for 10- and 30-year intervals as well. So what did the study reveal?
The two researchers could find "no evidence of future prediction skill delivered by past performance-based model selection," noting that "there seems to be little persistence in relative model skill." As for why this was so, they speculated that "the cause of this behavior is the non-stationarity of climate feedback strengths," which they explain by stating that "models that respond accurately in one period are likely to have the correct feedback strength at that time," but that "the feedback strength and forcing is not stationary, favoring no particular model or groups of models consistently."
Given such findings, the UK physicists conclude their analysis of the subject by stating that "the common investment advice that 'past performance is no guarantee of future returns' and to 'own a portfolio' appears also to be relevant to climate projections." Or as we might put it in more simple terms, "there's strength in numbers." Even then, however, there is still no guarantee of success, as all can be wrong together.