DLR-Logo -> http://www.dlr.de
DLR Portal Home | Imprint | Contact | Deutsch
Fontsize: [-] Text [+]

Comment on "Quantitative performance metrics for stratospheric-resolving chemistry-climate models" by Waugh and Eyring (2008)

Grewe, Volker and Sausen, Robert (2009) Comment on "Quantitative performance metrics for stratospheric-resolving chemistry-climate models" by Waugh and Eyring (2008). Atmospheric Chemistry and Physics, 9, pp. 9101-9110. Copernicus. ISSN 1680-7316.

[img] PDF - Registered users only - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader

Official URL: http://www.atmos-chem-phys.net/9/9101/2009/


This comment focuses on the statistical limitations of a model grading, as applied by D. Waugh and V. Eyring (2008) (WE08). The grade g is calculated for a specific diagnostic, which basically relates the difference of means of model and observational data to the standard deviation in the observational dataset. We performed Monte Carlo simulations, which show that this method has the potential to lead to large 95%-confidence intervals for the grade. Moreover, the difference between two model grades often has to be very large to become statistically significant. Since the confidence intervals were not considered in detail for all diagnostics, the grading in WE08 cannot be interpreted, without further analysis. The results of the statistical tests performed in WE08 agree with our findings. However, most of those tests are based on special cases, which implicitely assume that observations are available without any errors and that the interannual variability of the observational data and the model data are equal. Without these assumptions, the 95%- confidence intervals become even larger. Hence, the case, where we assumed perfect observations (ignored errors), provides a good estimate for an upper boundary of the threshold, below that a grade becomes statistically significant. Examples have shown that the 95%-confidence interval may even span the whole grading interval [0, 1]. Without considering confidence intervals, the grades presented in WE08 do not allow to decide whether a model result significantly deviates from reality. Neither in WE08 nor in our comment it is pointed out, which of the grades presented in WE08 inhibits such kind of significant deviation. However, our analysis of the grading method demonstrates the unacceptably high potential for these grades to be insignificant. This implies that the grades given by WE08 can not be interpreted by the reader. We further show that the inclusion of confidence intervals into the grading approach is necessary, since otherwise even a perfect model may get a low grade.

Document Type:Article
Title:Comment on "Quantitative performance metrics for stratospheric-resolving chemistry-climate models" by Waugh and Eyring (2008)
AuthorsInstitution or Email of Authors
Grewe, VolkerDLR, IPA
Sausen, RobertDLR, IPA
Date:1 December 2009
Journal or Publication Title:Atmospheric Chemistry and Physics
Refereed publication:Yes
In Open Access:Yes
In ISI Web of Science:Yes
Page Range:pp. 9101-9110
Keywords:Climate-Chemistry Model Evaluation Grade
HGF - Research field:Aeronautics, Space and Transport (old)
HGF - Program:Space (old)
HGF - Program Themes:W EO - Erdbeobachtung
DLR - Research area:Space
DLR - Program:W EO - Erdbeobachtung
DLR - Research theme (Project):W - Vorhaben Atmosphären- und Klimaforschung (old)
Location: Oberpfaffenhofen
Institutes and Institutions:Institute of Atmospheric Physics > Atmospheric Dynamics
Deposited By: Dr.rer.nat. Volker Grewe
Deposited On:02 Dec 2009 17:52
Last Modified:23 Jul 2015 18:50

Repository Staff Only: item control page

Help & Contact
electronic library is running on EPrints 3.3.12
Copyright © 2008-2012 German Aerospace Center (DLR). All rights reserved.