DLR-Logo -> http://www.dlr.de
DLR Portal Home | Imprint | Privacy Policy | Contact | Deutsch
Fontsize: [-] Text [+]

Comment on "Quantitative performance metrics for stratospheric-resolving chemistry-climate models" by Waugh and Eyring (2008)

Grewe, Volker and Sausen, Robert (2009) Comment on "Quantitative performance metrics for stratospheric-resolving chemistry-climate models" by Waugh and Eyring (2008). Atmospheric Chemistry and Physics, 9, pp. 9101-9110. Copernicus Publications. ISSN 1680-7316

[img] PDF - Registered users only

Official URL: http://www.atmos-chem-phys.net/9/9101/2009/


This comment focuses on the statistical limitations of a model grading, as applied by D. Waugh and V. Eyring (2008) (WE08). The grade g is calculated for a specific diagnostic, which basically relates the difference of means of model and observational data to the standard deviation in the observational dataset. We performed Monte Carlo simulations, which show that this method has the potential to lead to large 95%-confidence intervals for the grade. Moreover, the difference between two model grades often has to be very large to become statistically significant. Since the confidence intervals were not considered in detail for all diagnostics, the grading in WE08 cannot be interpreted, without further analysis. The results of the statistical tests performed in WE08 agree with our findings. However, most of those tests are based on special cases, which implicitely assume that observations are available without any errors and that the interannual variability of the observational data and the model data are equal. Without these assumptions, the 95%- confidence intervals become even larger. Hence, the case, where we assumed perfect observations (ignored errors), provides a good estimate for an upper boundary of the threshold, below that a grade becomes statistically significant. Examples have shown that the 95%-confidence interval may even span the whole grading interval [0, 1]. Without considering confidence intervals, the grades presented in WE08 do not allow to decide whether a model result significantly deviates from reality. Neither in WE08 nor in our comment it is pointed out, which of the grades presented in WE08 inhibits such kind of significant deviation. However, our analysis of the grading method demonstrates the unacceptably high potential for these grades to be insignificant. This implies that the grades given by WE08 can not be interpreted by the reader. We further show that the inclusion of confidence intervals into the grading approach is necessary, since otherwise even a perfect model may get a low grade.

Item URL in elib:https://elib.dlr.de/61283/
Document Type:Article
Title:Comment on "Quantitative performance metrics for stratospheric-resolving chemistry-climate models" by Waugh and Eyring (2008)
AuthorsInstitution or Email of AuthorsAuthors ORCID iD
Date:1 December 2009
Journal or Publication Title:Atmospheric Chemistry and Physics
Refereed publication:Yes
Open Access:Yes
Gold Open Access:Yes
In ISI Web of Science:Yes
Page Range:pp. 9101-9110
Publisher:Copernicus Publications
Keywords:Climate-Chemistry Model Evaluation Grade
HGF - Research field:Aeronautics, Space and Transport (old)
HGF - Program:Space (old)
HGF - Program Themes:W EO - Erdbeobachtung
DLR - Research area:Space
DLR - Program:W EO - Erdbeobachtung
DLR - Research theme (Project):W - Vorhaben Atmosphären- und Klimaforschung (old)
Location: Oberpfaffenhofen
Institutes and Institutions:Institute of Atmospheric Physics > Atmospheric Dynamics
Deposited By: Grewe, Prof. Dr. Volker
Deposited On:02 Dec 2009 17:52
Last Modified:02 May 2019 14:03

Repository Staff Only: item control page

Help & Contact
electronic library is running on EPrints 3.3.12
Copyright © 2008-2017 German Aerospace Center (DLR). All rights reserved.