DLR-Logo -> http://www.dlr.de
DLR Portal Home | Imprint | Privacy Policy | Contact | Deutsch
Fontsize: [-] Text [+]

Interrater Reliability at the Top End: Measures of Pilots’ Nontechnical Performance

Gontar, Patrick and Hörmann, Hans-Jürgen (2015) Interrater Reliability at the Top End: Measures of Pilots’ Nontechnical Performance. International Journal of Aviation Psychology, 25 (3-4), pp. 171-190. Taylor & Francis. doi: 10.1080/10508414.2015.1162636. ISSN 1050-8414.

[img] PDF

Official URL: http://dx.doi.org/10.1080/10508414.2015.1162636


Objective: The aim of this study was to analyze influences on interrater reliability and withingroup agreement within a highly experienced rater group when assessing pilots’ nontechnical skills. Background: Nontechnical skills of pilots are crucial for the conduct of safe flight operations. To train and assess these skills, reliable expert ratings are required. Literature shows to some degree that interrater reliability is influenced by factors related to the targets, scenarios, rating tools, or the raters themselves. Method: Thirty-seven type-rating examiners from a European airline assessed the performance of 4 flight crews based on video recordings using LOSA and adapted NOTECHS tools. We calculated rwg and ICC(3) to measure within-group agreement and interrater reliability. Results: The findings indicated that within-group agreement and interrater reliability were not always acceptable. It was shown that the performance of outstanding pilots was rated with the highest within-group agreement. For cognitive aspects of performance, interrater reliability was higher than for social aspects of performance. Agreement was lower on the pass–fail level than for the distinguished performance scales. Conclusion: These results suggest pass–fail decisions should not be based exclusively on nontechnical skill ratings. We furthermore recommend that regulatory authorities more systematically address interrater reliability in airline instructor training. Airlines as well as training facilities should be encouraged to demonstrate sufficient interrater reliability when using their rating tools.

Item URL in elib:https://elib.dlr.de/104261/
Document Type:Article
Title:Interrater Reliability at the Top End: Measures of Pilots’ Nontechnical Performance
AuthorsInstitution or Email of AuthorsAuthor's ORCID iD
Gontar, PatrickTechnische Univesität München, Institute of Ergonomicshttps://orcid.org/0000-0003-0345-2566
Hörmann, Hans-JürgenUNSPECIFIEDhttps://orcid.org/0000-0002-6358-7027
Journal or Publication Title:International Journal of Aviation Psychology
Refereed publication:Yes
Open Access:Yes
Gold Open Access:No
In ISI Web of Science:No
Page Range:pp. 171-190
Publisher:Taylor & Francis
Keywords:Interrater reliability, non-technical skills, CRM, NOTECHS, pilots, training
HGF - Research field:Aeronautics, Space and Transport
HGF - Program:Aeronautics
HGF - Program Themes:air traffic management and operations
DLR - Research area:Aeronautics
DLR - Program:L AO - Air Traffic Management and Operation
DLR - Research theme (Project):L - Human factors and safety in Aeronautics (old)
Location: Hamburg
Institutes and Institutions:Institute of Aerospace Medicine > Aviation and Space Psychology
Deposited By: Hörmann, Dr.phil. Hans-Jürgen
Deposited On:23 May 2016 14:43
Last Modified:06 Sep 2019 15:22

Repository Staff Only: item control page

Help & Contact
electronic library is running on EPrints 3.3.12
Website and database design: Copyright © German Aerospace Center (DLR). All rights reserved.