elib
DLR-Header
DLR-Logo -> http://www.dlr.de
DLR Portal Home | Imprint | Privacy Policy | Contact | Deutsch
Fontsize: [-] Text [+]

Interrater Reliability at the Top End: Measures of Pilots’ Nontechnical Performance

Gontar, Patrick and Hörmann, Hans-Jürgen (2015) Interrater Reliability at the Top End: Measures of Pilots’ Nontechnical Performance. International Journal of Aviation Psychology, 25 (3-4), pp. 171-190. Taylor & Francis. DOI: 10.1080/10508414.2015.1162636 ISSN 1050-8414

[img] PDF
654kB

Official URL: http://dx.doi.org/10.1080/10508414.2015.1162636

Abstract

Objective: The aim of this study was to analyze influences on interrater reliability and withingroup agreement within a highly experienced rater group when assessing pilots’ nontechnical skills. Background: Nontechnical skills of pilots are crucial for the conduct of safe flight operations. To train and assess these skills, reliable expert ratings are required. Literature shows to some degree that interrater reliability is influenced by factors related to the targets, scenarios, rating tools, or the raters themselves. Method: Thirty-seven type-rating examiners from a European airline assessed the performance of 4 flight crews based on video recordings using LOSA and adapted NOTECHS tools. We calculated rwg and ICC(3) to measure within-group agreement and interrater reliability. Results: The findings indicated that within-group agreement and interrater reliability were not always acceptable. It was shown that the performance of outstanding pilots was rated with the highest within-group agreement. For cognitive aspects of performance, interrater reliability was higher than for social aspects of performance. Agreement was lower on the pass–fail level than for the distinguished performance scales. Conclusion: These results suggest pass–fail decisions should not be based exclusively on nontechnical skill ratings. We furthermore recommend that regulatory authorities more systematically address interrater reliability in airline instructor training. Airlines as well as training facilities should be encouraged to demonstrate sufficient interrater reliability when using their rating tools.

Item URL in elib:https://elib.dlr.de/104261/
Document Type:Article
Title:Interrater Reliability at the Top End: Measures of Pilots’ Nontechnical Performance
Authors:
AuthorsInstitution or Email of AuthorsAuthors ORCID iD
Gontar, PatrickTechnische Univesität München, Institute of Ergonomicshttps://orcid.org/0000-0003-0345-2566
Hörmann, Hans-Jürgenhans.hoermann (at) dlr.dehttps://orcid.org/0000-0002-6358-7027
Date:2015
Journal or Publication Title:International Journal of Aviation Psychology
Refereed publication:Yes
Open Access:Yes
Gold Open Access:No
In SCOPUS:Yes
In ISI Web of Science:No
Volume:25
DOI :10.1080/10508414.2015.1162636
Page Range:pp. 171-190
Publisher:Taylor & Francis
ISSN:1050-8414
Status:Published
Keywords:Interrater reliability, non-technical skills, CRM, NOTECHS, pilots, training
HGF - Research field:Aeronautics, Space and Transport
HGF - Program:Aeronautics
HGF - Program Themes:air traffic management and operations
DLR - Research area:Aeronautics
DLR - Program:L AO - Air Traffic Management and Operation
DLR - Research theme (Project):L - Human factors and safety in Aeronautics
Location: Hamburg
Institutes and Institutions:Institute of Aerospace Medicine > Aviation and Space Psychology
Deposited By: Hörmann, Dr.phil. Hans-Jürgen
Deposited On:23 May 2016 14:43
Last Modified:06 Sep 2019 15:22

Repository Staff Only: item control page

Browse
Search
Help & Contact
Information
electronic library is running on EPrints 3.3.12
Copyright © 2008-2017 German Aerospace Center (DLR). All rights reserved.