Tell a friend
Print
Gontar, Patrick and Hörmann, Hans-Jürgen (2015) Interrater Reliability at the Top End: Measures of Pilots’ Nontechnical Performance. International Journal of Aviation Psychology, 25 (3-4), pp. 171-190. Taylor & Francis. doi: 10.1080/10508414.2015.1162636. ISSN 1050-8414.
![[img]](https://elib.dlr.de/style/images/fileicons/application_pdf.png) |
PDF
654kB |
Official URL: http://dx.doi.org/10.1080/10508414.2015.1162636
Abstract
Objective: The aim of this study was to analyze influences on interrater reliability and withingroup agreement within a highly experienced rater group when assessing pilots’ nontechnical skills. Background: Nontechnical skills of pilots are crucial for the conduct of safe flight operations. To train and assess these skills, reliable expert ratings are required. Literature shows to some degree that interrater reliability is influenced by factors related to the targets, scenarios, rating tools, or the raters themselves. Method: Thirty-seven type-rating examiners from a European airline assessed the performance of 4 flight crews based on video recordings using LOSA and adapted NOTECHS tools. We calculated rwg and ICC(3) to measure within-group agreement and interrater reliability. Results: The findings indicated that within-group agreement and interrater reliability were not always acceptable. It was shown that the performance of outstanding pilots was rated with the highest within-group agreement. For cognitive aspects of performance, interrater reliability was higher than for social aspects of performance. Agreement was lower on the pass–fail level than for the distinguished performance scales. Conclusion: These results suggest pass–fail decisions should not be based exclusively on nontechnical skill ratings. We furthermore recommend that regulatory authorities more systematically address interrater reliability in airline instructor training. Airlines as well as training facilities should be encouraged to demonstrate sufficient interrater reliability when using their rating tools.
| Item URL in elib: | https://elib.dlr.de/104261/ | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Document Type: | Article | |||||||||
| Title: | Interrater Reliability at the Top End: Measures of Pilots’ Nontechnical Performance | |||||||||
| Authors: |
| |||||||||
| Date: | 2015 | |||||||||
| Journal or Publication Title: | International Journal of Aviation Psychology | |||||||||
| Refereed publication: | Yes | |||||||||
| Open Access: | Yes | |||||||||
| Gold Open Access: | No | |||||||||
| In SCOPUS: | Yes | |||||||||
| In ISI Web of Science: | No | |||||||||
| Volume: | 25 | |||||||||
| DOI : | 10.1080/10508414.2015.1162636 | |||||||||
| Page Range: | pp. 171-190 | |||||||||
| Publisher: | Taylor & Francis | |||||||||
| ISSN: | 1050-8414 | |||||||||
| Status: | Published | |||||||||
| Keywords: | Interrater reliability, non-technical skills, CRM, NOTECHS, pilots, training | |||||||||
| HGF - Research field: | Aeronautics, Space and Transport | |||||||||
| HGF - Program: | Aeronautics | |||||||||
| HGF - Program Themes: | air traffic management and operations | |||||||||
| DLR - Research area: | Aeronautics | |||||||||
| DLR - Program: | L AO - Air Traffic Management and Operation | |||||||||
| DLR - Research theme (Project): | L - Human factors and safety in Aeronautics (old) | |||||||||
| Location: | Hamburg | |||||||||
| Institutes and Institutions: | Institute of Aerospace Medicine > Aviation and Space Psychology | |||||||||
| Deposited By: | Hörmann, Dr.phil. Hans-Jürgen | |||||||||
| Deposited On: | 23 May 2016 14:43 | |||||||||
| Last Modified: | 06 Sep 2019 15:22 |
Repository Staff Only: item control page
