DLR-Logo -> http://www.dlr.de
DLR Portal Home | Imprint | Privacy Policy | Contact | Deutsch
Fontsize: [-] Text [+]

Measuring Speech Recognition And Understanding Performance in Air Traffic Control Domain Beyond Word Error Rates

Helmke, Hartmut and Shetty, Shruthi and Kleinert, Matthias and Ohneiser, Oliver and Prasad, Amrutha and Motlicek, Petr and Cerna, Aneta and Windisch, Christian (2021) Measuring Speech Recognition And Understanding Performance in Air Traffic Control Domain Beyond Word Error Rates. 11th SESAR Innovation Days, 2021-12-07 - 2021-12-09, Virtual.

Full text not available from this repository.


Applying Automatic Speech Recognition (ASR) in the domain of analogue voice communication between air traffic controllers (ATCo) and pilots has more end user requirements than just transforming spoken words into text. It is useless for, e.g., read-back error detection support, if word recognition is perfect, as long as the semantic interpretation is wrong. For an ATCo it is of almost no importance if the words of a greeting are correctly recognized. A wrong recognition of a greeting should, however, not disturb the correct recognition of, e.g., a “descend” com-mand. More important is the correct semantic interpretation. What, however, is the correct semantic interpretation especially when ATCos or pilot, deviate more of less from published standard phraseology? For comparing performance of different speech recognition applications, 14 European partners from Air Traffic Management (ATM) domain have recently agreed on a common set of rules, i.e., an ontology on how to annotate the speech utterances of an ATCo on semantic level. This paper first presents the new metric of “unclassified word rate”, extends the ontology to pilot utterances, and introduces the metrics of com-mand recognition rate, command recognition error rate, and command recognition rejection rate. This enables the compari-son of different speech recognition and understanding instances on semantic level. The implementation used in this paper achieves a command recognition rate better than 96% for Pra-gue Approach, even if word error rate is above 2.5% based on more than 12,000 ATCo commands – recorded in both opera-tional and lab environment. This outperforms previous pub-lished rates by 2% absolute.

Item URL in elib:https://elib.dlr.de/145679/
Document Type:Conference or Workshop Item (Speech)
Title:Measuring Speech Recognition And Understanding Performance in Air Traffic Control Domain Beyond Word Error Rates
AuthorsInstitution or Email of AuthorsAuthor's ORCID iDORCID Put Code
Helmke, HartmutUNSPECIFIEDhttps://orcid.org/0000-0002-1939-0200UNSPECIFIED
Kleinert, MatthiasUNSPECIFIEDhttps://orcid.org/0000-0002-0782-4147UNSPECIFIED
Ohneiser, OliverUNSPECIFIEDhttps://orcid.org/0000-0002-5411-691XUNSPECIFIED
Refereed publication:Yes
Open Access:No
Gold Open Access:No
In ISI Web of Science:No
Keywords:word error rate, command recognition rate, language understanding, air traffic control, ATC, unclassified word rate
Event Title:11th SESAR Innovation Days
Event Location:Virtual
Event Type:international Conference
Event Start Date:7 December 2021
Event End Date:9 December 2021
Organizer:SESAR Joint Undertaking
HGF - Research field:Aeronautics, Space and Transport
HGF - Program:Aeronautics
HGF - Program Themes:other
DLR - Research area:Aeronautics
DLR - Program:L - no assignment
DLR - Research theme (Project):L - Managementaufgaben Luftfahrt
Location: Braunschweig
Institutes and Institutions:Institute of Flight Guidance > Controller Assistance
Deposited By: Kleinert, Matthias
Deposited On:13 Dec 2021 09:46
Last Modified:24 Apr 2024 20:44

Repository Staff Only: item control page

Help & Contact
electronic library is running on EPrints 3.3.12
Website and database design: Copyright © German Aerospace Center (DLR). All rights reserved.