elib
DLR-Header
DLR-Logo -> http://www.dlr.de
DLR Portal Home | Impressum | Datenschutz | Kontakt | English
Schriftgröße: [-] Text [+]

The impact of averaging logits over probabilities on ensembles of neural networks

Njieutcheu Tassi, Cedrique Rovile und Gawlikowski, Jakob und Fitri, Auliya Unnisa und Triebel, Rudolph (2022) The impact of averaging logits over probabilities on ensembles of neural networks. In: 2022 Workshop on Artificial Intelligence Safety, AISafety 2022, 3215 (19). AISafety 2022: Workshop on Artificial Intelligence Safety, 2022-07-23 - 2022-07-25, Vienna, Austria. ISSN 1613-0073.

[img] PDF - Nur DLR-intern zugänglich
2MB

Offizielle URL: http://ceur-ws.org/Vol-3215/19.pdf

Kurzfassung

Model averaging has become a standard for improving neural networks in terms of accuracy, calibration, and the ability to detect false predictions (FPs). However, recent findings show that model averaging does not necessarily lead to calibrated confidences, especially for underconfident networks. While existing methods for improving the calibration of combined networks focus on recalibrating, building, or sampling calibrated models, we focus on the combination process. Specifically, we evaluate the impact of averaging logits instead of probabilities on the quality of confidence (QoC). We compare combined logits instead of probabilities of members (networks) for models such as ensembles, Monte Carlo Dropout (MCD), and Mixture of Monte Carlo Dropout (MMCD). Comparison is done using experimental results on three datasets using three different architectures. We show that averaging logits instead of probabilities increase the confidence thereby improving the confidence calibration for underconfident models. For example, for MCD evaluated on CIFAR10, averaging logits instead of probabilities reduces the expected calibration error (ECE) from 12.03% to 5.44%. However, the increase in confidence can bring harm to confidence calibration for overconfident models and the separability between true predictions (TPs) and FPs. For example, for MMCD evaluated on MNIST, the average confidence on FPs due to the noisy data increases from 51.31% to 94.58% when averaging logits instead of probabilities. While averaging logits can be applied with underconfident models to improve the calibration on test data, we suggest to average probabilities for safety- and mission-critical applications where the separability of TPs and FPs is of paramount importance.

elib-URL des Eintrags:https://elib.dlr.de/188833/
Dokumentart:Konferenzbeitrag (Vortrag)
Titel:The impact of averaging logits over probabilities on ensembles of neural networks
Autoren:
AutorenInstitution oder E-Mail-AdresseAutoren-ORCID-iDORCID Put Code
Njieutcheu Tassi, Cedrique RovileCedrique.NjieutcheuTassi (at) dlr.deNICHT SPEZIFIZIERTNICHT SPEZIFIZIERT
Gawlikowski, JakobJakob.Gawlikowski (at) dlr.deNICHT SPEZIFIZIERTNICHT SPEZIFIZIERT
Fitri, Auliya UnnisaAuliya.Fitri (at) dlr.deNICHT SPEZIFIZIERTNICHT SPEZIFIZIERT
Triebel, RudolphRudolph.Triebel (at) dlr.dehttps://orcid.org/0000-0002-7975-036XNICHT SPEZIFIZIERT
Datum:2022
Erschienen in:2022 Workshop on Artificial Intelligence Safety, AISafety 2022
Referierte Publikation:Ja
Open Access:Nein
Gold Open Access:Nein
In SCOPUS:Ja
In ISI Web of Science:Nein
Band:3215
Name der Reihe:CEUR Workshop Proceedings (CEUR-WS.org)
ISSN:1613-0073
Status:veröffentlicht
Stichwörter:Model averaging, Combination process, Logit averaging, Probability averaging, Ensemble, Monte Carlo Dropout (MCD), Mixture of Monte Carlo Dropout (MMCD), Quality of confidence (QoC), Confidence calibration, Separating true predictions (TPs) and false predictions (FPs)
Veranstaltungstitel:AISafety 2022: Workshop on Artificial Intelligence Safety
Veranstaltungsort:Vienna, Austria
Veranstaltungsart:Workshop
Veranstaltungsbeginn:23 Juli 2022
Veranstaltungsende:25 Juli 2022
Veranstalter :IJCAI-ECAI 2022
HGF - Forschungsbereich:keine Zuordnung
HGF - Programm:keine Zuordnung
HGF - Programmthema:keine Zuordnung
DLR - Schwerpunkt:Digitalisierung
DLR - Forschungsgebiet:D IAS - Innovative autonome Systeme
DLR - Teilgebiet (Projekt, Vorhaben):D - SKIAS, R - Multisensorielle Weltmodellierung (RM) [RO]
Standort: Berlin-Adlershof
Institute & Einrichtungen:Institut für Optische Sensorsysteme
Institut für Robotik und Mechatronik (ab 2013)
Institut für Datenwissenschaften
Hinterlegt von: Njieutcheu Tassi, Cedrique Rovile
Hinterlegt am:12 Okt 2022 07:54
Letzte Änderung:24 Apr 2024 20:50

Nur für Mitarbeiter des Archivs: Kontrollseite des Eintrags

Blättern
Suchen
Hilfe & Kontakt
Informationen
electronic library verwendet EPrints 3.3.12
Gestaltung Webseite und Datenbank: Copyright © Deutsches Zentrum für Luft- und Raumfahrt (DLR). Alle Rechte vorbehalten.