elib
DLR-Header
DLR-Logo -> http://www.dlr.de
DLR Portal Home | Imprint | Privacy Policy | Contact | Deutsch
Fontsize: [-] Text [+]

Kronecker-Factored Optimal Curvature

Schnaus, Dominik and Lee, Jongseok and Triebel, Rudolph (2021) Kronecker-Factored Optimal Curvature. In: Bayesian Deep Learning NeurIPS 2021 Workshop. Bayesian Deep Learning NeurIPS 2021 Workshop, Virtual.

[img] PDF
318kB

Abstract

The current scalable Bayesian methods for Deep Neural Networks (DNNs) often rely on the Fisher Information Matrix (FIM). For the tractable computations of the FIM, the Kronecker-Factored Approximate Curvature (K-FAC) method is widely adopted, which approximates the true FIM by a layer-wise block-diagonal matrix, and each diagonal block is then Kronecker-factored. In this paper, we propose an alternative formulation to obtain the Kronecker-factored FIM. The key insight is to cast the given FIM computations into an optimization problem over the sums of Kronecker products. In particular, we prove that this formulation is equivalent to the best rank-one approximation problem, where the well-known power iteration method is guaranteed to converge to an optimal rank-one solution - resulting in our novel algorithm: the Kronecker-Factored Optimal Curvature (K-FOC). In a proof-of-concept experiment, we show that the proposed algorithm can achieve more accurate estimates of the true FIM when compared to the K-FAC method.

Item URL in elib:https://elib.dlr.de/145806/
Document Type:Conference or Workshop Item (Poster)
Title:Kronecker-Factored Optimal Curvature
Authors:
AuthorsInstitution or Email of AuthorsAuthor's ORCID iDORCID Put Code
Schnaus, DominikUNSPECIFIEDUNSPECIFIEDUNSPECIFIED
Lee, JongseokUNSPECIFIEDhttps://orcid.org/0000-0002-0960-0809UNSPECIFIED
Triebel, RudolphUNSPECIFIEDhttps://orcid.org/0000-0002-7975-036XUNSPECIFIED
Date:14 December 2021
Journal or Publication Title:Bayesian Deep Learning NeurIPS 2021 Workshop
Refereed publication:Yes
Open Access:Yes
Gold Open Access:No
In SCOPUS:No
In ISI Web of Science:No
Status:Published
Keywords:Information theory, Deep Neural Networks, Fisher Information Matrix, Kronecker Factorization
Event Title:Bayesian Deep Learning NeurIPS 2021 Workshop
Event Location:Virtual
Event Type:Workshop
HGF - Research field:Aeronautics, Space and Transport
HGF - Program:Space
HGF - Program Themes:Robotics
DLR - Research area:Raumfahrt
DLR - Program:R RO - Robotics
DLR - Research theme (Project):R - Intelligent Mobility (RM) [RO]
Location: Oberpfaffenhofen
Institutes and Institutions:Institute of Robotics and Mechatronics (since 2013)
Institute of Robotics and Mechatronics (since 2013) > Perception and Cognition
Deposited By: Lee, Jongseok
Deposited On:22 Nov 2021 17:33
Last Modified:22 Nov 2021 17:33

Repository Staff Only: item control page

Browse
Search
Help & Contact
Information
electronic library is running on EPrints 3.3.12
Website and database design: Copyright © German Aerospace Center (DLR). All rights reserved.