Decentralized Federated Learning using Transformer-based Language Models

Kurzfassung

When simulating federated learning scenarios, most studies use a centralized learning architecture in which a global server performs the orchestration of the individual clients and the aggregation of the associated weights. Decentralized federated learning (DFL) clusters, on the other hand, are rarely investigated, although they can counteract disadvantages such as "single point of failure" or limited scaling options. Furthermore, Transformer-based language models have not yet been investigated in DFL clusters, although they are still the de facto standard in the field of natural language processing. In order to investigate how such model architectures can be trained in a decentralized federated manner and which influencing factors play a decisive role, this study presents the implementation of three peer-to-peer topologies (Ring, Wheel, Fully connected) for a DFL cluster and their evaluation using the well-known text classification problem "Contradiction Detection". The results of this case study show that the use of the "Fully connected" topology leads to faster and more stable convergence (model divergence of 0.38), while "Ring" and "Wheel" topologies initially exhibit greater fluctuations (model divergence of up to 0.53).