Highly scaled Federated Learning Simulations for Text Classification

Abstract

When simulating federated learning scenarios, most studies use a small number of clients that have comparatively large amounts of local data. In this study, we investigated how the classification accuracy of a language model fine-tuned using federated learning changes when the same amount of data is distributed over an increasing number of clients (up to 1,000), so that the amount of data per client is steadily reduced. To this end, we conducted several experiments using an example of the wellknown "contradiction detection" classification task, which showed that the model accuracy decreased with an increasing number of clients when the number of federated training rounds remained the same. To counteract this effect and ensure that each client participates equally often in the training, we dynamically adjusted the number of federated training rounds and modified the widely used "FedAvg" method to allow a controlled client selection per training round instead of a random selection. In this way, a Bert model trained on 1,000 clients (with only 391 data instances each) achieved an accuracy that was 0.81% higher than that of a Bert model trained on 100 clients (with 3,910 data instances each) and only 0.18% below the accuracy of a Bert model trained conventionally (non-federated). These results can be relevant for all federated learning use cases where model accuracy losses caused by a high number of clients need to be compensated, especially in the case of transformer-based language models such as Bert.