In the Jungle of Federated Learning Frameworks

In this guest blog post, we are delighted to have Pascal Riedel, Data Scientist at the University of Ulm, describe his recent research. The study provides a detailed comparative analysis of 15 open-source Federated Learning frameworks, aiming to guide practitioners and researchers in selecting the most suitable framework for their needs.

Based on our study, we identified Flower as the top FL-framework, scoring a total of 84.75% and excelling in most evaluation categories.

Share on Twitter

Federated Learning

Federated Learning (FL) is a semi-distributed Machine Learning (ML) concept that has gained popularity in recent years, addressing data privacy concerns associated with centralized ML. In FL, a predefined number of clients with sensitive training data and a coordinator server jointly train a global model, while the local training data remains on the original client and is isolated from other clients. Each client trains a model obtained by the coordinator server with the client’s local training data. The locally updated model weights are sent back to the coordinator server, where an updated global model is computed using an aggregation strategy.

FL Frameworks

The advent of FL has spurred the development of various FL frameworks aimed at facilitating the deployment of FL applications, offering standardized functionalities and enhanced usability.

Why a comparative analysis of federated learning frameworks is necessary

With the rise of FL and its numerous applications spanning from the healthcare industry to the private energy sector, the demand for user-friendly frameworks has become more pressing. Implementing an FL system presents several challenges: How do you connect the server to the clients? How are models trained and aggregated in a distributed manner? What security methods are implemented? These are critical questions that need addressing.

Recognizing the need to simplify FL implementation, some framework developers have focused on creating interfaces that bridge the gap between the complex FL backend and the user-friendly frontend. Consequently, there has been a surge in the development of open-source FL frameworks in recent years.

As advocates of the open-source movement and spirit, we decided to delve deeper into the market and conduct a comprehensive study on these FL frameworks.

Study Overview

Our study aims to provide a thorough comparison of 15 open-source FL frameworks. We evaluated these frameworks based on 15 qualitative and quantitative criteria, focusing on the features, interoperability, and user-friendliness of each framework. The goal was to guide practitioners and researchers in selecting the most suitable FL framework for their specific needs.

Key Findings

Based on our study, we identified Flower as the top FL-framework, scoring a total of 84.75% and excelling in most evaluation categories. Especially Flower convinces with a high user friendliness and interoperability.

A few comparison criteria where Flower dominated most other frameworks are:

• Development Effort: For a purely open-source based Python framework, Flower requires only moderate development effort, providing an intuitive setup process.
• Documentation Level: In contrast to some other competitors, Flower provides an extensive API documentation, sample applications, and basic tutorials, aiding developers in effectively using the framework.
• Rollout Edge Devices: Flower supports an simple deployment process on edge devices (e.g. Nvidia Jetson DevKits), making it highly suitable for real-world applications with low computational power.

Although some FL frameworks score better than Flower in certain criteria, Flower offers the best overall package for an open-source FL framework. The constant growth of the framework, the actively used Slack channel, and the up-to-date documentation should also be emphasized.

Conclusion

Our novel FL framework comparison suite serves as a valuable tool for selecting FL frameworks and is designed to be adaptable, allowing for the inclusion of new frameworks and evolving requirements. The detailed analysis currently highlights Flower as the leading open-source FL framework, offering the most added value to users. It is exciting to see what Flower will provide next!

For more details, refer to the full paper available in the International Journal of Machine Learning and Cybernetics.