Federated Learning with HuggingFace Transformers and Flower (Quickstart Example)¶

View on GitHub

This introductory example to using 🤗Transformers with Flower. The training script closely follows the HuggingFace course, so you are encouraged to check that out for a detailed explanation of the transformer pipeline.

In this example, we will federated the training of a BERT-tiny model on the IMDB dataset. The data will be downloaded and partitioned using Flower Datasets. This example runs best when a GPU is available.

Set up the project¶

Fetch the app¶

Install Flower:

pip install flwr

Fetch the app:

flwr new @flwrlabs/quickstart-huggingface

This will create a new directory called quickstart-huggingface containing the following files:

quickstart-huggingface
├── huggingface_example
│   ├── __init__.py
│   ├── client_app.py   # Defines your ClientApp
│   ├── server_app.py   # Defines your ServerApp
│   └── task.py         # Defines your model, training and data loading
├── pyproject.toml      # Project metadata like dependencies and configs
└── README.md

Install dependencies and project¶

Install the dependencies defined in pyproject.toml as well as the huggingface_example package.

pip install -e .

Run the Example¶

You can run your Flower project in both simulation and deployment mode without making changes to the code. If you are starting with Flower, we recommend you using the simulation mode as it requires fewer components to be launched manually. By default, flwr run will make use of the Simulation Engine.

Run with the Simulation Engine¶

This example is designed to run with 100 virtual SuperNodes. First we need to change the configuration of the Simulation Runtime (which by default uses 10 nodes and only CPU). This guide assumes your default SuperLink connection points to one ready for simulations. If you aren’t sure, please refer to the How-to run Flower locally guide.

flwr federation simulation-config --num-supernodes=100

# Run with the default federation (CPU only)
flwr run .  --stream

[!TIP] This example runs faster when the ClientApps have access to a GPU. If your system has one, adjust the settings of your simulations as shown below. Check the Simulation Engine documentation to learn more about Flower simulations and how to optimize them.

flwr federation simulation-config \
    --client-resources-num-cpus=4 \ # each ClientApp assumes to use 4CPUs
--client-resources-num-gpus=0.25  # at most 4 ClientApp will run in a given GPU

And then run the app

# Run with the `local-gpu` settings
flwr run .  --stream

You can also override some of the settings for your ClientApp and ServerApp defined in pyproject.toml. For example

flwr run . --run-config "num-server-rounds=5 fraction-train=0.1"  --stream

[!TIP] For a more detailed walk-through check our quickstart 🤗Transformers tutorial

Run with the Deployment Engine¶

Follow this how-to guide to run the same app in this example but with Flower’s Deployment Engine. After that, you might be intersted in setting up secure TLS-enabled communications and SuperNode authentication in your federation.

If you are already familiar with how the Deployment Engine works, you may want to learn how to run it using Docker. Check out the Flower with Docker documentation.