---
title: Federated Self-training for Semi-supervised Audio Recognition
url: https://dl.acm.org/doi/10.1145/3520128
labels: [Audio Classification, Semi Supervised learning]
dataset: [Ambient Context, Speech Commands]
---
# FedStar: Federated Self-training for Semi-supervised Audio Recognition
[](https://github.com/adap/flower/blob/main/baselines/fedstar)
> Note: If you use this baseline in your work, please remember to cite the original authors of the paper as well as the Flower paper.
**Paper:** [dl.acm.org/doi/10.1145/3520128](https://dl.acm.org/doi/10.1145/3520128)
**Authors:** Vasileios Tsouvalas, Aaqib Saeed, Tanir Özcelebi
**Abstract:** Federated Learning is a distributed machine learning paradigm dealing with decentralized and personal datasets. Since data reside on devices such as smartphones and virtual assistants, labeling is entrusted to the clients or labels are extracted in an automated way. Specifically, in the case of audio data, acquiring semantic annotations can be prohibitively expensive and time-consuming. As a result, an abundance of audio data remains unlabeled and unexploited on users’ devices. Most existing federated learning approaches focus on supervised learning without harnessing the unlabeled data. In this work, we study the problem of semi-supervised learning of audio models via self-training in conjunction with federated learning. We propose FedSTAR to exploit large-scale on-device unlabeled data to improve the generalization of audio recognition models. We further demonstrate that self-supervised pre-trained models can accelerate the training of on-device models, significantly improving convergence within fewer training rounds. We conduct experiments on diverse public audio classification datasets and investigate the performance of our models under varying percentages of labeled and unlabeled data. Notably, we show that with as little as 3% labeled data available, FedSTAR on average can improve the recognition rate by 13.28% compared to the fully supervised federated model.
## About this baseline
**What’s implemented:** The code is structured in such a way that all experiments for ambient context and speech commands can be derived.
**Datasets:** Ambient Context, Speech Commands
**Hardware Setup:** These experiments were run on a linux server with 56 CPU threads with 325 GB Ram with A10 GPU in it. Any machine with 16 CPU cores and 32 GB memory would be able to run experiments with small number of clients in a reasonable amount of time. For context, a machine with 24 cores and a RTX3090Ti ran the Speech Commands experiment in Table 3 with 10 clients in 1h. For this experiment 30GB of RAM was used and clients required ~1.4GB of VRAM each. The same experiment but with the Ambient Context dataset too 13minutes.
**Contributors:** Raj Parekh [GitHub](https://github.com/Raj-Parekh24), [Mail](rajparekhwc@gmail.com)
## Environment Setup
```bash
# Set python version
pyenv local 3.10.6
# Tell poetry to use python 3.10
poetry env use 3.10.6
# Now install the environment
poetry install
# Start the shell to activate your environment.
poetry shell
```
Next, you'll need to download the datasets. In the case of SpeechCommands, some preprocessing is also required:
```bash
# Make the shell script executable
chmod +x setup_datasets.sh
# The below script will download the datasets and create a directory structure require to run this experiment.
./setup_datasets.sh
# If you want to run the SpeechCommands experiment, pre-process the dataset
# This will generate a few training example from the _silence_ category
python -m fedstar.dataset_preparation
# Please note the above will make following changes:
# * Add new files to datasets/speech_commands/Data/Train/_silence_
# * Add new entries to data_splits/speech_commands/train_split.txt
# Therefore the above command should only be run once. If you want to run it again
# after making modifications to the script, please either revert the changes outlined
# above or erase the dataset and repeat the download + preprocessing as defined in setup_datasets.sh script.
```
## Setting up GPU Memory
**Note:** The experiment is designed to run on both GPU and CPU, but runs better on a system with GPU (specially when using the SpeechCommands dataset). If you wish to use GPU, make sure you have installed the [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads). This baseline has been tested with CUDA 12.3. By default, it will run only on the CPU. Please update the value of the list `gpu_total_mem` with the corresponding memory for each GPU in your machine that you want to expose to the experiment. The variable is in the `distribute_gpus` function inside the `clients.py`. Reference is shown below.
```python
# For Eg:- We have a system with two GPUs with 8GB and 4GB VRAM.
# The modified variable will looks like below.
gpu_free_mem = [8000,4000]
```
## Running the Experiments
By default, the `Ambient Context` experiment in Table 3 with 10 clients will be run.
```bash
python -m fedstar.server
python -m fedstar.clients
```
You can change the dataset, number of clients and number of rounds like this:
```bash
python -m fedstar.server num_clients=5 dataset_name=speech_commands server.rounds=20
python -m fedstar.clients num_clients=5 dataset_name=speech_commands
```
To run experiments for Table 4, you should pass a different config file (i.e. that in `fedstar/conf/table4.yaml`). You can do this as follows:
```bash
# by default will run FedStar with Ambient Context and L=3%
python -m fedstar.server --config-name table4
python -m fedstar.clients --config-name table4
```
To modify the ratio of labelled data do so as follows:
```bash
# To use a different L setting
python -m fedstar.server --config-name table4 L=L5 # {L3, L5, L20, L50}
# same for fedstar.clients
```
To run in supervised mode, pass `fedstar=false` to any of the commands above (when launching both the server and clients). Naturally, you can also override any other setting, like `dataset_name` and `num_clients` if desired.
## Expected Results
This section indicates the commands to execute to obtain the results shown below in Table 3 and Table 4. While both configs fix the number of rounds to 100, in many settings fewer rounds are enough for the model to reach the accuracy shown in the tables. The commands below make use of Hydra's `--multirun` to run multiple experiments. This is better suited when using Flower simulations. Here they work fine but, if you encounter any issues, you can always "unroll" the multirun and run one configuration at a time. If you do this, results won't go into the `multirun/` directory, instead to the default `outputs/` directory.
### Table 3
Results will be stored in `multirun/Table3//N_//
](https://github.com/adap/flower/blob/main/baselines/fedstar)
> Note: If you use this baseline in your work, please remember to cite the original authors of the paper as well as the Flower paper.
**Paper:** [dl.acm.org/doi/10.1145/3520128](https://dl.acm.org/doi/10.1145/3520128)
**Authors:** Vasileios Tsouvalas, Aaqib Saeed, Tanir Özcelebi
**Abstract:** Federated Learning is a distributed machine learning paradigm dealing with decentralized and personal datasets. Since data reside on devices such as smartphones and virtual assistants, labeling is entrusted to the clients or labels are extracted in an automated way. Specifically, in the case of audio data, acquiring semantic annotations can be prohibitively expensive and time-consuming. As a result, an abundance of audio data remains unlabeled and unexploited on users’ devices. Most existing federated learning approaches focus on supervised learning without harnessing the unlabeled data. In this work, we study the problem of semi-supervised learning of audio models via self-training in conjunction with federated learning. We propose FedSTAR to exploit large-scale on-device unlabeled data to improve the generalization of audio recognition models. We further demonstrate that self-supervised pre-trained models can accelerate the training of on-device models, significantly improving convergence within fewer training rounds. We conduct experiments on diverse public audio classification datasets and investigate the performance of our models under varying percentages of labeled and unlabeled data. Notably, we show that with as little as 3% labeled data available, FedSTAR on average can improve the recognition rate by 13.28% compared to the fully supervised federated model.
## About this baseline
**What’s implemented:** The code is structured in such a way that all experiments for ambient context and speech commands can be derived.
**Datasets:** Ambient Context, Speech Commands
**Hardware Setup:** These experiments were run on a linux server with 56 CPU threads with 325 GB Ram with A10 GPU in it. Any machine with 16 CPU cores and 32 GB memory would be able to run experiments with small number of clients in a reasonable amount of time. For context, a machine with 24 cores and a RTX3090Ti ran the Speech Commands experiment in Table 3 with 10 clients in 1h. For this experiment 30GB of RAM was used and clients required ~1.4GB of VRAM each. The same experiment but with the Ambient Context dataset too 13minutes.
**Contributors:** Raj Parekh [GitHub](https://github.com/Raj-Parekh24), [Mail](rajparekhwc@gmail.com)
## Environment Setup
```bash
# Set python version
pyenv local 3.10.6
# Tell poetry to use python 3.10
poetry env use 3.10.6
# Now install the environment
poetry install
# Start the shell to activate your environment.
poetry shell
```
Next, you'll need to download the datasets. In the case of SpeechCommands, some preprocessing is also required:
```bash
# Make the shell script executable
chmod +x setup_datasets.sh
# The below script will download the datasets and create a directory structure require to run this experiment.
./setup_datasets.sh
# If you want to run the SpeechCommands experiment, pre-process the dataset
# This will generate a few training example from the _silence_ category
python -m fedstar.dataset_preparation
# Please note the above will make following changes:
# * Add new files to datasets/speech_commands/Data/Train/_silence_
# * Add new entries to data_splits/speech_commands/train_split.txt
# Therefore the above command should only be run once. If you want to run it again
# after making modifications to the script, please either revert the changes outlined
# above or erase the dataset and repeat the download + preprocessing as defined in setup_datasets.sh script.
```
## Setting up GPU Memory
**Note:** The experiment is designed to run on both GPU and CPU, but runs better on a system with GPU (specially when using the SpeechCommands dataset). If you wish to use GPU, make sure you have installed the [CUDA Toolkit](https://developer.nvidia.com/cuda-downloads). This baseline has been tested with CUDA 12.3. By default, it will run only on the CPU. Please update the value of the list `gpu_total_mem` with the corresponding memory for each GPU in your machine that you want to expose to the experiment. The variable is in the `distribute_gpus` function inside the `clients.py`. Reference is shown below.
```python
# For Eg:- We have a system with two GPUs with 8GB and 4GB VRAM.
# The modified variable will looks like below.
gpu_free_mem = [8000,4000]
```
## Running the Experiments
By default, the `Ambient Context` experiment in Table 3 with 10 clients will be run.
```bash
python -m fedstar.server
python -m fedstar.clients
```
You can change the dataset, number of clients and number of rounds like this:
```bash
python -m fedstar.server num_clients=5 dataset_name=speech_commands server.rounds=20
python -m fedstar.clients num_clients=5 dataset_name=speech_commands
```
To run experiments for Table 4, you should pass a different config file (i.e. that in `fedstar/conf/table4.yaml`). You can do this as follows:
```bash
# by default will run FedStar with Ambient Context and L=3%
python -m fedstar.server --config-name table4
python -m fedstar.clients --config-name table4
```
To modify the ratio of labelled data do so as follows:
```bash
# To use a different L setting
python -m fedstar.server --config-name table4 L=L5 # {L3, L5, L20, L50}
# same for fedstar.clients
```
To run in supervised mode, pass `fedstar=false` to any of the commands above (when launching both the server and clients). Naturally, you can also override any other setting, like `dataset_name` and `num_clients` if desired.
## Expected Results
This section indicates the commands to execute to obtain the results shown below in Table 3 and Table 4. While both configs fix the number of rounds to 100, in many settings fewer rounds are enough for the model to reach the accuracy shown in the tables. The commands below make use of Hydra's `--multirun` to run multiple experiments. This is better suited when using Flower simulations. Here they work fine but, if you encounter any issues, you can always "unroll" the multirun and run one configuration at a time. If you do this, results won't go into the `multirun/` directory, instead to the default `outputs/` directory.
### Table 3
Results will be stored in `multirun/Table3/