Deploy SuperNodes using Slurm

This guide shows how to deploy a SuperNode as a Slurm job with one of four deployment patterns:

  1. Default subprocess isolation

  2. process isolation with a separate SuperExec

  3. process isolation with a GPU-enabled SuperExec

  4. process isolation with scheduler-selected nodes

Each example connects a SuperNode to SuperGrid. Before you continue, register a separate key pair for every SuperNode that you plan to deploy. See Connect SuperNodes to SuperGrid for instructions.

You will need:

  • Access to a Slurm cluster and a compute partition

  • Flower installed on the compute nodes

  • A registered SuperNode private key that is readable from the compute node

  • All ClientApp dependencies installed in the execution environment

In the examples, replace each value in angle brackets with the corresponding value for your cluster. The first two process-isolation examples select one compute node that can run both jobs at the same time. The final example lets Slurm place each job.

Note

You can install ClientApp dependencies in advance or let Flower install them when an app starts. For the available options and their network requirements, see Install Flower App dependencies at runtime.

Use default subprocess isolation

By default, the SuperNode starts each ClientApp as a subprocess. This model needs only one Slurm job, and the command does not require an --isolation option.

Create supernode-subprocess.sbatch:

#!/bin/bash
#SBATCH --job-name=flower-supernode
#SBATCH --partition=<partition>
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=4G
#SBATCH --time=01:00:00
#SBATCH --output=flower-supernode-%j.log

set -Eeuo pipefail

export FLWR_HOME="${SLURM_TMPDIR:-/tmp}/flower-${SLURM_JOB_ID}"

exec flower-supernode \
    --superlink fleet-supergrid.flower.ai:443 \
    --auth-supernode-private-key <path-to-private-key>

Submit the batch script to deploy the SuperNode:

$ sbatch supernode-subprocess.sbatch

Each ClientApp process that the SuperNode starts can use the CPU and memory assigned to this job.

Use process isolation

With process isolation, Slurm manages the SuperNode and SuperExec as separate jobs. The SuperNode receives tasks from SuperGrid, while the SuperExec connects to the SuperNode Runtime API and starts the ClientApp processes.

The Runtime API in this example listens on 127.0.0.1, so both jobs must use the same compute node.

Warning

The process-isolation examples below are minimal configurations for a dedicated or otherwise trusted compute node. On a multi-tenant node, binding the Runtime API to 127.0.0.1 does not prevent other local jobs from connecting to it. Another local process could claim ClientApp tasks and access their inputs. Do not use these examples unchanged on a shared or production compute node. Security hardening for those environments is outside the scope of this guide.

Create supernode-process.sbatch:

#!/bin/bash
#SBATCH --job-name=flower-supernode
#SBATCH --partition=<partition>
#SBATCH --nodelist=<compute-node>
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=1G
#SBATCH --time=01:00:00
#SBATCH --output=flower-supernode-%j.log

set -Eeuo pipefail

export FLWR_HOME="${SLURM_TMPDIR:-/tmp}/flower-${SLURM_JOB_ID}"

exec flower-supernode \
    --superlink fleet-supergrid.flower.ai:443 \
    --auth-supernode-private-key <path-to-private-key> \
    --isolation process \
    --host 127.0.0.1 \
    --port 9094

Create superexec-clientapp.sbatch:

#!/bin/bash
#SBATCH --job-name=flower-superexec
#SBATCH --partition=<partition>
#SBATCH --nodelist=<compute-node>
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=4G
#SBATCH --time=01:00:00
#SBATCH --output=flower-superexec-%j.log

set -Eeuo pipefail

export FLWR_HOME="${SLURM_TMPDIR:-/tmp}/flower-${SLURM_JOB_ID}"

# Wait up to 60 seconds for the SuperNode Runtime API.
for _ in {1..30}; do
    if bash -c '</dev/tcp/127.0.0.1/9094' 2>/dev/null; then
        exec flower-superexec \
            --insecure \
            --plugin-type clientapp \
            --runtime-api-address 127.0.0.1:9094
    fi
    sleep 2
done

echo "SuperNode Runtime API did not start on 127.0.0.1:9094." >&2
exit 1

Submit the SuperNode job first, then submit the SuperExec job:

$ sbatch supernode-process.sbatch
$ sbatch superexec-clientapp.sbatch

The SuperExec waits for ClientApp tasks from the SuperNode. The ClientApp processes inherit the CPU and memory allocation of the SuperExec job.

Note

--insecure applies only to the local Runtime API connection between SuperExec and SuperNode. The connection from SuperNode to SuperGrid still uses TLS. To protect Runtime API traffic outside a trusted host, configure TLS as described in Enable TLS connections.

Use process isolation with a GPU ClientApp

To run a ClientApp on a GPU, use the supernode-process.sbatch script from the previous section without adding a GPU request. The SuperNode coordinates the work but does not execute the ClientApp, so it does not need access to the GPU.

Request the GPU only in the SuperExec job. The ClientApp process then inherits the SuperExec environment, including CUDA_VISIBLE_DEVICES. Configure both batch scripts with the same <partition> and <compute-node> values so that they run together on the GPU-capable node.

Create superexec-clientapp-gpu.sbatch:

#!/bin/bash
#SBATCH --job-name=flower-superexec-gpu
#SBATCH --partition=<partition>
#SBATCH --nodelist=<compute-node>
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=8G
#SBATCH --gres=gpu:1
#SBATCH --time=01:00:00
#SBATCH --output=flower-superexec-gpu-%j.log

set -Eeuo pipefail

export FLWR_HOME="${SLURM_TMPDIR:-/tmp}/flower-${SLURM_JOB_ID}"

if [[ -z "${CUDA_VISIBLE_DEVICES:-}" ]]; then
    echo "Slurm did not assign a GPU to this job." >&2
    exit 1
fi

# Wait up to 60 seconds for the SuperNode Runtime API.
for _ in {1..30}; do
    if bash -c '</dev/tcp/127.0.0.1/9094' 2>/dev/null; then
        exec flower-superexec \
            --insecure \
            --plugin-type clientapp \
            --runtime-api-address 127.0.0.1:9094
    fi
    sleep 2
done

echo "SuperNode Runtime API did not start on 127.0.0.1:9094." >&2
exit 1

Submit the unchanged process-isolated SuperNode job first, then submit the GPU SuperExec job:

$ sbatch supernode-process.sbatch
$ sbatch superexec-clientapp-gpu.sbatch

The ClientApp environment must include a GPU-enabled version of its machine learning framework and the required GPU libraries. For example, a PyTorch ClientApp can use the device that Slurm exposes through CUDA_VISIBLE_DEVICES.

Discover the SuperNode address from its job ID

The preceding process-isolation examples use --nodelist to place both jobs on the same node. You can remove this constraint when the compute nodes can reach each other. In this model, a submission script passes the SuperNode job ID to a dependent SuperExec job. After the SuperNode starts, SuperExec uses that ID to discover the assigned node's address.

Create supernode-process-dynamic.sbatch:

#!/bin/bash
#SBATCH --job-name=flower-supernode
#SBATCH --partition=<partition>
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem=1G
#SBATCH --time=01:00:00
#SBATCH --output=flower-supernode-%j.log

set -Eeuo pipefail

export FLWR_HOME="${SLURM_TMPDIR:-/tmp}/flower-${SLURM_JOB_ID}"

exec flower-supernode \
    --superlink fleet-supergrid.flower.ai:443 \
    --auth-supernode-private-key <path-to-private-key> \
    --isolation process \
    --host 0.0.0.0 \
    --port 9094

The SuperNode listens on all network interfaces because Slurm can place SuperExec on a different node. Create superexec-clientapp-dynamic.sbatch without a --nodelist directive:

#!/bin/bash
#SBATCH --job-name=flower-superexec
#SBATCH --partition=<partition>
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=8G
#SBATCH --gres=gpu:1
#SBATCH --time=01:00:00
#SBATCH --output=flower-superexec-%j.log

set -Eeuo pipefail

: "${SUPERNODE_JOB_ID:?SuperNode job ID is required.}"

export FLWR_HOME="${SLURM_TMPDIR:-/tmp}/flower-${SLURM_JOB_ID}"

node_list="$(squeue \
    --noheader \
    --jobs="${SUPERNODE_JOB_ID}" \
    --format='%N')"
node_list="${node_list//[[:space:]]/}"

if [[ -z "${node_list}" || "${node_list}" == "(null)" ]]; then
    echo "SuperNode job has no assigned node." >&2
    exit 1
fi

supernode_node="$(scontrol show hostnames "${node_list}" | head -n 1)"
node_record="$(scontrol show node "${supernode_node}" --oneliner)"
node_address="$(printf '%s\n' "${node_record}" \
    | tr ' ' '\n' \
    | awk -F= '$1 == "NodeAddr" {print $2; exit}')"

if [[ -z "${node_address}" ]]; then
    echo "Could not resolve the SuperNode address." >&2
    exit 1
fi

runtime_api_address="${node_address}:9094"
runtime_host="${runtime_api_address%:*}"
runtime_port="${runtime_api_address##*:}"

for _ in {1..30}; do
    if bash -c "</dev/tcp/${runtime_host}/${runtime_port}" 2>/dev/null; then
        exec flower-superexec \
            --insecure \
            --plugin-type clientapp \
            --runtime-api-address "${runtime_api_address}"
    fi
    sleep 2
done

echo "SuperNode Runtime API did not become ready." >&2
exit 1

This example requests one GPU for SuperExec. Remove #SBATCH --gres=gpu:1 when the ClientApp does not need a GPU.

Create submit-process-pair.sh to submit both jobs:

#!/usr/bin/env bash

set -Eeuo pipefail

supernode_job="$(sbatch --parsable supernode-process-dynamic.sbatch)"
supernode_job_id="${supernode_job%%;*}"
echo "Submitted SuperNode job ${supernode_job_id}."

superexec_job="$(sbatch \
    --parsable \
    --dependency="after:${supernode_job_id}" \
    --export="ALL,SUPERNODE_JOB_ID=${supernode_job_id}" \
    superexec-clientapp-dynamic.sbatch)"

echo "Submitted SuperExec job ${superexec_job%%;*}."

Run the submission script:

$ bash submit-process-pair.sh

The first sbatch command returns the SuperNode job ID. The second job uses an after:<job-id> dependency, so Slurm holds it until the SuperNode job starts. The SuperExec script then uses squeue and scontrol to resolve the assigned NodeAddr. The short readiness loop accounts for the time between job startup and the Runtime API becoming available.

Warning

This minimal example exposes an unauthenticated and unencrypted Runtime API on the compute-node network. Use it only in a trusted, isolated test environment. Do not use it unchanged on a shared or production cluster. Security hardening for those environments is outside the scope of this guide.

For more information about the two isolation modes and the Runtime API, see Flower Network Communication. After the SuperNode is online, see Run Flower Apps on SuperGrid.