> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/kyutai-labs/unmute/llms.txt
> Use this file to discover all available pages before exploring further.

# Docker Swarm Deployment

> Scale Unmute across multiple machines with Docker Swarm for production deployments

Docker Swarm is used for production deployments like [unmute.sh](https://unmute.sh). While Docker Compose runs on a single machine, Docker Swarm scales across multiple nodes. Think of it as "multi-node Docker Compose."

<Warning>
  This deployment method is provided to show how Unmute.sh scales in production. Due to the complexity of multi-node debugging, the Kyutai team cannot provide support for Swarm deployments. Use at your own risk.
</Warning>

## When to Use Docker Swarm

Choose Docker Swarm when you need:

* **High availability**: Multiple replicas ensure uptime during updates or crashes
* **Horizontal scaling**: Distribute load across multiple GPUs/machines
* **Production features**: HTTPS, monitoring, authentication, load balancing
* **Traffic handling**: Support for many concurrent users

| Feature | Docker Compose | Docker Swarm |
| - | - | - |
| Machines | 1 | 1 to \~100 |
| GPUs | 1+ | 1 to \~100 |
| Difficulty | Very easy | Medium |
| HTTPS | No | Yes |
| Monitoring | No | Yes (Prometheus, Grafana) |
| Load Balancing | No | Yes |
| High Availability | No | Yes |

## Architecture

### Main Application

```mermaid theme={null}
graph LR
    UB[User browser] --> T((Traefik))
    T --> B(Backend)
    T --> F(Frontend)
    B --> TTS(TTS)
    B --> STT(STT)
    B --> LLM(LLM)
    B --> R(Redis)
    B --> V(Voice cloning)
```

### Monitoring Stack

```mermaid theme={null}
graph LR
    UB[Dev browser] --> T((Traefik))
    T --> TFA(Traefik forward auth)
    TFA --> G(Grafana)
    TFA --> Port(Portainer)
    TFA --> Prom(Prometheus)
    G --> Prom
    Port --> A(Agent)
    Prom --> B(Backend)
    Prom --> STT(STT)
    Prom --> TTS(TTS)
    Prom --> LLM(LLM)
    Prom --> C(Cadvisor)
```

## Setup Instructions

<Warning>
  All commands should be executed from a **client machine** with access to your swarm nodes, not directly on the swarm nodes themselves.
</Warning>

<Steps>
  <Step title="Prepare GPU Nodes">
    Set up each GPU node in your swarm:

    ```bash theme={null}
    # Copy setup script to the node
    scp setup_gpu_swarm_node.py llm-wrapper-gpu000:/root/

    # Run setup on the node
    ssh llm-wrapper-gpu000 python3 /root/setup_gpu_swarm_node.py
    ```

    This script installs Docker, NVIDIA drivers, and configures the node for Swarm.
  </Step>

  <Step title="Initialize Swarm Manager">
    Designate one node as the manager (only needed once):

    ```bash theme={null}
    docker -H ssh://llm-wrapper-gpu000 swarm init
    ```

    The manager node coordinates the swarm and must run certain services like Prometheus.
  </Step>

  <Step title="Add Worker Nodes">
    Get the join command from the manager:

    ```bash theme={null}
    docker -H ssh://llm-wrapper-gpu000 swarm join-token worker
    ```

    This outputs a command like:

    ```bash theme={null}
    docker swarm join --token SWMTKN-1-xxxxx... llm-wrapper-gpu000:2377
    ```

    Run this command on each worker node to join the swarm.
  </Step>

  <Step title="Configure Environment Variables">
    Set up required environment variables on your client machine:

    ```bash theme={null}
    # Required: Hugging Face token for model access
    export HUGGING_FACE_HUB_TOKEN=hf_...

    # Required: Google OAuth for monitoring access
    export PROVIDERS_GOOGLE_CLIENT_SECRET=...

    # Optional: For the "Dev (news)" character
    export NEWSAPI_API_KEY=...
    ```

    **How to generate tokens:**

    * **HUGGING\_FACE\_HUB\_TOKEN**: [Create a token](https://huggingface.co/docs/hub/en/security-tokens) with access to [gemma-3-12b-it](https://huggingface.co/google/gemma-3-12b-it)
    * **PROVIDERS\_GOOGLE\_CLIENT\_SECRET**: [Set up Google OAuth](https://github.com/thomseddon/traefik-forward-auth?tab=readme-ov-file#google) for monitoring authentication
    * **NEWSAPI\_API\_KEY**: [Get an API key](https://newsapi.org/) (optional)
  </Step>

  <Step title="Deploy the Stack">
    Run the deployment script:

    ```bash theme={null}
    # For production
    ./bake_deploy_prod.sh

    # For staging
    ./bake_deploy_staging.sh
    ```

    These scripts build images and deploy the stack defined in `swarm-deploy.yml`.
  </Step>
</Steps>

## Production URLs

Once deployed, services are available at:

* **Main app**: [https://unmute.sh](https://unmute.sh)
* **Traefik dashboard**: [https://traefik.unmute.sh](https://traefik.unmute.sh)
* **Grafana**: [https://grafana.unmute.sh](https://grafana.unmute.sh)
* **Prometheus**: [https://prometheus.unmute.sh](https://prometheus.unmute.sh)
* **Portainer**: [https://portainer.unmute.sh](https://portainer.unmute.sh)

Monitoring services require Google authentication (configured via `traefik-forward-auth`).

## Key Configuration Details

### HTTPS with Let's Encrypt

Traefik automatically obtains and renews SSL certificates:

```yaml swarm-deploy.yml theme={null}
traefik:
  command:
    - "--entrypoints.web.address=:80"
    - "--entrypoints.websecure.address=:443"
    # Redirect all HTTP to HTTPS
    - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
    - "--entrypoints.web.http.redirections.entryPoint.scheme=https"
    # Let's Encrypt configuration
    - "--certificatesResolvers.letsencrypt_resolver.acme.httpChallenge.entryPoint=web"
    - "--certificatesResolvers.letsencrypt_resolver.acme.storage=/letsencrypt/acme.json"
    - "--certificatesResolvers.letsencrypt_resolver.acme.email=your-email@example.com"
```

### GPU Resource Allocation

Services reserve GPUs using generic resources:

```yaml swarm-deploy.yml theme={null}
tts:
  deploy:
    replicas: 3
    resources:
      reservations:
        generic_resources:
          - discrete_resource_spec:
              kind: gpu
              value: 1  # Reserve 1 GPU per replica
```

With 3 TTS replicas, this requires 3 GPUs total.

### Service Replicas

Swarm configuration on unmute.sh:

* **Frontend**: 5 replicas (no GPU needed)
* **Backend**: 16 replicas (no GPU needed)
* **LLM**: 2 replicas (2 GPUs total)
* **TTS**: 3 replicas (3 GPUs total)
* **STT**: 1 replica (1 GPU)
* **Voice cloning**: 2 replicas (no GPU)

### Load Balancing

The backend uses manual load balancing for TTS/STT via `tasks.<service_name>`:

```yaml swarm-deploy.yml theme={null}
backend:
  environment:
    # Returns all replica IPs for manual load balancing
    - KYUTAI_STT_URL=ws://tasks.stt:8080
    - KYUTAI_TTS_URL=ws://tasks.tts:8080
    # Single endpoint (Swarm handles load balancing)
    - KYUTAI_LLM_URL=http://llm:8000
```

## Scaling Operations

### Adding More Resources

<Steps>
  <Step title="Add New Node to Swarm">
    ```bash theme={null}
    # On manager, get join token
    docker -H ssh://llm-wrapper-gpu000 swarm join-token worker

    # Run the output command on the new node
    ```
  </Step>

  <Step title="Scale Service">
    ```bash theme={null}
    docker -H ssh://llm-wrapper-gpu000 service scale llm-wrapper_llm=10
    ```

    This increases the LLM service to 10 replicas.
  </Step>
</Steps>

<Note>
  Swarm does not automatically rebalance containers across nodes. To redistribute containers after adding nodes, force a service restart.
</Note>

### Restarting a Service

Force update to restart all replicas:

```bash theme={null}
docker -H ssh://llm-wrapper-gpu000 service update --force llm-wrapper_tts
```

Useful when:

* New voices are added to `voices.yaml`
* Configuration changes need to propagate
* Rebalancing containers across nodes

### Updating a Single Service

Update just the frontend image without touching other services:

```bash theme={null}
docker -H ssh://llm-wrapper-gpu000 service update \
  --image your-registry/unmute-frontend:latest \
  --with-registry-auth \
  llm-wrapper_frontend
```

Other useful updates:

```bash theme={null}
# Add a volume
docker service update \
  --mount-add type=volume,source=new-volume,target=/data \
  llm-wrapper_frontend

# Change environment variable
docker service update \
  --env-add KYUTAI_LLM_MODEL=mistralai/Mistral-7B-Instruct-v0.3 \
  llm-wrapper_backend
```

## Monitoring

The swarm deployment includes a complete monitoring stack:

### Prometheus

Metrics collection configured via Docker socket:

```yaml swarm-deploy.yml theme={null}
prometheus:
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock:ro
  user: root
  deploy:
    placement:
      constraints:
        - node.role == manager  # Must run on manager
```

Services expose metrics via the `prometheus-port` label:

```yaml swarm-deploy.yml theme={null}
backend:
  deploy:
    labels:
      - "prometheus-port=80"
```

### Grafana

Dashboards are built into the Docker image:

```yaml swarm-deploy.yml theme={null}
grafana:
  build:
    context: services/grafana
```

Changes to dashboards in the UI are lost on restart unless exported and added to the build context.

### Cadvisor

Collects container metrics on every node:

```yaml swarm-deploy.yml theme={null}
cadvisor:
  deploy:
    mode: global  # One container per node
```

## Advanced Configuration

### Changing Docker Data Directory

If you need more disk space, change Docker's data location:

1. Edit `/etc/docker/daemon.json` on each node:
   ```json theme={null}
   {
     "data-root": "/new/docker-data"
   }
   ```

2. Restart Docker:
   ```bash theme={null}
   service docker restart
   ```

### Network Encryption

The overlay network is encrypted for security:

```yaml swarm-deploy.yml theme={null}
networks:
  default:
    driver: overlay
    attachable: true
    driver_opts:
      encrypted: "true"
```

This is important when nodes communicate over the public internet.

### Redis for State Management

Unlike single-machine deployments, Swarm uses Redis for shared state:

```yaml swarm-deploy.yml theme={null}
redis:
  image: redis:latest

backend:
  environment:
    - KYUTAI_REDIS_URL=redis://redis:6379
```

## Troubleshooting

### Service Won't Start

Check service logs:

```bash theme={null}
docker -H ssh://llm-wrapper-gpu000 service logs llm-wrapper_tts
```

Check service status:

```bash theme={null}
docker -H ssh://llm-wrapper-gpu000 service ps llm-wrapper_tts
```

### Not Enough Resources

If services can't find GPUs or have insufficient CPU/memory:

```bash theme={null}
# List available nodes and resources
docker -H ssh://llm-wrapper-gpu000 node ls

# Inspect node details
docker -H ssh://llm-wrapper-gpu000 node inspect llm-wrapper-gpu001
```

Adjust resource reservations in `swarm-deploy.yml`.

### Network Issues

Use the debugger service to test connectivity:

```bash theme={null}
docker -H ssh://llm-wrapper-gpu000 exec \
  $(docker service ps llm-wrapper_debugger -q) \
  ping tts
```

## Differences from Docker Compose

| Feature | Docker Compose | Docker Swarm |
| - | - | - |
| Configuration syntax | Very similar | Nearly identical |
| Service placement | Same machine | Across nodes |
| Scaling | Manual container creation | `docker service scale` |
| Updates | `docker compose up` | `docker service update` |
| Load balancing | None | Automatic |
| Secrets management | .env files | Docker secrets |
| Health checks | Limited | Full support |

## Migration Path

To migrate from Docker Compose to Swarm:

1. **Start with one node**: Initialize swarm on your existing machine
2. **Convert compose file**: Most syntax is compatible, add `deploy` sections
3. **Test locally**: Deploy to single-node swarm
4. **Add monitoring**: Set up Prometheus, Grafana, Traefik
5. **Scale out**: Add worker nodes and increase replicas

## Next Steps

* Review the [swarm-deploy.yml](https://github.com/kyutai-labs/unmute/blob/main/swarm-deploy.yml) file for complete configuration
* Learn about [Docker Swarm documentation](https://docs.docker.com/engine/swarm/)
* Understand [Traefik with Swarm](https://doc.traefik.io/traefik/providers/docker/)
* Explore simpler options: [Docker Compose](/deployment/docker-compose) or [Dockerless](/deployment/dockerless)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.