> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/kyutai-labs/unmute/llms.txt
> Use this file to discover all available pages before exploring further.

# Multi-GPU Setup

> Configure Unmute to use multiple GPUs for improved performance and reduced latency

## Overview

Unmute can leverage multiple GPUs to distribute the workload across Speech-to-Text (STT), Text-to-Speech (TTS), and Language Model (LLM) services. Running these services on separate GPUs significantly improves latency compared to a single-GPU setup.

**Performance improvement:** On production deployments like [unmute.sh](https://unmute.sh), running on separate GPUs reduces TTS latency from \~750ms (single L40S GPU) to \~450ms (multi-GPU setup).

## GPU Memory Requirements

Each service requires specific GPU memory:

* **STT (Speech-to-Text):** 2.5GB VRAM
* **TTS (Text-to-Speech):** 5.3GB VRAM
* **LLM (Language Model):** 6.1GB VRAM (for Llama-3.2-1B)

**Total recommended:** At least 16GB VRAM (can run on single GPU) or 3+ GPUs for optimal performance.

## Docker Compose Configuration

### Default Configuration (All GPUs)

By default, `docker-compose.yml` allocates all available GPUs to each service:

```yaml theme={null}
deploy:
  resources:
    reservations:
      devices:
        - driver: nvidia
          count: all
          capabilities: [gpu]
```

This works well for single-GPU setups but doesn't optimize multi-GPU systems.

### Multi-GPU Configuration (Recommended)

For systems with 3+ GPUs, configure each service to use a dedicated GPU. Modify the `stt`, `tts`, and `llm` service definitions in `docker-compose.yml`:

```yaml theme={null}
stt:
  image: moshi-server:latest
  command: ["worker", "--config", "configs/stt.toml"]
  # ... other configuration ...
  deploy:
    resources:
      reservations:
        devices:
          - driver: nvidia
            count: 1
            capabilities: [gpu]

tts:
  image: moshi-server:latest
  command: ["worker", "--config", "configs/tts.toml"]
  # ... other configuration ...
  deploy:
    resources:
      reservations:
        devices:
          - driver: nvidia
            count: 1
            capabilities: [gpu]

llm:
  image: vllm/vllm-openai:v0.11.0
  command:
    [
      "--model=meta-llama/Llama-3.2-1B-Instruct",
      "--max-model-len=1536",
      "--dtype=bfloat16",
      "--gpu-memory-utilization=0.4",
    ]
  # ... other configuration ...
  deploy:
    resources:
      reservations:
        devices:
          - driver: nvidia
            count: 1
            capabilities: [gpu]
```

With `count: 1`, Docker Swarm automatically distributes services across available GPUs.

## GPU Memory Optimization

If running into memory issues on a single GPU, adjust these parameters in `docker-compose.yml`:

### LLM Memory Settings

```yaml theme={null}
llm:
  command:
    [
      "--model=meta-llama/Llama-3.2-1B-Instruct",
      # Reduce max conversation length
      "--max-model-len=1536",  # Lower this value (default: 1536)
      "--dtype=bfloat16",
      # Reduce GPU memory allocation
      "--gpu-memory-utilization=0.4",  # Lower this value (range: 0.0-1.0)
    ]
```

**Parameters:**

* `--max-model-len`: Maximum conversation length in tokens. Lower values = less memory but shorter conversations
* `--gpu-memory-utilization`: Percentage of GPU memory to use (0.4 = 40%). Lower values leave room for other services

## Verifying GPU Usage

Check GPU allocation with `nvidia-smi`:

```bash theme={null}
# On the host machine
nvidia-smi

# Inside a container
docker exec -it <container_name> nvidia-smi
```

You should see separate processes for STT, TTS, and LLM services on different GPUs (if configured for multi-GPU).

## Docker Swarm Multi-Node Setup

For production deployments across multiple machines, Docker Swarm provides advanced GPU scheduling. See the [swarm-deploy.yml](https://github.com/kyutai-labs/unmute/blob/main/swarm-deploy.yml) configuration:

```yaml theme={null}
tts:
  deploy:
    replicas: 8  # Multiple replicas for load balancing
    resources:
      reservations:
        devices:
          - driver: nvidia
            count: 1
            capabilities: [gpu]
    placement:
      constraints:
        - node.labels.gpu==true  # Only nodes with GPU label
      max_replicas_per_node: 1   # One replica per node
```

**Benefits:**

* Horizontal scaling across multiple GPU nodes
* Automatic failover and load balancing
* Independent scaling of each service

For full swarm deployment instructions, see [SWARM.md](https://github.com/kyutai-labs/unmute/blob/main/SWARM.md).

## Troubleshooting

### Service fails to start

**Error:** `could not select device driver "nvidia" with capabilities: [[gpu]]`

**Solution:** Install [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html):

```bash theme={null}
# Verify installation
sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi
```

### Out of memory errors

**Symptoms:** Container crashes or `CUDA out of memory` errors

**Solutions:**

1. Use a smaller LLM model (e.g., Llama-3.2-1B instead of larger models)
2. Reduce `--gpu-memory-utilization` for the LLM service
3. Lower `--max-model-len` to reduce context window
4. Add more GPUs and use the multi-GPU configuration

### Poor performance despite multiple GPUs

**Check:**

1. Verify each service is on a different GPU with `nvidia-smi`
2. Ensure `count: 1` is set in deploy configuration (not `count: all`)
3. Monitor GPU utilization - underutilized GPUs may indicate bottlenecks elsewhere

## Next Steps

* [Performance Tuning](/guides/performance-tuning) - Optimize latency and throughput
* [Monitoring](/guides/monitoring) - Track GPU metrics with Prometheus and Grafana
* [Debugging](/guides/debugging) - Troubleshoot GPU-related issues


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.