> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/kyutai-labs/unmute/llms.txt
> Use this file to discover all available pages before exploring further.

# Using External LLM Servers

> Connect Unmute to OpenAI, Ollama, or other OpenAI-compatible LLM providers

Unmute is designed to work with any OpenAI-compatible LLM server. By default, it uses a local VLLM instance, but you can easily switch to external providers like OpenAI, Mistral, Ollama, or any other compatible API.

## How It Works

The Unmute backend communicates with the LLM server using the OpenAI-compatible API format. The LLM generates text responses based on the conversation history, which are then passed to the TTS system for speech synthesis.

## Configuration

LLM configuration is set through environment variables in the `backend` service:

```yaml theme={null}
backend:
  environment:
    - KYUTAI_LLM_URL=http://llm:8000
    - KYUTAI_LLM_MODEL=meta-llama/Llama-3.2-1B-Instruct
    - KYUTAI_LLM_API_KEY=your-api-key
```

### Environment Variables

<ParamField path="KYUTAI_LLM_URL" type="string" required>
  The base URL of the OpenAI-compatible API endpoint
</ParamField>

<ParamField path="KYUTAI_LLM_MODEL" type="string" required>
  The model identifier to use for text generation
</ParamField>

<ParamField path="KYUTAI_LLM_API_KEY" type="string">
  API key for authentication (if required by the provider)
</ParamField>

## Provider Examples

### OpenAI

To use OpenAI's GPT models:

<Steps>
  <Step title="Get an API Key">
    Create an API key at [platform.openai.com/api-keys](https://platform.openai.com/api-keys)
  </Step>

  <Step title="Update docker-compose.yml">
    Modify the `backend` service configuration:

    ```yaml theme={null}
    backend:
      image: unmute-backend:latest
      environment:
        - KYUTAI_STT_URL=ws://stt:8080
        - KYUTAI_TTS_URL=ws://tts:8080
        - KYUTAI_LLM_URL=https://api.openai.com/v1
        - KYUTAI_LLM_MODEL=gpt-4.1
        - KYUTAI_LLM_API_KEY=sk-...
        - NEWSAPI_API_KEY=$NEWSAPI_API_KEY
    ```
  </Step>

  <Step title="Remove VLLM Service (Optional)">
    Since you're using an external LLM, you can remove the `llm` service from `docker-compose.yml` to save resources:

    ```yaml theme={null}
    # Remove or comment out this entire section:
    # llm:
    #   image: vllm/vllm-openai:v0.11.0
    #   ...
    ```
  </Step>

  <Step title="Restart Services">
    Apply the changes:

    ```bash theme={null}
    docker compose up --build
    ```
  </Step>
</Steps>

### Ollama (Local)

To use a locally running Ollama instance:

<Steps>
  <Step title="Install and Start Ollama">
    Install Ollama from [ollama.ai](https://ollama.ai) and pull a model:

    ```bash theme={null}
    ollama pull gemma3
    ```
  </Step>

  <Step title="Update docker-compose.yml">
    Configure the backend to connect to your host machine:

    ```yaml theme={null}
    backend:
      image: unmute-backend:latest
      environment:
        - KYUTAI_STT_URL=ws://stt:8080
        - KYUTAI_TTS_URL=ws://tts:8080
        - KYUTAI_LLM_URL=http://host.docker.internal:11434
        - KYUTAI_LLM_MODEL=gemma3
        - KYUTAI_LLM_API_KEY=ollama
        - NEWSAPI_API_KEY=$NEWSAPI_API_KEY
      extra_hosts:
        - "host.docker.internal:host-gateway"
    ```

    The `extra_hosts` section allows Docker containers to access services running on the host machine.
  </Step>

  <Step title="Remove VLLM Service (Optional)">
    Remove the `llm` service definition from `docker-compose.yml` to free up GPU memory.
  </Step>

  <Step title="Restart Services">
    ```bash theme={null}
    docker compose up --build
    ```
  </Step>
</Steps>

### Mistral AI

To use Mistral's API:

```yaml theme={null}
backend:
  environment:
    - KYUTAI_LLM_URL=https://api.mistral.ai/v1
    - KYUTAI_LLM_MODEL=mistral-small-latest
    - KYUTAI_LLM_API_KEY=your-mistral-api-key
```

### Custom VLLM Server

If you're running your own VLLM server elsewhere:

```yaml theme={null}
backend:
  environment:
    - KYUTAI_LLM_URL=http://your-vllm-server:8000
    - KYUTAI_LLM_MODEL=meta-llama/Llama-3.2-1B-Instruct
    # API key may not be needed for private VLLM instances
```

## Dockerless Configuration

For dockerless deployments, set the environment variables before starting the backend:

```bash theme={null}
export KYUTAI_LLM_URL=https://api.openai.com/v1
export KYUTAI_LLM_MODEL=gpt-4.1
export KYUTAI_LLM_API_KEY=sk-...

./dockerless/start_backend.sh
```

Or modify the `dockerless/start_backend.sh` script to include these variables.

## Model Selection

The LLM is displayed in conversation system prompts. The backend automatically formats the model name for readability (e.g., `meta-llama/Llama-3.2-1B-Instruct` becomes "meta llama Llama 3.2 1B Instruct").

### Recommended Models

**For local VLLM deployment:**

* `meta-llama/Llama-3.2-1B-Instruct` - Lightweight, 16GB VRAM (default)
* `mistralai/Mistral-Small-3.2-24B-Instruct-2506` - Higher quality, requires more VRAM
* `google/gemma-3-12b-it` - Good balance of quality and performance

**For external APIs:**

* OpenAI: `gpt-4o`, `gpt-4.1`, or `gpt-3.5-turbo`
* Mistral: `mistral-small-latest` or `mistral-large-latest`
* Ollama: `gemma3`, `llama3`, `mistral`, etc.

## LLM Requirements for Unmute

Unmute works best with models that:

<Check>
  Support streaming text generation for low-latency responses
</Check>

<Check>
  Are optimized for conversational tasks
</Check>

<Check>
  Follow instruction formats well
</Check>

<Check>
  Generate concise responses suitable for speech
</Check>

## System Prompt Integration

The LLM receives a detailed system prompt that includes:

* Base instructions for voice conversation format
* Character-specific personality traits (from `voices.yaml`)
* Language instructions (English/French support)
* Context about the Unmute system itself
* Current date, time, and timezone (for relevant character types)

The system prompt is generated by `unmute/llm/system_prompt.py` and automatically includes the LLM model name.

## Performance Considerations

<Warning>
  **Latency matters**: Voice conversation requires low-latency LLM responses. Local VLLM or nearby servers will provide better user experience than distant cloud APIs.
</Warning>

### Latency Optimization Tips

1. **Use streaming**: Unmute starts TTS generation before the LLM finishes the full response
2. **Choose faster models**: Smaller models (1-12B parameters) respond faster than large models
3. **Local deployment**: Running VLLM locally eliminates network latency
4. **Adjust max tokens**: Configure `--max-model-len` in VLLM to balance context vs. speed

### VLLM Configuration

If you're running your own VLLM instance, optimize these parameters:

```yaml theme={null}
llm:
  image: vllm/vllm-openai:v0.11.0
  command:
    - "--model=meta-llama/Llama-3.2-1B-Instruct"
    - "--max-model-len=1536"  # Shorter conversations = less memory
    - "--dtype=bfloat16"      # Faster than float32
    - "--gpu-memory-utilization=0.4"  # Adjust based on your GPU
```

## Testing Your Configuration

After configuring your external LLM:

<Steps>
  <Step title="Check Backend Logs">
    Verify the backend connects successfully:

    ```bash theme={null}
    docker compose logs backend
    ```

    Look for successful LLM initialization messages.
  </Step>

  <Step title="Test a Conversation">
    Open Unmute in your browser and start a conversation. The LLM model name will be mentioned in the character's system prompt.
  </Step>

  <Step title="Monitor Response Times">
    Use the dev mode (press `D` after enabling in `useKeyboardShortcuts.ts`) to see timing information.
  </Step>
</Steps>

## Troubleshooting

<AccordionGroup>
  <Accordion title="Connection refused errors">
    * For localhost Ollama: Ensure `extra_hosts` is configured correctly
    * For external APIs: Check your network can reach the API URL
    * Verify the API endpoint is correct (usually ends with `/v1`)
  </Accordion>

  <Accordion title="Authentication errors">
    * Verify your API key is correct and has not expired
    * Check that the API key has proper permissions
    * For Ollama, any string works as the API key (e.g., "ollama")
  </Accordion>

  <Accordion title="Model not found errors">
    * Verify the model name is correct for your provider
    * For Ollama: Ensure you've pulled the model with `ollama pull <model>`
    * For OpenAI: Check available models at platform.openai.com/docs/models
  </Accordion>

  <Accordion title="Slow response times">
    * Consider using a smaller/faster model
    * For cloud APIs: Check your network latency
    * For local VLLM: Increase `--gpu-memory-utilization` if you have spare GPU memory
    * Reduce `--max-model-len` to decrease memory usage
  </Accordion>
</AccordionGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.