> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/kyutai-labs/unmute/llms.txt
> Use this file to discover all available pages before exploring further.

# Hugging Face Setup

> Configure Hugging Face Hub access for downloading LLM models

Unmute uses models from Hugging Face Hub, including the default LLM and the speech-to-text/text-to-speech models. To access these models, you need to set up a Hugging Face access token.

## Why You Need a Token

The default `docker-compose.yml` configuration uses [Llama 3.2 1B Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct) as the LLM, which requires only 16GB of GPU memory. This model is freely available but is gated, meaning you need to:

1. Accept the model's usage conditions
2. Authenticate with a Hugging Face token

Alternative models like [Mistral Small 3.2 24B](https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506) (\~24GB VRAM) and [Gemma 3 12B](https://huggingface.co/google/gemma-3-12b-it) also have similar requirements.

## Setup Steps

<Steps>
  <Step title="Create a Hugging Face Account">
    If you don't already have one, [sign up for Hugging Face](https://huggingface.co/join).
  </Step>

  <Step title="Accept Model Conditions">
    Visit the model page and accept the usage conditions:

    * [Llama 3.2 1B Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct) (default in docker-compose.yml)
    * Or [Mistral Small 3.2 24B](https://huggingface.co/mistralai/Mistral-Small-3.2-24B-Instruct-2506) (alternative, better quality)
    * Or [Gemma 3 12B](https://huggingface.co/google/gemma-3-12b-it) (alternative)
  </Step>

  <Step title="Create an Access Token">
    1. Go to [Settings → Access Tokens](https://huggingface.co/settings/tokens)
    2. Click "New token"
    3. Select **Fine-grained** token type
    4. Grant permission: "Read access to contents of all public gated repos you can access"
    5. Copy the token (starts with `hf_`)

    <Warning>
      **Do not use tokens with write access** when deploying publicly. If the server is compromised, an attacker would gain write access to your Hugging Face models and datasets.
    </Warning>
  </Step>

  <Step title="Add Token to Environment">
    Add the token to your shell configuration file (`~/.bashrc`, `~/.zshrc`, or equivalent):

    ```bash theme={null}
    export HUGGING_FACE_HUB_TOKEN=hf_your_token_here
    ```

    Reload your shell configuration:

    ```bash theme={null}
    source ~/.bashrc  # or ~/.zshrc
    ```
  </Step>

  <Step title="Verify the Token">
    Confirm the token is set correctly:

    ```bash theme={null}
    echo $HUGGING_FACE_HUB_TOKEN
    ```

    This should print your token starting with `hf_`.
  </Step>
</Steps>

## Using the Token

### Docker Compose

The `docker-compose.yml` file automatically passes the token to services that need it:

```yaml theme={null}
services:
  llm:
    environment:
      - HUGGING_FACE_HUB_TOKEN=$HUGGING_FACE_HUB_TOKEN
  
  tts:
    environment:
      - HUGGING_FACE_HUB_TOKEN=$HUGGING_FACE_HUB_TOKEN
  
  stt:
    environment:
      - HUGGING_FACE_HUB_TOKEN=$HUGGING_FACE_HUB_TOKEN
```

Make sure the environment variable is set in your shell before running `docker compose up`.

### Dockerless Deployment

For dockerless setups, the token is automatically read from your environment when you run the startup scripts:

```bash theme={null}
./dockerless/start_llm.sh   # Uses $HUGGING_FACE_HUB_TOKEN
./dockerless/start_stt.sh
./dockerless/start_tts.sh
```

## Troubleshooting

<AccordionGroup>
  <Accordion title="Token not found error">
    If you see errors about missing authentication:

    1. Verify the token is set: `echo $HUGGING_FACE_HUB_TOKEN`
    2. Make sure you've reloaded your shell after adding it to `.bashrc`
    3. For Docker, ensure the token was set *before* running `docker compose up`
  </Accordion>

  <Accordion title="Access denied to gated model">
    If you get permission errors:

    1. Confirm you've accepted the model conditions on Hugging Face
    2. Wait a few minutes for the permissions to propagate
    3. Verify your token has "Read access to contents of all public gated repos"
  </Accordion>

  <Accordion title="Token with write permissions">
    If you accidentally created a token with write access:

    1. Go to [Settings → Access Tokens](https://huggingface.co/settings/tokens)
    2. Revoke the old token
    3. Create a new fine-grained token with read-only access
    4. Update your environment variable
  </Accordion>
</AccordionGroup>

## Security Best Practices

<Warning>
  Never commit your Hugging Face token to version control. Use environment variables only.
</Warning>

* Use **fine-grained tokens** with minimal permissions
* Create separate tokens for development and production
* Rotate tokens periodically
* Revoke tokens immediately if compromised
* Never use write-access tokens for public deployments


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.