Voices and Characters

Customize conversation personalities and voice characteristics in Unmute

Unmute allows you to create custom characters with unique voices and personalities. Characters are defined in the voices.yaml configuration file, which combines voice samples with conversation instructions.

Understanding Character Configuration

Each character in Unmute consists of two main components:

1. Voice Sample: An audio file that defines how the character sounds
2. Instructions: System prompts that define the character’s personality and behavior

The voices.yaml File

The voices.yaml file is located in the root of the Unmute repository. Each character is defined with the following structure:

- name: Watercooler
  good: true
  instructions:
    type: smalltalk
  source:
    source_type: file
    path_on_server: unmute-prod-website/p329_022.wav
    description: From the Device Recorded VCTK dataset.
    description_link: https://datashare.ed.ac.uk/handle/10283/3038

Configuration Fields

name · string · required

Display name for the character that appears in the UI

good · boolean

Quality flag - set to true to mark as production-ready, false for testing, or null for undecided

comment · string

Internal notes about the character (e.g., “man, UK, skeptical”)

instructions · object · required

Defines the character’s conversation style and personality. See Instruction Types below.

source · object · required

Specifies the voice audio source. Can be either file or freesound type.

Instruction Types

Unmute provides several built-in instruction types that define character behavior:

Smalltalk

For general conversation with context-aware greetings:

instructions:
  type: smalltalk
  language: en  # optional: en, fr, en/fr, or fr/en

This type automatically includes the current date, time, and timezone in the system prompt, and randomly selects conversation starters.

Constant

For custom character personalities with specific instructions:

instructions:
  type: constant
  text: "Offer life advice. Be kind and sympathetic. Your name is Gertrude."
  language: en  # optional

Example - Gertrude (Life Advice Counselor):

- name: Gertrude
  good: true
  instructions:
    type: constant
    text: Offer life advice. Be kind and sympathetic. Your name is Gertrude.
  source:
    source_type: file
    path_on_server: unmute-prod-website/gertrude.wav

Quiz Show

For a game show host personality:

instructions:
  type: quiz_show
  language: en  # optional

Randomly selects 5 questions from a predefined question bank and acts as a snarky quiz show host.

News

For tech news discussions using live headlines from The Verge:

instructions:
  type: news
  language: en  # optional

Requires the NEWSAPI_API_KEY environment variable to be set.

Guess Animal

For playing a guessing game:

instructions:
  type: guess_animal
  language: en  # optional

Unmute Explanation

For answering questions about Unmute itself:

instructions:
  type: unmute_explanation

Voice Sources

Unmute supports multiple ways to source voice samples:

File-based Voices

Reference voice files stored on the server:

source:
  source_type: file
  path_on_server: unmute-prod-website/developer-1.mp3
  description: This is the voice of a Kyutai developer
  description_link: https://example.com  # optional

Freesound Voices

Automatically download voices from Freesound.org:

source:
  source_type: freesound
  url: https://freesound.org/people/InspectorJ/sounds/519189/
  path_on_server: unmute-prod-website/freesound/519189.mp3

Available Voices

Browse the Kyutai TTS voice repository for hundreds of available voices, including:

• Voices from open datasets (VCTK, Expresso, etc.)
• Community-donated voices from the Voice Donation Project
• Freesound.org voice samples

To use a voice from the repository, set the path_on_server to the relative path:

path_on_server: voice-donations/Haku.wav

Adding a New Character

1. Choose or Create a Voice Sample

   Select a voice from the voice repository or prepare your own 10-second audio sample in WAV or MP3 format.

2. Edit voices.yaml

   Open voices.yaml in the root of the Unmute repository and add your character definition:

   - name: My Character
     good: true
     instructions:
       type: constant
       text: You are a friendly robot assistant. Be helpful and enthusiastic.
     source:
       source_type: file
       path_on_server: voice-donations/robot-voice.wav
   
3. Restart the Backend

   The voices.yaml file is cached when the backend starts:

   # For Docker Compose
   docker compose restart backend
   
   # For dockerless deployment
   # Stop and restart the backend service
   
4. Test Your Character

   Open Unmute in your browser and select your new character from the voice selector.

Language Support

The TTS model supports English and French. Configure language support using the language field:

• en: Primarily English (default)
• fr: Primarily French
• en/fr: Fluent in both, English preference
• fr/en: Fluent in both, French preference

Example for a bilingual character:

- name: Bilingual Assistant
  good: true
  instructions:
    type: smalltalk
    language: en/fr
  source:
    source_type: file
    path_on_server: unmute-prod-website/bilingual.wav

Advanced Customization

For more complex character behaviors, you can create custom instruction types by modifying unmute/llm/system_prompt.py. See the existing instruction classes for examples:

• SmalltalkInstructions: Adds dynamic date/time context
• QuizShowInstructions: Randomly selects questions
• NewsInstructions: Fetches live news data

Each instruction type must implement a make_system_prompt() method that returns the final system prompt string.

Voice Quality Tips

For best results, voice samples should:

• Be 5-10 seconds long
• Contain clear speech without background noise
• Be in WAV or MP3 format at 24kHz (will be resampled if needed)
• Represent the desired speaking style

The voices.yaml file is only loaded when the backend starts. You must restart the backend service after making changes for them to take effect.