> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/kyutai-labs/unmute/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitoring

> Track Unmute's performance and health with Prometheus and Grafana

## Overview

Unmute includes comprehensive monitoring capabilities using Prometheus for metrics collection and Grafana for visualization. The monitoring stack tracks latency, throughput, error rates, and resource utilization across all services.

## Architecture

```mermaid theme={null}
graph LR
    B[Backend] -->|/metrics| P[Prometheus]
    STT[STT Service] -->|/metrics| P
    TTS[TTS Service] -->|/metrics| P
    LLM[LLM Service] -->|/metrics| P
    T[Traefik] -->|/metrics| P
    P -->|Query| G[Grafana]
    User -->|View Dashboards| G
```

Prometheus scrapes metrics from all services and stores time-series data. Grafana queries Prometheus to display real-time dashboards.

## Metrics Collection

### Available Metrics

Unmute exposes metrics via the Prometheus client library (defined in `unmute/metrics.py`):

#### Session Metrics

```python theme={null}
SESSIONS = Counter("worker_sessions", "")  # Total sessions
ACTIVE_SESSIONS = Gauge("worker_active_sessions", "")  # Current connections
SESSION_DURATION = Histogram("worker_session_duration", "")
```

#### STT (Speech-to-Text) Metrics

```python theme={null}
STT_SESSIONS = Counter("worker_stt_sessions", "")
STT_ACTIVE_SESSIONS = Gauge("worker_stt_active_sessions", "")
STT_TTFT = Histogram("worker_stt_ttft", "")  # Time-to-first-token
STT_PING_TIME = Histogram("worker_stt_ping_time", "")
STT_RECV_WORDS = Counter("worker_stt_recv_words", "")
STT_MISSES = Counter("worker_stt_misses", "")  # Connection failures
```

#### TTS (Text-to-Speech) Metrics

```python theme={null}
TTS_SESSIONS = Counter("worker_tts_sessions", "")
TTS_ACTIVE_SESSIONS = Gauge("worker_tts_active_sessions", "")
TTS_TTFT = Histogram("worker_tts_ttft", "")  # Time-to-first-audio
TTS_INTERRUPT = Counter("worker_tts_interrupt", "")  # User interruptions
TTS_AUDIO_DURATION = Histogram("worker_tts_audio_duration", "")
TTS_GEN_DURATION = Histogram("worker_tts_gen_duration", "")
```

#### LLM (Language Model) Metrics

```python theme={null}
VLLM_SESSIONS = Counter("worker_vllm_sessions", "")
VLLM_ACTIVE_SESSIONS = Gauge("worker_vllm_active_sessions", "")
VLLM_TTFT = Histogram("worker_vllm_ttft", "")  # Time-to-first-token
VLLM_REQUEST_LENGTH = Histogram("worker_vllm_request_length", "")
VLLM_REPLY_LENGTH = Histogram("worker_vllm_reply_length", "")
VLLM_INTERRUPTS = Counter("worker_vllm_interrupt", "")
```

#### Error Metrics

```python theme={null}
HARD_ERRORS = Counter("worker_hard_errors", "")  # Fatal errors
SERVICE_MISSES = Counter("worker_service_misses", "")  # Service unavailable
FATAL_SERVICE_MISSES = Counter("worker_fatal_service_misses", "")
FORCE_DISCONNECTS = Counter("worker_force_disconnects", "")
```

### Histogram Buckets

Metrics use predefined buckets for accurate percentile calculations:

```python theme={null}
# STT latency buckets (milliseconds)
TTFT_BINS_STT_MS = [10.0, 15.0, 25.0, 50.0, 75.0, 100.0]

# TTS latency buckets (milliseconds)  
TTFT_BINS_TTS_MS = [200.0, 250.0, 300.0, 350.0, 400.0, 450.0, 500.0, 550.0]

# LLM latency buckets (milliseconds)
TTFT_BINS_VLLM_MS = [50.0, 75.0, 100.0, 150.0, 200.0, 250.0, 300.0, 400.0, 500.0, 750.0, 1000.0]

# Session duration buckets (seconds)
SESSION_DURATION_BINS = [1.0, 10.0, 30.0, 60.0, 120.0, 240.0, 480.0, 960.0, 1920.0]
```

## Prometheus Setup

### Docker Compose Configuration

For production deployments, add Prometheus to your Docker Swarm stack:

```yaml theme={null}
prometheus:
  image: prom/prometheus:latest
  volumes:
    - ./services/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
    - /var/run/docker.sock:/var/run/docker.sock:ro
    - prometheus-data:/prometheus
  command:
    - '--config.file=/etc/prometheus/prometheus.yml'
    - '--storage.tsdb.path=/prometheus'
  ports:
    - "9090:9090"
```

### Prometheus Configuration

Create `services/prometheus/prometheus.yml`:

```yaml theme={null}
scrape_configs:
  - job_name: 'dockerswarm'
    scrape_interval: 5s
    dockerswarm_sd_configs:
      - host: unix:///var/run/docker.sock
        role: tasks
    relabel_configs:
      # Keep only running tasks
      - source_labels: [__meta_dockerswarm_task_desired_state]
        regex: running
        action: keep
      
      # Keep only tasks with prometheus-port label
      - source_labels: [__meta_dockerswarm_service_label_prometheus_port]
        regex: .+
        action: keep
      
      # Set job name from service name
      - source_labels: [__meta_dockerswarm_service_name]
        regex: .*_(.+)
        replacement: $1
        target_label: job
      
      # Set scrape address from prometheus-port label
      - source_labels: [__address__, __meta_dockerswarm_service_label_prometheus_port]
        regex: ([^:]+):\d+;(\d+)
        replacement: $1:$2
        target_label: __address__
```

### Service Labels

Label services to expose metrics:

```yaml theme={null}
backend:
  labels:
    - "prometheus-port=80"  # Backend exposes metrics on port 80

traefik:
  labels:
    - "prometheus-port=8080"  # Traefik metrics on port 8080
```

## Grafana Setup

### Docker Configuration

```yaml theme={null}
grafana:
  image: grafana/grafana:latest
  volumes:
    - grafana-data:/var/lib/grafana
    - ./services/grafana/grafana.ini:/etc/grafana/grafana.ini
    - ./services/grafana/provisioning:/etc/grafana/provisioning
    - ./services/grafana/dashboards:/etc/grafana/dashboards
  environment:
    - GF_SECURITY_ADMIN_PASSWORD=your_secure_password
  ports:
    - "3000:3000"
  depends_on:
    - prometheus
```

### Data Source Configuration

Create `services/grafana/provisioning/datasources/datasources.yaml`:

```yaml theme={null}
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    orgId: 1
    url: http://prometheus:9090
    isDefault: true
    editable: true
```

### Dashboard Provisioning

Create `services/grafana/provisioning/dashboards/dashboards.yaml`:

```yaml theme={null}
apiVersion: 1

providers:
  - name: 'default'
    orgId: 1
    folder: ''
    type: file
    disableDeletion: false
    editable: true
    updateIntervalSeconds: 10
    options:
      path: /etc/grafana/dashboards
```

## Key Dashboards

### System Overview Dashboard

**Active Sessions:**

```promql theme={null}
worker_active_sessions
```

**Total Sessions (Rate):**

```promql theme={null}
rate(worker_sessions_total[5m])
```

**Error Rate:**

```promql theme={null}
rate(worker_hard_errors_total[5m])
```

### Latency Dashboard

**STT Latency (p95):**

```promql theme={null}
histogram_quantile(0.95, rate(worker_stt_ttft_bucket[5m]))
```

**TTS Latency (p95):**

```promql theme={null}
histogram_quantile(0.95, rate(worker_tts_ttft_bucket[5m]))
```

**LLM Latency (p95):**

```promql theme={null}
histogram_quantile(0.95, rate(worker_vllm_ttft_bucket[5m]))
```

**Average STT Latency:**

```promql theme={null}
rate(worker_stt_ttft_sum[5m]) / rate(worker_stt_ttft_count[5m])
```

### Throughput Dashboard

**STT Words Per Second:**

```promql theme={null}
rate(worker_stt_recv_words_total[5m])
```

**TTS Words Per Second:**

```promql theme={null}
rate(worker_tts_recv_words_total[5m])
```

**LLM Tokens Per Second:**

```promql theme={null}
rate(worker_vllm_recv_words_total[5m])
```

### Service Health Dashboard

**STT Service Misses:**

```promql theme={null}
rate(worker_stt_misses_total[5m])
```

**TTS Service Misses:**

```promql theme={null}
rate(worker_tts_misses_total[5m])
```

**Active STT Sessions:**

```promql theme={null}
worker_stt_active_sessions
```

**Active TTS Sessions:**

```promql theme={null}
worker_tts_active_sessions
```

**Active LLM Sessions:**

```promql theme={null}
worker_vllm_active_sessions
```

### User Behavior Dashboard

**Session Duration (Average):**

```promql theme={null}
rate(worker_session_duration_sum[5m]) / rate(worker_session_duration_count[5m])
```

**Interruption Rate:**

```promql theme={null}
rate(worker_tts_interrupt_total[5m])
```

**Average Request Length:**

```promql theme={null}
rate(worker_vllm_request_length_sum[5m]) / rate(worker_vllm_request_length_count[5m])
```

**Average Reply Length:**

```promql theme={null}
rate(worker_vllm_reply_length_sum[5m]) / rate(worker_vllm_reply_length_count[5m])
```

## Accessing Dashboards

### Local Development

Access Grafana at `http://localhost:3000` (default credentials: admin/admin).

### Production Deployment

For [unmute.sh](https://unmute.sh) deployment with Traefik:

```yaml theme={null}
grafana:
  deploy:
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.grafana.rule=Host(`grafana.${DOMAIN}`)"
      - "traefik.http.routers.grafana.middlewares=traefik-forward-auth"
      - "traefik.http.routers.grafana.entrypoints=websecure"
      - "traefik.http.routers.grafana.tls=true"
      - "traefik.http.services.grafana.loadbalancer.server.port=3000"
```

Access at: `https://grafana.unmute.sh`

## Alerting

### Example Alert Rules

Create `services/prometheus/alerts.yml`:

```yaml theme={null}
groups:
  - name: unmute_alerts
    interval: 30s
    rules:
      - alert: HighErrorRate
        expr: rate(worker_hard_errors_total[5m]) > 0.1
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate detected"
          description: "Error rate is {{ $value }} errors/sec"
      
      - alert: HighLatency
        expr: histogram_quantile(0.95, rate(worker_tts_ttft_bucket[5m])) > 1.0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High TTS latency"
          description: "P95 TTS latency is {{ $value }}s"
      
      - alert: ServiceUnavailable
        expr: rate(worker_fatal_service_misses_total[5m]) > 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Backend services unavailable"
```

Add to Prometheus configuration:

```yaml theme={null}
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093

rule_files:
  - /etc/prometheus/alerts.yml
```

## Health Checks

### Backend Health Endpoint

Unmute exposes a health check endpoint:

```bash theme={null}
curl http://localhost:8000/v1/health
```

**Response:**

```json theme={null}
{
  "ok": true,
  "services": {
    "stt": "available",
    "tts": "available",
    "llm": "available"
  }
}
```

### Load Testing

Use the built-in load test client to validate monitoring:

```bash theme={null}
uv run unmute/loadtest/loadtest_client.py \
  --server-url ws://localhost:8000 \
  --n-workers 10 \
  --n-conversations 50
```

Watch metrics in Grafana during the test to verify collection.

## Production Monitoring URLs

From [unmute.sh](https://unmute.sh) deployment:

* **Main app:** [https://unmute.sh](https://unmute.sh)
* **Grafana:** [https://grafana.unmute.sh](https://grafana.unmute.sh)
* **Prometheus:** [https://prometheus.unmute.sh](https://prometheus.unmute.sh)
* **Traefik:** [https://traefik.unmute.sh](https://traefik.unmute.sh)
* **Portainer:** [https://portainer.unmute.sh](https://portainer.unmute.sh)

All monitoring services are protected by OAuth authentication (Google) via Traefik Forward Auth.

## Best Practices

1. **Set appropriate scrape intervals:** 5s for real-time, 30s for cost savings
2. **Use retention policies:** Configure Prometheus to retain data for 30-90 days
3. **Monitor percentiles, not just averages:** p95 and p99 reveal tail latencies
4. **Set up alerts:** Proactive notification prevents outages
5. **Archive long-term data:** Export to long-term storage (e.g., S3) for historical analysis

## Troubleshooting

### Metrics not appearing

**Check:**

1. Service has `prometheus-port` label
2. Prometheus can reach the service (check targets page)
3. Metrics endpoint returns data: `curl http://backend/metrics`

### High cardinality warnings

**Cause:** Too many unique label combinations

**Solution:** Avoid using user IDs or session IDs as labels. Use counters instead.

### Missing histograms

**Check:** Bucket configuration matches expected latency ranges. Add buckets if values exceed defined ranges.

## Next Steps

* [Performance Tuning](/guides/performance-tuning) - Optimize based on metrics
* [Debugging](/guides/debugging) - Use metrics to identify issues
* [Multi-GPU Setup](/guides/multi-gpu) - Monitor GPU-specific metrics


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.