> ## Documentation Index
> Fetch the complete documentation index at: https://docs.bota.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Multi-Modal AI (Coming Soon)

> Transcription, summarization, and visual AI processing for Bota recordings

<Info>
  Multi-Modal AI features are **Coming Soon** and not yet available.
</Info>

Bota's AI pipeline processes recordings through three stages: **transcription** (audio → text), **summarization** (text → structured output), and **multi-modal analysis** (text + visual media → context-aware output).

## Transcription

Bota transcribes audio recordings using Automatic Speech Recognition (ASR). Transcription is asynchronous — you submit a job and receive results via webhook or polling.

```mermaid theme={null}
sequenceDiagram
    participant You as Your Server
    participant Bota as Bota API
    participant ASR as ASR Provider

    You->>Bota: Create transcription job
    Bota-->>You: 201 (status: pending)
    Bota->>ASR: Send audio for processing
    ASR-->>Bota: Transcription result
    Bota-->>You: Webhook: transcription.completed
    You->>Bota: Fetch transcription
    Bota-->>You: Full transcript with segments
```

1. **Upload a recording** — see [Quickstart](/quickstart)
2. **Create a transcription** — specify the recording and optional language hint
3. **Wait for completion** — poll or listen for the `transcription.completed` webhook
4. **Retrieve results** — structured output with timestamps, speaker labels, and confidence scores

### Output Format

A completed transcription includes a full text string and time-stamped segments with speaker diarization:

```json theme={null}
{
  "id": "txn_abc123",
  "status": "completed",
  "full_text": "Good morning. I'd like to discuss the project timeline...",
  "segments": [
    {
      "start": 0.0,
      "end": 1.2,
      "text": "Good morning.",
      "speaker": "Speaker 1",
      "confidence": 0.95
    },
    {
      "start": 1.5,
      "end": 4.8,
      "text": "I'd like to discuss the project timeline.",
      "speaker": "Speaker 1",
      "confidence": 0.92
    }
  ],
  "word_count": 42,
  "confidence": 0.93,
  "language": "en"
}
```

Each segment includes:

| Field           | Description                                    |
| --------------- | ---------------------------------------------- |
| `start` / `end` | Timestamps in seconds                          |
| `text`          | Transcribed text for this segment              |
| `speaker`       | Speaker label (e.g., `Speaker 1`, `Speaker 2`) |
| `confidence`    | Per-segment confidence score (0–1)             |

### ASR Providers

| Provider     | Best For                              |
| ------------ | ------------------------------------- |
| `whisper`    | General purpose, multilingual support |
| `deepgram`   | Low latency, real-time processing     |
| `assemblyai` | Speaker diarization, content analysis |
| `elevenlabs` | High accuracy transcription           |

You can specify a provider when creating a transcription, or let Bota use the default configured for your project.

### Language Support

Transcription supports 50+ languages. Provide a language hint (e.g., `en`, `es`, `zh`) to improve accuracy, or omit it for automatic detection.

### Transcription API Reference

* [Create Transcription](/api-reference/ai/transcriptions/create)
* [Get Transcription](/api-reference/ai/transcriptions/get)

## Summarization

Bota generates structured summaries from transcriptions using LLM providers. Use built-in templates for common formats (SOAP notes, sales calls, legal memos) or provide custom prompts.

```mermaid theme={null}
sequenceDiagram
    participant You as Your Server
    participant Bota as Bota API
    participant LLM as LLM Provider

    You->>Bota: Create summary job
    Bota-->>You: 201 (status: pending)
    Bota->>LLM: Send transcript + template
    LLM-->>Bota: Structured summary
    Bota-->>You: Webhook: summary.completed
    You->>Bota: Fetch summary
    Bota-->>You: Structured output
```

### Templates vs Custom Prompts

* **Template** — Use a built-in template for standardized, structured output. Best for repeatable workflows.
* **Custom Prompt** — Provide your own instructions for flexible, ad-hoc summarization.

<Note>
  Provide either a template or a custom prompt, not both.
</Note>

### Built-in Templates

#### General Notes

Ideal for meetings, discussions, and team syncs. Extracts key points, action items, decisions, and participants.

```json theme={null}
{
  "overview": "Team discussed Q2 roadmap priorities...",
  "key_points": ["Launch new API version by March", "Hire 2 engineers"],
  "action_items": [
    { "task": "Draft API migration guide", "owner": "Sarah", "deadline": "2025-02-01" }
  ],
  "decisions": ["Postpone mobile app to Q3"],
  "participants": ["Sarah", "Mike", "Lisa"]
}
```

#### Sales Call

Captures pain points, budget, next steps, and deal sentiment from sales conversations.

```json theme={null}
{
  "pain_points": ["Current solution too slow", "No API access"],
  "budget": { "range": "$50k-75k", "timeline": "Q2 2025" },
  "next_steps": ["Send proposal by Friday", "Schedule demo with CTO"],
  "key_quotes": [
    { "quote": "We need this integrated by April", "speaker": "Prospect", "context": "Timeline discussion" }
  ],
  "sentiment": "positive",
  "deal_probability": 0.7
}
```

#### Clinical SOAP

Generates structured SOAP notes from healthcare encounters.

```json theme={null}
{
  "chief_complaint": "Patient reports persistent lower back pain for 2 weeks",
  "subjective": "Pain rated 6/10, worse with sitting...",
  "objective": "BP 120/80, ROM limited in lumbar flexion...",
  "assessment": "Lumbar strain, likely mechanical origin",
  "plan": "Physical therapy 2x/week, NSAIDs as needed, follow up in 2 weeks"
}
```

#### Legal Memo

Summarizes legal proceedings, depositions, and client meetings into structured memos with facts, issues, and analysis.

#### Template Reference

| Template      | ID                   | Use Case              |
| ------------- | -------------------- | --------------------- |
| General Notes | `tmpl_general_notes` | Meetings, discussions |
| Sales Call    | `tmpl_sales_call`    | Sales conversations   |
| Clinical SOAP | `tmpl_clinical_soap` | Healthcare encounters |
| Legal Memo    | `tmpl_legal_memo`    | Legal proceedings     |

### LLM Providers

| Provider | Best For                              |
| -------- | ------------------------------------- |
| `gemini` | Fast processing, good general quality |
| `openai` | High accuracy, structured output      |
| `claude` | Nuanced analysis, long transcripts    |

### Summarization API Reference

* [Create Summary](/api-reference/ai/summaries/create)
* [Get Summary](/api-reference/ai/summaries/get)

## Multi-Modal Analysis

Multi-Modal extends the pipeline with visual context from the [Bota Pin Pro](/hardware/pin-pro). The Pin Pro captures images and video alongside audio, enabling AI that understands both what was said and what was seen.

```mermaid theme={null}
sequenceDiagram
    participant Device as Pin Pro
    participant Bota as Bota API
    participant AI as AI Provider

    Device->>Bota: Upload audio recording
    Device->>Bota: Upload images & video
    Note over Bota: Audio + media linked to same recording
    Bota->>AI: Transcript + visual context
    AI-->>Bota: Multi-modal results
    Bota-->>Device: Webhook notification
```

### Media Types

| Type            | Format      | Best For                                                                  |
| --------------- | ----------- | ------------------------------------------------------------------------- |
| **Images**      | JPEG, PNG   | Periodic snapshots, whiteboard captures, document scans, equipment photos |
| **Video clips** | MP4 (H.264) | Short scene captures, demonstrations, walkthroughs                        |

Media is captured based on configurable triggers:

| Trigger      | Description                                                          |
| ------------ | -------------------------------------------------------------------- |
| **Periodic** | Capture at fixed intervals (e.g., every 30 seconds, every 5 minutes) |
| **Motion**   | Capture when significant scene change is detected                    |
| **Manual**   | Capture on button press                                              |

### Video Summary

Generates a visual summary from video clips by identifying key frames, generating captions, and producing a timeline of visual highlights. Useful for quickly reviewing long recordings without watching the entire video.

See [Create Video Summary](/api-reference/ai/video-summaries/create) for the API reference.

### Use Cases

<CardGroup cols={2}>
  <Card title="Field Inspection" icon="clipboard-check">
    Inspector narrates findings while the camera captures equipment and damage. Video summary highlights key visual moments alongside the transcript.
  </Card>

  <Card title="Clinical Encounter" icon="stethoscope">
    Doctor-patient conversation captured alongside video of the examination. Transcript + video summary provide a complete record.
  </Card>

  <Card title="Meeting + Whiteboard" icon="chalkboard">
    Discussion transcript combined with video of whiteboard diagrams. Video summary extracts key frames for quick review.
  </Card>

  <Card title="Training Session" icon="graduation-cap">
    Trainer's spoken instructions paired with video of demonstrations. Video summary creates a visual timeline of the session.
  </Card>
</CardGroup>

## End-to-End Flow

A typical multi-modal workflow:

1. **Record** — End user wears Pin Pro, presses button to start. Audio records continuously; camera captures video.
2. **Upload** — Device uploads audio and video via the [Upload URL](/api-reference/uploads/create-url) endpoint (repeated per file), then calls [Complete Upload](/api-reference/uploads/complete).
3. **Transcribe** — Create a [transcription](/api-reference/ai/transcriptions/create) from the audio.
4. **Summarize** — Create a [summary](/api-reference/ai/summaries/create) from the transcript.
5. **Video Summary** — Create a [video summary](/api-reference/ai/video-summaries/create) for visual highlights.
6. **Deliver** — Results delivered via webhook or polling.

## BYO API Keys

All AI processing supports bringing your own provider API keys. This gives you control over costs, rate limits, and model selection.

1. Register your provider API key through the [Integrations API](/api-reference/introduction)
2. Test the key to verify it works
3. Bota automatically uses your key when you select that provider

Keys are encrypted at rest (AES-256-GCM) and never exposed in API responses. You can rotate or delete keys at any time.

## Webhooks

| Event                     | Description                         |
| ------------------------- | ----------------------------------- |
| `transcription.completed` | Transcription finished successfully |
| `transcription.failed`    | Transcription encountered an error  |
| `summary.completed`       | Summary generated successfully      |
| `summary.failed`          | Summary encountered an error        |

See [Webhook Events](/webhooks/events) for payload details.

## Related

* [Bota Pin Pro](/hardware/pin-pro) — Multi-modal hardware
* [Upload API Reference](/api-reference/uploads/create-url) — File upload documentation
* [Streaming Upload](/guides/streaming-upload) — Upload while recording for faster processing