# Realtime API (/connect/realtime-api)

Cuelo serves the [OpenAI Realtime API](https://developers.openai.com/api/reference/resources/realtime/client-events)
(GA) over WebSocket at:

```text
ws://127.0.0.1:4141/v1/realtime
```

Existing Realtime clients and SDKs work unchanged. Point them at Cuelo instead
of `api.openai.com`. Beta clients that send `OpenAI-Beta: realtime=v1` get the
beta event names.

## Audio [#audio]

PCM16, mono, **24 kHz**, base64 in `input_audio_buffer.append` and
`response.output_audio.delta`. The engine resamples internally for each
provider.

## What's different from OpenAI [#whats-different-from-openai]

- **Turn detection is Cuelo's.** The agent's controllers decide end of turn,
  interruptions and backchannels. The client's `turn_detection` settings are
  accepted but not applied. You still get `input_audio_buffer.speech_started`
  and `speech_stopped`.
- **Server tools are invisible.** Tools from `config.cuelo` run on the server.
  The client only hears the result.
- **Interruptions.** When the agent is interrupted, the server sends
  `input_audio_buffer.speech_started` so clients stop playback. Then it ends the
  response with status `cancelled`. Drop any audio you haven't played.
- **Backchannels** arrive as normal output audio while the user is speaking.

## Client events [#client-events]

| Event                        | Supported                                                       |
| ---------------------------- | --------------------------------------------------------------- |
| `session.update`             | `instructions`, `tools`, `audio.output.voice`                   |
| `input_audio_buffer.append`  | yes                                                             |
| `input_audio_buffer.commit`  | yes: asks the agent to reply now (push-to-talk)                 |
| `input_audio_buffer.clear`   | acknowledged with `input_audio_buffer.cleared`                  |
| `conversation.item.create`   | user text messages, and `function_call_output` for client tools |
| `conversation.item.truncate` | acknowledged; the engine tracks interrupted replies itself      |
| `response.create`            | yes: makes the agent speak, e.g. first, before the user         |
| `response.cancel`            | yes                                                             |

## Server events [#server-events]

`session.created`, `session.updated`, `conversation.item.added`/`done`,
`input_audio_buffer.speech_started`/`speech_stopped`/`committed`/`cleared`,
`conversation.item.input_audio_transcription.completed`, `response.created`,
`response.output_item.added`/`done`, `response.content_part.added`/`done`,
`response.output_audio.delta`/`done`, `response.output_audio_transcript.delta`/`done`,
`response.function_call_arguments.done`, `response.done` and `error`.

## Tools [#tools]

Tools a client declares in `session.update` run in the client:

1. The response ends with a `function_call` output item (`call_id`, `name`,
   `arguments`).
2. The client runs it and sends:

   ```json
   {
     "type": "conversation.item.create",
     "item": { "type": "function_call_output", "call_id": "call_123", "output": "{\"ok\":true}" }
   }
   ```

3. The agent continues on its own. A following `response.create` is accepted
   and ignored.

## Examples [#examples]

<Tabs items={["Python (openai)","Browser"]}>
<Tab value="Python (openai)">
```python
import asyncio, base64
from openai import AsyncOpenAI

async def main():
    client = AsyncOpenAI(base_url="http://127.0.0.1:4141/v1", api_key="unused")
    async with client.realtime.connect(model="cuelo") as conn:
        await conn.session.update(session={
            "type": "realtime",
            "instructions": "You are a concise barista.",
        })
        await conn.response.create()          # speak first
        async for event in conn:
            if event.type == "response.output_audio.delta":
                play(base64.b64decode(event.delta))   # PCM16 24 kHz
            elif event.type == "input_audio_buffer.speech_started":
                stop_playback()                       # interrupted

asyncio.run(main())
```
</Tab>

<Tab value="Browser">
```js
// Browsers can't set WebSocket headers: the API key (if any) goes in a subprotocol.
const protocols = ["realtime"];
if (key) protocols.push(`openai-insecure-api-key.${key}`);
const ws = new WebSocket("wss://agent.example.com/v1/realtime", protocols);

ws.onopen = () => {
  ws.send(JSON.stringify({ type: "session.update",
    session: { instructions: "You are a concise barista." } }));
};
ws.onmessage = (e) => {
  const ev = JSON.parse(e.data);
  if (ev.type === "response.output_audio.delta") playPcm16(ev.delta);
  if (ev.type === "input_audio_buffer.speech_started") flushPlayback();
};
// mic: capture at 24 kHz, convert to PCM16, base64, then
// ws.send(JSON.stringify({ type: "input_audio_buffer.append", audio }))
```
</Tab>
</Tabs>

The dashboard's Talk page (`observer/src/demo/realtime.ts`) is a complete
browser client with AudioWorklets for capture and playback, a good starting
point.

## Turning it off [#turning-it-off]

`serve.api: none` serves only your own routes from `src/api.py`. See
[Custom endpoints](/connect/custom-endpoints).