Realtime API

Cuelo serves the OpenAI Realtime API (GA) over WebSocket at:

ws://127.0.0.1:4141/v1/realtime

Existing Realtime clients and SDKs work unchanged. Point them at Cuelo instead of api.openai.com. Beta clients that send OpenAI-Beta: realtime=v1 get the beta event names.

AudioCopy Anchor Link

PCM16, mono, 24 kHz, base64 in input_audio_buffer.append and response.output_audio.delta. The engine resamples internally for each provider.

What's different from OpenAICopy Anchor Link

• Turn detection is Cuelo's. The agent's controllers decide end of turn, interruptions and backchannels. The client's turn_detection settings are accepted but not applied. You still get input_audio_buffer.speech_started and speech_stopped.
• Server tools are invisible. Tools from config.cuelo run on the server. The client only hears the result.
• Interruptions. When the agent is interrupted, the server sends input_audio_buffer.speech_started so clients stop playback. Then it ends the response with status cancelled. Drop any audio you haven't played.
• Backchannels arrive as normal output audio while the user is speaking.

Client eventsCopy Anchor Link

Server eventsCopy Anchor Link

session.created, session.updated, conversation.item.added/done, input_audio_buffer.speech_started/speech_stopped/committed/cleared, conversation.item.input_audio_transcription.completed, response.created, response.output_item.added/done, response.content_part.added/done, response.output_audio.delta/done, response.output_audio_transcript.delta/done, response.function_call_arguments.done, response.done and error.

ToolsCopy Anchor Link

Tools a client declares in session.update run in the client:

1. The response ends with a function_call output item (call_id, name, arguments).

2. The client runs it and sends:

   Copy Text

   {
     "type": "conversation.item.create",
     "item": { "type": "function_call_output", "call_id": "call_123", "output": "{\"ok\":true}" }
   }

3. The agent continues on its own. A following response.create is accepted and ignored.

ExamplesCopy Anchor Link

import asyncio, base64
from openai import AsyncOpenAI

async def main():
    client = AsyncOpenAI(base_url="http://127.0.0.1:4141/v1", api_key="unused")
    async with client.realtime.connect(model="cuelo") as conn:
        await conn.session.update(session={
            "type": "realtime",
            "instructions": "You are a concise barista.",
        })
        await conn.response.create()          # speak first
        async for event in conn:
            if event.type == "response.output_audio.delta":
                play(base64.b64decode(event.delta))   # PCM16 24 kHz
            elif event.type == "input_audio_buffer.speech_started":
                stop_playback()                       # interrupted

asyncio.run(main())

The dashboard's Talk page (observer/src/demo/realtime.ts) is a complete browser client with AudioWorklets for capture and playback, a good starting point.

Turning it offCopy Anchor Link

serve.api: none serves only your own routes from src/api.py. See Custom endpoints.