01Start
Concepts
The pieces of a Cuelo agent, the life of one turn, and where the latency goes.
The pieces
| Piece | What it does | Lives in |
|---|---|---|
| Endpoint | Speaks a wire protocol (OpenAI Realtime, raw PCM, your own) and turns it into harness calls. | cuelo.endpoint |
| Harness | Opens one Session per call, builds a fresh Pipeline for it, records events and audio. | cuelo.runtime |
| STT | Streaming speech-to-text. Audio goes in continuously and interim and final transcripts come out. | cuelo.stt |
| Controllers | Turn-taking models. They listen to user audio, transcripts and (duplex ones) the agent's audio, and emit signals. | cuelo.controllers |
| Policy | Turns signals into actions: respond, stop talking, keep talking, play a backchannel. | cuelo.policies |
| LLM | The talker. It streams the reply text and can call tools or consult the thinker. | cuelo.llms |
| TTS | Streams audio for the reply as text arrives. | cuelo.tts |
| Store | SQLite, Postgres or MySQL: sessions, turns, events, summaries. Recordings go to disk. | cuelo.database |
Each piece is a slot with a small Python interface. Providers implement it, and you can swap any of them in config or in code. See Python API.
The life of a turn
- Audio in. The client streams PCM16 at 24 kHz. The harness resamples it for each consumer (16 kHz for most STT and controllers) and fans it out.
- Transcribe. The STT streams interim results (
stt.partial) and finals (stt.final). - Decide. Every 20 to 80 ms each controller scores the audio. When the
end-of-turn model is confident the user is done, it emits
turn_end. The policy then asks the STT to finalize, which flushes the transcript for the turn right away instead of waiting for the provider's own endpointing. - Think. The LLM gets the system prompt, the conversation and the tools.
Text streams back as
llm.delta. Tool calls run (Python, HTTP or forwarded to the client) and the model continues with their results. - Speak. Text streams into the TTS. Streaming providers take tokens as they arrive. Non-streaming ones get one sentence at a time. Agent audio streams back to the client.
- Listen while speaking. While the agent talks, the barge-in detector tells
a real interruption ("wait, no") from a backchannel ("yeah"). On an
interruption the harness cancels LLM and TTS, emits
agent.interrupted, and the client drops any audio it hasn't played. - Backchannel. While the user talks, the backchannel model can fire. The
agent then plays a short "mhm" from a cache of pre-synthesized phrases in its
own voice (
assets/phrases/), so a backchannel costs no TTS round trip.
Every step emits an event. Events are stored, streamed
to the dashboard and passed to your on_event hook.
Where latency goes
The dashboard splits each agent turn into stages. The number that matters is TTFB: from the moment the user stopped speaking to the first agent audio.
| Stage | Meaning | Rough range |
|---|---|---|
vad_tail | Silence the end-of-turn model needed before it was sure. | 100–200 ms |
finalize | STT final transcript after the finalize request. | 50–150 ms |
llm_ttft | LLM time to first token. | 150–400 ms (Cerebras) |
tts_ttfa | TTS time to first audio for the first sentence. | 100–250 ms |
ttfb | All of the above, from user stop to first agent audio. | 500–900 ms |
The ranges are a rough guide. They depend on providers, region and prompt
length, so measure your own in the dashboard. Tool calls and the thinker add
their own time, shown as tool. Use a filler
("let me check") so the caller isn't left in silence. See
Thinker.
Concurrency
The engine is asyncio. Controllers are synchronous model code, and the harness
runs each one on its own worker thread. One process handles many calls. The
heavy part is the turn-taking models: the MaAI stack costs about 23 ms of CPU
per 80 ms step per call. Run cuelo machine check to measure your hardware,
and see Deploy for workers.