# Add a turn-taking model (/contributing/controllers)

Turn-taking models run locally inside the engine. Integrating one means three
things:

1. a **catalog entry** in `schema/models.json` that pins the weights,
2. a **controller** in `engine/src/cuelo/controllers/` that runs the model and
   emits signals,
3. optionally, **config wiring**, so `turns:` in `config.cuelo` can select it.

## 1. Catalog entry [#1-catalog-entry]

`schema/models.json` is the shared list of every model Cuelo knows about. The
CLI (`cuelo models`), the engine (`weights.py`) and the [Models](/reference/models)
page all read it.

```json title="schema/models.json"
{
  "name": "acme-turn",
  "role": "turn",
  "description": "Streaming end of turn from audio, a score every 40 ms.",
  "status": "evaluating",
  "config": "turns.controller: acme-turn",
  "repo": "acme/acme-turn",
  "revision": "3f9c2e…full commit sha…",
  "license": "Apache-2.0",
  "languages": "English, Spanish",
  "params": "8M",
  "cpu": "~2 ms / 40 ms step",
  "files": [{ "path": "onnx/acme-turn.onnx", "size": 32104448, "sha256": "…" }]
}
```

- **Pin a commit**, not a branch. `revision` must be a full Hugging Face commit
  sha, and every file needs its exact `size` and `sha256`. Cuelo refuses a file
  that doesn't match, so a changed upstream file can't be loaded silently.
- **`role`** is one of `vad`, `turn`, `text_turn`, `barge`, `agent_backchannel`,
  `duplex` or `component`.
- **`status`** starts at `evaluating`. It becomes `integrated` once a controller
  ships and you have numbers from live calls.
- **`requires`** lists other catalog entries a model depends on. The MaAI
  models require `maai-mimi-encoder`, for example.
- **Check the license.** Weights are downloaded, never bundled, but users still
  need to know what they're running. Note attribution requirements in `license`.

Run `make schema` to copy it into the engine and CLI.

Get the size and hash with:

```bash
curl -sL https://huggingface.co/acme/acme-turn/resolve/<sha>/onnx/acme-turn.onnx -o m.onnx
stat -f%z m.onnx      # Linux: stat -c%s
shasum -a 256 m.onnx
```

## 2. The controller [#2-the-controller]

A controller is synchronous code that consumes audio and transcripts and
returns `ControlEvent`s. The harness runs each controller on its own worker
thread, feeds it in order, and resamples audio to its `sample_rate`.

```python title="engine/src/cuelo/controllers/acme_turn.py"
"""Acme Turn: streaming end of turn from audio.

Scores every 40 ms; the turn ends when P(end) crosses `threshold` after at
least `min_silence_ms` of quiet, or after `max_silence_ms` regardless.

Model: https://huggingface.co/acme/acme-turn (Apache-2.0)
"""

from typing import ClassVar

import numpy as np

from cuelo.controllers.interface import ControlEvent, Controller, Signal
from cuelo.controllers.vad import VAD, SpeechTracker, default_vad
from cuelo.runtime.audio import Audio

STEP = 640  # 40 ms at 16 kHz

class AcmeTurnModel:
    """The ONNX model alone: audio in, probabilities out. Easy to fake in tests."""

    def __init__(self) -> None:
        from cuelo.controllers import weights

        self.session = weights.session(weights.weights("acme-turn"))   # downloads + verifies once
        self.reset()

    def reset(self) -> None:
        self._pending = np.zeros(0, np.float32)

    def push(self, audio: np.ndarray) -> np.ndarray:
        self._pending = np.concatenate([self._pending, audio])
        out = []
        while self._pending.size >= STEP:
            step, self._pending = self._pending[:STEP], self._pending[STEP:]
            (p,) = self.session.run(["p_end"], {"audio": step[None]})
            out.append(float(p.reshape(-1)[0]))
        return np.asarray(out, np.float32)

class AcmeTurnController(Controller):
    emits: ClassVar[frozenset[Signal]] = frozenset({Signal.TURN_END})
    sample_rate = 16_000

    def __init__(
        self,
        vad: VAD | None = None,
        *,
        model: AcmeTurnModel | None = None,
        threshold: float = 0.5,
        min_silence_ms: float = 80,
        max_silence_ms: float = 1500,
    ) -> None:
        super().__init__()
        self.model = model or AcmeTurnModel()
        self.tracker = SpeechTracker(vad or default_vad(), self.sample_rate, gap_ms=max_silence_ms)
        self.threshold = threshold
        self.min_silence_s = min_silence_ms / 1000
        self.t = 0.0           # seconds of user audio consumed
        self.fired = False

    def push_user(self, audio: Audio) -> list[ControlEvent]:
        samples = np.frombuffer(audio.pcm, np.int16).astype(np.float32) / 32768
        self.t += audio.duration
        ...  # update the speech tracker, run the model, and when the user is
        ...  # quiet long enough and P(end) >= threshold, return once:
        # return [ControlEvent(Signal.TURN_END, t=self.t, confidence=p, speech_end_t=end)]
        return []

    def reset(self) -> None:
        self.model.reset()
        self.t, self.fired = 0.0, False
```

The rules:

- **Time is audio time.** `t` is seconds of user audio consumed, never wall
  clock, so the same audio always gives the same events. That makes tests and
  replays deterministic.
- **Declare `emits`.** The harness fills gaps with other controllers. A
  controller that only does end of turn leaves interruptions to the barge-in
  detector.
- **Duplex models** set `duplex = True` and get `push_agent(audio)` with the
  agent's audio as it plays. Non-duplex controllers can read
  `self.agent_speaking`.
- **Transcript models** (`text_turn`) implement `push_transcript()`.
- **Fire once per turn.** Return `TURN_END` once, then re-arm when the user
  speaks again. Set `speech_end_t` (when the user actually stopped) so latency
  metrics are right.
- **Keep the model separate** from the controller (`AcmeTurnModel` above), so
  tests can pass a fake model and don't need the weights.
- **Mind the CPU.** It runs on every audio frame of every call. Measure with
  `cuelo-bench --controllers your.module:factory` and put the per-step cost in
  the catalog's `cpu` field. Share heavy encoders across controllers in a call,
  the way `MaaiCall` shares one Mimi encoder pair between the MaAI controllers.

Register it in the schema's `x-cuelo-providers.controller` so tooling can find
it:

```json
"acme-turn": { "class": "cuelo.controllers.acme_turn.AcmeTurnController", "extra": "silero", "env": null }
```

Use `silero` as the extra for ONNX models (onnxruntime and numpy), and `maai`
for PyTorch.

## 3. Config wiring (optional) [#3-config-wiring-optional]

At this point anyone can use the controller from `src/turns.py`:

```python
from cuelo.controllers.acme_turn import AcmeTurnController
from cuelo.controllers.interface import Signal

def controllers(defaults):
    return [AcmeTurnController(), *[c for c in defaults if Signal.TURN_END not in c.emits]]
```

To make it a first-class choice (`turns.controller: acme-turn`):

1. Add `acme-turn` to the `turns.controller` enum in
   `schema/config.cuelo.schema.json`, with a sentence in its description. Then
   run `make schema`.
2. Map its config to constructor arguments in `cli/internal/gen/plan.go`
   (the `switch t.Controller` block). Pass `min_silence`, `max_silence` and
   `threshold` the same way `turn-1-mini` does.
3. Add a case to the default agent's `CUELO_TURNS` in
   `engine/src/cuelo/agents/default.py`.
4. Update the Go generator test expectations (`cli/internal/gen/gen_test.go`).

## Tests and evaluation [#tests-and-evaluation]

- **Unit tests** with a fake model: feed synthetic speech and silence and assert
  when `TURN_END` fires. See `engine/tests/test_model_controllers.py`.
- **A weights test** that loads the real model behind a skip when the weights
  aren't cached. See `test_model_weights.py`.
- **Live calls.** Before proposing `integrated`, run real conversations
  with `cuelo start` and compare against turn-1-mini in the dashboard's latency
  tab: `vad_tail`, premature turn ends and flagged turns. Include the numbers in
  the PR.