The Collaboration Bottleneck in Current AI Systems
Most AI models today are optimized for autonomous operation rather than human-in-the-loop collaboration. A frontier model card cited by Thinking Machines Lab noted that interactive, synchronous use produced less clear benefits than autonomous, long-running agents. This reveals a design philosophy that inadvertently sidelines real-time human participation.
Real work rarely unfolds in isolated monologues. It involves messaging, talking, listening, seeing, showing, and interjecting. Yet current models experience reality in a single thread: they wait until the user finishes input, then freeze perception during generation. This creates a narrow channel that limits the transfer of knowledge, intent, and judgment. The bandwidth bottleneck is not just technical—it is conceptual, treating interaction as an afterthought rather than a native capability.
How Interaction Models Work Mechanically
The proposed solution makes interactivity part of the model itself. At the core is a multi-stream, micro-turn design that continuously processes audio, video, and text in real time. The model interleaves 200ms chunks of input processing and output generation without artificial turn boundaries. Input and output tokens are treated as parallel streams, enabling near real-time concurrency across modalities.
Architecturally, the system uses encoder-free early fusion. Audio signals are processed as dMel via a lightweight embedding layer. Images are split into 40×40 patches encoded by an hMLP. Audio decoding employs a flow head. All components are co-trained from scratch with the transformer, making interactivity a fundamental property that scales with model intelligence rather than a harness bolted on afterward.
The Dual-Model Architecture: Coordination Between Responsiveness and Reasoning
The full system splits labor between two specialized models. An interaction model maintains a constant two-way exchange with the user, operating in real time across all modalities. A background model runs asynchronously for sustained reasoning, tool use, and longer-horizon tasks.
When the interaction model delegates work, it sends a complete conversation context package. Results from the background model stream back and are interleaved at appropriate moments in the live conversation. This architecture lets users benefit from both immediate, fluid responsiveness and the full intelligence of reasoning models without sacrificing either. The coordination mechanism ensures that deep computation never interrupts the natural rhythm of dialogue.
Benchmarking the Interactivity Frontier
The model, TML-Interaction-Small, is a 276B parameter mixture-of-experts architecture with 12B active parameters. Benchmarks reveal significant advances in real-time performance while maintaining competitive intelligence.
| Benchmark | Metric | TML-Small (instant) | GPT-realtime-2.0 (xhigh) | Gemini-3.1-flash-live (high) |
|---|---|---|---|---|
| FD-bench V1 Latency (s) | Audio | 0.40 | 1.63 | 0.94 |
| FD-bench V1.5 Average | Audio | 77.8 | 47.8 | 45.5 |
| FD-bench V3 Pass@1 (%) | Audio+Tools | 68.0 | 58.0 | 48.0 |
| BigBench Audio Accuracy (%) | Audio | 75.7 / 96.5* | 96.6 | 96.6 |
Turn-taking latency drops to 400ms—substantially faster than competing real-time systems. On FD-bench V3, which tests response quality with simultaneous tool use, the model achieves 68% Pass@1, exceeding alternatives. An asterisk denotes results computed with the background model engaged, showing how the dual architecture lifts performance on knowledge-intensive tasks.
Inference Engineering: Latency and Alignment at Scale
Meeting the 200ms latency constraint required deep inference optimization. Each chunk is sent as a separate request and appended to a persistent sequence in GPU memory, avoiding costly reallocations. This streaming session design was upstreamed to SGLang. For MoE kernels, a gather+gemv strategy replaced grouped GEMM to reduce latency.
Bitwise trainer-sampler alignment was achieved with less than 5% end-to-end overhead. Two kernel innovations stand out. All-reduce and reduce-scatter operations use NVLS for deterministic communication on Blackwell hardware, ensuring bitwise alignment between Sequence and Tensor Parallelism. Attention kernels maintain consistent accumulation order for Split-KV by splitting batches consistently with left-aligned 4096-token chunks. These details ensure that training behavior precisely mirrors inference.
Safety and Refusal Design for Real-Time Speech
Safety mechanisms were rebuilt for the voice modality. Refusal training data was generated using text-to-speech to produce refusals that are colloquial and firm rather than robotic or evasive. Multi-turn refusal examples were created through an automated red-teaming harness, ensuring the model maintains behavioral parity with text-based safety standards when speaking aloud.
This approach addresses a subtle failure mode in voice AI: refusals that sound unnatural or hesitant can undermine user trust. By training directly on spoken refusal patterns, the model learns to decline requests in ways that feel appropriate to conversational context—brief, clear, and tonally consistent with the ongoing dialogue.
Practical Implications and Future Directions
Interaction models unlock capabilities previously requiring separate software harnesses. Seamless dialog management tracks whether a speaker is thinking, yielding, or inviting response. Verbal and visual interjections happen based on context, not rigid turn boundaries. Simultaneous speech enables use cases like live translation. Time-awareness gives the model a direct sense of elapsed seconds, and tool calls, search, and generative UI can run concurrently with speaking and listening.
The split between interaction and background models points toward a future where AI assistants feel less like transactional tools and more like collaborators. As this architecture scales, improvements in intelligence and interactivity compound together. Open questions remain about how these models handle adversarial interruptions, accented speech, or environments with multiple speakers. The research preview demonstrates that interactivity can be a first-class model property, not a post-hoc interface layer.
