Turn-Taking Is Part of Character Design: Why AI Characters Need Scenario-Specific Timing

AI characters should not all use the same interruption, pause and acknowledgement policy. A tutor, interviewer and negotiator may use the same voice model, but they do not hold the conversational floor in the same way. Turn-taking belongs in the scene design because the character's role changes when it should listen, encourage, interrupt or respond.

That is the useful production lesson from DuplexGen, a new arXiv preprint on scenario-adaptive human-AI turn-taking. The paper does not test 3D characters or prove that one model will make every conversation feel natural. It does provide evidence against a common shortcut: treating turn-taking as one global model behaviour regardless of context.

For teams building games, XR experiences or role-play training, the next step is practical. Define the turn-taking policy for each important interaction, test it with representative dialogue and make sure the runtime decision reaches the visible character performance coherently.

What DuplexGen changes

The researchers studied six conversational tasks. Three were cooperative: Socratic teaching, mixed-initiative planning and interviewing. Three were more competitive: negotiation, persuasion and social chat.

At candidate moments inside a user's utterance, the system had three possible actions:

• Listen: let the user continue.

• Backchannel: acknowledge without claiming the floor, with a brief response such as "uh-huh."

• Take the floor: begin the assistant's turn, including interruption-like entries where the scenario supports them.

The team collected slot-level preferences from 248 English-speaking participants. Each sample received five annotations, preserving disagreement as a distribution rather than forcing one majority label. A calibration set covered 20 dialogues per task, while the evaluation set covered 50 dialogues per task.

Human preferences varied systematically. Backchannels appeared more often in the cooperative settings. Taking the floor appeared more often in the competitive settings. Nearly every cross-scenario comparison in the paper was statistically different, while the competitive tasks were more similar to one another.

The model result matters too. A small scenario-specific calibration set aligned turn-taking predictions more closely with human annotations than uncalibrated prompting or training on a much larger generic human conversation corpus. In a follow-on full-duplex experiment, a model trained with DuplexGen-generated data showed more scenario variation and achieved the highest average turn-taking naturalness score of the three tested systems.

This does not mean prompts are useless or generic speech data has no value. It means neither automatically captures what people expect from a particular role.

The production lesson is bigger than the model

Teams often frame conversational timing as an engineering problem: reduce latency, detect the end of a turn and stop the character from speaking over the user. Those are necessary concerns, but they do not answer the design question.

How patient should this character be? Is a quick acknowledgement supportive or distracting? Can it interrupt to correct a dangerous misunderstanding? Does silence signal reflection, uncertainty or a broken system? If two characters use identical timing rules, they can feel mechanically related even when their scripts describe different personalities and jobs.

Our interpretation of the paper is that turn-taking needs a scene contract. That contract should be understandable to writers, interaction designers, audio teams, animators and runtime engineers. It should also be testable without asking whether the whole conversation simply "feels natural."

Use a Turn-Taking Scene Matrix

The matrix below is a Snippets3D production framework, not part of the DuplexGen study. Use it to describe one representative interaction before tuning a model or connecting a runtime stack.

The point is not to create a thick specification for every line. Start with interactions where floor control changes trust, clarity, pressure or pacing. A negotiation scene may need a more assertive entry policy. A teaching scene may need longer listening windows and restrained acknowledgements. A safety-critical correction may justify interruption even inside an otherwise patient role.

Those are hypotheses until users review them. The matrix makes the hypotheses visible.

Put the timing decision in the right layer

Scenario-specific turn-taking also exposes an architecture boundary.

In an autonomous conversation, the surrounding runtime system usually owns speech recognition, endpointing, dialogue choice, safety, scenario state and the decision to listen, backchannel or take the floor. A character-performance layer then has to realize that decision through voice timing, lip sync, facial and body animation, gaze and text.

Snippets is a controllable production layer for creating and delivering synchronized 3D character performances in Unity. It can turn approved dialogue into reusable performance assets that combine voice, lip sync, animation and text. It is not the live speech recognizer, turn detector, branching system or autonomous NPC brain.

That boundary is especially important for hybrid experiences. A runtime system may choose what happens next, while controlled performances handle reviewed moments that need exact wording and delivery. Current complete-performance generation in Snippets is batch-oriented, so a product that requires low-latency freeform responses needs a separate real-time conversation and performance path.

If you are deciding where that split belongs, read Scripted Characters vs Autonomous NPCs and What Snippets Does and Does Not Replace in a Unity Character Stack.

Test one scene before scaling

You do not need a new model training programme to learn whether the global timing policy is wrong. Run a small scene test first:

1. Choose one role-sensitive interaction with realistic user speech.

2. Mark several candidate moments as listen, backchannel or take the floor.

3. Have multiple reviewers label the moments independently. Preserve disagreement instead of forcing consensus too early.

4. Play the timing choices with the intended voice and visible character performance.

5. Test one failure: a false interruption, late response or acknowledgement that lands on the wrong phrase.

6. Record which system owns detection, recovery and the final performance handoff.

This is not a substitute for user research. It is a way to arrive at that research with explicit choices instead of a vague naturalness question.

What the paper does not prove

DuplexGen is a promising preprint, not a finished production recipe. The authors identify three important limitations.

First, the generated data covers a limited range of scenarios and interaction types. Second, full-duplex behaviour remained sensitive to training data and hyperparameters such as turn-switching latency and pause duration. Third, the calibration was grounded in incrementally revealed text. It did not directly condition on prosody, intonation, speech rate, acoustic overlap or visual cues.

That last limit matters for 3D characters. A spoken backchannel and a visible nod can reinforce one another, compete for attention or arrive out of sync. The research does not tell us how body language changes the preferred speech action. It also does not test whether the reported gains transfer to a game, XR simulation or training deployment.

The safe conclusion is narrower and more useful: one universal timing norm is a weak design assumption when characters occupy different roles.

Design the pause, not just the line

Dialogue content tells us what a character says. Turn-taking tells us how the character participates.

Before scaling an interactive character system, use the matrix on one representative scene. Decide when the character should listen, acknowledge, enter and recover. Then test whether the conversation system and the visible performance deliver the same choice.

If the timing policy cannot explain the difference between a tutor and a negotiator, the character design is not finished.