Skip to content

[MiniCPM-o-4.5] tts_bound selects a history turn's <|tts_eos|> when current speech is truncated, yielding an empty TTS span #1141

Open

Description

@jaminmei

Hi! I hit an edge case in modeling_minicpmo.py of openbmb/MiniCPM-o-4_5 while testing multi-turn speech chat.

chat() takes the last <|tts_bos|> but the globally last <|tts_eos|> over prompt + generated tokens:

tts_bos_idx = tts_bos_indices[-1] if tts_bos_indices else -1
tts_eos_idx = tts_eos_indices[-1] if tts_eos_indices else None

If the history contains a finished speech turn (prompt has an old <|tts_eos|>) and the current reply stops before its own marker (e.g. a small max_new_tokens), the end index lands before the start index, full_sequences[0][start:end] is empty, and the talker gets an empty condition — no audio, even though the turn had speakable tokens. The bare except around _generate_speech_non_streaming hides the failure, so you just get text with no wav. The None fallback only covers "no <|tts_eos|> anywhere"; it can't help when an old marker sits before the last <|tts_bos|>.

The text-side extraction in the same file already does this right (text.split("<|tts_bos|>")[-1].split("<|tts_eos|>")[0]), so maybe just scope the token path the same way:

start = tts_bos_indices[-1] + 1
eos_in_segment = [i for i in tts_eos_indices if i >= start]
tts_eos_idx = eos_in_segment[0] if eos_in_segment else None
tts_bound = (start, tts_eos_idx)

Single-turn and completed-turn behavior stay unchanged.

(One more thing, probably a separate issue: truncated turns also hit an off-by-one between the token span and the hidden states. In _generate_speech_non_streaming (hidden_text_merge path), llm_tokens is sliced from full_sequences, but hidden_embeds comes from the per-step hidden stack, which only covers positions consumed by a forward pass — the last generated token of a truncated turn is never fed back in, so it has no hidden row, and llm_embeds + hidden_embeds would raises RuntimeError (swallowed by the bare except). Completed turns don't hit this because the tokens generated after the span, like <|tts_eos|>, consume the last span token as input.)

Activity

  1. tc-mb commented on Sep 20, 2026

    @tc-mb
    Collaborator

    cool, thank you very much for your issue.
    this is indeed a corner case, we will fix it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions