Repository navigation
fix(tokenize): ignore stray closing tags in the XML depth check - #7742
Merged
xianshijing-lk merged 1 commit intoOct 14, 2026
Merged
Conversation
…h check _has_unclosed_xml_tags decremented its depth counter on every closing tag, so a closing tag whose opener is not in the text (already emitted in an earlier chunk, or never sent by the LLM) drove it negative and cancelled out a later opening tag. The stream then treated an open wrapping tag as closed and split its sentences across TTS chunks. Clamp the counter at zero.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
ANDROIDHASSAN
changed the title
ANDROIDHASSAN
force-pushed
the
fix/xml-stray-closing-tag
branch
from
October 13, 2026 07:32
ff94feb to
bca4cb3
Compare
xianshijing-lk
approved these changes
Oct 14, 2026
xianshijing-lk
left a comment
xianshijing-lk
left a comment
There was a problem hiding this comment.
this looks safe, thanks for the fix
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem: with
xml_aware=True, the sentence stream won't emit a token for which_has_unclosed_xml_tagsis true, so a wrapping tag like<emotion>…</emotion>stays in one chunk. The depth counter in that check is decremented on every closing tag, including one whose opener isn't in the text. That happens when the opener was emitted in an earlier chunk, or the LLM never sent it. The counter then goes negative and cancels out the next real opening tag, so the wrapping tag looks closed and its sentences are split across TTS chunks.Streaming
"Okay then.</b> Now <emotion value='sad'>I am sorry. Truly sorry.</emotion> Bye."word by word throughblingfire.SentenceTokenizer(min_sentence_len=1, stream_context_len=5, xml_aware=True)gives onmain:Without the stray
</b>, the same text keeps the<emotion>span in one chunk.Fix: clamp the counter at zero, so a closing tag without an opener in the text no longer affects the check.
Tests
One new case in
tests/test_tokenizer_xml_markup.py:TestStreamingTokenizer::test_stray_closing_tag_does_not_unbalance_a_later_wrapping_tag. It checks_has_unclosed_xml_tagsdirectly, and it checks that the streamed<emotion>span stays in one chunk. Both assertions fail onmain.ruff formatandruff checkare clean.Scope: one line in
_has_unclosed_xml_tags, and nothing else reads its depth counter. Text without a stray closing tag gets the same result as before.