[BUG] Reasoning tag removal fails when <think> is part of the input prompt #1453

Activity

1. Tialo

   added

   bug

   on Oct 27, 2026
2. hamarshehmhmd commented on Oct 27, 2026

   

   hamarshehmhmd

   on Oct 27, 2026

   More actions

   I can confirm this, and the fix can stay entirely inside remove_reasoning_tags (src/lighteval/utils/utils.py). Once the paired tags have been stripped, an orphan closing tag that has no opening tag before it means the opening tag came from the prompt. Everything up to and including that closing tag is reasoning.

   --- a/src/lighteval/utils/utils.py
   +++ b/src/lighteval/utils/utils.py
   @@ def remove_reasoning_tags(text: str, tag_pairs: list[tuple[str, str]]) -> str:
                else:
                    break
    
   +        # The opening tag may have been part of the prompt (e.g. chat templates
   +        # that end with "<think>\n", like Qwen3.5), so the generation contains
   +        # only the closing tag. Drop everything up to that orphan closing tag.
   +        end = result.find(end_tag)
   +        if end != -1 and start_tag not in result[:end]:
   +            result = result[end + len(end_tag) :]
   +
        return result

   Behaviour before/after (I ran the existing docstring examples plus the new cases):

   generated text	before	after
   <think> Reasoning </think> Answer	Answer	Answer
   <reasoning>Step 1</reasoning>Answer<reasoning>Step 2</reasoning>	Answer	Answer
   Plain answer	unchanged	unchanged
   <think> unfinished reasoning (truncated)	unchanged	unchanged
   Okay, let me think...\n</think>\n\nB (the Qwen3.5 case)	unchanged	\n\nB
   reasoning</think>A<think>more</think>	reasoning</think>A	A

   Outputs without a closing tag are left as they are, so non-reasoning models aren't affected. Truncated reasoning (an opening tag but no closing one) is also unchanged, same as today. This is simpler than passing the prompt into post-processing the way Transformers' response parsing does, and covers the common chat-template case. Happy to send a PR with these as unit tests in tests/unit/utils/test_utils.py.