fix(telemetry): record a LiteLlm streamed answer once, not chunks plus aggregate by ishan-1010 · Pull Request #7694 · google/adk-python

ishan-1010 · 2026-10-23T11:56:57Z

Link to Issue or Description of Change

Closes: LiteLlm with SSE streaming records the answer twice in gen_ai.output.messages #7630

Problem:
LiteLlm streams chunks without a finish reason and then yields their aggregate with one. record_llm_response keeps recording until a finish reason arrives, so gen_ai.output.messages held every chunk and then the whole answer again.

Solution:
As suggested in #7630: when a complete response with a finish reason follows only partial chunks without one, drop the chunks' output messages and keep the aggregate. Streams whose last chunk carries the finish reason (Gemini) and streams that fail before their aggregate keep their chunks.

Testing Plan

Unit Tests:

I have added or updated unit tests for my change.
All unit tests pass locally.

New tests in tests/unittests/telemetry/test_instrumentation.py:

test_aggregate_after_unfinished_chunks_is_recorded_once (fails on main: 3 messages)
test_tool_call_aggregate_replaces_its_partial_call_chunks (fails on main: a call with arguments: null plus the real one)
test_stream_cut_off_before_its_aggregate_keeps_its_chunks
test_stream_whose_last_chunk_finishes_keeps_its_chunks (Gemini shape)

tests/unittests/telemetry: 1023 passed, 9 xfailed (functional goldens unchanged)
tests/unittests/models + flows: 2818 passed
pre-commit: passed

Manual End-to-End (E2E) Tests:

gen_ai.output.messages on main vs this PR, LiteLlm with StreamingMode.SSE:

stream	main	this PR
groq `openai/gpt-oss-20b`, text answer	137	1
groq `openai/gpt-oss-20b`, tool call	36 incl. `tool_call` with `arguments: null`	1: text + `tool_call({"city": "Oslo"})`
openai provider vs a local SSE server, text in 5 chunks	6	1
same, tool call args in 3 pieces	5	1

Same counts in the completion log with EVENT_ONLY.

Checklist

I have read the CONTRIBUTING.md document.
I have performed a self-review of my own code.
I have commented my code, particularly in hard-to-understand areas.
I have added tests that prove my fix is effective or that my feature works.
New and existing unit tests pass locally with my changes.
I have manually tested my changes end-to-end.
Any dependent changes have been merged and published in downstream modules.

Additional context

Out of scope: with the v1 schema (ADK_TELEMETRY_SCHEMA_VERSION_OPT_IN=1) or stable semconv, trace_inference_result emits one gen_ai.choice log per response, and those are already emitted when the aggregate arrives. On the local SSE server that is 6 logs for a 5-chunk text answer and 5 for a streamed tool call, the same on main and on this branch. That needs a different fix.

The silent cut-off from #7630 (a dropped stream is finalized as stop) is tracked separately in #7693.

…s aggregate LiteLlm streams chunks without a finish reason and then yields their aggregate with one. record_llm_response kept recording until that finish reason, so gen_ai.output.messages held every chunk and then the whole answer again (a 5-chunk answer was 6 messages, a streamed tool call 4 half-built calls plus the real one). When a complete response with a finish reason follows only partial chunks without one, drop the chunks' output messages and keep the aggregate. Streams whose last chunk carries the finish reason (Gemini) and streams that fail before their aggregate keep their chunks. Fixes google#7630

GBX-Max1220 · 2026-10-23T15:58:09Z

Thanks, Ishan—I’ve attached two supplemental native tests against 2c5773f0:

An interrupted stream propagates the original exception, retains partial messages on the finished span, and records ERROR status.
EVENT_ONLY records the completed aggregate once.

The full test_instrumentation.py passes locally: 104 tests. I set the test process’s trace sampler to always_on because this environment defaults to 1% sampling.
ADK_7694_Supplemental_Native_Tests_20261023.zip

I’m not claiming a full telemetry-suite pass: the broader run hit a missing SOCKS dependency in the functional tests. The patch only changes tests and was AI-assisted.

GBX-Max1220 · 2026-10-26T18:50:43Z

Hi Ishan — I’ve now ported the two supplemental tests from the attached patch onto your PR head (2c5773f0) in a local branch. The full test_instrumentation.py passes with 104 tests, including both additions. The changes only touch the telemetry test file. Since I can’t push to your branch, could you apply the test-only patch to this PR if you’d like to include the extra coverage?

ishan-1010 · 2026-10-27T08:08:51Z

@surajksharma07 added the v1/stable path to the description as out of scope. it logs 6 gen_ai.choice for a 5-chunk answer and 5 for a streamed tool call, same on main and this branch. marked ready for review.

@GBX-Max1220 you can open a PR against ishan-1010/adk-python, branch fix-litellm-stream-output-once. i'll merge it into this one so the tests show up on this diff.

RKest · 2026-10-28T16:00:59Z

Hi @ishan-1010, thank you for the contribution,

This looks a lot like a cross-functional issue, having to do with LLM model implementations rather than just telemetry. Telemetry is only a symptom.

From what I gathered, Gemini when streaming emits last meaningful chunk with the finish_reason set.
LiteLLM and OpenAI-themed APIs, return one last empty chunk with finish_reason set.

I'm wondering if this can be fixed on the Model-level, so that both are compatible, instead of patching this in Telemetry @GWeale could you please look into this🙏

ishan-1010 · 2026-10-28T16:28:01Z

Hi @RKest, thanks for looking into this. You're right, telemetry is just where it shows up. Gemini puts the finish reason on its last chunk, so telemetry stops there. LiteLLM sends it in a separate empty chunk, so LiteLlm only has it on the aggregate at the end, and both get recorded.

I can move the fix into LiteLlm, for example by holding each chunk back by one so the last one carries the finish reason like Gemini. One thing to check first: telemetry would then record one message per chunk, like it does for Gemini, so around 136 for the Groq example instead of 1. Is that what you'd want, or should a LiteLlm stream end up as one message?

RKest · 2026-10-29T14:22:05Z

One thing to check first: telemetry would then record one message per chunk, like it does for Gemini, so around 136 for the Groq example instead of 1. Is that what you'd want, or should a LiteLlm stream end up as one message?

Well, I wanted to follow the example of https://github.com/open-telemetry/opentelemetry-python-genai which used to emit one message per chunk.
But ~2 weeks ago it changed to instead emit one message per-stream
open-telemetry/opentelemetry-python-genai#908

In-hindsight this seems like a better approach

I'll work on a comprehensive solution to this, because it looks very severe, as to my knowledge all models except Gemini are impacted.

ishan-1010 · 2026-10-29T14:51:10Z

Sounds good, thanks. Should I close this once yours is up, or keep it as the LiteLlm part?

RKest · 2026-10-29T17:59:32Z

Sounds good, thanks. Should I close this once yours is up, or keep it as the LiteLlm part?

Could you please verify this PR here fixes your issue:
#7815 🙏

I'll try to have it merged later today or tomorrow. Monday at the latest

… not its chunks The inference span recorded every streamed chunk as its own output message, ending at the first response with a finish reason. Model adapters put that finish reason in different places: Gemini on an empty partial chunk before its aggregate, so the span recorded the chunks; LiteLlm on the aggregate, so the span recorded the chunks and then the whole answer again. Both yield the whole answer as a `partial=False` aggregate after the `partial=True` chunks. Telemetry now skips the `partial=True` chunks and records that aggregate. A `BaseLlm` that leaves `partial` unset ran no aggregator, so its chunks are still recorded up to the finish reason, but now as one output message whose parts are the chunks' parts, as opentelemetry-instrumentation-google-genai reports a stream. Text parts are not joined: thought and answer text are both `text` parts in the output messages. Addresses the LiteLlm duplicate reported in #7694. Co-authored-by: Baixin Guo <141702260+GBX-Max1220@users.noreply.github.com> Co-authored-by: Max Ind <maxind@google.com> PiperOrigin-RevId: 1009687738

ishan-1010 · 2026-10-30T05:10:49Z

Tried it locally with LiteLLM, every case records one message now. Looks good, I'll close this one once yours is merged.

my run

== main
text output_messages: 6
tool output_messages: 5
cutoff output_messages: 3
== #7815
text output_messages: 1
tool output_messages: 1
cutoff output_messages: 1

… not its chunks The inference span recorded every streamed chunk as its own output message, ending at the first response with a finish reason. Model adapters put that finish reason in different places: Gemini on an empty partial chunk before its aggregate, so the span recorded the chunks; LiteLlm on the aggregate, so the span recorded the chunks and then the whole answer again. Both yield the whole answer as a `partial=False` aggregate after the `partial=True` chunks. Telemetry now skips the `partial=True` chunks and records that aggregate. A `BaseLlm` that leaves `partial` unset ran no aggregator, so its chunks are still recorded up to the finish reason, but now as one output message whose parts are the chunks' parts, as opentelemetry-instrumentation-google-genai reports a stream. Text parts are not joined: thought and answer text are both `text` parts in the output messages. Addresses the LiteLlm duplicate reported in #7694. Co-authored-by: Baixin Guo <141702260+GBX-Max1220@users.noreply.github.com> Co-authored-by: Max Ind <maxind@google.com> PiperOrigin-RevId: 1012105751

RKest · 2026-11-02T15:54:57Z

Hi @ishan-1010,

#7815 is merged

Thank you for your help with testing and the patience🙏

ishan-1010 mentioned this pull request Oct 23, 2026

LiteLlm with SSE streaming records the answer twice in gen_ai.output.messages #7630

Open

adk-bot assigned jawoszek Oct 23, 2026

ishan-1010 marked this pull request as ready for review October 27, 2026 08:08

GBX-Max1220 mentioned this pull request Oct 27, 2026

test(telemetry): add two supplemental tests for #7694 ishan-1010/adk-python#1

Open

copybara-service Bot mentioned this pull request Oct 29, 2026

fix(telemetry): Record the aggregated answer of a streamed inference, not its chunks #7815

Merged

Navigation Menu

Repository navigation

fix(telemetry): record a LiteLlm streamed answer once, not chunks plus aggregate - #7694

fix(telemetry): record a LiteLlm streamed answer once, not chunks plus aggregate#7694
ishan-1010 wants to merge 1 commit into
google:mainfrom
ishan-1010:fix-litellm-stream-output-once

ishan-1010 commented Oct 23, 2026 •

edited

Loading

Uh oh!

GBX-Max1220 commented Oct 23, 2026

Uh oh!

GBX-Max1220 commented Oct 26, 2026

Uh oh!

ishan-1010 commented Oct 27, 2026

Uh oh!

RKest commented Oct 28, 2026

Uh oh!

ishan-1010 commented Oct 28, 2026

Uh oh!

RKest commented Oct 29, 2026

Uh oh!

ishan-1010 commented Oct 29, 2026

Uh oh!

RKest commented Oct 29, 2026 •

edited

Loading

Uh oh!

ishan-1010 commented Oct 30, 2026

Uh oh!

RKest commented Nov 2, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

ishan-1010 commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Link to Issue or Description of Change

Testing Plan

Checklist

Additional context

Uh oh!

GBX-Max1220 commented Oct 23, 2026

Uh oh!

GBX-Max1220 commented Oct 26, 2026

Uh oh!

ishan-1010 commented Oct 27, 2026

Uh oh!

RKest commented Oct 28, 2026

Uh oh!

ishan-1010 commented Oct 28, 2026

Uh oh!

RKest commented Oct 29, 2026

Uh oh!

ishan-1010 commented Oct 29, 2026

Uh oh!

RKest commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ishan-1010 commented Oct 30, 2026

Uh oh!

RKest commented Nov 2, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ishan-1010 commented Oct 23, 2026 •

edited

Loading

RKest commented Oct 29, 2026 •

edited

Loading