Skip to content

fix(telemetry): Record the aggregated answer of a streamed inference, not its chunks - #7815

Open
copybara-service[bot] wants to merge 1 commit into
mainfrom
copybara/1009687738
Open

copybara-service[bot] wants to merge 1 commit into
mainfrom
copybara/1009687738

Conversation

@copybara-service

@copybara-service copybara-service Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

fix(telemetry): Record the aggregated answer of a streamed inference, not its chunks

The inference span recorded every streamed chunk as its own output message,
ending at the first response with a finish reason. Model adapters put that
finish reason in different places: Gemini on an empty partial chunk before its
aggregate, so the span recorded the chunks; LiteLlm on the aggregate, so the
span recorded the chunks and then the whole answer again.

Both yield the whole answer as a partial=False aggregate after the
partial=True chunks. Telemetry now skips the partial=True chunks and
records that aggregate.

A BaseLlm that leaves partial unset ran no aggregator, so its chunks are
still recorded up to the finish reason, but now as one output message whose
parts are the chunks' parts, as opentelemetry-instrumentation-google-genai
reports a stream. Text parts are not joined: thought and answer text are both
text parts in the output messages.

Addresses the LiteLlm duplicate reported in
#7694.

Co-authored-by: Baixin Guo 141702260+GBX-Max1220@users.noreply.github.com
Co-authored-by: Max Ind maxind@google.com

… not its chunks

The inference span recorded every streamed chunk as its own output message,
ending at the first response with a finish reason. Model adapters put that
finish reason in different places: Gemini on an empty partial chunk before its
aggregate, so the span recorded the chunks; LiteLlm on the aggregate, so the
span recorded the chunks and then the whole answer again.

Both yield the whole answer as a `partial=False` aggregate after the
`partial=True` chunks. Telemetry now skips the `partial=True` chunks and
records that aggregate.

A `BaseLlm` that leaves `partial` unset ran no aggregator, so its chunks are
still recorded up to the finish reason, but now as one output message whose
parts are the chunks' parts, as opentelemetry-instrumentation-google-genai
reports a stream. Text parts are not joined: thought and answer text are both
`text` parts in the output messages.

Addresses the LiteLlm duplicate reported in
#7694.

Co-authored-by: Baixin Guo <141702260+GBX-Max1220@users.noreply.github.com>
Co-authored-by: Max Ind <maxind@google.com>
PiperOrigin-RevId: 1009687738
@GBX-Max1220

Copy link
Copy Markdown

Thanks for taking the broader fix forward! I tried the supplemental tests from ishan-1010#1 against this PR (2200585e).

The completed EVENT_ONLY aggregate test passes. The interrupted-stream test behaves differently: if a partial=True response arrives and the stream raises before the aggregate, the original exception propagates and the span is marked ERROR, but gen_ai.output.messages is absent, so the partial text isn't recorded. The same test passes on #7694's head (2c5773f0).

Is dropping partial output on an interrupted stream the intended behavior here, or should it be retained when no aggregate arrives?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants