Repository navigation
fix(telemetry): Record the aggregated answer of a streamed inference, not its chunks - #7815
Open
copybara-service[bot] wants to merge 1 commit into
Open
copybara-service[bot] wants to merge 1 commit into
copybara-service[bot] wants to merge 1 commit into
Conversation
8 of 9 tasks
… not its chunks The inference span recorded every streamed chunk as its own output message, ending at the first response with a finish reason. Model adapters put that finish reason in different places: Gemini on an empty partial chunk before its aggregate, so the span recorded the chunks; LiteLlm on the aggregate, so the span recorded the chunks and then the whole answer again. Both yield the whole answer as a `partial=False` aggregate after the `partial=True` chunks. Telemetry now skips the `partial=True` chunks and records that aggregate. A `BaseLlm` that leaves `partial` unset ran no aggregator, so its chunks are still recorded up to the finish reason, but now as one output message whose parts are the chunks' parts, as opentelemetry-instrumentation-google-genai reports a stream. Text parts are not joined: thought and answer text are both `text` parts in the output messages. Addresses the LiteLlm duplicate reported in #7694. Co-authored-by: Baixin Guo <141702260+GBX-Max1220@users.noreply.github.com> Co-authored-by: Max Ind <maxind@google.com> PiperOrigin-RevId: 1009687738
copybara-service
Bot
force-pushed
the
copybara/1009687738
branch
from
October 29, 2026 18:02
8a03d9d to
2200585
Compare
|
Thanks for taking the broader fix forward! I tried the supplemental tests from ishan-1010#1 against this PR ( The completed Is dropping partial output on an interrupted stream the intended behavior here, or should it be retained when no aggregate arrives? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(telemetry): Record the aggregated answer of a streamed inference, not its chunks
The inference span recorded every streamed chunk as its own output message,
ending at the first response with a finish reason. Model adapters put that
finish reason in different places: Gemini on an empty partial chunk before its
aggregate, so the span recorded the chunks; LiteLlm on the aggregate, so the
span recorded the chunks and then the whole answer again.
Both yield the whole answer as a
partial=Falseaggregate after thepartial=Truechunks. Telemetry now skips thepartial=Truechunks andrecords that aggregate.
A
BaseLlmthat leavespartialunset ran no aggregator, so its chunks arestill recorded up to the finish reason, but now as one output message whose
parts are the chunks' parts, as opentelemetry-instrumentation-google-genai
reports a stream. Text parts are not joined: thought and answer text are both
textparts in the output messages.Addresses the LiteLlm duplicate reported in
#7694.
Co-authored-by: Baixin Guo 141702260+GBX-Max1220@users.noreply.github.com
Co-authored-by: Max Ind maxind@google.com