Skip to content

[BUG] Translated OpenAI streams keep the first nonzero usage counters #4263

Open

Description

@oxura

The translated OpenAI streaming path keeps the first nonzero usage count, even when the provider sends a later cumulative snapshot.

I reproduced this through the actual proxy and installed AnyLLM/OpenAI SDK, using a loopback OpenAI-compatible streaming server. It sends these usage snapshots on successive content chunks:

{"prompt_tokens":10,"completion_tokens":1,"total_tokens":11,"prompt_tokens_details":{"cached_tokens":2}}
{"prompt_tokens":13,"completion_tokens":5,"total_tokens":18,"prompt_tokens_details":{"cached_tokens":9}}

On main fbf7bc3278c11975299b9bd0a2ecf200dfd25a05, /stats records 10 input / 1 output token, and /metrics records 2 cache-read tokens. The client receives both frames intact, so this is accounting after delivery, not a missing upstream event.

The fix records the final 13 / 5 / 9 counts. Usage frames update only fields they actually contain; an absent field leaves the last known value alone, while an explicit zero replaces it. The stateful SSE parser already handles completion-only frames, so the redundant single-chunk parser and its implementation-only tests are removed.

The eight replacement regressions exercise progressive, partial, zero, and absent usage through stream finalization, in Chat and Responses shapes. Six fail before the fix; all eight pass after it. The related streaming/outcome suites pass 71 tests; the shared parser's malformed-event, billed-input, cache-TTL and handler-helper suites pass another 469. The same real HTTP replay reports the corrected counts after the patch.

I'll keep the branch on the fork until I'm below the ten-open-PR limit.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions