Skip to content

feat(httpclient): optional read-only observations for upstream attempts and response completion (#8209) - #8269

Open
konlue wants to merge 2 commits into
apache:masterfrom
konlue:feat/http-attempt-observations
Open

konlue wants to merge 2 commits into
apache:masterfrom
konlue:feat/http-attempt-observations

Conversation

@konlue

@konlue konlue commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Part of #8199. Fixes #8209.

Problem

The standard HTTP forwarding path makes one or more real transport calls per logical request (CURRENT resubscribes the same publisher; failover executes against another selected target), but a single enclosing observation cannot describe each attempt: headers readiness is not response completion, because the body is consumed later by NettyClientMessageWriter / WebClientMessageWriter. There is no read-only, opt-in way to observe each actual attempt — including its body — without changing routing, retry eligibility, circuit-breaker policy or response-writing behavior.

Change

  • UpstreamAttemptObservation (plugin-base): read-only facts of one attempt — identity (unique per transport subscription), target address (no query, no credentials), method, reused trace id, headers-ready timing/status, terminal outcome (COMPLETED / ERROR / CANCELLED, recorded exactly once).
  • UpstreamAttemptObserver + UpstreamAttemptObserverHolder: a small opt-in adapter with a no-op default, registered by observability consumers at startup (mirroring MetricsReporter). With no observer registered the transport path performs one volatile read, returns the same publisher and allocates nothing.
  • AbstractHttpClientPlugin / DefaultRetryStrategy: every transport subscription creates its observation lazily at subscription time (CURRENT resubscriptions and failover attempts each get their own identity), reports headers-ready when the response arrives, and finishes on body completion, error or downstream cancellation. Each attempt finishes from its own subscription closure, so an inline failover resend cannot finish or replace the next attempt's record; the exchange attribute only identifies the serving attempt whose body is still streaming.
  • A rejected status drained without retries is a failed attempt, not a successful one; pre-header failures finish with no status.
  • The observation reuses the trace id the gateway span already published and never creates a span or a second trace context. Response byte counts stay owned by the response writers, so this adapter never buffers or subscribes to a body. Consumers own bounded delivery and must not use the trace id, target or per-request values as unbounded metric labels.
  • All callbacks are failure-isolated and nonblocking; observations carry no credentials, raw queries or request ids.
  • Half-open probes are untouched: they forbid retries and record outcomes exclusively through the existing probe path.

Tests

  • UpstreamAttemptObservationTest drives the real NettyHttpClientPlugin and WebClientPlugin against real local HttpServers: success reports started / headers-ready / completed exactly once; a pre-header connection failure finishes with an error and no status; a delayed chunked body does not finish the observation before the body completes; downstream cancellation records CANCELLED; an observer that throws cannot fail the request; with no observer registered nothing is reported.
  • Concurrent requests each finish exactly once with distinct identities — state belongs to the request, not to shared mutable fields.
  • Local request rejection (no resolved URI) produces no observation.
  • CURRENT retries and failover attempts get separate identities: a disconnected subscription records ERROR, the retried subscription on the same publisher records COMPLETED, and with the default inline failover resend the initial attempt keeps its ERROR record while the standby attempt records COMPLETED. Both attempts reuse the trace id of the logical request.
  • Existing retry, timeout, body-replay and aggregation tests are adapted only where the abstract plugin gained the observation hook.

Verification

shenyu-plugin-httpclient: 177/177
shenyu-plugin-base: 359/359
shenyu-plugin-divide: 158/158
shenyu-common: 663 (4 skipped)
shenyu-loadbalancer: 86/86

Checkstyle clean on the touched modules as part of the reactor validate phase.

@konlue
konlue force-pushed the feat/http-attempt-observations branch from e8734c0 to ace8097 Compare November 1, 2026 07:58

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Task][HTTP Client] Add optional observations for upstream attempts and response completion

1 participant