Repository navigation
fix(divide): retain resolved canary partition during HTTP failover - #8076
Conversation
Aias00
left a comment
There was a problem hiding this comment.
Review (Aias00)
Verdict: APPROVE. The bug is real, the fix is scoped to the failover lookup, and the new suite fails 6/12 when the retry-path change is reverted, so it is pinned rather than decorative.
What I verified
mvn -pl shenyu-plugin/shenyu-plugin-proxy/shenyu-plugin-divide,shenyu-plugin/shenyu-plugin-httpclient -am test: 234 tests, 0 failures, 0 errors, 0 skipped (divide 114, httpclient 120), includingDivideCanaryFailoverTest12/12.- Reverted only the
resend()lookup hunk (leaving everything else, including the new context class, in place) and re-ran:Tests run: 12, Failures: 6— all fourfailoverRetainsResolvedPartitionAfterRefreshvariants plus bothexhaustedPartitionCannotFallBackAfterBackendFailurecases. The six that fail are exactly the ones that assert partition retention, which is the behavior under change. - Checked the identity assumption that the whole fix depends on:
UpstreamRoutingContextis built withselector.getId()inDividePlugin.selectPartition, and the retry path compares it againstConstants.DIVIDE_SELECTOR_ID, whichDividePlugin:212sets fromselector.getId()as well. Same source, sofiltercannot reject a legitimate retry. - Checked the health semantics of the switch from
findUpstreamListBySelectorIdtofindAllUpstreamListBySelectorId+ label filter.findAll...readstask.getHealthyUpstream(), i.e. still the healthy snapshot; the only difference is that it does not apply the legacy gray-exclusive projection. That is exactly what the fix needs — the legacy projection is what could route a retry outside the selected label partition — and health filtering is not lost. - Confirmed the empty-result path is safe:
filterreturns an empty list rather than null on a selector mismatch, andresend()already maps an empty candidate list toShenyuException→503 CANNOT_FIND_HEALTHY_UPSTREAM_URL_AFTER_FAILOVER. No NPE, no silent retry against a stale URI.
Design notes
- Capturing the labels after the canary decision, including the fallback branch, is the correct place: the fallback is itself a resolved decision, and re-deciding on retry is precisely what produced the cross-partition retry. The test that mutates the shared
CanaryConfigmid-request (config.setStableLabels(Map.of("release", "changed"))) is a good way to prove the request-scoped copy is used instead of the live config — that is the failure mode a naive implementation would have. UpstreamRoutingContext.filtercarrying the selector identity is the right guard. Without it, an exchange re-entered against a different selector could silently inherit a stale partition.- Clearing the attribute at the top of
executeRequestis defensive and correct for a re-used exchange; it also makes the "no canary decision happened" case distinguishable inlegacyFailoverKeepsGrayExclusivePool, which asserts the attribute is null and the gray-exclusive pool still applies. That legacy-preservation assertion is the one I most wanted to see, since the easy version of this fix would have changed legacy behavior too.
Nits (non-blocking)
- The
pom.xmlchange promotesshenyu-plugin-httpclientfrom thedivide-gateway-benchmarkprofile into the module's default test scope. It works and there is no cycle (httpclient does not depend on divide), but it now resolves for everydividebuild, including plain-DskipTestscompiles, purely because one new test needsDefaultRetryStrategy. Consider either keeping it in the profile and adding that profile to the CI invocation, or a one-line comment in the POM saying why the scope moved — the next person to trim test dependencies will otherwise try to put it back. UpstreamRoutingContext.filterboth filters by labels and enforces selector identity. The name reads as "apply the partition";filterFor(selectorId, upstreams)or a short javadoc sentence on the identity check would make the second job visible at the call site, where it currently looks like a plain label filter.DividePlugin.executeRequestremoving the attribute at entry is not self-explanatory. A comment like "a re-entered exchange must not inherit a previous selector's partition" would stop someone from deleting it as dead code.exhaustedPartitionCannotFallBackAfterBackendFailureassertshttp.attempts.isEmpty(), which is the right assertion, but it does not distinguish "the retry never ran" from "the retry ran and selected nothing before touching the client". Given the 503 assertion above it, the distinction is covered in practice — just noting the assertion is doing less than it appears to.
Interaction with #8035 / #8078
No overlap in files with the gRPC work; this is divide + httpclient only. Independent to merge.
CI
Green on 07d6e02c: build, changes, pr_build, rat_check, check-license-header, registry_dependencies, build (shenyu-integrated-test-http), build (shenyu-integrated-test-k8s-gateway-api-http), CodeQL Analyze (java), e2e all pass.
Aias00
left a comment
There was a problem hiding this comment.
Approved on the technical merits, but please do not merge this before it is coordinated with #8088 — that PR implements the same fix (same DefaultRetryStrategy.resend call site, same DividePlugin partition snapshot, same exchange-attribute carrier) from a different author. Only one of the two can land, and whichever loses should be closed rather than rebased. I have flagged the same thing over there.
Verified locally
shenyu-plugin-divide114/114 andshenyu-plugin-httpclient120/120 green;checkstyle:checkclean ondivide,httpclientandbase.- Rollback: reverting only the two integration points (
DividePlugin.selectPartition,DefaultRetryStrategy.resend) while keepingUpstreamRoutingContextfails 6 of the 12 new cases, and the failure messages are exactly the bug:expected: <[http://second:8080/test]> but was: <[http://opposite:8080/test]>— the retry escaped into the other partition — plusexpectErrorSatisfies ... actual: onComplete()for the exhausted-partition case, which used to silently succeed instead of returning 503. The non-discriminating parameterisations (stable/default, non-canary) stay green, so the test is honest about what it pins. - I checked the health semantics that the retry path relies on:
findAllUpstreamListBySelectorIdreturnstask.getHealthyUpstream()(healthy only), andfindUpstreamListBySelectorIdis that plus the legacy gray-exclusive filter. So filtering the all list by the resolved labels keeps health filtering intact and only drops the legacy gray restriction — which is correct here, becauseDividePluginalready resolves its canary pool from the all-list. UpstreamRoutingContextcopies the label map into a newHashMapand wraps it unmodifiable, so the mid-flightconfig.setStableLabels(...)mutation infailoverRetainsResolvedPartitionAfterRefreshreally is testing snapshot semantics and not accidentally passing. Good test design.DIVIDE_SELECTOR_IDis written withselector.getId()(DividePlugin:213) and the context is created with the same value, so thefilterguard cannot false-negative on the normal path.
Non-blocking
- When
UpstreamRoutingContext.filterrejects a selector mismatch it returnsCollections.emptyList(), which the caller reports as the genericCANNOT_FIND_HEALTHY_UPSTREAM_URL_AFTER_FAILOVER. The pool was healthy; the selector just changed under the request. A distinct message would make that traceable in production. - No
RELEASE-NOTES.mdentry, while other recently merged behaviour changes (#8043) added one. This changes retry routing for canary rules, so it is worth a line. - Minor:
shenyu-plugin-httpclientmoved from the profile block to the top-level test dependency block. Harmless (test scope) but it now applies to every build of the module — worth a word in the PR description so reviewers do not wonder whydividetests need the HTTP client.
Nice catch on removing the attribute at the top of executeRequest; re-entered exchanges inheriting a stale partition would have been an ugly follow-up bug.
| * @param upstreams latest healthy upstreams | ||
| * @return matching nodes, or no nodes if the selector no longer matches | ||
| */ | ||
| public List<Upstream> filter(final String currentSelectorId, final List<Upstream> upstreams) { |
There was a problem hiding this comment.
Returning an empty list on selector mismatch makes the caller fall through to CANNOT_FIND_HEALTHY_UPSTREAM_URL_AFTER_FAILOVER, which claims the pool has no healthy upstream. That is not what happened here — the pool was fine, the selector changed underneath the request.
Since this is the only branch that can produce it, consider making the mismatch visible (a debug log with both selector ids, or a dedicated message on the caller side) so an operator debugging a 503 does not chase a non-existent health problem.
Sync to current master to resolve the stable shenyu-integrated-test-http CI failure (CachePluginTest.testMemoryModle/testRedisModle). Root cause: this branch predated #8079, whose pre-fix CachedResponse.capture() refused to store any response carrying a Vary header, so the cached endpoint was never reused and both requests hit the backend. #8079 keys Vary responses on the selecting request headers instead. Also pulls in the divide/loadbalancer work (#8076, #7172, #7986, #8005, #7821, #8033, #8090). #7172 removeStaleUpstreams reconciles via removeFromMap/triggerRemoveOne, which republish the immutable healthy snapshot under the same lock, so the read-only snapshot model stays correct with no stale retained instances.
Fixes #8019.
Stable and canary requests currently fail over through the legacy gray-exclusive pool, which can route a retry outside the selected label partition. Capture immutable, selector-scoped labels after Divide resolves its initial partition or fallback, then apply those labels to the latest healthy full snapshot before each failover selection. An exhausted partition fails with 503; a backend failure does not select a new fallback. Legacy rules retain their existing gray-exclusive pool.
Validation on OpenJDK 17.0.20:
Full-repository and Docker E2E suites were not run. Local Mockito tests use the existing Byte Buddy agent because self-attachment is unavailable.