Skip to content

core: Reset configSelector on realChannel while entering IDLE state - #12832

Merged
kannanjgithub merged 3 commits into
grpc:masterfrom
kkalin68:xds-fix-channel-idle-limbo
Sep 4, 2026
Merged

kannanjgithub merged 3 commits into
grpc:masterfrom
kkalin68:xds-fix-channel-idle-limbo

Conversation

@kkalin68

@kkalin68 kkalin68 commented May 27, 2026 •

Copy link
Copy Markdown
Contributor

ManagedChannel will be stuck in IDLE state when xDS control plane doesn't have a resource anymore.
The scenario is following:

  1. Channel is open for a xds resource.
  2. XdsNameResolver subscribes to the resource on xDS control plane.
  3. The resource is removed from xDS control plane for extended period of time (unhealthy for more than the idle timeout on Channel)
  4. Channel enters into TRANSIENT_FAILURE
  5. The Idle timeout triggers and Channel shutdowns XdsNameResolver and other resources. xDS watchers are removed.
  6. The resource comes back online on xDS control plane.
  7. A new GRPC call is executed targeting the channel.
  8. The channel stays in IDLE state and reports: "io.grpc.StatusRuntimeException: UNAVAILABLE: LDS resource xxxx does not exist nodeID: yyyy" because realChannel.configSelector still points to old state.
public <ReqT, RespT> ClientCall<ReqT, RespT> newCall(
        MethodDescriptor<ReqT, RespT> method, CallOptions callOptions) {
      if (configSelector.get() != INITIAL_PENDING_SELECTOR) {
        return newClientCall(method, callOptions);
      }
...

@linux-foundation-easycla

linux-foundation-easycla Bot commented May 27, 2026 •

Copy link
Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.

  • ✅ login: kkalin68 / name: Konstantin Kalin (f09b7d9)

@AgraVator

AgraVator commented Jun 2, 2026 •

Copy link
Copy Markdown
Contributor

Hey,
Please fix the failing test cases. Thanks.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a ManagedChannel idle-mode edge case where a stale InternalConfigSelector can prevent the channel from properly exiting IDLE after xDS resources disappear and later reappear.

Changes:

  • Reset realChannel’s config selector to INITIAL_PENDING_SELECTOR when entering IDLE, forcing the next call to trigger resolver/LB restart.
  • Prevent pending-call reprocessing when updateConfigSelector() is called with INITIAL_PENDING_SELECTOR (only reprocess when transitioning away from the initial pending selector).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread core/src/main/java/io/grpc/internal/ManagedChannelImpl.java
@kkalin68

kkalin68 commented Jun 2, 2026

Copy link
Copy Markdown
Contributor Author

Hey, Please fix the failing test cases. Thanks.

The tests are passing locally. Also the failures in tests(11) and tests(17) runs are unrelated to changes I made. And same tests passed in tests(8) and test(21). Also I don't see an option to retrigger the failed tests.

@kkalin68

kkalin68 commented Jun 8, 2026

Copy link
Copy Markdown
Contributor Author

Is it anything else I should do to get the change reviewed ? We face this issue occasionally in production and it requires a service restart to reset GRPC channel

@kannanjgithub

Copy link
Copy Markdown
Contributor

Is it anything else I should do to get the change reviewed ? We face this issue occasionally in production and it requires a service restart to reset GRPC channel

Can you address the Copilot review comment above for adding a new test?

@kkalin68

Copy link
Copy Markdown
Contributor Author

Is it anything else I should do to get the change reviewed ? We face this issue occasionally in production and it requires a service restart to reset GRPC channel

Can you address the Copilot review comment above for adding a new test?

I added tests. Please note, I used Copilot to navigate and create the tests

@kkalin68

Copy link
Copy Markdown
Contributor Author

Failed test(11) is not related to my changes. Same test passed for other Java versions

@kkalin68

Copy link
Copy Markdown
Contributor Author

Is it anything else I should do to get the change reviewed ? We face this issue occasionally in production and it requires a service restart to reset GRPC channel

Can you address the Copilot review comment above for adding a new test?

Is there anything I should do to get this merged ?

@kannanjgithub kannanjgithub added the kokoro:force-run Add this label to a PR to tell Kokoro to re-run all tests. Not generally necessary label Aug 18, 2026
@grpc-kokoro grpc-kokoro removed the kokoro:force-run Add this label to a PR to tell Kokoro to re-run all tests. Not generally necessary label Aug 18, 2026
@kkalin68

kkalin68 commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

@AgraVator Could you please take a look at the change ?

@AgraVator

Copy link
Copy Markdown
Contributor

@AgraVator Could you please take a look at the change ?

Hey @kkalin68 feel free to merge as @kannanjgithub has already approved the changes

@kkalin68

kkalin68 commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor Author

@AgraVator @kannanjgithub GH tells me I cannot merge since PR needs 2 reviewers and only one is approved. Or may be the issue is in failed build for Java 11.

@kannanjgithub kannanjgithub changed the title Reset configSelector on realChannel while entering IDLE state core: Reset configSelector on realChannel while entering IDLE state Sep 4, 2026
@kannanjgithub
kannanjgithub merged commit fc43144 into grpc:master Sep 4, 2026
16 of 18 checks passed
AgraVator added a commit to AgraVator/grpc-java that referenced this pull request Sep 17, 2026
Resolves one conflict in ManagedChannelImpl.updateConfigSelector().

Upstream grpc#12832 resets the config selector when the channel enters IDLE, by
calling realChannel.updateConfigSelector(INITIAL_PENDING_SELECTOR) from
enterIdleMode(), and guards the method so that this reset does not reprocess
the queued calls. This branch had rewritten the same method to release the
pending calls through releasePendingCalls() and no longer has the prevConfig
check that the guard was added to, so the two changes could not be combined
textually.

The reset is now handled by an early return that keeps upstream's behaviour:
the new config selector is stored, but nothing is released. It also clears
initialConfigResolved and lastResolutionError, because entering IDLE shuts the
name resolver down and the channel really is back to the state it was in before
its first resolution. Leaving initialConfigResolved set would make a resolution
failure after the idle period skip the default service config in
onConfigError(), and a stale lastResolutionError would be reported as the delay
reason for a call queued after the channel went idle.

Everything else merged cleanly.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants