Skip to content

fix: honor MaxConsecutiveFailures for validateHealth failures - #2006

Merged
gianlucam76 merged 1 commit into
projectsveltos:mainfrom
gianlucam76:max-retry
Sep 29, 2026
Merged

gianlucam76 merged 1 commit into
projectsveltos:mainfrom
gianlucam76:max-retry

Conversation

@gianlucam76

Copy link
Copy Markdown
Member

MaxConsecutiveFailures bounds retries for deploy failures, but a failing validateHealth check was exempted from it entirely. handleDeployerError special-cased *clusterops.HealthCheckError to always return as retriable, skipping the maxNumberOfConsecutiveFailureReached check that every other error type goes through. The result: FeatureSummary.Status reported Failed with ConsecutiveFailures climbing in status, MaxConsecutiveFailures configured right next to it, and no enforcement ever happening: a health check that never passes retried forever regardless of the configured limit.

handleDeployerError's HealthCheckError branch now checks maxNumberOfConsecutiveFailureReached before deciding whether to keep retrying, exactly like the generic error path already does. Once the limit is reached, the feature transitions to FailedNonRetriable the same way any other exhausted-retry error does. The transition logic (status update + event) is factored into a shared markMaxConsecutiveFailuresReached helper used by both the health-check branch and the existing generic fallthrough, so there is a single place that decides "budget exhausted" instead of two that could disagree.

proceedDeployingClusterSummary needs no change: once the budget is spent, handleDeployerError no longer returns a *HealthCheckError, so its retry-pacing carve-out simply stops applying on its own.

Below the limit, the health-check branch no longer re-applies FeatureStatusFailed itself: proceedDeployingFeature already records the failure with this same error before calling handleDeployerError, so doing it again there was double-counting every consecutive failure and tripping the limit at roughly half the configured value.

MaxConsecutiveFailures bounds retries for deploy failures, but a failing
validateHealth check was exempted from it entirely. handleDeployerError
special-cased *clusterops.HealthCheckError to always return as retriable,
skipping the maxNumberOfConsecutiveFailureReached check that every other
error type goes through. The result: FeatureSummary.Status reported
Failed with ConsecutiveFailures climbing in status, MaxConsecutiveFailures
configured right next to it, and no enforcement ever happening: a health
check that never passes retried forever regardless of the configured limit.

handleDeployerError's HealthCheckError branch now checks
maxNumberOfConsecutiveFailureReached before deciding whether to keep
retrying, exactly like the generic error path already does. Once the limit
is reached, the feature transitions to FailedNonRetriable the same way
any other exhausted-retry error does. The transition logic (status update +
event) is factored into a shared markMaxConsecutiveFailuresReached helper
used by both the health-check branch and the existing generic fallthrough,
so there is a single place that decides "budget exhausted" instead of two
that could disagree.

proceedDeployingClusterSummary needs no change: once the budget is spent,
handleDeployerError no longer returns a *HealthCheckError, so its
retry-pacing carve-out simply stops applying on its own.

Below the limit, the health-check branch no longer re-applies
FeatureStatusFailed itself: proceedDeployingFeature already records the
failure with this same error before calling handleDeployerError, so doing
it again there was double-counting every consecutive failure and tripping
the limit at roughly half the configured value.
@gianlucam76
gianlucam76 merged commit 9698b62 into projectsveltos:main Sep 29, 2026
12 checks passed
@gianlucam76
gianlucam76 deleted the max-retry branch September 29, 2026 09:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant