Repository navigation
fix(system-rule): clamp and smooth process CPU usage snapshot to prevent false-positive blocking - #3670
Open
xu-xiaofei wants to merge 86 commits into
Open
fix(system-rule): clamp and smooth process CPU usage snapshot to prevent false-positive blocking#3670xu-xiaofei wants to merge 86 commits into
xu-xiaofei wants to merge 86 commits into
Conversation
Signed-off-by: Eric Zhao <sczyh16@gmail.com>
…uster-server-envoy-rls module (alibaba#2921) Bumps [snakeyaml](https://bitbucket.org/snakeyaml/snakeyaml) from 1.26 to 1.32. - [Commits](https://bitbucket.org/snakeyaml/snakeyaml/branches/compare/snakeyaml-1.32..snakeyaml-1.26) --- updated-dependencies: - dependency-name: org.yaml:snakeyaml dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Rename: DefaultDegradeSlot -> DefaultCircuitBreakerSlot * Refine DefaultCircuitBreakerRuleManager * Add test cases Signed-off-by: Eric Zhao <sczyh16@gmail.com>
…he-dubbo (alibaba#2923) Bumps [dubbo](https://github.com/apache/dubbo) from 2.7.3 to 2.7.18. - [Release notes](https://github.com/apache/dubbo/releases) - [Changelog](https://github.com/apache/dubbo/blob/3.1/CHANGES.md) - [Commits](apache/dubbo@dubbo-2.7.3...dubbo-2.7.18) --- updated-dependencies: - dependency-name: org.apache.dubbo:dubbo dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Add dockerfile for dashboard with doc update
Signed-off-by: Eric Zhao <sczyh16@gmail.com>
… 1000 * Rename to ThrottlingController * Improve accuracy: use nanoseconds when necessary and support maxQps threshold > 1000 (i.e. wait < 1ms) Signed-off-by: Eric Zhao <sczyh16@gmail.com>
* fix thread blocked problem of etcd watcher in the previous version of EtcdDataSource, when the flowRules defined in the etcd change and trigger etcd watcher, there will be a following error: 2022-12-07 14:36:03.592 [vertx-blocked-thread-checker] WARN io.vertx.core.impl.BlockedThreadChecker [?:?] - Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 2336 ms, time limit is 2000 ms 2022-12-07 14:36:04.592 [vertx-blocked-thread-checker] WARN io.vertx.core.impl.BlockedThreadChecker [?:?] - Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 3337 ms, time limit is 2000 ms 2022-12-07 14:36:05.592 [vertx-blocked-thread-checker] WARN io.vertx.core.impl.BlockedThreadChecker [?:?] - Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 4337 ms, time limit is 2000 ms 2022-12-07 14:36:06.595 [vertx-blocked-thread-checker] WARN io.vertx.core.impl.BlockedThreadChecker [?:?] - Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 5337 ms, time limit is 2000 ms io.vertx.core.VertxException: Thread blocked at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707) at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3334) at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) at com.htsc.fst.saas.infrastructure.component.EtcdDataSource.readSource(EtcdDataSource.java:92) at com.htsc.fst.saas.infrastructure.component.EtcdDataSource.readSource(EtcdDataSource.java:23) at com.alibaba.csp.sentinel.datasource.AbstractDataSource.loadConfig(AbstractDataSource.java:44) at com.htsc.fst.saas.infrastructure.component.EtcdDataSource.lambda$initWatcher$0(EtcdDataSource.java:74) at com.htsc.fst.saas.infrastructure.component.EtcdDataSource$$Lambda$480/1375681611.accept(Unknown Source) at io.etcd.jetcd.Watch$1.onNext(Watch.java:183) at io.etcd.jetcd.impl.WatchImpl$WatcherImpl.onNext(WatchImpl.java:310) at io.etcd.jetcd.impl.WatchImpl$WatcherImpl$$Lambda$488/2088661957.handle(Unknown Source I guess the reason is : when the watcher hold etcd Client, you can't use etcd Client in the Watch's consumer function. Besides the problem above, use getKeyValue() in the watchEvent can reduce once invoke with etcd server to getting the latest flowRules :) * Update EtcdDataSource.java Add charset adaptation -- accept the suggestion from @LProDreamAll
… macOS aarch64 compile error (alibaba#2993)
…-source (alibaba#2963) * Fix issue to upgrade curator-recipes and related API. * Resolve test failure: ClassNotFoundException of ZooKeeperAdmin. * Remove unused property in pom.
Signed-off-by: frank-zsy <syzhao1988@126.com>
Signed-off-by: Eric Zhao <sczyh16@gmail.com>
Signed-off-by: Eric Zhao <sczyh16@gmail.com>
Signed-off-by: Eric Zhao <sczyh16@gmail.com>
…3375) Bumps [org.springframework:spring-expression](https://github.com/spring-projects/spring-framework) from 5.1.8.RELEASE to 5.2.24.RELEASE. - [Release notes](https://github.com/spring-projects/spring-framework/releases) - [Commits](spring-projects/spring-framework@v5.1.8.RELEASE...v5.2.24.RELEASE) --- updated-dependencies: - dependency-name: org.springframework:spring-expression dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…/blockHandler with different parameter types (alibaba#3395)
…a#3342) Bumps [com.fasterxml.jackson.core:jackson-databind](https://github.com/FasterXML/jackson) from 2.13.0 to 2.13.4.2. - [Commits](https://github.com/FasterXML/jackson/commits) --- updated-dependencies: - dependency-name: com.fasterxml.jackson.core:jackson-databind dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [org.springframework:spring-context](https://github.com/spring-projects/spring-framework) from 5.1.5.RELEASE to 5.2.21.RELEASE. - [Release notes](https://github.com/spring-projects/spring-framework/releases) - [Commits](spring-projects/spring-framework@v5.1.5.RELEASE...v5.2.21.RELEASE) --- updated-dependencies: - dependency-name: org.springframework:spring-context dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…baba#3440) * fix issue 2485 which occur oom when using async servlet request. * optimize imports * 1. fix the same issue in the webmvc-v6x 2. improve based on review comments
…pringCloud 2025 version (alibaba#3542) * feat: support Spring Cloud Alibaba 2025 * Add maven property to fix ci failure * doc: Update README.md
* Improved Date formatter * change in naming conv * change in datetime format * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* chore: update release configuration * Bump version to 1.8.9
Change-Id: Ic0556103ed65c4c760bb9b21132b72cd22770b68
…isabled, configurable)(alibaba#3565) * Simple rules should have higher priority than regular expression rules * Add optional switch to skip regex matching when simple rules already match, default: false, keeps backward compatibility. * fix RuleManagerTest unit test failure
…incorrect time unit in dashboard (alibaba#3569) * Fix the bugs mentioned in issue alibaba#3562 and alibaba#3563. Fix the issue in the spring-webmvc-v6x-adapter where, after configuring http-method-specify:true, the prefixes for different request methods were not concatenated. Fix incorrect time unit in the tooltip when adding a new circuit breaker rule with statistical window duration. * Fix the bugs mentioned in issue alibaba#3562 and alibaba#3563. Fix the issue in the spring-webmvc-v6x-adapter where, after configuring http-method-specify:true, the prefixes for different request methods were not concatenated. Fix incorrect time unit in the tooltip when adding a new circuit breaker rule with statistical window duration. * Fixed the issue where the original test class was affected by the concatenation of resource names after adding the logic for the HTTP prefix concatenation. * remove mseHttpMethodSpecify field and update the unit test as suggested. * retrigger CI
* chore: remove node_modules from git & add to .gitignore * fix(docs): normalize README table style (MD060) across adapters and cluster modules * docs: fix compact table style in parameter flow README * docs: fix markdownlint MD009 and MD060 in README files * docs: fix table format in parameter-flow-control README (MD060/MD009) * docs: fix table format in spring-webmvc README files (MD060/MD009) * fix: restore UTF-8 encoding and fix markdownlint errors * fix: wrap remaining bare URLs with angle brackets (MD034) * fix: remove trailing spaces from table rows (MD009) --------- Signed-off-by: 赖尧 <yujitang_2006@qq.com>
* fix: Endpoint#toString host formatting; add Endpoint unit test * test: remove empty EndpointTest.java (fix accidental PR changes) --------- Signed-off-by: 赖尧 <yujitang_2006@qq.com>
alibaba#3600) Co-developed-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: support Spring RestClient Change-Id: I49c8b1cf7a0a6f357b68b58bd352efc5981b76b2 Co-developed-by: OpenCode <noreply@opencode.ai> * add customized resourceExtractor and fallback test Change-Id: Ieacaf949222f5871f934d2f0aa3bd6d73d1a087b * fix: resolve markdown lint errors in README Change-Id: I2df54be42732b1c1786b4b9189639913b4598fd9 Co-developed-by: OpenCode <noreply@opencode.ai> * fix: skip tests on Java < 17 for Spring 6.x compatibility Change-Id: Ia9dceb36c2ef3a9a86a61110446c6aec4de331a9 Co-developed-by: OpenCode <noreply@opencode.ai> * fix: remove extra trailing blank line in README Change-Id: I10f1752fa921aa8980e640c4fe7726d3c1764f37 Co-developed-by: OpenCode <noreply@opencode.ai> * fix: address PR review comments for restclient adapter - DefaultRestClientFallback now throws SentinelRpcException for consistency with other client adapters (okhttp, apache-httpclient) - SentinelClientHttpResponse uses text/plain instead of application/json since the body is plain text - Update tests to reflect new exception-throwing behavior Change-Id: I778bb9bba1bd24435e667f996ebbff19d734b14a Co-developed-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor: remove SentinelClientHttpResponse class No longer needed since DefaultRestClientFallback now throws SentinelRpcException. Update README, ManualTest, and integration tests accordingly. Change-Id: Ic38af10b9a82f99205aeb1bccde8411ec97d618b Co-developed-by: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat: add Apache HttpClient 5.x adapter for Sentinel Add sentinel-apache-httpclient5-adapter module that integrates Sentinel flow control with Apache HttpClient 5.x outgoing HTTP requests. Features: - ExecChainHandler-based integration (SentinelApacheHttpClient5Handler) - Configurable resource name extraction (METHOD:URL format by default) - Query string and fragment stripping in resource names - Customizable fallback handling (throws SentinelRpcException by default) - Configurable resource name prefix (default: "httpclient:") Follows the same architecture pattern as the existing HttpClient 4.x adapter while adapting to HC5's ExecChainHandler API. Closes alibaba#3614 Change-Id: I966295da9419beac99451fc6c9c31b1d75a9e7e6 Co-developed-by: Cursor <noreply@cursor.com> * chore: bump version to 1.8.10-SNAPSHOT After the 1.8.9 release, several PRs have been merged but the development version was not updated. Bump to 1.8.10-SNAPSHOT for next development iteration. Change-Id: I2b7ba2b5b4e9e39d1f9749288d54b9926a512132 Co-developed-by: Cursor <noreply@cursor.com> * fix: fix markdown table format to pass document-lint Change-Id: I1fe4682e41185dc7c8ed30541989f2881ba87549 Co-developed-by: Cursor <noreply@cursor.com> * fix: update @author from qihuai.wyq to uuuyuqi Change-Id: Ie28f99434badfc00decc914ffd27f83c837d947d Co-developed-by: Cursor <noreply@cursor.com> * chore: remove accidentally committed .vscode/settings.json Change-Id: I60caec5b17e92a79827671e531a0818f61163b48 Co-developed-by: Cursor <noreply@cursor.com> * docs: add HC4 to HC5 migration notes for resource naming Document the default resource name format difference between HC4 and HC5 adapters so users understand that existing flow control rules will not match automatically after migration, and provide a snippet for preserving the legacy URI-only naming via a custom extractor. Change-Id: Iadacf3d3288acb4b23ce67441a88c0b8ad0f0b6e Co-developed-by: Claude <noreply@anthropic.com>
Bump version from 1.8.10-SNAPSHOT to 1.8.10 across pom.xml, Constants.java, Dockerfile, and README docs in preparation for the 1.8.10 release.
After the 1.8.10 release, bump to 1.8.11-SNAPSHOT for next development iteration.
…ent false-positive blocking The raw 1-second processCpuUsage snapshot can exceed the physical bound of 1.0 in containerized environments (observed up to 28.75 in production while the real process CPU usage was below 0.05), because delta(processCpuTime) may arrive in coarse jumps (~10s read granularity) while the listener ticks every second. The snapshot is consumed directly by SystemRuleManager #checkSystem, so a single anomalous second blocks all inbound requests. - clamp the raw value to [0.0, 1.0] (the cgroup quota caps the process at availableProcessors CPU-seconds per second, so anything above is a measurement anomaly) - publish a 10-sample sliding average instead of the raw snapshot - keep the last smoothed value when the sample is unusable (negative deltas) - keep the existing Math.max(processCpuUsage, systemCpuUsage) semantics See alibaba#3669 for production data and analysis.
|
Xu seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe what this PR does / why we need it
The system-rule CPU check (
SystemRuleManager#checkSystem) consumes the raw per-secondprocessCpuUsagesnapshot computed inSystemStatusListenerwith no smoothing and no bounds.In containerized environments (cgroup CPU quota +
-XX:+UseContainerSupport), we observed that this raw snapshot can exceed the physical bound of 1.0 by large factors: in production, threshold-exceeding snapshots reached 28.75 (i.e. 2875% CPU), and 80.1% of the threshold-exceeding samples were above 1.0, while the real process CPU usage (micrometerprocess.cpu.usage, which reads the samegetProcessCpuLoad()family of metrics) stayed at 0.00-0.31 at those exact seconds.Since the container's cgroup quota caps the whole process at
availableProcessorsCPU-seconds per second, any value outside [0.0, 1.0] is physically impossible. A single anomalous sample is consumed directly bycheckSystem()and blocks all inbound requests for that second (HTTP 429), producing large false-positive rate-limiting bursts: in our production this caused tens of thousands of spurious 429s across several services during traffic peaks.Production data and analysis: #3669
Does this pull request fix one issue?
Fixes #3669
Describe how you did it
In
SystemStatusListener:availableProcessorsCPU-seconds/second, values outside this range can only be measurement anomalies.getCpuUsage(); a single anomalous sample is diluted to at most 1/10 of the window. We measured a ~10-second read granularity ofgetProcessCpuTime()in the affected environments (threshold-exceeding values sit strictly on a 1.25 = 10,000ms/1,000ms/8-cores grid), so a 10-sample window covers a full read cycle.processCpuTime) by keeping the last smoothed value.Math.max(processCpuUsage, systemCpuUsage)semantics ofcurrentCpuUsageunchanged.No changes to rules, slots, configuration or the
checkSystemlogic. The public contract (getCpuUsage()returning the CPU usage) is unchanged - only the value becomes bounded and smoothed. Worst-case behavior change: detection of a real sustained overload is delayed by at most a couple of ticks (covered by unit tests).Describe how to verify it
SystemStatusListenerTest(6 cases): clamping of an impossible 28.75 sample, dilution of a single spike (0.4,0.4,0.4,0.4 + spike -> 0.52), negative-delta keeps last value, sustained overload (0.9 x10) still triggers, bounded ramp-up delay, partial-window averaging.delta(processCpuTime) = 230,000mson a 1s tick with 8 cores (raw = 28.75) between normal 0.4 samples - before this patch the snapshot blocks all traffic for that second; after this patch the smoothed value stays at 0.52.Special notes for reviews
run()is still invoked by the single-threaded scheduler inSystemRuleManager;currentCpuUsageremainsvolatile.