Skip to content

fix(system-rule): clamp and smooth process CPU usage snapshot to prevent false-positive blocking - #3670

Open
xu-xiaofei wants to merge 86 commits into
alibaba:masterfrom
xu-xiaofei:fix/system-rule-cpu-usage-clamp-and-smoothing
Open

xu-xiaofei wants to merge 86 commits into
alibaba:masterfrom
xu-xiaofei:fix/system-rule-cpu-usage-clamp-and-smoothing

Conversation

@xu-xiaofei

Copy link
Copy Markdown

Describe what this PR does / why we need it

The system-rule CPU check (SystemRuleManager#checkSystem) consumes the raw per-second processCpuUsage snapshot computed in SystemStatusListener with no smoothing and no bounds.

In containerized environments (cgroup CPU quota + -XX:+UseContainerSupport), we observed that this raw snapshot can exceed the physical bound of 1.0 by large factors: in production, threshold-exceeding snapshots reached 28.75 (i.e. 2875% CPU), and 80.1% of the threshold-exceeding samples were above 1.0, while the real process CPU usage (micrometer process.cpu.usage, which reads the same getProcessCpuLoad() family of metrics) stayed at 0.00-0.31 at those exact seconds.

Since the container's cgroup quota caps the whole process at availableProcessors CPU-seconds per second, any value outside [0.0, 1.0] is physically impossible. A single anomalous sample is consumed directly by checkSystem() and blocks all inbound requests for that second (HTTP 429), producing large false-positive rate-limiting bursts: in our production this caused tens of thousands of spurious 429s across several services during traffic peaks.

Production data and analysis: #3669

Does this pull request fix one issue?

Fixes #3669

Describe how you did it

In SystemStatusListener:

  1. Clamp the raw process CPU usage to [0.0, 1.0] - with the cgroup quota capping the process at availableProcessors CPU-seconds/second, values outside this range can only be measurement anomalies.
  2. Smooth the clamped value with a sliding window over the latest 10 samples (~10 seconds at the 1s tick) before publishing it via getCpuUsage(); a single anomalous sample is diluted to at most 1/10 of the window. We measured a ~10-second read granularity of getProcessCpuTime() in the affected environments (threshold-exceeding values sit strictly on a 1.25 = 10,000ms/1,000ms/8-cores grid), so a 10-sample window covers a full read cycle.
  3. Guard against unusable samples (negative deltas / unreadable processCpuTime) by keeping the last smoothed value.
  4. Keep the existing Math.max(processCpuUsage, systemCpuUsage) semantics of currentCpuUsage unchanged.

No changes to rules, slots, configuration or the checkSystem logic. The public contract (getCpuUsage() returning the CPU usage) is unchanged - only the value becomes bounded and smoothed. Worst-case behavior change: detection of a real sustained overload is delayed by at most a couple of ticks (covered by unit tests).

Describe how to verify it

  • New unit tests SystemStatusListenerTest (6 cases): clamping of an impossible 28.75 sample, dilution of a single spike (0.4,0.4,0.4,0.4 + spike -> 0.52), negative-delta keeps last value, sustained overload (0.9 x10) still triggers, bounded ramp-up delay, partial-window averaging.
  • Reproduction of the incident shape: feed one sample with delta(processCpuTime) = 230,000ms on a 1s tick with 8 cores (raw = 28.75) between normal 0.4 samples - before this patch the snapshot blocks all traffic for that second; after this patch the smoothed value stays at 0.52.

Special notes for reviews

  • The window size (10) was chosen to cover the observed ~10s read granularity; it is a constant and easy to tune.
  • The threading model is unchanged: run() is still invoked by the single-threaded scheduler in SystemRuleManager; currentCpuUsage remains volatile.
  • If maintainers prefer, the clamping alone (one line) already removes the physically-impossible values; the sliding window additionally protects against GC/JIT burst seconds that stay within [0,1] but still exceed typical thresholds for a single tick.

sczyh30 and others added 30 commits December 27, 2023 20:40
Signed-off-by: Eric Zhao <sczyh16@gmail.com>
…uster-server-envoy-rls module (alibaba#2921)

Bumps [snakeyaml](https://bitbucket.org/snakeyaml/snakeyaml) from 1.26 to 1.32.
- [Commits](https://bitbucket.org/snakeyaml/snakeyaml/branches/compare/snakeyaml-1.32..snakeyaml-1.26)

---
updated-dependencies:
- dependency-name: org.yaml:snakeyaml
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Rename: DefaultDegradeSlot -> DefaultCircuitBreakerSlot
* Refine DefaultCircuitBreakerRuleManager
* Add test cases

Signed-off-by: Eric Zhao <sczyh16@gmail.com>
…he-dubbo (alibaba#2923)

Bumps [dubbo](https://github.com/apache/dubbo) from 2.7.3 to 2.7.18.
- [Release notes](https://github.com/apache/dubbo/releases)
- [Changelog](https://github.com/apache/dubbo/blob/3.1/CHANGES.md)
- [Commits](apache/dubbo@dubbo-2.7.3...dubbo-2.7.18)

---
updated-dependencies:
- dependency-name: org.apache.dubbo:dubbo
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Add dockerfile for dashboard with doc update
Signed-off-by: Eric Zhao <sczyh16@gmail.com>
… 1000

* Rename to ThrottlingController
* Improve accuracy: use nanoseconds when necessary and support maxQps threshold > 1000 (i.e. wait < 1ms)

Signed-off-by: Eric Zhao <sczyh16@gmail.com>
* fix thread blocked problem of etcd watcher

in the previous version of EtcdDataSource, when the flowRules defined in the etcd change and trigger etcd watcher, there will be a following error:
2022-12-07 14:36:03.592 [vertx-blocked-thread-checker] WARN io.vertx.core.impl.BlockedThreadChecker [?:?] -  Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 2336 ms, time limit is 2000 ms
2022-12-07 14:36:04.592 [vertx-blocked-thread-checker] WARN io.vertx.core.impl.BlockedThreadChecker [?:?] -  Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 3337 ms, time limit is 2000 ms
2022-12-07 14:36:05.592 [vertx-blocked-thread-checker] WARN io.vertx.core.impl.BlockedThreadChecker [?:?] -  Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 4337 ms, time limit is 2000 ms
2022-12-07 14:36:06.595 [vertx-blocked-thread-checker] WARN io.vertx.core.impl.BlockedThreadChecker [?:?] -  Thread Thread[vert.x-eventloop-thread-0,5,main] has been blocked for 5337 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
 at sun.misc.Unsafe.park(Native Method)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
 at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3334)
 at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
 at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
 at com.htsc.fst.saas.infrastructure.component.EtcdDataSource.readSource(EtcdDataSource.java:92)
 at com.htsc.fst.saas.infrastructure.component.EtcdDataSource.readSource(EtcdDataSource.java:23)
 at com.alibaba.csp.sentinel.datasource.AbstractDataSource.loadConfig(AbstractDataSource.java:44)
 at com.htsc.fst.saas.infrastructure.component.EtcdDataSource.lambda$initWatcher$0(EtcdDataSource.java:74)
 at com.htsc.fst.saas.infrastructure.component.EtcdDataSource$$Lambda$480/1375681611.accept(Unknown Source)
 at io.etcd.jetcd.Watch$1.onNext(Watch.java:183)
 at io.etcd.jetcd.impl.WatchImpl$WatcherImpl.onNext(WatchImpl.java:310)
 at io.etcd.jetcd.impl.WatchImpl$WatcherImpl$$Lambda$488/2088661957.handle(Unknown Source

I guess the reason is : when the watcher hold etcd Client, you can't use etcd Client in the Watch's consumer function. 
Besides the problem above, use getKeyValue() in the watchEvent can reduce once invoke with etcd server to getting the latest flowRules :)

* Update EtcdDataSource.java

Add charset adaptation -- accept the suggestion from @LProDreamAll
…-source (alibaba#2963)

* Fix issue to upgrade curator-recipes and related API.
* Resolve test failure: ClassNotFoundException of ZooKeeperAdmin.
* Remove unused property in pom.
Signed-off-by: Eric Zhao <sczyh16@gmail.com>
Signed-off-by: Eric Zhao <sczyh16@gmail.com>
Signed-off-by: Eric Zhao <sczyh16@gmail.com>
cdfive and others added 29 commits April 28, 2024 09:53
…3375)

Bumps [org.springframework:spring-expression](https://github.com/spring-projects/spring-framework) from 5.1.8.RELEASE to 5.2.24.RELEASE.
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](spring-projects/spring-framework@v5.1.8.RELEASE...v5.2.24.RELEASE)

---
updated-dependencies:
- dependency-name: org.springframework:spring-expression
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…a#3342)

Bumps [com.fasterxml.jackson.core:jackson-databind](https://github.com/FasterXML/jackson) from 2.13.0 to 2.13.4.2.
- [Commits](https://github.com/FasterXML/jackson/commits)

---
updated-dependencies:
- dependency-name: com.fasterxml.jackson.core:jackson-databind
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [org.springframework:spring-context](https://github.com/spring-projects/spring-framework) from 5.1.5.RELEASE to 5.2.21.RELEASE.
- [Release notes](https://github.com/spring-projects/spring-framework/releases)
- [Commits](spring-projects/spring-framework@v5.1.5.RELEASE...v5.2.21.RELEASE)

---
updated-dependencies:
- dependency-name: org.springframework:spring-context
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…baba#3440)

* fix issue 2485 which occur oom when using async servlet request.

* optimize imports

* 1. fix the same issue in the webmvc-v6x
2. improve based on review comments
…pringCloud 2025 version (alibaba#3542)

* feat: support Spring Cloud Alibaba 2025

* Add maven property to fix ci failure

* doc: Update README.md
* Improved Date formatter

* change in naming conv

* change in datetime format

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* chore: update release configuration

* Bump version to 1.8.9
Change-Id: Ic0556103ed65c4c760bb9b21132b72cd22770b68
…isabled, configurable)(alibaba#3565)

* Simple rules should have higher priority than regular expression rules

* Add optional switch to skip regex matching when simple rules already match, default: false, keeps backward compatibility.

* fix RuleManagerTest unit test failure
…incorrect time unit in dashboard (alibaba#3569)

* Fix the bugs mentioned in issue alibaba#3562 and alibaba#3563.
Fix the issue in the spring-webmvc-v6x-adapter where, after configuring http-method-specify:true, the prefixes for different request methods were not concatenated.
Fix incorrect time unit in the tooltip when adding a new circuit breaker rule with statistical window duration.

* Fix the bugs mentioned in issue alibaba#3562 and alibaba#3563.
Fix the issue in the spring-webmvc-v6x-adapter where, after configuring http-method-specify:true, the prefixes for different request methods were not concatenated.
Fix incorrect time unit in the tooltip when adding a new circuit breaker rule with statistical window duration.

* Fixed the issue where the original test class was affected by the concatenation of resource names after adding the logic for the HTTP prefix concatenation.

* remove mseHttpMethodSpecify field and update the unit test as suggested.

* retrigger CI
* chore: remove node_modules from git & add to .gitignore

* fix(docs): normalize README table style (MD060) across adapters and cluster modules

* docs: fix compact table style in parameter flow README

* docs: fix markdownlint MD009 and MD060 in README files

* docs: fix table format in parameter-flow-control README (MD060/MD009)

* docs: fix table format in spring-webmvc README files (MD060/MD009)

* fix: restore UTF-8 encoding and fix markdownlint errors

* fix: wrap remaining bare URLs with angle brackets (MD034)

* fix: remove trailing spaces from table rows (MD009)

---------

Signed-off-by: 赖尧 <yujitang_2006@qq.com>
* fix: Endpoint#toString host formatting; add Endpoint unit test

* test: remove empty EndpointTest.java (fix accidental PR changes)

---------

Signed-off-by: 赖尧 <yujitang_2006@qq.com>
alibaba#3600)

Co-developed-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: support Spring RestClient

Change-Id: I49c8b1cf7a0a6f357b68b58bd352efc5981b76b2
Co-developed-by: OpenCode <noreply@opencode.ai>

* add customized resourceExtractor and fallback test

Change-Id: Ieacaf949222f5871f934d2f0aa3bd6d73d1a087b

* fix: resolve markdown lint errors in README

Change-Id: I2df54be42732b1c1786b4b9189639913b4598fd9
Co-developed-by: OpenCode <noreply@opencode.ai>

* fix: skip tests on Java < 17 for Spring 6.x compatibility

Change-Id: Ia9dceb36c2ef3a9a86a61110446c6aec4de331a9
Co-developed-by: OpenCode <noreply@opencode.ai>

* fix: remove extra trailing blank line in README

Change-Id: I10f1752fa921aa8980e640c4fe7726d3c1764f37
Co-developed-by: OpenCode <noreply@opencode.ai>

* fix: address PR review comments for restclient adapter

- DefaultRestClientFallback now throws SentinelRpcException for consistency with other client adapters (okhttp, apache-httpclient)
- SentinelClientHttpResponse uses text/plain instead of application/json since the body is plain text
- Update tests to reflect new exception-throwing behavior

Change-Id: I778bb9bba1bd24435e667f996ebbff19d734b14a
Co-developed-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor: remove SentinelClientHttpResponse class

No longer needed since DefaultRestClientFallback now throws
SentinelRpcException. Update README, ManualTest, and integration
tests accordingly.

Change-Id: Ic38af10b9a82f99205aeb1bccde8411ec97d618b
Co-developed-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat: add Apache HttpClient 5.x adapter for Sentinel

Add sentinel-apache-httpclient5-adapter module that integrates Sentinel
flow control with Apache HttpClient 5.x outgoing HTTP requests.

Features:
- ExecChainHandler-based integration (SentinelApacheHttpClient5Handler)
- Configurable resource name extraction (METHOD:URL format by default)
- Query string and fragment stripping in resource names
- Customizable fallback handling (throws SentinelRpcException by default)
- Configurable resource name prefix (default: "httpclient:")

Follows the same architecture pattern as the existing HttpClient 4.x
adapter while adapting to HC5's ExecChainHandler API.

Closes alibaba#3614

Change-Id: I966295da9419beac99451fc6c9c31b1d75a9e7e6
Co-developed-by: Cursor <noreply@cursor.com>

* chore: bump version to 1.8.10-SNAPSHOT

After the 1.8.9 release, several PRs have been merged but
the development version was not updated. Bump to 1.8.10-SNAPSHOT
for next development iteration.

Change-Id: I2b7ba2b5b4e9e39d1f9749288d54b9926a512132
Co-developed-by: Cursor <noreply@cursor.com>

* fix: fix markdown table format to pass document-lint

Change-Id: I1fe4682e41185dc7c8ed30541989f2881ba87549
Co-developed-by: Cursor <noreply@cursor.com>

* fix: update @author from qihuai.wyq to uuuyuqi

Change-Id: Ie28f99434badfc00decc914ffd27f83c837d947d
Co-developed-by: Cursor <noreply@cursor.com>

* chore: remove accidentally committed .vscode/settings.json

Change-Id: I60caec5b17e92a79827671e531a0818f61163b48
Co-developed-by: Cursor <noreply@cursor.com>

* docs: add HC4 to HC5 migration notes for resource naming

Document the default resource name format difference between HC4 and HC5
adapters so users understand that existing flow control rules will not
match automatically after migration, and provide a snippet for preserving
the legacy URI-only naming via a custom extractor.

Change-Id: Iadacf3d3288acb4b23ce67441a88c0b8ad0f0b6e
Co-developed-by: Claude <noreply@anthropic.com>
Bump version from 1.8.10-SNAPSHOT to 1.8.10 across pom.xml,
Constants.java, Dockerfile, and README docs in preparation
for the 1.8.10 release.
After the 1.8.10 release, bump to 1.8.11-SNAPSHOT
for next development iteration.
…ent false-positive blocking

The raw 1-second processCpuUsage snapshot can exceed the physical bound of
1.0 in containerized environments (observed up to 28.75 in production while
the real process CPU usage was below 0.05), because delta(processCpuTime)
may arrive in coarse jumps (~10s read granularity) while the listener ticks
every second. The snapshot is consumed directly by SystemRuleManager
#checkSystem, so a single anomalous second blocks all inbound requests.

- clamp the raw value to [0.0, 1.0] (the cgroup quota caps the process at
  availableProcessors CPU-seconds per second, so anything above is a
  measurement anomaly)
- publish a 10-sample sliding average instead of the raw snapshot
- keep the last smoothed value when the sample is unusable (negative deltas)
- keep the existing Math.max(processCpuUsage, systemCpuUsage) semantics

See alibaba#3669 for production data and analysis.
@CLAassistant

CLAassistant commented Oct 23, 2026 •

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
36 out of 37 committers have signed the CLA.

✅ kirklin
✅ EzrealOf
✅ sczyh30
✅ xiaozzzi
✅ wenshao
✅ garroshh
✅ pandaapo
✅ wuwen5
✅ quguai
✅ JosephZhang3
✅ wilsonwu
✅ frank-zsy
✅ Yan1025
✅ shichaoyuan
✅ robberphex
✅ saaivenk
✅ LearningGp
✅ clara0
✅ DepUpdater
✅ cdfive
✅ Linzyoo
✅ marmot-z
✅ mawen12
✅ dowenliu-xyz
✅ 24kpure
✅ karl-sy
✅ ruansheng8
✅ ylnxwlp
✅ hermya
✅ xierunzi
✅ XenoAmess
✅ manojks1999
✅ Layau-code
✅ gaoyf
✅ z521598
✅ uuuyuqi
❌ Xu


Xu seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.