Repository navigation
branch-4.1: [fix](test) run search score TopN case in nonConcurrent group - #67363
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
COMPILE has failed twice here (builds 1035610 and 1035744) and both landed on the same TeamCity agent, This PR changes exactly one line in one regression .groovy file (adding the nonConcurrent group tag), so it cannot affect the BE C++ thrift build. Meanwhile seven other PRs compiled green in the same window on other agents (builds 1035762, 1035766, 1035768, 1035774, 1035782, 1035786, 1035788), so the branch tip is fine and this looks like a stale/incompatible thrift in that one agent's thirdparty. Re-triggering so it can be scheduled elsewhere. |
|
run compile |
|
Correction to my earlier comment: this is not specific to one agent, it is a fleet-wide thirdparty problem on the CI right now. COMPILE has now failed four times here (1035610, 1035744, 1035807, 1035947) across four different agents (172.16.0.152 twice, .215, .42), always at: And other unrelated PRs are failing the same way in the same window: build 1035948 (#67398) fails at the identical thrift line, while 1036005 (#67214) and 1036002 (#66913) fail on the paimon This PR changes one line in one regression .groovy file, so it cannot be the cause. I am not re-triggering any further; this needs the CI/thirdparty owners to refresh the prebuilt thirdparty on the build fleet. |
|
Correction to the reasoning in this PR's description: I wrote that branch-4.1 "lacks master's prevent_search_row_fallback path (added by #66052), so on this branch a search that does not take the index surfaces the internal error directly instead of falling back". That is wrong. So master is exposed to the same flakiness, and I have opened #67402 to apply the same nonConcurrent grouping there. Everything else in the description — the measured 2-of-4 failure rate on branch-4.1, the local 1-BE/3-BE runs passing, and the fix itself — is unchanged. |
|
run compile |
|
Narrowing this down further: the breakage is specific to branch-4.1, not the CI fleet as a whole. Evidence from the last few hours:
So the prebuilt thirdparty currently used for branch-4.1 builds ships a thrift that no longer provides I have stopped re-triggering. This needs the CI/thirdparty owners to fix the branch-4.1 build environment (or branch-4.1 to be updated for the newer thrift API). |
cfda33f to
d2bb529
Compare
|
run buildall |
…roup test_search_score_topn_predicates fails intermittently on the shared P0 pipeline with '[INTERNAL_ERROR]SearchExpr should not be executed without inverted index' on its first query, right after CREATE TABLE + INSERT + sync. Observed 2 failures out of 4 runs on branch-4.1 CI (builds 1034808 and 1035495 failed, 1035120 and 1035335 passed), so it randomly reddens unrelated branch-4.1 PRs. The case never fails when run alone: it passes locally against both a 1-BE and a 3-BE cluster, and passed on the CI rerun 1035120 with the identical code, which points at contention on the shared P0 runner rather than at the query itself. branch-4.1 also lacks master's prevent_search_row_fallback path (added by apache#66052), so a search that does not take the index reports this error instead of falling back. Move it to the nonConcurrent group, as 35 other inverted_index_p0 cases on this branch already do, so it runs serialized. Verified: suite passes locally with the tag applied.
d2bb529 to
badc2c6
Compare
|
run buildall |
Related PR: #67363 (same fix on branch-4.1), #67327 (backported the suite to branch-4.1), #65821 (introduced the suite on master) Problem Summary: `test_search_score_topn_predicates` (added on master by #65821) runs in the shared **concurrent** P0 group. On branch-4.1 it fails intermittently on its very first query, right after `CREATE TABLE` + `INSERT` + `sync`, with: ``` errCode = 2, detailMessage = [INTERNAL_ERROR]SearchExpr should not be executed without inverted index ``` This was observed on branch-4.1 CI (2 failures out of 4 runs) and fixed there by #67363, which moves the case into the `nonConcurrent` group. master carries the **identical** suite grouping (`"p0"`), so it is exposed to the same intermittent failure when the case races other P0 cases. This PR applies the same one-line fix to master to keep it out of the concurrent group. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…t_generation (#70308) ### Background An auto-partition table with `partition.retention_count = N` keeps only its N latest historical partitions: the FE's `DynamicPartitionScheduler` drops older ones on every run, once per `dynamic_partition_check_interval_seconds` (600 by default). `partition_p0/auto_partition/test_auto_new_recycle` lowers that interval to 1 s and checks the partition count after each load or ALTER. `audit/test_sql_digest_generation` checks that the audit log gets an `sql_digest` only for queries slower than the FE config `sql_digest_generation_threshold_ms`, once with the threshold at 0 and once at 100000. Both suites are in group `nonConcurrent`. ### What problem does this PR solve? Issue Number: None Related PR: #65375 and #65646 (the branch-4.1 recycle-suite fixes by @shuke987, ported here), #65343 (the branch-4.1 digest-suite fix by @shuke987, ported here), #65435 (master's earlier move of the recycle suite's day-boundary guard) PR Type: Test Problem Summary: Both suites rely on timing the product does not promise. The recycle suite checks the partition count after a fixed `sleep(8000)` (5 s after an ALTER), while the drop happens asynchronously at an unbounded time after the config change; branch-4.1's NonConcurrent pipeline failed at the first such check repeatedly (#65375). The digest suite's threshold-0 check needs the query's elapsed time to be above 0 ms. This PR gives both suites their branch-4.1/4.2 form: the recycle suite polls the partition count every second, for at most 1200 s, and widens its day-boundary guard to cover the poll; the digest query adds `sleep(1)`. Both files become identical to branch-4.2. ### Details - **Failure:** On master's public NonConcurrent pipeline neither suite has failed on its own since mid-July: of about 8,000 runs each, the recycle suite failed 3 times and the digest suite 87 times, all in builds whose cluster had died (most suites failing). The failures that motivated the fixes are on branch-4.1's pipeline (#65375); #65343 made the digest query deterministic there. - **Cause:** `ADMIN SET FRONTEND CONFIG` sets the scheduler's new interval at once, but the scheduler sleeps in 5-second slices and notices the change only at the next slice. It then runs `executeDynamicPartition` over every registered table before the drop. So the drop lands 0 to 5 s plus one full pass after the config change, and a pass longer than about 3 s fails the 8-second check. In the digest suite, `AuditLogHelper` writes a digest only when `elapseMs > sql_digest_generation_threshold_ms`, and nothing bounds the elapsed time of `select *` on a two-row table from below (9 to 15 ms locally). With `sleep(1)` on each row it takes about 2 s, still far below the second check's 100000 ms. - **Stability evidence:** The recycle checks now wait for the drop and fail only when the count misses the target for 1200 s. The poll runs on the suite thread, because the framework sets `Awaitility.pollInSameThread()` (#68338), so it does not leak connections. Shortening one of master's waits to 1 s reproduces the branch-4.1 failure shape, and the ported suite passes; mutants show that both suites still catch wrong results (see Testing). ### Testing All runs used `./run-regression-test.sh --run -d partition_p0/auto_partition -s test_auto_new_recycle` or `--run -d audit -s test_sql_digest_generation` against a local single-FE, single-BE cluster built from master. | Suite | Version | Result | |---|---|---| | `test_auto_new_recycle` | master | passes | | `test_auto_new_recycle` | this PR | passes; the polls reached the target after 1, 2, 1, 3 and 3 s | | `test_auto_new_recycle` | master, with the wait before the 10-partition check cut to 1 s | fails as #65375 describes: `expected: <101> but was: <10>` (master's `assertEquals` has its arguments swapped, so 101 partitions were still there) | | `test_auto_new_recycle` | this PR, expecting 11 partitions, poll capped at 20 s | fails as it should: `ConditionTimeoutException` | | `test_sql_digest_generation` | master | passes | | `test_sql_digest_generation` | this PR | passes; the query took 2,024 and 2,025 ms | | `test_sql_digest_generation` | this PR, second threshold 1000 ms | fails as it should: `Count of sql_digest changed` | The PR NonConcurrent pipeline runs both suites; cloud_p0 skips `nonConcurrent` suites. ### Notes for Reviewers - The day-boundary guard before the final `now()`-based section goes from 100 s to 1300 s, as in #65646, so that it covers the 1200 s poll plus the insert and checks. It sleeps only when that section would otherwise start within 1300 s of midnight. - Four more suites fixed in the same branch-4.1 batch need nothing on master: #65340 (master #68449 enables the publish debug point on all FEs, superseding the open port #65556), #65929 (master #68329 fixed the actual cause, a connection leak), #65885 (master #69723 removed the V1 scanner whose message it also accepted), and #67363 (ported as #67402). ### Release note None ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: Claude Code (Claude Opus 5.5) - Generated: the port (both suites taken from branch-4.2), the local verification runs, and this description - Human review: not stated - Prompts: The author asked to continue moving branch-4.2's changes back to master, where six branch-4.1 test fixes were recorded as one task. After the check showed that master already covers four of them and has not failed on the other two, the author chose to port those two exactly as branch-4.2 has them and to verify them on a local cluster. Co-authored-by: shuke <shuke@selectdb.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
What problem does this PR solve?
Issue Number: N/A
Related PR: #67327 (which backported #65821 and introduced this suite)
Problem Summary:
test_search_score_topn_predicatesfails intermittently on the shared P0 pipeline of branch-4.1. It always fails on its very first query (line 59), right afterCREATE TABLE+INSERT+sync, with:Observed on branch-4.1 CI, 2 failures out of 4 runs — builds
1034808(PR #67327) and1035495(PR #67342) failed, while1035120(rerun of #67327, identical code, 5530 tests passed) and1035335(PR #67340, after the suite landed) passed. At roughly a 50% hit rate it randomly reddens unrelated branch-4.1 PRs, so it needs to stop running in the concurrent P0 group.The case is not wrong on its own: it passes locally against both a 1-BE and a 3-BE cluster built from this branch, and the CI rerun with identical code passed as well. That points at contention on the shared P0 runner (the index of the freshly inserted rowset not being usable yet at query time) rather than at the query. branch-4.1 also lacks master's
prevent_search_row_fallbackpath (added by #66052 together with the SNII format), so on this branch a search that does not take the index surfaces the internal error directly instead of falling back.Fix: move the suite to the
nonConcurrentgroup so it runs serialized, as 35 otherinverted_index_p0cases on this branch already do. No test logic or expected output changes.Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)