Repository navigation
[fix](be) Score regexp and wildcard patterns with a constant like Elasticsearch - #69815
Merged
Merged
Conversation
…sticsearch ### What problem does this PR solve? Issue Number: None Related PR: apache#69052 Problem Summary: score() of pattern queries disagrees with Elasticsearch and with itself: -- 'Inverted indexes are suitable for full text search', -- 'The parser splits text into multiple terms', ... SELECT score() FROM document WHERE SEARCH('content:/.*ext/'); -- BM25 sum SELECT score() FROM document WHERE SEARCH('content:*ext'); -- BM25 sum SELECT score() FROM document WHERE SEARCH('content:te*'); -- 1 SELECT score() FROM document WHERE content MATCH_REGEXP 'ext'; -- 0 Elasticsearch scores a query_string /regexp/, wildcard or prefix, and a regexp, wildcard or prefix query, with a constant 1 per matching row (default rewrite constant_score_blended); measured on 9.5.5 with the rows above: 1.0 for both matching rows. Only an explicit "rewrite": "scoring_boolean" sums the BM25 of the expanded terms. Next to a scored clause in a bool must, a regexp adds exactly 1. apache#69052 made SEARCH regexp and wildcard sum the BM25 of every expanded term, i.e. scoring_boolean always on, while prefix stayed constant. That sum also takes idf and avgdl from the current segment, not from the tablet statistics MATCH uses: two inserts of the same text scored 0.2876821 and 1.3862942 for identical 'apple' rows, while MATCH 'apple' scored 0.6931471 for both. MATCH_REGEXP never requested a scoring context, so it scored 0 like a filter. Fix: 1. Revert the scoring part of apache#69052: ExpandWeight scores every pattern with the constant scorer again. The test fixture extensions that apache#69051 built on are kept. 2. MATCH_REGEXP requests scoring on every index; the regexp expansion then publishes the constant 1. The MATCH statistics collector skips it, since a constant reads no statistics and the pattern is not text to analyze. ### Release note score() of SEARCH regexp and wildcard clauses is a constant 1 per matching row again, like prefix clauses and Elasticsearch's default. score() of MATCH_REGEXP is 1 instead of 0. ### Check List (For Author) - Test: Regression test / Unit Test / Manual test - Regression test: new search/test_search_score_pattern_constant (V2/V3/SNII, two segments, keyword and analyzed fields; identical on all three formats); 24 existing score suites in inverted_index_p0, search and variant_p0 pass unchanged - Unit Test: 477 tests (IndexReaderHelperTest, CollectionStatistics*, regexp/wildcard/prefix engine tests restored by the revert, FunctionSearchTest, SNII query tests). Mutation checks: the gate test fails without the gate change, the collector test fails without the regexp skip - Manual test: Elasticsearch 9.5.5 on the same rows for every query shape above; two-segment table now scores 1 for every pattern hit - Built and run on master 62e3d81 plus this change: the current master tip does not compile (function_datasketches_hll.cpp after apache#69679), and none of the files touched here changed in between - Behavior changed: Yes. SEARCH regexp/wildcard scores and MATCH_REGEXP scores change as described in the release note. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Member
Author
|
run buildall |
Member
Author
|
/review |
airborne12
marked this pull request as ready for review
October 29, 2026 11:58
airborne12
requested review from
csun5285,
eldenmoon,
gavinchou and
yiguolei
as code owners
October 29, 2026 11:58
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Static review of PR #69815 at exact head bdde929 against ee6969e found no blocking or actionable defect.
The requested behavior is implemented coherently: expanded regexp, wildcard, prefix, suffix, and contains leaves enumerate matching terms into a deduplicated row bitmap and publish one constant score per matching row; Boolean AND/OR/MUST_NOT, multi-segment, top-k, nullable, candidate, phrase-prefix fallback, CLucene/V2/V3, and SNII paths preserve the expected score and truth-set composition. MATCH_REGEXP is admitted to score-enabled execution on analyzed and keyword indexes, while statistics collection intentionally skips the raw pattern; score-disabled execution retains the existing listing and cache behavior. No new concurrency, lifecycle, persistence, protocol, configuration, memory-accounting, or error-propagation risk was found.
The changed unit tests and the new regression suite cover the intended constant-score behavior across V2, V3, and SNII, analyzed and keyword fields, two segments, direct MATCH_REGEXP, mixed MATCH plus regexp scoring, and equality baselines. Existing inline review threads are empty and there is no additional user focus. This was a static-only review; no builds or tests were run.
Review converged after 1 round(s).
Member
Author
|
run external |
Contributor
TPC-H: Total hot run time: 28387 ms |
Contributor
TPC-DS: Total hot run time: 151178 ms |
Contributor
ClickBench: Total hot run time: 24.3 s |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
csun5285
approved these changes
Oct 30, 2026
yiguolei
pushed a commit
that referenced
this pull request
Oct 30, 2026
…ant like Elasticsearch (#69825) ### What problem does this PR solve? Issue Number: None Related PR: #69815, #69269 Problem Summary: score() of pattern queries disagrees with Elasticsearch and with itself on branch-4.1: -- 'Inverted indexes are suitable for full text search', -- 'The parser splits text into multiple terms', ... SELECT score() FROM document WHERE SEARCH('content:/.*ext/'); -- BM25 sum SELECT score() FROM document WHERE SEARCH('content:*ext'); -- BM25 sum SELECT score() FROM document WHERE content MATCH_REGEXP 'ext'; -- 0 Elasticsearch scores a query_string /regexp/, wildcard or prefix, and a regexp, wildcard or prefix query, with a constant 1 per matching row (default rewrite constant_score_blended); measured on 9.5.5 with the rows above: 1.0 for both matching rows. Only an explicit "rewrite": "scoring_boolean" sums the BM25 of the expanded terms. Next to a scored clause in a bool must, a regexp adds exactly 1. #69269 (the branch-4.1 pick of #69052) made SEARCH regexp and wildcard sum the BM25 of every expanded term, i.e. scoring_boolean always on, with idf and avgdl taken from the current segment rather than the tablet statistics MATCH uses, so identical rows in two segments scored differently. MATCH_REGEXP never requested a scoring context, so it scored 0 like a filter. Fix, the same as #69815 on master: 1. Revert #69269: RegexpWeight scores every pattern with the constant scorer again; its suite that pinned the BM25 sums is removed. 2. MATCH_REGEXP requests scoring on every index, and the legacy RegexpQuery records 1 for every matching row when score() is requested. The MATCH statistics collector skips it, since a constant reads no statistics and the pattern is not text to analyze. A MATCH_REGEXP on a keyword (untokenized) index also scores 1 once the keyword reader requests scoring (#69791); until then it keeps 0. ### Release note score() of SEARCH regexp and wildcard clauses is a constant 1 per matching row again, like prefix clauses and Elasticsearch's default. score() of MATCH_REGEXP on a tokenized index is 1 instead of 0. ### Check List (For Author) - Test: Regression test / Unit Test / Manual test - Regression test: new search/test_search_score_pattern_constant (V2/V3, two segments); all 18 blocks equal the master PR's V2/V3 blocks; 16 existing score suites in inverted_index_p0 and search pass unchanged - Unit Test: 130 tests (IndexReaderHelperTest, CollectionStatistics*, RegexpQueryTest and the query_v2 regexp/wildcard tests restored by the revert) - Manual test: Elasticsearch 9.5.5 on the same rows; a two-segment table now scores 1 for every pattern hit on a branch-4.1 ASAN build - Behavior changed: Yes. SEARCH regexp/wildcard scores and MATCH_REGEXP scores change as described in the release note. - Does this need documentation: No 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
github-actions Bot
pushed a commit
that referenced
this pull request
Nov 1, 2026
…sticsearch (#69815) ### What problem does this PR solve? Issue Number: None Related PR: #69052 Problem Summary: score() of pattern queries disagrees with Elasticsearch and with itself: -- 'Inverted indexes are suitable for full text search', -- 'The parser splits text into multiple terms', ... SELECT score() FROM document WHERE SEARCH('content:/.*ext/'); -- BM25 sum SELECT score() FROM document WHERE SEARCH('content:*ext'); -- BM25 sum SELECT score() FROM document WHERE SEARCH('content:te*'); -- 1 SELECT score() FROM document WHERE content MATCH_REGEXP 'ext'; -- 0 Elasticsearch scores a query_string /regexp/, wildcard or prefix, and a regexp, wildcard or prefix query, with a constant 1 per matching row (default rewrite constant_score_blended); measured on 9.5.5 with the rows above: 1.0 for both matching rows. Only an explicit "rewrite": "scoring_boolean" sums the BM25 of the expanded terms. Next to a scored clause in a bool must, a regexp adds exactly 1. #69052 made SEARCH regexp and wildcard sum the BM25 of every expanded term, i.e. scoring_boolean always on, while prefix stayed constant. That sum also takes idf and avgdl from the current segment, not from the tablet statistics MATCH uses: two inserts of the same text scored 0.2876821 and 1.3862942 for identical 'apple' rows, while MATCH 'apple' scored 0.6931471 for both. MATCH_REGEXP never requested a scoring context, so it scored 0 like a filter. Fix: 1. Revert the scoring part of #69052: ExpandWeight scores every pattern with the constant scorer again. The test fixture extensions that #69051 built on are kept. 2. MATCH_REGEXP requests scoring on every index; the regexp expansion then publishes the constant 1. The MATCH statistics collector skips it, since a constant reads no statistics and the pattern is not text to analyze.
Open
6 of 10 tasks
yiguolei
pushed a commit
that referenced
this pull request
Nov 5, 2026
…ARCH #69790 (#70008) Backport of #69790 to branch-4.2. ### What problem does this PR solve? Issue Number: None Related PR: #69790, #69051 Problem Summary: score() of a MATCH on an untokenized (keyword) inverted index is always 0, while SEARCH on the same field scores BM25: INDEX idx_redirect (redirect) USING INVERTED -- one row ('BP') SELECT score() FROM t WHERE redirect MATCH 'BP'; -- 0 SELECT score() FROM t WHERE SEARCH('redirect:BP'); -- 0.2876821 Elasticsearch scores match, term and match_phrase on a keyword field with BM25 in query context, so the score is the idf of the whole value. #69051 (already on 4.2) made the engine score keyword postings but kept MATCH out. This makes a keyword index eligible for MATCH scoring and collects its whole value as one term, the one the reader looks up. It also carries the follow-up fixes from #69790's review: - A keyword MATCH_PHRASE drops its ` ~N` slop before collecting. - A MATCH on a non-string leaf such as `CAST(n AS STRING)` collects nothing. - A VARIANT path that a segment indexed as BKD, or does not hold, adds no statistics. This covers the BKD searcher coming from the searcher cache, and SNII. - A SNII index kind is looked up without copying the directory. ### Branch adaptations - `can_score_match` keeps 4.2's analyzed rule (support_phrase only); branch-4.2 has no lowercase-normalizer clause. - `test_omit_norms` does not exist on 4.2, so its `.out` change is dropped. The branch was cut before #69922 (the 4.2 backport of #69815) and then merged with branch-4.2 to resolve its conflict in `index_reader_helper_test.cpp`: both tests are kept, as on master. - `test_match_keyword_score` inserts its VARIANT rows as JSON strings: `parse_to_variant` in `VALUES` fails on 4.2 with a max-subcolumns conversion error. The expected output is master's file, unchanged. - **#69901 (BM25 statistics per index) is not on 4.2.** A MATCH on a keyword index and a MATCH on an analyzed index of the same column in one `score()` query (`name MATCH 'apple' USING ANALYZER english AND name MATCH 'apple' USING ANALYZER none`) now fails with "Scoring predicates for one field select different inverted indexes". Master behaved the same before #69901. Before this change, that query ran with the keyword clause scoring 0. Each clause alone scores fine. Two analyzed indexes on one column already fail this way on 4.2. Backporting #69901 lifts this. ### Validation On this branch merged with branch-4.2 c2aaed9, ASAN build: - BE UT: 1611 tests in `IndexReaderHelperTest`, `CollectionStatistics*` and `*Snii*` pass. - Regression: `test_match_keyword_score` passes against master's `.out` on V2/V3/SNII. The following also pass: `test_match_phrase_prefix_score`, `test_bm25_score`, `test_score_filter_matrix`, `test_score_filter_usage`, `test_search_score_topn_predicates`, `test_multi_analyzer_query`, `test_match_using_analyzer`, `test_single_column_multi_index` (with debug points on), `test_search_score_keyword_filters`, `test_search_score_raw_index`, `test_search_score_pattern_constant`. - Local note: the local thirdparty predates the lance-c upgrade, so `lance_reader.cpp`'s `lance_scanner_set_scalar_index_auto_selection` call was stubbed for the local build only. It is not part of this PR. ### Release note score() of MATCH / MATCH_ALL / MATCH_PHRASE on a keyword (untokenized) inverted index now returns the BM25 score of the value, the same as SEARCH, instead of 0. ### Check List (For Author) - Test: Regression test / Unit Test (see Validation) - Behavior changed: Yes. Keyword MATCH contributes its BM25 score to score() instead of 0. - Does this need documentation: No 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: #69052
Problem Summary: score() of pattern queries disagrees with Elasticsearch
and with itself:
Elasticsearch scores a query_string /regexp/, wildcard or prefix, and a
regexp, wildcard or prefix query, with a constant 1 per matching row
(default rewrite constant_score_blended); measured on 9.5.5 with the
rows above: 1.0 for both matching rows. Only an explicit
"rewrite": "scoring_boolean" sums the BM25 of the expanded terms. Next
to a scored clause in a bool must, a regexp adds exactly 1.
#69052 made SEARCH regexp and wildcard sum the BM25 of every expanded
term, i.e. scoring_boolean always on, while prefix stayed constant. That
sum also takes idf and avgdl from the current segment, not from the
tablet statistics MATCH uses: two inserts of the same text scored
0.2876821 and 1.3862942 for identical 'apple' rows, while MATCH 'apple'
scored 0.6931471 for both. MATCH_REGEXP never requested a scoring
context, so it scored 0 like a filter.
Fix:
with the constant scorer again. The test fixture extensions that
[fix](be) Support BM25 scoring on keyword SEARCH indexes #69051 built on are kept.
then publishes the constant 1. The MATCH statistics collector skips
it, since a constant reads no statistics and the pattern is not text
to analyze.
Release note
score() of SEARCH regexp and wildcard clauses is a constant 1 per
matching row again, like prefix clauses and Elasticsearch's default.
score() of MATCH_REGEXP is 1 instead of 0.
Check List (For Author)
(V2/V3/SNII, two segments, keyword and analyzed fields; identical
on all three formats); 24 existing score suites in
inverted_index_p0, search and variant_p0 pass unchanged
regexp/wildcard/prefix engine tests restored by the revert,
FunctionSearchTest, SNII query tests). Mutation checks: the gate
test fails without the gate change, the collector test fails
without the regexp skip
shape above; two-segment table now scores 1 for every pattern hit
master tip does not compile (function_datasketches_hll.cpp after
[refine](column) Normalize constant expression results #69679), and none of the files touched here changed in between
scores change as described in the release note.
🤖 Generated with Claude Code