Repository navigation
branch-4.2: [fix](be) Score regexp and wildcard patterns with a constant like Elasticsearch #69815 - #69922
Merged
Conversation
…sticsearch (#69815) ### What problem does this PR solve? Issue Number: None Related PR: #69052 Problem Summary: score() of pattern queries disagrees with Elasticsearch and with itself: -- 'Inverted indexes are suitable for full text search', -- 'The parser splits text into multiple terms', ... SELECT score() FROM document WHERE SEARCH('content:/.*ext/'); -- BM25 sum SELECT score() FROM document WHERE SEARCH('content:*ext'); -- BM25 sum SELECT score() FROM document WHERE SEARCH('content:te*'); -- 1 SELECT score() FROM document WHERE content MATCH_REGEXP 'ext'; -- 0 Elasticsearch scores a query_string /regexp/, wildcard or prefix, and a regexp, wildcard or prefix query, with a constant 1 per matching row (default rewrite constant_score_blended); measured on 9.5.5 with the rows above: 1.0 for both matching rows. Only an explicit "rewrite": "scoring_boolean" sums the BM25 of the expanded terms. Next to a scored clause in a bool must, a regexp adds exactly 1. #69052 made SEARCH regexp and wildcard sum the BM25 of every expanded term, i.e. scoring_boolean always on, while prefix stayed constant. That sum also takes idf and avgdl from the current segment, not from the tablet statistics MATCH uses: two inserts of the same text scored 0.2876821 and 1.3862942 for identical 'apple' rows, while MATCH 'apple' scored 0.6931471 for both. MATCH_REGEXP never requested a scoring context, so it scored 0 like a filter. Fix: 1. Revert the scoring part of #69052: ExpandWeight scores every pattern with the constant scorer again. The test fixture extensions that #69051 built on are kept. 2. MATCH_REGEXP requests scoring on every index; the regexp expansion then publishes the constant 1. The MATCH statistics collector skips it, since a constant reads no statistics and the pattern is not text to analyze.
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
|
run buildall |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
yiguolei
approved these changes
Nov 2, 2026
Contributor
Author
|
PR approved by anyone and no changes requested. |
github-actions
Bot
added
the
approved
Contributor
Author
|
PR approved by at least one committer and no changes requested. |
yiguolei
pushed a commit
that referenced
this pull request
Nov 5, 2026
…ARCH #69790 (#70008) Backport of #69790 to branch-4.2. ### What problem does this PR solve? Issue Number: None Related PR: #69790, #69051 Problem Summary: score() of a MATCH on an untokenized (keyword) inverted index is always 0, while SEARCH on the same field scores BM25: INDEX idx_redirect (redirect) USING INVERTED -- one row ('BP') SELECT score() FROM t WHERE redirect MATCH 'BP'; -- 0 SELECT score() FROM t WHERE SEARCH('redirect:BP'); -- 0.2876821 Elasticsearch scores match, term and match_phrase on a keyword field with BM25 in query context, so the score is the idf of the whole value. #69051 (already on 4.2) made the engine score keyword postings but kept MATCH out. This makes a keyword index eligible for MATCH scoring and collects its whole value as one term, the one the reader looks up. It also carries the follow-up fixes from #69790's review: - A keyword MATCH_PHRASE drops its ` ~N` slop before collecting. - A MATCH on a non-string leaf such as `CAST(n AS STRING)` collects nothing. - A VARIANT path that a segment indexed as BKD, or does not hold, adds no statistics. This covers the BKD searcher coming from the searcher cache, and SNII. - A SNII index kind is looked up without copying the directory. ### Branch adaptations - `can_score_match` keeps 4.2's analyzed rule (support_phrase only); branch-4.2 has no lowercase-normalizer clause. - `test_omit_norms` does not exist on 4.2, so its `.out` change is dropped. The branch was cut before #69922 (the 4.2 backport of #69815) and then merged with branch-4.2 to resolve its conflict in `index_reader_helper_test.cpp`: both tests are kept, as on master. - `test_match_keyword_score` inserts its VARIANT rows as JSON strings: `parse_to_variant` in `VALUES` fails on 4.2 with a max-subcolumns conversion error. The expected output is master's file, unchanged. - **#69901 (BM25 statistics per index) is not on 4.2.** A MATCH on a keyword index and a MATCH on an analyzed index of the same column in one `score()` query (`name MATCH 'apple' USING ANALYZER english AND name MATCH 'apple' USING ANALYZER none`) now fails with "Scoring predicates for one field select different inverted indexes". Master behaved the same before #69901. Before this change, that query ran with the keyword clause scoring 0. Each clause alone scores fine. Two analyzed indexes on one column already fail this way on 4.2. Backporting #69901 lifts this. ### Validation On this branch merged with branch-4.2 c2aaed9, ASAN build: - BE UT: 1611 tests in `IndexReaderHelperTest`, `CollectionStatistics*` and `*Snii*` pass. - Regression: `test_match_keyword_score` passes against master's `.out` on V2/V3/SNII. The following also pass: `test_match_phrase_prefix_score`, `test_bm25_score`, `test_score_filter_matrix`, `test_score_filter_usage`, `test_search_score_topn_predicates`, `test_multi_analyzer_query`, `test_match_using_analyzer`, `test_single_column_multi_index` (with debug points on), `test_search_score_keyword_filters`, `test_search_score_raw_index`, `test_search_score_pattern_constant`. - Local note: the local thirdparty predates the lance-c upgrade, so `lance_reader.cpp`'s `lance_scanner_set_scalar_index_auto_selection` call was stubbed for the local build only. It is not part of this PR. ### Release note score() of MATCH / MATCH_ALL / MATCH_PHRASE on a keyword (untokenized) inverted index now returns the BM25 score of the value, the same as SEARCH, instead of 0. ### Check List (For Author) - Test: Regression test / Unit Test (see Validation) - Behavior changed: Yes. Keyword MATCH contributes its BM25 score to score() instead of 0. - Does this need documentation: No 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picked from #69815