[airborne12]

Backport of #69790 to branch-4.2.

What problem does this PR solve?

Issue Number: None

Related PR: #69790, #69051

Problem Summary: score() of a MATCH on an untokenized (keyword) inverted index is always 0, while SEARCH on the same field scores BM25:

INDEX idx_redirect (redirect) USING INVERTED
-- one row ('BP')
SELECT score() FROM t WHERE redirect MATCH 'BP';  -- 0
SELECT score() FROM t WHERE SEARCH('redirect:BP'); -- 0.2876821

Elasticsearch scores match, term and match_phrase on a keyword field with BM25 in query context, so the score is the idf of the whole value. #69051 (already on 4.2) made the engine score keyword postings but kept MATCH out. This makes a keyword index eligible for MATCH scoring and collects its whole value as one term, the one the reader looks up. It also carries the follow-up fixes from #69790's review:

• A keyword MATCH_PHRASE drops its ~N slop before collecting.
• A MATCH on a non-string leaf such as CAST(n AS STRING) collects nothing.
• A VARIANT path that a segment indexed as BKD, or does not hold, adds no statistics. This covers the BKD searcher coming from the searcher cache, and SNII.
• A SNII index kind is looked up without copying the directory.

Branch adaptations

• can_score_match keeps 4.2's analyzed rule (support_phrase only); branch-4.2 has no lowercase-normalizer clause.
• test_omit_norms does not exist on 4.2, so its .out change is dropped. The branch was cut before branch-4.2: [fix](be) Score regexp and wildcard patterns with a constant like Elasticsearch #69815 #69922 (the 4.2 backport of [fix](be) Score regexp and wildcard patterns with a constant like Elasticsearch #69815) and then merged with branch-4.2 to resolve its conflict in index_reader_helper_test.cpp: both tests are kept, as on master.
• test_match_keyword_score inserts its VARIANT rows as JSON strings: parse_to_variant in VALUES fails on 4.2 with a max-subcolumns conversion error. The expected output is master's file, unchanged.
• [fix](be) Keep BM25 statistics per index for MATCH on one column #69901 (BM25 statistics per index) is not on 4.2. A MATCH on a keyword index and a MATCH on an analyzed index of the same column in one score() query (name MATCH 'apple' USING ANALYZER english AND name MATCH 'apple' USING ANALYZER none) now fails with "Scoring predicates for one field select different inverted indexes". Master behaved the same before [fix](be) Keep BM25 statistics per index for MATCH on one column #69901. Before this change, that query ran with the keyword clause scoring 0. Each clause alone scores fine. Two analyzed indexes on one column already fail this way on 4.2. Backporting [fix](be) Keep BM25 statistics per index for MATCH on one column #69901 lifts this.

Validation

On this branch merged with branch-4.2 c2aaed9, ASAN build:

• BE UT: 1611 tests in IndexReaderHelperTest, CollectionStatistics* and *Snii* pass.
• Regression: test_match_keyword_score passes against master's .out on V2/V3/SNII. The following also pass: test_match_phrase_prefix_score, test_bm25_score, test_score_filter_matrix, test_score_filter_usage, test_search_score_topn_predicates, test_multi_analyzer_query, test_match_using_analyzer, test_single_column_multi_index (with debug points on), test_search_score_keyword_filters, test_search_score_raw_index, test_search_score_pattern_constant.
• Local note: the local thirdparty predates the lance-c upgrade, so lance_reader.cpp's lance_scanner_set_scalar_index_auto_selection call was stubbed for the local build only. It is not part of this PR.

Release note

score() of MATCH / MATCH_ALL / MATCH_PHRASE on a keyword (untokenized) inverted index now returns the BM25 score of the value, the same as SEARCH, instead of 0.

Check List (For Author)

• Test: Regression test / Unit Test (see Validation)
• Behavior changed: Yes. Keyword MATCH contributes its BM25 score to score() instead of 0.
• Does this need documentation: No

🤖 Generated with Claude Code

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[airborne12]

run buildall

[airborne12]

run buildall

[airborne12]

run buildall

[hello-stephen]

BE UT Coverage Report

Increment line coverage 88.68% (47/53) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	64.81% (30378/46876)
Line Coverage	49.71% (321751/647293)
Region Coverage	45.46% (259156/570043)
Branch Coverage	47.17% (122617/259966)

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 89.66% (52/58) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	76.34% (34756/45528)
Line Coverage	61.23% (393051/641897)
Region Coverage	58.02% (331213/570902)
Branch Coverage	58.99% (153181/259656)

[github-actions]

PR approved by anyone and no changes requested.

[github-actions]

PR approved by at least one committer and no changes requested.