Repository navigation
[fix](be) Match SNII wildcards by UTF-8 code point - #66861
Merged
Merged
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Member
Author
|
/review |
Member
Author
|
run buildall |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Automated review status: converged after two rounds; requesting changes for one data-correctness issue.
The UTF-8 matcher is correct for valid inputs, and the added unit and integration coverage demonstrates SNII/V3 ? parity. The inline finding is a separate compatibility regression: legal raw keyword values containing malformed UTF-8 are silently omitted by nontrivial wildcards.
Critical checkpoints:
- Goal and proof: code-point matching for valid UTF-8 is implemented and exercised by matcher, logical-index, and ordered SNII/V3 SEARCH tests; malformed stored-term compatibility is not preserved.
- Scope and clarity: the production diff is focused on wildcard matching and invalid-pattern handling; no unrelated production change was found.
- Concurrency: the matcher is request-local; existing query-cache and single-flight keys include the raw normalized pattern, query type, and expansion cap, with no new lock or shared-state defect.
- Lifecycle and static initialization: the pattern view remains live for synchronous enumeration, scratch storage is request-scoped, and no cross-TU static dependency is added.
- Configuration: no configuration item is added or changed.
- Compatibility: there is no storage-format or protocol change, but existing and newly written raw keyword indexes can contain legal non-UTF-8 bytes; the inline issue changes their results and can publish/cache an incomplete bitmap.
- Parallel paths: standard and Lucene SEARCH parsing, analyzed and non-analyzed PREFIX/WILDCARD routing, SNII and V3 execution, raw and escaped dictionaries, cache hit/miss, null masking, and expansion caps were checked.
- Conditions and errors: valid UTF-8 boundary validation makes width jumps safe; invalid patterns fail before enumeration, but invalid stored terms are incorrectly converted to successful non-matches as described inline.
- Test coverage: valid 2-, 3-, and 4-byte code points, literals, star/question combinations, malformed matcher inputs, allocation reuse, union ordering, nulls, and end-to-end SNII/V3 results are covered. Add a writer-to-SEARCH regression for the accepted issue.
- Test results: the checked-in ordered output matches the suite, and the PR reports 22/22 targeted ASAN unit tests, the named regression, and a full BE build. This review did not run builds or tests because the review prompt forbids them.
- Observability: existing timers, cache counters, status propagation, and VLOGs are adequate for valid execution; the accepted silent omission is precisely the missing failure signal.
- Transactions and persistence: no transaction, EditLog, visibility-version, MoW, or metadata-persistence behavior is changed.
- Data writes and crash behavior: the production writer and on-disk bytes are unchanged; no new write atomicity or cleanup path is introduced.
- FE-BE variables and compatibility carriers: no query enum, thrift field, function symbol, or transmitted variable is added.
- Performance and other checks: prefix seeking and two reusable DP rows remain intact; validation is linear per visited term and introduces no per-term heap allocation. No second correctness, lifecycle, or performance issue survived verification.
User focus: no additional focus was provided. All candidates were independently verified, deduplicated, accepted, or dismissed; Round 2 returned NO_NEW_VALUABLE_FINDINGS on both full-review tracks and the separate risk track.
Contributor
TPC-H: Total hot run time: 17392 ms |
Contributor
TPC-DS: Total hot run time: 81270 ms |
airborne12
force-pushed
the
fix-snii-utf8-wildcard
branch
from
August 18, 2026 06:56
73af19e to
bd9a2b5
Compare
Member
Author
|
run buildall |
Member
Author
|
/review |
Contributor
|
Codex automated review failed and did not complete. Error: All Codex review accounts are usage-limited; earliest retry is 2026-08-20T03:35:00Z. Please trigger /review again after that time. |
### What problem does this PR solve? Issue Number: None Related PR: apache#66052 Problem Summary: SNII evaluated wildcard question marks byte by byte, so a pattern such as a?b could not match a猫b although the V3 inverted index treats the question mark as one Unicode character. Switching all terms to strict UTF-8 initially caused a compatibility regression: keyword indexes can contain arbitrary VARCHAR bytes, and already-written malformed terms such as 61FF stopped matching a*, **, and *. The matcher now advances valid terms by UTF-8 code point while falling back to the legacy byte semantics for malformed stored terms. Query patterns remain strict UTF-8 and invalid patterns return INVALID_ARGUMENT. The existing reusable DP scratch remains bounded. ### Release note SNII wildcard question marks now match one UTF-8 code point for valid text while preserving wildcard matches for existing raw keyword bytes. ### Check List (For Author) - Test: Unit Test and Regression test - ASAN BE unit tests: SniiWildcardQueryTest.* and SniiPatternQuery.* (23 passed) - Regression test: test_storage_format_snii_utf8_wildcard (generated output and normal run passed) - Full BE build with -j 192 - Behavior changed: Yes. SNII wildcard matching is code-point based for valid UTF-8 and retains legacy byte matching for malformed stored terms; the storage format and write path are unchanged. - Does this need documentation: No
airborne12
force-pushed
the
fix-snii-utf8-wildcard
branch
from
August 24, 2026 03:20
bd9a2b5 to
6dcf551
Compare
Member
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 17395 ms |
Contributor
TPC-DS: Total hot run time: 84784 ms |
Contributor
ClickBench: Total hot run time: 14.8 s |
Member
Author
|
/review |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
There was a problem hiding this comment.
Automated review status: converged after one round; no new inline findings. Approval opinion: no blocking issue remains at head 6dcf5513e7a82e70b8b6a01d4a8c23fb260d798c.
The previously reported malformed raw-keyword regression is fixed: valid dictionary terms use UTF-8 code-point transitions, malformed stored terms retain the legacy byte matcher, and query patterns remain strict UTF-8. The current writer-reader and SQL regression coverage exercises the original 61FF case for a*, **, and *.
Critical checkpoints:
- Goal and proof: the change makes SNII
?consume one UTF-8 code point and preserves existing raw keyword bytes. Direct matcher tests, logical-index writer-reader tests, and ordered SNII/V3 SEARCH regression results cover the intended behavior. - Scope and clarity: the production change is focused on wildcard matching, pattern validation, and documentation; the remaining changes are targeted unit/regression tests and generated output.
- Concurrency: each query owns its matcher and scratch rows. Existing result-cache and single-flight identities include raw query bytes, query type, and expansion cap; no new shared mutation, lock, lock-order, or race was introduced.
- Lifecycle and static initialization: the pattern view and decoded-term scratch remain alive throughout synchronous enumeration, matcher buffers are request-scoped, and no global/static dependency or special teardown path is added.
- Configuration: no configuration item is added or changed.
- Compatibility: there is no storage-format, writer, protocol, symbol, or persisted-metadata change. Existing valid and malformed SNII keyword indexes are handled in place; ASCII and malformed-term behavior remain compatible while valid UTF-8
?intentionally changes to V3 semantics. - Parallel paths: SEARCH dispatch, standalone-
*handling, all three plain-term key versions, escaped-term decoding, cache hit/miss and follower paths, null masking, max expansion, and the V3 Hyperscan wildcard path were checked. - Conditions and errors: whole-string validation makes code-point width jumps and literal
memcmpbounds safe. Invalid patterns returnINVALID_ARGUMENTbefore enumeration or cache insertion; malformed stored terms take the documented whole-match byte fallback. - Test coverage: coverage includes empty input, ASCII exhaustiveness, 2/3/4-byte code points, UTF-8 literals, mixed and consecutive wildcards, malformed terms, invalid patterns, matcher reuse/allocation bounds, posting union/order, max expansion, null arguments, real writer-reader behavior, and SNII/V3 SQL parity.
- Test results: checked-in ordered output matches the SQL suite. The current PR reports 20 completed passing checks, including BE UT, compile, formatter, P0/nonConcurrent regression, performance, and coverage; incremental changed-line coverage is 100% (43/43). No builds or test suites were run by this review because the review prompt prohibits them; the pure-text BE build-hygiene gate passed.
- Observability: existing Status propagation, query profiles, cache counters, and logs cover this request-local query path; no new long-lived operational state needs a metric.
- Transactions and persistence: no transaction, EditLog, visible-version, delete-bitmap, or metadata-persistence behavior is changed.
- Data writes and crash behavior: the production writer and on-disk bytes are unchanged, so no new write atomicity, crash-recovery, or cleanup path is introduced.
- FE-BE carriers: no thrift field, query enum, function symbol, or transmitted variable is added.
- Performance and other checks: prefix seeking and two reusable DP rows remain intact with no per-term scratch allocation. Per-term UTF-8 validation is linear and uses the existing architecture-specific validator; no material regression was substantiated. Header layering, include closure, PCH, extern-template, and unity-skip hygiene passed.
User focus: no additional focus was provided. Both normal full-review tracks and the separate risk-focused track returned NO_NEW_VALUABLE_FINDINGS; the main-agent final sweep found no unresolved candidate and no new issue distinct from the already-fixed thread.
Contributor
|
PR approved by anyone and no changes requested. |
github-actions
Bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
5 of 9 tasks
airborne12
added a commit
to airborne12/apache-doris
that referenced
this pull request
Sep 29, 2026
Manually adapt the merged SNII feature and its follow-up fixes from master for the branch-4.2 FE, BE, cloud, and regression APIs. Remove CommonGrams as in the final master state. Keep the target branch's existing index behavior. Preserve existing SNII wire tags for statistics and section references. Bundle a compatibility corpus and verify its query results in the reader test. Conflicts: - Use target FE index definition and session variable APIs. - Use target BE segment and ANN query APIs. - Share the target branch's existing norms policy with SNII writers. - Adapt variant regressions to branch-4.2 planner settings. Backport-of: 96fd1e9 (apache#66052) Backport-of: 5870fb2 (apache#66855) Backport-of: 1db4b1d (apache#66864) Backport-of: e33e048 (apache#66961) Backport-of: 72a587f (apache#66856) Backport-of: 74b4418 (apache#66869) Backport-of: 511a8a0 (apache#66874) Backport-of: 80c1a3a (apache#66861) Backport-of: 40e1463 (apache#66877) Backport-of: fa95a70 (apache#67135) Backport-of: 40ff995 (apache#67171) Backport-of: 9f3b084 (apache#67537) Backport-of: 5985916 (apache#67808) Backport-of: 47b2e1d (apache#67180) Related: apache#68039 SNII-specific norms policy (manually adapted) Related: apache#67451 ARM64 CRC32C guard (manually adapted)
airborne12
added a commit
to airborne12/apache-doris
that referenced
this pull request
Sep 29, 2026
Manually adapt the merged SNII feature and its follow-up fixes from master for the branch-4.2 FE, BE, cloud, and regression APIs. Remove CommonGrams as in the final master state. Keep the target branch's existing index behavior. Preserve existing SNII wire tags for statistics and section references. Bundle a compatibility corpus and verify its query results in the reader test. Conflicts: - Use target FE index definition and session variable APIs. - Use target BE segment and ANN query APIs. - Share the target branch's existing norms policy with SNII writers. - Adapt variant regressions to branch-4.2 planner settings. Backport-of: 96fd1e9 (apache#66052) Backport-of: 5870fb2 (apache#66855) Backport-of: 1db4b1d (apache#66864) Backport-of: e33e048 (apache#66961) Backport-of: 72a587f (apache#66856) Backport-of: 74b4418 (apache#66869) Backport-of: 511a8a0 (apache#66874) Backport-of: 80c1a3a (apache#66861) Backport-of: 40e1463 (apache#66877) Backport-of: fa95a70 (apache#67135) Backport-of: 40ff995 (apache#67171) Backport-of: 9f3b084 (apache#67537) Backport-of: 5985916 (apache#67808) Backport-of: 47b2e1d (apache#67180) Related: apache#68039 SNII-specific norms policy (manually adapted) Related: apache#67451 ARM64 CRC32C guard (manually adapted)
airborne12
added a commit
to airborne12/apache-doris
that referenced
this pull request
Sep 29, 2026
Manually adapt the merged SNII feature and its follow-up fixes from master for the branch-4.2 FE, BE, cloud, and regression APIs. Remove CommonGrams as in the final master state. Keep the target branch's existing index behavior. Preserve existing SNII wire tags for statistics and section references. Bundle a compatibility corpus and verify its query results in the reader test. Conflicts: - Use target FE index definition and session variable APIs. - Use target BE segment and ANN query APIs. - Share the target branch's existing norms policy with SNII writers. - Adapt variant regressions to branch-4.2 planner settings. Backport-of: 96fd1e9 (apache#66052) Backport-of: 5870fb2 (apache#66855) Backport-of: 1db4b1d (apache#66864) Backport-of: e33e048 (apache#66961) Backport-of: 72a587f (apache#66856) Backport-of: 74b4418 (apache#66869) Backport-of: 511a8a0 (apache#66874) Backport-of: 80c1a3a (apache#66861) Backport-of: 40e1463 (apache#66877) Backport-of: fa95a70 (apache#67135) Backport-of: 40ff995 (apache#67171) Backport-of: 9f3b084 (apache#67537) Backport-of: 5985916 (apache#67808) Backport-of: 47b2e1d (apache#67180) Related: apache#68039 SNII-specific norms policy (manually adapted) Related: apache#67451 ARM64 CRC32C guard (manually adapted)
yiguolei
pushed a commit
that referenced
this pull request
Sep 29, 2026
…68584) ### What problem does this PR solve? Issue Number: N/A (branch backport) Related PRs: #66052, #66855, #66856, #66861, #66864, #66869, #66874, #66877, #66961, #67135, #67171, #67180, #67537, #67808. Backport the merged SNII storage format and its follow-up fixes from master to `branch-4.2`. The FE, BE, cloud, and regression interfaces differ, so the source commits were adapted together. Their exact SHAs are recorded in the backport commit. The target branch already contains the generic norms policy and IK custom analyzers from #68550. The backport preserves the exact analyzer-key binding and invalid-policy validation introduced there. It also adapts the SNII-specific norms policy and ARM64 CRC32C guard to this branch. The protobuf audit compared all shared fields with current master: 379 fields in `olap_file.proto` and 32 in `snii.proto` have identical numbers, with no collisions. Existing statistics, section-reference, and legacy CommonGrams tags remain reserved for their original meanings. The reader test includes a neutral compatibility corpus (eight index/expect pairs, 513 observations, zero mismatches). No audit document, unrelated array test data, customer-specific directory name, or CommonGrams implementation is included in this commit. #67538 (gram indexes for LIKE/REGEXP) will be backported separately. #67657 is outside this PR. ### Backport audit The local source-hunk audit records 3,188 source diff hunks: 2,962 ported, 174 adapted, and 52 inapplicable. Fifty inapplicable hunks were removed by the later selected master PR #67537; one concerns a build helper absent on this branch, and one is an unused CommonGrams-only import. The audit artifact is intentionally excluded from this commit. | Source PR | Ported | Adapted | Inapplicable | | --- | ---: | ---: | ---: | | #66052 | 935 | 124 | 27 | | #66855 | 10 | 1 | 0 | | #66856 | 94 | 1 | 0 | | #66861 | 23 | 0 | 0 | | #66864 | 13 | 0 | 0 | | #66869 | 13 | 5 | 0 | | #66874 | 5 | 1 | 0 | | #66877 | 25 | 2 | 0 | | #66961 | 24 | 0 | 0 | | #67135 | 6 | 2 | 0 | | #67171 | 13 | 1 | 0 | | #67180 | 203 | 2 | 0 | | #67537 | 1,500 | 26 | 25 | | #67808 | 98 | 9 | 0 | The main adaptations retain the branch's FE index and session interfaces, BE segment and ANN APIs, cloud APIs, and existing IK policy behavior. The moved similarity collector and its tests follow the current index layout. ### Verification - FE production build passed; 62 focused FE unit tests passed. - BE production build passed; ASAN BE test binary built. Focused BE tests (80), broad SNII tests (1,334), source-related tests (703), and adapted-path tests (98) passed. These sets overlap. - Compatibility corpus readback passed all 513 observations. - All 13 affected cloud-mode regression suites passed on the updated head, including storage format, cloud BUILD INDEX, phrase pushdown, and VARIANT index/compaction coverage. - Complete PR diff English gate passed (509 source files, 22 reviewed Unicode input fixtures). Clang-format 16, FE Checkstyle, and the merged-tree format gate passed. - The first regression run lost its BE during a background auto-analysis `MIN/MAX` query in an unchanged branch-4.2 statistics iterator. Two VARIANT suites failed while the BE was unavailable. On the isolated validation cluster, disabling automatic analysis and restarting the BE allowed all 13 suites to pass. The initial logs were retained for diagnosis; this PR does not change that iterator. ### Release note Backport SNII inverted index storage format support to `branch-4.2`. ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - [x] Compatibility corpus readback - Behavior changed: - [x] Yes. SNII storage format support is available on this branch. - Does this need documentation? - [x] No. This PR backports the merged source feature without adding new syntax. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: #66052
Problem Summary:
SNII evaluated wildcard question marks byte by byte, so
a?bcould not matcha猫b, while V3 converts?to a RE2 dot and matches one Unicode code point. The SNII matcher now validates UTF-8 and advances its existing reusable dynamic-programming rows only at code-point boundaries. This preserves the bounded two-buffer scratch design, makes invalid patterns returnINVALID_ARGUMENT, and aligns SNII query results with V3 without changing the writer or stored bytes.Release note
SNII wildcard question marks now match one UTF-8 code point, consistent with V3.
Check List (For Author)
?now consumes exactly one UTF-8 code point. This is query-only; the storage format and write path are unchanged, so existing SNII indexes are upgrade-compatible.Tests
SniiWildcardQueryTest.*:SniiPatternQuery.*(22/22 passed)inverted_index_p0/storage_format/test_storage_format_snii_utf8_wildcard./build.sh --be -j 192Reviewer Checklist
type(scope): [Issue #xxx] summary