Repository navigation
[fix](external) Preserve ES residual filters and repair scanner regression setup - #69650
Merged
Gabriel39 merged 1 commit intoOct 27, 2026
Merged
Conversation
…ssion setup ### What problem does this PR solve? Related PR: apache#69614 `external_table_p2/es/test_es_file_scanner_v2.groovy` fails on its first legacy-scanner baseline query with `Table [test1] does not exist in database [default_db]`. Its interpolated port key is a Groovy `GString`, while `otherConfigs` is a `Properties` containing `String` keys. `Properties.get(Object)` returns null even when the port is configured, producing `http://<es-host>:null`. Convert the key to `String` and fail explicitly when the configuration is absent. The failure happens before any V2 result comparison or BE scan. The coverage review also found a query correctness issue shared by both scanners: a nested AND can return a partial ES DSL and record an unsupported child, but the single-expression path ignored that residual tracking. For `(a = 1 AND local_predicate) OR a = 2`, the engine could drop the local predicate; `NOT(a = 1 AND local_predicate)` could also become an unsafe `NOT(a = 1)` at the source. Retain the complete expression locally whenever conversion records a residual. Fully pushable expressions and independent top-level AND conjuncts retain their existing pushdown behavior. Expand coverage in the same change: - ES 5/6/7/8 V1/V2 comparisons across source/docvalue scans, parallel/whole-index scrolling, and batch sizes 1/1024. Add scalar/date/array/JSON, IN/NOT IN, null, raw ES query, residual filtering, aggregation and LIMIT cases; restore every changed session variable. - An independent equivalent-query oracle for nested residual OR, since a V1/V2 comparison alone cannot detect a shared FE error. Compare cardinality for unordered LIMIT rather than document identity across BEs. - FE tests for residual tracking, docvalue eligibility and field-count boundaries, alias/context serialization, shard planning and safe early-stop parameters. - BE tests for source-only fallback, missing/null docvalues and empty arrays, real HTTP empty results, early close and scroll 404/500 cleanup. Add an opt-in authenticated live-reader smoke test. ### Release note Preserve local filtering for Elasticsearch compound predicates that cannot be completely pushed down. ### Check List (For Author) - Test - [x] FE: all 127 ES connector tests pass. The two new nested OR/NOT tests fail before the residual-tracking fix. - [x] BE: all 27 selected ES ASAN tests pass against authenticated Elasticsearch 9.0.0 and 9.5.5, including source/docvalue reads with batch size 1. - [x] FE Checkstyle for the ES module and its reactor dependencies; clang-format 16, clang-tidy on changed C++ lines, and BE build-hygiene checks. - [x] Groovy 4.0.19: reproduce the null port before the fix; verify all four port lookups, suite setup/matrix execution, missing configuration, disabled execution and restoration after an injected query failure. The setup probe substitutes SQL execution and does not validate SQL results. - [ ] Full external ES 5/6/7/8 SQL regression run; requires the external Docker fixtures and a Doris cluster. - Behavior changed: - [x] Yes: residual ES predicates remain enforced; regression setup reads configured ports correctly. - Does this need documentation? - [x] No: existing catalog properties and scanner session variables are unchanged. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
7 of 12 tasks
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Static review of all five changed files at 06d1850f3aa5e05a089c27624065c88e308ff340 found no introduced issue requiring an inline finding. There are no existing inline findings to retain. No additional user focus was supplied.
The changed single-expression guard preserves the complete OR/NOT whenever conversion records an unsupported nested child. I traced the residual index through the connector result, FE conjunct retention and predicate-slot materialization, both scanner implementations, and source/scanner LIMIT decisions. Independent FE, BE/regression, and predicate-flow reviews all returned NO_NEW_VALUABLE_FINDINGS; all candidates and initial risks have evidence-backed dispositions.
Critical checkpoint conclusions:
- Goal and proof: the recorded-partial-predicate fix and String port-key lookup match their stated failure paths. New unit cases cover partial AND, OR/NOT fallback and fully pushable controls; the regression suite adds an independent nested-OR oracle. These tests were inspected, not executed.
- Scope and reuse: the production change is one condition with a concrete explanatory example. Expanded tests reuse existing connector fakes and reader helpers.
- Concurrency and locks: predicate conversion uses local accumulators and introduces no shared mutation or lock. HTTP test counters are atomic; handler options are set before workers start. No new lock-order or heavy-work-under-lock issue was found.
- Lifecycle and memory: test declaration order keeps handlers alive until server workers stop and join, with readers closed first. Early close, empty results and 404/500 paths exercise scroll cleanup. No new production allocation, ownership or static-initialization mechanism is added.
- Configuration: no configuration item is added. The three changed session variables are captured and restored in finally; missing port configuration fails explicitly.
- Compatibility: no function symbol, plugin SPI or storage-format change is introduced. Existing ES properties and residual-index contracts remain in use.
- Thrift/protobuf: no definitions or field IDs change; new tests inspect existing ES Thrift fields.
- Parallel paths: the single OR/NOT guard now agrees with existing partial-child handling under a top-level AND. Both scanners consume the same FE result; source/docvalue and parallel/whole-index paths are covered by the test matrix.
- Conditional checks: the changed guard documents the concrete unsafe NOT example; tests cover both rejection and successful-pushdown cases.
- Coverage: inspected FE builder/planning tests, BE projection/null/empty-array/error/cleanup tests, and the ES5/6/7/8 matrix at batch sizes 1 and 1024. Version-specific fixtures and query fields match their guards. The optional live-reader test remains opt-in.
- Expected results: new assertion values agree with mapping, routing, serialization and parser code. Ordering is deterministic for the fixtures; unordered LIMIT compares cardinality. No golden result files change.
- Observability: no new operational stage or metric is needed for this guard. Existing query DSL, scanner profiles and error paths cover the affected flow.
- Transactions and persistence: no transaction, visible-version, delete-bitmap, EditLog or failover behavior is changed by this read-planning fix.
- Writes and atomicity: no production data-write path is changed; HTTP test cleanup is explicit and checked.
- FE/BE variables: no new transmitted variable or sending point is added. Retained predicates prevent the all-conjuncts-pushed flag, so the existing source early-stop gate withholds LIMIT; scanner limits count filtered survivors.
- Performance: the new emptiness check adds no traversal. Wider source reads for incompletely pushable compounds are required to preserve local filtering. Test expansion is bounded by small fixtures.
- Additional recheck: inspected unsupported converters, predicate-only columns, source-only fields, JSONB sorting, empty/error pages and parallel paths. Computed IS NULL tracking and mixed source-only ESQUERY/local composition limitations predate this change and were not promoted to introduced findings. No unresolved static candidate remains.
Review converged in one round. Validation is static only: no builds, tests, formatting tools, benchmarks or live ES/Doris execution were run. The author's reported results and full external ES5/6/7/8 runtime interoperability were not independently verified.
Review converged after 1 round(s).
924060929
approved these changes
Oct 27, 2026
Contributor
TPC-H: Total hot run time: 28248 ms |
Contributor
TPC-DS: Total hot run time: 150410 ms |
Contributor
ClickBench: Total hot run time: 24.33 s |
Contributor
FE UT Coverage ReportIncrement line coverage `` 🎉 |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Gabriel39
added a commit
that referenced
this pull request
Oct 27, 2026
) ### What problem does this PR solve? Related PR: #69650 (reader regression coverage) With `nodes_discovery=true`, ES metadata loading follows `EsMetadataFetcher.fetchShards -> EsConnectorRestClient.getHttpNodes -> EsNodeInfo -> EsMajorVersion.parse`. The parser accepts only majors through 8, so a 9.x node makes discovery fail with an unsupported-version exception. The configured-seed path with discovery disabled does not parse node versions and is not affected in the same way. Add the explicit ES 9 major, comparisons and latest-supported bound while continuing to reject unvalidated future majors with a clear diagnostic. Add an HTTP unit test containing mixed 8.19/9.0 nodes, IPv6 routing and a node without HTTP, plus an opt-in live compatibility suite rather than relying on version parsing alone. The live suite creates isolated temporary indices and checks the production connector's `_nodes/http`, `_mapping`, `_search_shards`, index listing and metadata fetching with discovery enabled/disabled. It also verifies authenticated source/docvalue scroll requests, missing values, pagination, scroll cleanup and invalid-credential failures. Enable it with `DORIS_ES9_TEST_URL`, `DORIS_ES9_TEST_USER` and `DORIS_ES9_TEST_PASSWORD`. ### Release note Allow Elasticsearch 9.x nodes during ES catalog node discovery. ### Check List (For Author) - Test - [x] Version parsing and mixed-node tests reproduce the unsupported-9.x failure before the change. - [x] All 124 ES connector tests pass with live authenticated Elasticsearch 9.0.0 and 9.5.5, including the three opt-in compatibility tests on each version. - [x] Default unit-test execution passes with the live suite disabled when no endpoint is configured. - [x] FE Checkstyle for the ES module and its reactor dependencies. - [x] The BE ES reader also reads the authenticated three-row fixture through source/docvalue scans with batch size 1 on both versions, using the reader coverage change in the related ES regression PR. - [ ] Full Doris SQL end-to-end validation against ES 9; the live tests above validate connector metadata and HTTP reader compatibility. - Behavior changed: - [x] Yes: discovery accepts ES 9 nodes; later unverified majors remain rejected. - Does this need documentation? - [x] No new catalog properties or configuration are introduced. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
4 tasks done
Gabriel39
added a commit
to Gabriel39/incubator-doris
that referenced
this pull request
Oct 29, 2026
Apply the outstanding changes in master merge order: apache#69545 7d16729 apache#69551 7224a2d apache#69612 61d1aef apache#69615 8d2bd1e apache#69650 3dc7341 apache#69651 c00ecb1 apache#69646 e493c61 apache#69668 7865646 apache#69614 already exists on branch-4.2 through apache#69644. Adapt datasource APIs and tests to the release branch, retaining its existing scanner dispatch and statement-bound metadata contracts.
yiguolei
pushed a commit
that referenced
this pull request
Oct 30, 2026
### What problem does this PR solve? Backport the requested external-data and FileScannerV2 changes to `branch-4.2` in their actual master merge order: | Order | Master PR | Change | Backport | | --- | --- | --- | --- | | 1 | #69545 | Preserve Paimon schema options after field renames | Included | | 2 | #69551 | Expand Iceberg capability, transaction and snapshot tests | Included with 4.2 API adaptations | | 3 | #69614 | Elasticsearch HTTP scans in FileScannerV2 | Already present through #69644 (`2104b7509cda`); not applied twice | | 4 | #69612 | Check catalog visibility before SHOW DATABASES accesses metadata | Included | | 5 | #69615 | Publish Iceberg Parquet split offsets | Included | | 6 | #69650 | Preserve ES compound residuals and expand scanner coverage | Included | | 7 | #69651 | Accept Elasticsearch 9.x during node discovery | Included | | 8 | #69646 | Prioritize Parquet V2 row groups using TopN bounds | Included | | 9 | #69668 | Arrow loads and schema inference in FileScannerV2 | Included | The eight outstanding changes are combined in one commit. This branch retains the 4.2 FE datasource APIs instead of importing master's connector-module refactor. ### Backport adaptations - Port Paimon schema restoration, catalog-option provenance, fallback handling and tests to the existing table-copy/serialization lifecycle. Keep statement-bound fields when reader options are applied after a schema-only rename. - Exercise Iceberg's existing metadata and transaction entry points with real in-memory tables and retained-generation fixtures. Preserve the nine disabled known-bug reproductions; omit two connector-only lifecycle tests. Expiration checks verify live data-file sets and exact delete-file counters; 4.2's separate manifest-name/data-counter issue remains unchanged. The delete/history SQL suite checks visible rows and eager/late-materialization parity because 4.2 does not expose master's `_file`/`_pos` columns. The inventory labels upstream-only evidence explicitly. - Port ES residual tracking to the legacy planner and add compound OR/NOT, docvalue eligibility and Thrift-context tests there. Use Java 8-compatible HTTP APIs for opt-in ES 9 tests and the branch's supported node-address fixtures. - Keep the existing full-schema Iceberg metadata-table scan, which already supplies the file-size dependency for split offsets. Adapt the external write suite to verify offsets without requiring master-only distribution plans or golden output. - Use the existing Parquet statistics interface and physical row-location representation. Keep the TopN scheduling and pruning invariants. - Include the shared Arrow array normalizer required by the new reader, taken from the same master revision. Preserve 4.2 Lance/Avro dispatch; omit the schema-inference test for the unavailable Hive OpenCSV parser. ### Validation - FE production and test sources compile with the branch's Java 8 target. Across the targeted runs and focused reruns, 20 test classes report 181 discovered, 171 passed, zero failures/errors, and 10 skipped: nine intentionally disabled known-bug reproductions plus the opt-in live ES 9 class. - FE Checkstyle: passed with zero violations. - Clang compiler syntax checks: all 23 affected C++ translation units passed against the current sources and freshly generated Thrift, using cached third-party/generated build dependencies. Focused BE runtime validation is listed below. - clang-format 16: all 30 affected C++ source/header files passed. - Groovy 4.0.19 syntax and AST conversion: all seven changed suites passed. All 105 local inventory links resolve. - Complete staged-diff self-review, whitespace/conflict checks, and sensitive-information checks passed. A full BE rebuild and SQL/external-service regressions were not run locally. Live ES 9 tests require their opt-in endpoint. CI is requested for the combined backport. ### Release note Fix Paimon reads after field renames, external-catalog SHOW DATABASES authorization, and ES compound residual filtering. Support ES 9.x node discovery, publish Iceberg Parquet split offsets, improve Parquet V2 TopN row-group scheduling, and use FileScannerV2 for Arrow loads and text/JSON schema inference. ### Check List (For Author) - Test - [x] Unit Test - [x] Regression test additions and syntax checks; SQL execution pending CI - Behavior changed: - [x] Yes. Backport the changes listed above. - Does this need documentation? - [x] No. Existing interfaces and settings are retained; the test coverage inventory is included. ### Follow-up fixes - Set the Thrift `enable_file_scanner_v2` default to `true`, matching the session default. Preserve explicit `false` for legacy query scans and extend the existing load-selection test. - Keep the selected Paimon snapshot and schema together during relation projection. Historical sequence/aggregation options must not be combined with the latest field names. Read-only principals can resolve historical selectors without INSERT permission. - Read legacy append schemas with missing or empty bucket keys without bucket pruning, preserving every historical file. The regression covers both unaware and multiple fixed buckets. Follow-up validation: 169 targeted Paimon FE tests passed with zero failures/errors/skips. FE production compilation and Checkstyle passed. Both changed C++ test translation units passed compiler syntax checks and clang-format 16. Regenerated Java/C++ defaults are enabled; a Java 8-targeted protocol check verifies constructor/clear defaults, an omitted wire field, and an explicit-false round trip. Complete BE CI and live external SQL suites remain pending. The two new reproductions first failed with the same missing-bucket-key and renamed-sequence-field errors as CI. A separate read-only-principal reproduction also failed before the permission correction. Complete external regression execution remains pending CI. ### BE unit-test compatibility - Complete the native Arrow Variant input dependency for `branch-4.2`: accept binary storage and the Parquet Variant extension, validate metadata and payloads before appending, and retain owned data after Arrow buffers are released. Allow that ingestion path through the load reader while preserving query-format restrictions and source-default validation. - Preserve Variant V2 markers in the Arrow test fixtures, including nested types. Verify timestamp input against this branch's SerDe for positive, negative, and second-boundary values. Match the existing DATE range diagnostic in the TopN pruning test. - Add tests for sliced and empty reads, invalid ranges and carriers, malformed metadata/payloads, NULL children, atomic failure, buffer ownership, and the query capability boundary. Validation: all 13 previously failing BE tests passed. The expanded run passed all 309 tests across nine suites, including Arrow/JSON loads, load validation, Iceberg, table mapping, and Variant input/output. The focused executable rebuilds the affected readers and SerDe against cached 4.2 build dependencies; this is not a complete BE rebuild. All seven changed C++ translation units passed compiler checks with current headers/generated Thrift, and all 33 affected C/C++ files passed clang-format 16. The complete staged diff, whitespace, and sensitive-information checks passed. Full CI is requested with `run buildall` and `/review`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Related PR: #69614
external_table_p2/es/test_es_file_scanner_v2.groovyfails on its first legacy-scanner baseline query withTable [test1] does not exist in database [default_db]. Its interpolated port key is a GroovyGString, whileotherConfigsis aPropertiescontainingStringkeys.Properties.get(Object)returns null even when the port is configured, producinghttp://<es-host>:null. Convert the key toStringand fail explicitly when the configuration is absent. The failure happens before any V2 result comparison or BE scan.The coverage review also found a query correctness issue shared by both scanners: a nested AND can return a partial ES DSL and record an unsupported child, but the single-expression path ignored that residual tracking. For
(a = 1 AND local_predicate) OR a = 2, the engine could drop the local predicate;NOT(a = 1 AND local_predicate)could also become an unsafeNOT(a = 1)at the source. Retain the complete expression locally whenever conversion records a residual. Fully pushable expressions and independent top-level AND conjuncts retain their existing pushdown behavior.Expand coverage in the same change:
Release note
Preserve local filtering for Elasticsearch compound predicates that cannot be completely pushed down.
Check List (For Author)
Check List (For Reviewer who merge this PR)