Repository navigation
[fix](auth) Check external catalog visibility before SHOW DATABASES - #69612
Merged
Gabriel39 merged 1 commit intoOct 26, 2026
Merged
Conversation
Gabriel39
requested review from
924060929,
englefly,
morrySnow and
starocean999
as code owners
October 26, 2026 06:45
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Reviewed the complete two-file PR at commit 199dce7. No actionable new findings or existing blocking comments were identified.
Checkpoint conclusions:
- Goal and evidence: external catalog visibility is checked before remote database listing and WHERE planning. Unit cases cover explicit/default/LIKE/WHERE rejection, database filtering, internal bypass, missing catalogs, and native database/table/column grants; inspected only.
- Scope and reuse: the focused command/test patch reuses existing authorization and error APIs.
- Concurrency and locks: no new threads, shared state, or locks; existing native authorization read locks end before connector metadata access.
- Lifecycle and initialization: no new owned resource, cleanup requirement, or static initialization dependency.
- Configuration: no additions; existing skip_catalog_priv_check behavior is preserved.
- Compatibility: no signature or storage-format change. Access-denied errors for invisible external catalogs are intentional; internal and inherited native visibility remain supported.
- Thrift/protobuf: unchanged; field-ID compatibility review is not applicable.
- Parallel paths: plain, LIKE, default-catalog, and WHERE forms share the gate. It matches SHOW CATALOGS visibility policy. Global Ranger inherits descendant visibility; catalog-bound sources retain documented catalog-check/skip behavior. Cloud and classic use the same command path.
- Special conditions: the internal bypass is explained and tested; missing catalogs fail explicitly. Using the supplied ctx for database checks matches command invocation ownership.
- Coverage: meaningful negative and filtered-result tests are present. Static tracing also covered permitted WHERE planning, selected-catalog schema scanning, and database filtering; no unresolved code candidate remains.
- Expected results: asserted visible rows are consistent with filtering and deterministic ordering; no golden output files changed.
- Observability and errors: existing error 5087 names the user/catalog and propagates through the command error boundary; no added long-running operation requires metrics.
- Transactions, persistence, and failover: no affected write or EditLog path.
- Data writes and atomicity: no data mutation introduced.
- FE-BE variable propagation: no new field or send path.
- Performance: one catalog authorization check per external invocation; no new remote metadata request or asymptotic regression identified.
- Other invariants: no optimizer-rule, slot, physical-property, partition-version, delete-bitmap, or BE memory-ownership change. Denial follows the existing failure-reporting convention.
User focus: no additional focus points were supplied.
Validation limits: static review only, as instructed. No builds or tests were run. The added unit tests do not execute a permitted WHERE query or live Ranger/connector integration, and no runtime verification is claimed. Both independent reviewers returned NO_NEW_VALUABLE_FINDINGS; all initial risk points have evidence-backed dispositions and no static coverage gaps remain.
Review converged after 1 round(s).
Contributor
TPC-H: Total hot run time: 28716 ms |
Contributor
TPC-DS: Total hot run time: 150401 ms |
Contributor
ClickBench: Total hot run time: 24.34 s |
CalvinKirs
approved these changes
Oct 26, 2026
924060929
approved these changes
Oct 26, 2026
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
|
PR approved by anyone and no changes requested. |
github-actions
Bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
4 tasks done
Gabriel39
added a commit
to Gabriel39/incubator-doris
that referenced
this pull request
Oct 29, 2026
Apply the outstanding changes in master merge order: apache#69545 7d16729 apache#69551 7224a2d apache#69612 61d1aef apache#69615 8d2bd1e apache#69650 3dc7341 apache#69651 c00ecb1 apache#69646 e493c61 apache#69668 7865646 apache#69614 already exists on branch-4.2 through apache#69644. Adapt datasource APIs and tests to the release branch, retaining its existing scanner dispatch and statement-bound metadata contracts.
yiguolei
pushed a commit
that referenced
this pull request
Oct 30, 2026
### What problem does this PR solve? Backport the requested external-data and FileScannerV2 changes to `branch-4.2` in their actual master merge order: | Order | Master PR | Change | Backport | | --- | --- | --- | --- | | 1 | #69545 | Preserve Paimon schema options after field renames | Included | | 2 | #69551 | Expand Iceberg capability, transaction and snapshot tests | Included with 4.2 API adaptations | | 3 | #69614 | Elasticsearch HTTP scans in FileScannerV2 | Already present through #69644 (`2104b7509cda`); not applied twice | | 4 | #69612 | Check catalog visibility before SHOW DATABASES accesses metadata | Included | | 5 | #69615 | Publish Iceberg Parquet split offsets | Included | | 6 | #69650 | Preserve ES compound residuals and expand scanner coverage | Included | | 7 | #69651 | Accept Elasticsearch 9.x during node discovery | Included | | 8 | #69646 | Prioritize Parquet V2 row groups using TopN bounds | Included | | 9 | #69668 | Arrow loads and schema inference in FileScannerV2 | Included | The eight outstanding changes are combined in one commit. This branch retains the 4.2 FE datasource APIs instead of importing master's connector-module refactor. ### Backport adaptations - Port Paimon schema restoration, catalog-option provenance, fallback handling and tests to the existing table-copy/serialization lifecycle. Keep statement-bound fields when reader options are applied after a schema-only rename. - Exercise Iceberg's existing metadata and transaction entry points with real in-memory tables and retained-generation fixtures. Preserve the nine disabled known-bug reproductions; omit two connector-only lifecycle tests. Expiration checks verify live data-file sets and exact delete-file counters; 4.2's separate manifest-name/data-counter issue remains unchanged. The delete/history SQL suite checks visible rows and eager/late-materialization parity because 4.2 does not expose master's `_file`/`_pos` columns. The inventory labels upstream-only evidence explicitly. - Port ES residual tracking to the legacy planner and add compound OR/NOT, docvalue eligibility and Thrift-context tests there. Use Java 8-compatible HTTP APIs for opt-in ES 9 tests and the branch's supported node-address fixtures. - Keep the existing full-schema Iceberg metadata-table scan, which already supplies the file-size dependency for split offsets. Adapt the external write suite to verify offsets without requiring master-only distribution plans or golden output. - Use the existing Parquet statistics interface and physical row-location representation. Keep the TopN scheduling and pruning invariants. - Include the shared Arrow array normalizer required by the new reader, taken from the same master revision. Preserve 4.2 Lance/Avro dispatch; omit the schema-inference test for the unavailable Hive OpenCSV parser. ### Validation - FE production and test sources compile with the branch's Java 8 target. Across the targeted runs and focused reruns, 20 test classes report 181 discovered, 171 passed, zero failures/errors, and 10 skipped: nine intentionally disabled known-bug reproductions plus the opt-in live ES 9 class. - FE Checkstyle: passed with zero violations. - Clang compiler syntax checks: all 23 affected C++ translation units passed against the current sources and freshly generated Thrift, using cached third-party/generated build dependencies. Focused BE runtime validation is listed below. - clang-format 16: all 30 affected C++ source/header files passed. - Groovy 4.0.19 syntax and AST conversion: all seven changed suites passed. All 105 local inventory links resolve. - Complete staged-diff self-review, whitespace/conflict checks, and sensitive-information checks passed. A full BE rebuild and SQL/external-service regressions were not run locally. Live ES 9 tests require their opt-in endpoint. CI is requested for the combined backport. ### Release note Fix Paimon reads after field renames, external-catalog SHOW DATABASES authorization, and ES compound residual filtering. Support ES 9.x node discovery, publish Iceberg Parquet split offsets, improve Parquet V2 TopN row-group scheduling, and use FileScannerV2 for Arrow loads and text/JSON schema inference. ### Check List (For Author) - Test - [x] Unit Test - [x] Regression test additions and syntax checks; SQL execution pending CI - Behavior changed: - [x] Yes. Backport the changes listed above. - Does this need documentation? - [x] No. Existing interfaces and settings are retained; the test coverage inventory is included. ### Follow-up fixes - Set the Thrift `enable_file_scanner_v2` default to `true`, matching the session default. Preserve explicit `false` for legacy query scans and extend the existing load-selection test. - Keep the selected Paimon snapshot and schema together during relation projection. Historical sequence/aggregation options must not be combined with the latest field names. Read-only principals can resolve historical selectors without INSERT permission. - Read legacy append schemas with missing or empty bucket keys without bucket pruning, preserving every historical file. The regression covers both unaware and multiple fixed buckets. Follow-up validation: 169 targeted Paimon FE tests passed with zero failures/errors/skips. FE production compilation and Checkstyle passed. Both changed C++ test translation units passed compiler syntax checks and clang-format 16. Regenerated Java/C++ defaults are enabled; a Java 8-targeted protocol check verifies constructor/clear defaults, an omitted wire field, and an explicit-false round trip. Complete BE CI and live external SQL suites remain pending. The two new reproductions first failed with the same missing-bucket-key and renamed-sequence-field errors as CI. A separate read-only-principal reproduction also failed before the permission correction. Complete external regression execution remains pending CI. ### BE unit-test compatibility - Complete the native Arrow Variant input dependency for `branch-4.2`: accept binary storage and the Parquet Variant extension, validate metadata and payloads before appending, and retain owned data after Arrow buffers are released. Allow that ingestion path through the load reader while preserving query-format restrictions and source-default validation. - Preserve Variant V2 markers in the Arrow test fixtures, including nested types. Verify timestamp input against this branch's SerDe for positive, negative, and second-boundary values. Match the existing DATE range diagnostic in the TopN pruning test. - Add tests for sliced and empty reads, invalid ranges and carriers, malformed metadata/payloads, NULL children, atomic failure, buffer ownership, and the query capability boundary. Validation: all 13 previously failing BE tests passed. The expanded run passed all 309 tests across nine suites, including Arrow/JSON loads, load validation, Iceberg, table mapping, and Variant input/output. The focused executable rebuilds the affected readers and SerDe against cached 4.2 build dependencies; this is not a complete BE rebuild. All seven changed C++ translation units passed compiler checks with current headers/generated Thrift, and all 33 affected C/C++ files passed clang-format 16. The complete staged diff, whitespace, and sensitive-information checks passed. Full CI is requested with `run buildall` and `/review`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
SHOW DATABASEScould call an external catalog's metadata service before checking whether the caller could see that catalog. A caller with no privileges could trigger remote initialization and receive connection errors.Check catalog visibility before listing databases or planning the
WHEREform. Reuse the existingSHOWpredicate, which includes database, table and column grants and honors the configured authorization controller. Keep the existing per-database filtering and internal-catalog behavior.Release note
Check external catalog visibility before accessing metadata for
SHOW DATABASES.Validation
ShowDatabasesCommandTesttests passed.Check List (For Author)