[Feature][Tracking] Native MongoDB Catalog #70251

Description

Track the implementation of a native, read-only MongoDB Catalog on Doris master, from client dependencies and the first single-cursor query to safe pushdown, parallel scans, and release qualification.

Design: Complete design PDF — Chinese, 49 pages, 16 chapters, four diagrams.

The PDF covers the architecture, FE/BE contracts, Schema management, BSON-to-Doris mapping, predicate semantics, split planning, consistency, resource ownership, SQL analytics, regression coverage, and the implementation roadmap. It is an RFC: proposed interfaces and settings are not claims of implemented functionality. The code-reading baseline is c5bfd6819f393bde37b73147abddc2f9a44641cb.

Related PR: #70249 prepares the client dependencies and packaging. The dependency preparation does not register a usable Catalog or Scanner. Keep D00 unchecked until integration and artifact validation pass.

Use cases and initial scope

Query MongoDB collections through SQL for filters, joins, aggregation, windows, JSON/array analysis, and pattern matching. Doris evaluates operators that are not safely pushed down. Repeated full scans and indexed text analytics can use an explicit import into a Doris internal table.

• FE: a MongoDB connector using the existing Connector SPI and Java Sync Driver for metadata, Schema discovery, and scan planning.
• BE: a native C++ Reader using libmongoc/libbson with RAII; BSON conversion belongs in DataTypeSerDe.
• Initial qualification targets: ordinary collections, standard views, and time-series collections on MongoDB 7/8, replica sets and mongos. Basic reads for all three object kinds are required for the initial release; parallel scans, snapshot reads, and pushdown are qualified per object kind.
• Read-only access. Writes, source DDL, CDC, and cross-cluster global snapshots are outside the initial release.

Initial-release object support

View and time-series basic reads are part of M0/M1 development and mandatory release acceptance. An unimplemented optimization must not defer the entire object kind. Schema/metadata privileges are documented separately from query privileges; unavailable identity/version metadata must not be fabricated.

Source constraints: standard views and permissions/snapshot isolation, view collation, and time-series limitations.

Work items

M0 — Minimal single-cursor path

• D00 — Client dependencies and build artifacts (W1; PR [chore](build) Add native MongoDB client dependencies #70249): land the dependency preparation, validate pinned drivers and transitive dependencies, licenses, complete FE/BE builds, plugin packaging, supported architectures, TLS/SASL/SRV, and compression. A dependency archive alone must not activate an unfinished connector.
• D01 — Catalog and Schema management (W1–W4): register the FE connector and distinguish collection/view/timeseries metadata. Map ordinary-collection validators; support explicit/document Schema for view output and time-series measurements; record view collation/available definition metadata and timeField/metaField. Retain source logical types, pin Schema/configuration versions, and define refresh/evolution. Do not reuse a base-collection validator as a view-output Schema, require a time-series validator, or treat a validator as proof about historical rows.
• D02 — Versioned query and single-Split contract (W1–W3): serialize object kind, namespace, available UUID/definition identity, stable configuration/Schema versions, field bindings, projection/filter descriptions, collation, and read policy. Wire the scan descriptors and capability negotiation. Active cursors, driver sessions, and client objects remain local to their owners.
• D03 — Native BE Reader (W2–W4): integrate C++ pool leases, cursor/getMore, batching, deadlines, cancellation, error translation, ScanNode execution, and Profile metrics for ordinary collections, standard views, and time-series collections. Query each logical namespace directly: MongoDB evaluates view pipelines and unpacks time-series measurements. Do not bypass view access controls or read system.buckets directly. Release leases/cursors on every success and failure path.
• D04a — Basic BSON DataTypeSerDe support (W2–W4): implement initial scalar/Nullable conversions and ObjectId as 12 raw bytes in VARBINARY(12); preserve source type/subtype metadata and copy borrowed BSON bytes before their backing storage is released.

M0 gate: Catalog creation, object-kind discovery, and minimal SELECT work for ordinary collections, standard views, and time-series collections with one Split/cursor. Basic Schema/type, authentication, cancellation, error, cleanup, and build/package checks pass. Filters remain in Doris until their pushdown contracts are verified; M1 completes object-specific qualification.

M1 — Complete single-cursor queries

• D04b — Remaining first-release type mappings (W5–W6): complete the design's numeric/Decimal128, UTC date, binary, Extended JSON, array, struct, and missing/null rules; validate precision/overflow, recursive offsets, row-level rollback, and memory limits. Declare unsupported mappings explicitly.
• D05 — Safe predicate/LIKE pushdown and projection (W4–W7): classify translations as exact/superset/unsupported; preserve SQL three-valued logic, CAST, arrays, collation, and strict error semantics. Keep required residual predicates, implement conservative LIKE translation and column pruning, and gate behavior with Session variables. Include typed BSON Date time-range filtering for time-series collections. Standard-view SQL predicates remain local initially; do not override a view collation. Verify the existing SPI/applyFilter path cannot remove predicates incorrectly.
• D06 — Read policy, security, and resource control (W4–W7): implement Catalog defaults plus statement-pinned Session overrides for read preference/concern, sensitive configuration handling, TLS/credential versioning, pool isolation, batch/memory/time limits, and shared-source admission. Distinguish scanner-instance concurrency from query-wide and source-wide budgets.

M1 gate: basic reads of all three object kinds are mandatory. Validate view-output Schema, computed/projected/grouped fields, missing/non-unique _id, non-simple collation, and accounts that can query the view but cannot find the base collection. Validate time-series Schema without a validator, logical measurements, duplicate timestamps/metaField groups, BSON Date time filters, and cleanup. Result-multiset/error differential tests, permissions, pool isolation, and capability-rejection tests pass. Preview releases document single-cursor and consistency limits.

M2 — Parallel scans and a common snapshot

• D07a — Snapshot feasibility prototype (W1–W2, performed early): demonstrate a server-selected snapshot time T, its FE-to-BE representation, and the actual libmongoc command path that reads the same atClusterTime on replica sets and mongos. Majority reads alone do not satisfy this gate.
• D07b — Snapshot consistency implementation (W7–W9): pin the read contract across qualified Splits/objects in the same cluster; define history-expiration, identity/topology-change, cancellation, and retry behavior. Reject explicit common-snapshot queries containing standard views at planning time. Qualify time-series snapshot combinations against server/driver versions and concurrent-DDL restrictions separately. Basic view/time-series single-cursor reads remain required. Do not silently downgrade consistency.
• D08 — Bounded range/auto Split planning and execution (W8–W10): implement field/type-family selection, requested/admitted/actual Split counts, query/source budgets, bounded sampling, duplicate-boundary removal, and half-open ranges with unbounded ends. Preserve a disjoint complement for missing/null/array/other-type values when the scalar type family is not proven exhaustive. Freeze the complete plan before publishing any Split; expose decisions and fallback reasons in EXPLAIN/Profile. An already published plan must not fall back to a full single-Split rescan.

M2 gate: for qualified parallel-scan object kinds, every source document belongs to exactly one partition and the complete result multiset matches a single-Split read at the same snapshot. The required M2 parallel baseline covers ordinary collections; view/time-series basic reads continue through their declared single-cursor paths. Verify unsupported range/snapshot rejection, snapshot history failures, planning limits, scheduling, and source admission before enabling any additional mode.

Continuous validation and release qualification

• D09 — Regression and fault matrix (W1–W12, delivered with each implementation): provide controlled MongoDB services with environment-injected credentials; make ordinary-collection, standard-view, and time-series basic-read suites mandatory; cover view pipelines/output Schema/collation/least privileges and definition changes, time-series logical measurements/duplicate times/metaField/TTL/time filters, Schema sources/evolution, all mappings, SQL analytics, exact and superset pushdown, NULL/CAST/array counterexamples, malformed/deep/wide documents, duplicate/skewed split boundaries, complement partitions, authentication/TLS, topology changes, concurrent writes, timeout/cancellation, and resource cleanup. Add deterministic fixtures and reproducible differential checks.
• D10 — Performance, compatibility, and release evidence (W11–W12): establish repeatable single-cursor/pushdown/parallel baselines, source-side budgets, complete build/artifact and upgrade qualification, supported driver/server/authentication/architecture combinations, EXPLAIN/Profile coverage, and user/operations documentation. Target at least 1,000 query/cancel/failure cycles and a 24-hour mixed-workload soak without continuously growing memory, threads, cursors, or connections. Run FE Checkstyle, clang-format 16 on affected BE files, and the relevant UT/regression checks.

Release gate: mandatory basic reads for all three object kinds, ordinary-collection M2, explicit unsupported-mode rejection, and all applicable correctness/resource/compatibility cases pass; archive reproducible CI, fault, performance, and upgrade results. Week numbers are suggested relative development windows, not release commitments. The design reserves W13–W14 for blocking fixes and affected revalidation.

Dependency order

flowchart TD
    P["D00 dependencies + D02 query contract"] --> A["D01 Catalog / Schema: all three object kinds"]
    P --> B["D03 native Reader: logical namespaces"]
    P --> C["D04a basic DataTypeSerDe"]
    A --> M0["M0: single-cursor path"]
    B --> M0
    C --> M0
    P -. "validate in W1-W2" .-> S0["D07a snapshot prototype"]
    M0 --> T["D04b complete type semantics"]
    T --> F["D05 safe pushdown / projection"]
    M0 --> R["D06 read policy / resource admission"]
    F --> M1["M1: all three basic-read suites pass"]
    R --> M1
    M1 --> S["D07b fixed snapshot implementation"]
    S0 --> S
    M1 --> Q["D08 range / auto planner"]
    S --> M2["M2: qualified common snapshot + partitions + admission"]
    Q --> M2
    M2 --> RC["D10 release qualification"]

Independent branches may be developed in parallel. D01/D03/D05/D09 deliver view/time-series basic support by M1. Planner-only tests can start before snapshot implementation completes; enabling multi-Split reads requires both snapshot and partition/admission gates for the relevant object kind. D09 runs throughout all branches and milestones.

Follow-up capabilities, tracked separately after the initial release

• Advanced view/time-series optimizations only: separately prove safe view pushdown and timeField range planning, costs, and any supported time-series snapshot/DDL contract. Discovery, Schema, single-cursor reads, mappings, and basic filtering are already initial-release requirements. Source limitations on standard-view snapshot isolation remain explicit.
• Opt-in, bounded Schema inference with pinned versions; samples do not prove global type homogeneity.
• Whitelisted REGEXP, bounded runtime filters, LIMIT/TopN, and aggregation pushdown, each with a separate semantic contract, switch, differential suite, and benchmark.
• Source-side search interfaces; CDC, VARIANT document storage, and a large Schema registry require separate designs and lifecycles.

Tracking convention

Keep a work item unchecked until its implementation PRs and acceptance evidence are linked and verified. Link follow-up implementation PRs here and maintain the capability/support matrix as scope changes. The dependency preparation PR is not evidence that native MongoDB queries already work.