Repository navigation
[chore](build) Add native MongoDB client dependencies - #70249
Conversation
### What problem does this PR solve? Issue Number: None Problem Summary: Prepare the native MongoDB connector's FE and BE client dependencies. Build libmongoc/libbson together, expose explicit static CMake targets, and stage a version-aligned Java dependency archive with compression providers. Connector registration and query execution remain separate changes. ### Release note None ### Check List (For Author) - Test: Native dependency build, CMake/C++ and Java/JNI smoke checks, Maven package with Checkstyle, archive/staging inspection, and shell syntax checks. Full FE build was not completed because the existing third-party installation lacks Paimon artifacts. - Behavior changed: No query behavior change. - Does this need documentation: No user-facing change; dependency README included.
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
/review |
|
Codex automated review failed and did not complete. Error: Selected model is at capacity. Please try a different model. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
|
Dependency review of License compatibility: OK for the newly introduced components. The MongoDB client drivers use Apache-2.0; the bundled permissive licenses are compatible. The MongoDB Server's SSPL does not apply to these client dependencies. Please update these two dependency versions before merging:
No matching published advisory was found for MongoDB Java Driver 5.14.0 or zstd-jni 1.5.7-25 in the exact-version/vendor checks as of November 5, 2026. The module-local BOM, explicit runtime compression providers, logging exclusions and managed native dependency lookup look reasonable from static dependency inspection. Impact boundary: this PR prepares dependencies, stages an unopened FE archive and registers native targets with |
### Background The native MongoDB connector prepares C client libraries for BE and a Java dependency bundle for FE. Compression providers are included in the FE archive. ### What problem does this PR solve? Issue Number: None Related PR: apache#70249 PR Type: Release/Dependency Problem Summary: The new dependency bundle pins MongoDB C Driver 2.5.7 and inherits snappy-java 1.1.10.5, which contain known upstream security defects. Upgrade the C driver and libbson together to 2.5.8, and pin snappy-java 1.1.10.10 within the MongoDB module. ### Details - **Versions:** MongoDB C Driver/libbson 2.5.7 to 2.5.8; MongoDB module snappy-java 1.1.10.5 to 1.1.10.10. - **Upstream changes:** Include cursor-response, compressed-reply, GridFS, allocation, buffer-boundary and stream-handling fixes. - **Compatibility:** Preserve the existing native build options and module isolation; update the archive checksum and distribution version metadata together. - **Rollback:** Revert these dependency pins and distribution metadata. ### Testing - Build and install the native libraries with the third-party build script; link the BE CMake targets and run BSON/ObjectId, client lifecycle and snapshot-options smoke checks. Both native libraries report 2.5.8. - Package the MongoDB FE module with Checkstyle and the connector import gate; inspect the six runtime jars and archive staging, and run BSON/read-options and Snappy/Zstd JNI round trips. - Reproduce oversized allocation with snappy-java 1.1.10.5 and verify rejection using the packaged 1.1.10.10 jar, with a 64 MiB JVM heap. - Verify the official C archive digest, configured checksum, upstream notices, managed-prefix rejection, shell syntax and diff whitespace. - Full FE build and cross-platform coverage remain for CI. ### Release note None ### Check List (For Author) - Behavior changed: No query behavior change; update prepared dependency versions. - Does this need documentation: No. - Pre-submission status: Partially verified: scoped dependency builds and smoke checks pass; full Doris build and other platforms remain for CI. ### AI Disclosure - Tool: OpenAI Codex. - Generated: Dependency updates and this message. - Human review: Pending maintainer review; agent self-review completed.
|
run buildall |
|
/review |
|
Addressed the dependency review in the latest commit:
Validation: the native third-party build and a CMake/C++ consumer pass, with both native libraries reporting 2.5.8. The scoped Maven package passes Checkstyle and the connector import gate; the generated archive contains the expected six runtime jars and snappy-java 1.1.10.10. Java BSON/read-options and Snappy/Zstd JNI smoke checks, archive staging, the managed-prefix rejection check, shell syntax and diff checks pass. A local negative-input smoke check with a five-byte compressed input reproduces an oversized allocation/OOM on 1.1.10.5; the same input is rejected without the oversized allocation on the packaged 1.1.10.10 jar. Both JVM runs use a 64 MiB heap. These are dependency-level checks; no MongoDB SQL execution is claimed. Full Doris build and other platforms remain for CI. This addresses the review at #70249 (comment). |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Static review of PR head 5123cfa completed in 2 rounds and converged. Two verified issues remain: the dependency bundle omits the optional bson-record-codec jar that its README and license inventory promise, and FE packaging unconditionally copies an assembly zip that Maven's jar-only cache restore does not recreate. The native C driver targets, managed dependency cache, package ordering, checksums, notices, and root-archive/plugin-loader staging were independently checked and no additional defects were found. User review focus was empty, so no extra focus-specific issue applies. This was a static-only review under the contract: no Doris builds, tests, or runtime execution were performed; the managed native third-party tree was absent, so dynamic Linux/macOS package validation remains outside the evidence.
Review converged after 2 round(s).
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
|
/review |
|
Codex automated review failed and did not complete. Error: Review recovery stopped: PR base/head or open state changed; refusing to resume stale context Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
### Background The native MongoDB dependency preparation (#70249) provides the FE Java driver and BE C driver, but does not register a usable Catalog. Doris needs a native metadata and scan path that preserves BSON types and reads collection, view, and time-series namespaces directly. ### What problem does this PR solve? Issue Number: #70251 Related PR: #70249 (client dependencies) PR Type: Feature Problem Summary: Implement D01, D03, and D04 of the native MongoDB Catalog roadmap, plus the minimal single-Split descriptor needed to connect them. Users can create a `type=mongodb` Catalog and query ordinary collections, standard views, and time-series measurements through a native C++ reader. FE pins explicit, validator, or document schemas; concrete `DataTypeSerDe` implementations convert BSON without losing int64 precision, ObjectId bytes, or binary subtype identity. ### Details - **Change map:** 3,088 added production/build lines, 2,231 test/fixture lines, 350 generated regression-result lines, and 157 documentation lines (33 removed lines total). FE adds the Connector SPI provider, catalog-owned metadata client, schema resolution, stable handles, plugin packaging, and a single native scan range. BE adds the cursor/pool lifecycle and BSON view/context, with decoding in existing concrete SerDes. Tests include deterministic MongoDB fixtures, FE/BE unit tests, and a SQL regression suite with generated expected results. - **Scope:** read-only single-cursor scans of ordinary collections, standard views, and time-series collections. Views retain their own pipeline/collation and work with view-only `find` access; internal time-series buckets are hidden. All SQL filters, LIKE/REGEXP, joins, windows, aggregates, sorting, and LIMIT are evaluated in Doris. Predicate/projection pushdown, parallel range planning, common snapshots, and shared-source admission remain separate work items. - **Design:** FE sends stable namespace/configuration/schema values through `TMongoScanDesc`. BE owns bounded client-pool leases and cursors with RAII and closes them on EOF, early termination, cancellation, and errors. BSON conversions validate physical tags and exact numeric/date ranges; compound columns roll back on failure. The BSON helpers, MongoFileReader lifecycle, and all TMongoScanDesc fields document their ownership, units, scope, and constraints. Metadata pagination uses the pinned Java driver core operation and its server-affine cursor while retaining authorized/name-only listing and namespace kinds. BE inherits complete URI read preferences, including tags and max staleness. BSON validation uses a depth-bounded iterator stack; unsupported validator structures fall back as a whole in auto mode and fail clearly in validator mode. Canonical Extended JSON and raw BSON provide document access. Reader profiles expose read/decode/filter time, rows, and BSON bytes. - **Usage:** `CREATE CATALOG mongo_native PROPERTIES ("type"="mongodb", "mongodb.uri"="mongodb://mongo.example:27017", "mongodb.databases"="sample")`. See the connector README for explicit schemas, credentials, supported mappings, and fixture setup. ObjectId uses `VARBINARY(12)`; use `FROM_BINARY` for supported hex/string analysis because direct VARBINARY operators retain their existing engine restrictions. Declared VARBINARY length remains schema metadata; BSON decoding preserves full ObjectId, BinData, and document payloads. - **Default behavior:** `auto` chooses a namespace's explicit mapping, a fully representable ordinary-collection validator, or document mode; it never samples source rows. All mapped fields are nullable and are not declared keys. Mismatches fail by default; optional top-level null substitution retains atomic nested writes. Dates use UTC. Batch size and query timeout use session settings; connection/read defaults belong to the Catalog. - **Compatibility:** the new Catalog uses the dependencies introduced by #70249. Existing connectors and their serialization paths retain their behavior. Cross-version capability negotiation is outside this change. ### Testing All checks below ran locally: - Complete FE build (`build.sh --fe`) and complete ASAN BE build (`build.sh --be`). - FE connector unit tests: **31 passed**, including schema resolution, metadata/driver cursor cleanup, handle serialization, session policies and native scan descriptors. Related FE core and legacy MongoDB connector tests also passed through `run-fe-ut.sh`. - BE unit tests through `run-be-ut.sh`: **142 passed** across BSON SerDe/reader/real-driver error paths and existing decoded-value/name/VARBINARY coverage. The focused ObjectId/raw-BSON length tests failed before the fix and passed afterward. New coverage verifies bounded BSON validation memory, depth and element limits, and preservation of complete URI read preferences; the depth-boundary test was rerun after its final refactor. - Native MongoDB SQL regression against the deployed FE/BE and MongoDB 8.0.20: generated expected output, then independently compared it; final rerun **1 suite passed, 0 failed/fatal/skipped** after recreating the source fixtures. Coverage includes explicit BSON null versus missing fields, integer/decimal bounds, float subnormal/nonfinite/signed-zero values, datetime/timestamp boundaries, binary bytes/subtypes, strings, nested values, mismatch policies, and document limits. New fixtures cover unsupported tuple/pattern/dependency validators, a wide BSON document, and metadata pagination beyond 128 namespaces. - License headers: the PR incremental scan passed with skywalking-eyes v0.8.0 after adding the ASF header to the MongoDB ServiceLoader descriptor. - FE Checkstyle: zero violations. Clang-format **16.0.6** on all **31** affected C++ source/header files, build hygiene, and the repository clang-tidy check passed. Static analysis used compilation commands from the completed BE/UT builds, expanded from unity commands for individual files. - Two-mongos integration with 140 collections reproduced `CursorNotFound` on the old implementation; the updated implementation completed 20 consecutive full listings. Matching URI read-preference tags read successfully; unmatched tags correctly timed out instead of silently routing elsewhere. Unit tests also verify preservation of URI max staleness. - Full diff/self-review and private-context audit completed. ### Notes for Reviewers Start with `MongoSchema`, `MongoScanPlanProvider`, `mongodb_reader.cpp`, and the BSON SerDe tests. The regression suite covers all three logical object kinds, schema refresh, historical validator violations, source permissions, exact/special BSON values, nested rollback, SQL analysis, batch boundaries, and repeated early cursor closure. The current local integration environment uses MongoDB 8.0.20; MongoDB 7, full replica-set/mongos topology qualification (including failover and live staleness behavior), TLS/authentication matrices, performance, and soak evidence remain in the tracking issue. The two-mongos check above is a focused metadata-cursor and tag-routing test. ### Release note Add a native read-only MongoDB Catalog supporting explicit/validator/document schemas, native BSON decoding, and single-cursor reads of collections, standard views, and time-series measurements. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: Yes; connector and fixture READMEs are included. Public user-documentation and release qualification remain tracked in #70251. - Pre-submission status: Ready for review; local validation listed above, CI pending. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: implementation, tests, documentation, and this description. - Human review: pending maintainer review.
Background
Doris loads FE connectors in isolated plugin directories, while BE native clients are built through the shared third-party toolchain. A native MongoDB connector needs the synchronous Java client for FE metadata access and the C driver/BSON libraries for BE reads.
What problem does this PR solve?
Issue Number: None
Related PR: None
PR Type: Release/Dependency
Problem Summary:
Prepare the client dependencies and build outputs needed for native MongoDB catalog development. Build libmongoc and libbson together, expose explicit BE CMake targets, and package an isolated FE dependency bundle with aligned MongoDB jars and optional compression providers. The bundle is staged as an archive until a connector provider is implemented; no catalog or scanner is registered by this change.
Details
COMMON_THIRDPARTYuntil consumed by a reader.Testing
Locally on Linux x86_64:
bash thirdparty/build-thirdparty.sh -j 8 mongo_c_driver: builds and installs both static libraries successfully.mongoc2/bson2targets and runs BSON/ObjectId, client lifecycle and snapshot-session-options smoke checks. Required TLS/SASL/SRV/compression feature macros are enabled.mvn -f fe/pom.xml -pl :fe-connector-mongodb -am package -DskipTests -Dskip.doc=true -Dmaven.build.cache.enabled=false: succeeds, including FE Checkstyle and the connector import gate. No module unit tests are claimed.build.shstaging block and verify the copied archive hash and absence of an auto-loaded MongoDB plugin directory.OutOfMemoryErrorwith snappy-java 1.1.10.5 using a five-byte input declaring an oversized uncompressed length; the same check against the packaged 1.1.10.10 jar rejects the input without that allocation. The JVM is capped at 64 MiB for both checks.git diff --checkpass.The full
build.sh --febuild was not completed: the existing third-party installation lackslibpaimon_c.aandpaimon_rust/paimon.h, so its preflight starts an unrelated full third-party rebuild. That rebuild was stopped; the scoped validations above completed. Other platform and full-build coverage remain for CI.Release note
None
Check List (For Author)
AI Disclosure