[Tracking][Catalog] Catalog SPI migration — decouple built-in connectors from FE core into loadable plugins #65185

Development branch: branch-catalog-spi — all migration PRs target this branch; it merges to master through #64304.

Background

Today Apache Doris FE hard-codes every external data source (hive, iceberg, paimon,
hudi, trino-connector, maxcompute, jdbc, es, …) inside fe/fe-core/.../datasource/,
wired into the catalog and planner layers through switch-case and
instanceof *ExternalTable checks. Adding or changing a connector means touching FE
core.

This tracking issue covers the effort to decouple each concrete connector into an
independent, loadable plugin module under fe/fe-connector/*, leaving only generic
infrastructure and a stable catalog SPI bridge in fe-core.

End state: each connector ships as a self-contained plugin zip; fe-core has no
compile-time knowledge of any specific connector; the runtime discovers and loads
connectors through the catalog SPI (PluginDrivenExternalCatalog / …Table /
…ScanNode).

In scope: FE-side connector code under
datasource/{hive,iceberg,paimon,hudi,trinoconnector,maxcompute,jdbc,es,…}, plus the
reverse couplings (instanceof *ExternalTable, connector-specific sinks / scan-nodes)
scattered across nereids / planner / tablefunction / alter.

Out of scope: BE-side readers; fe-fs-spi file-system plugin work (separate
workstream); extension-spi; kafka / kinesis / odbc / Doris-to-Doris federation
(tracked separately). RemoteDoris stays an engine built-in catalog type; the
deprecated lakesoul type was removed rather than migrated (#65736).

Where the framework is documented

Prose docs now live in the tree and are the authoritative reference (they replaced the
branch's plan-doc/ corpus in #66251):

• fe/fe-connector/README.md — architecture, module map, how a connector runs
  (startup / CREATE CATALOG routing / per-statement metadata / classloading /
  packaging), how to read the API, and a step-by-step "adding a new connector" guide.
• fe/fe-connector/AGENTS.md — build & test recipes, the machine-checked obligations,
  and the invariants that no gate can express.

Design invariants (acceptance criteria)

These invariants must hold at every step:

1. One-way dependency fe-connector → fe-core. No connector module may
   import org.apache.doris.{catalog,common,datasource,qe,analysis,nereids,planner}.
   Enforced in the validate phase by build-support/check-fe-connector-imports.sh
   (self-test under build-support/tests/).
2. Single metadata entry point. Inside fe-core, only PluginDrivenMetadata may
   call Connector#getMetadata; exempt call sites carry an explicit marker and
   deleting a marker auto-tightens the gate. Enforced by
   build-support/check-fe-core-metadata-funnel.sh ([perf](catalog) fe-connector-iceberg hot-path caching + fe-core per-statement metadata funnel, with session=user cache isolation #65785).
3. Metadata backward compatibility. Old FE images holding GSON types such as
   IcebergExternalCatalog / PaimonExternalDatabase must deserialize and migrate
   transparently to PluginDrivenExternalCatalog. Verified by a unit-test suite whose
   fixtures are generated from real 4.1.3 bytecode ([test](catalog) verify 4.1.3 catalogs survive the connector-SPI cutover, with fixtures generated by real 4.1.3 bytecode #66214).
4. No user-visible behavior regression. SHOW CREATE CATALOG, SHOW TABLE STATUS,
   information_schema.tables, EXPLAIN output, error messages, and catalog type /
   engine names are all preserved.

Progress — phases

Each phase lands as one or more PRs against branch-catalog-spi.

• P0 — SPI foundation — fill SPI gaps (DDL / partition, procedures, MVCC
  snapshot, vended credentials, sys-tables, statistics, write-config) + connector
  import-gate CI. ([feat](connector) P0 SPI baseline + DDL/Partition + import gate (T03-T27) #63582)
• P1 — Scan-node consolidation — route plugin-driven scans first in the nereids
  translator; remove duplicated fe-core code. ([P1-T03-T05] route plugin-driven scans first in nereids translator #63641)
• P2 — trino-connector — first end-to-end connector migration; establishes the
  reusable migration playbook. ([feat](connector) P2 migrate trino-connector to catalog SPI (T01-T13) #64096)
• P3 — hudi (hybrid) — connector hardening + test baseline + DLA dispatch
  design; live cutover deferred to P7 (hudi is parasitic on HMS metadata) and
  landed there. ([feat](connector) P3 hudi connector hardening + test baseline + dispatch design (hybrid, T02-T08) #64143)
• P4 — maxcompute — migrate + remove legacy subsystem from fe-core (fe-core is
  now odps-free). ([feat](connector) P4 maxcompute connector full adoption + live cutover via catalog SPI (T01-T06) #64253, [refactor](connector) P4 maxcompute: remove legacy subsystem from fe-core + make fe-core odps-free (T07-T09) #64256)
• P5 — paimon — migrate + cutover + remove legacy subsystem (fe-core is now
  paimon-SDK-free). ([refactor](catalog) P5 paimon: migrate to catalog SPI + cutover (#64446) #64446, [refactor](catalog) P5 paimon: remove legacy subsystem from fe-core + make fe-core paimon-SDK-free (T29) #64653)
• P6 — iceberg — 7 catalog flavors + read metadata + scan / MVCC / cache /
  vended credentials + write path + procedures + sys-tables + views, then routing
  cutover. ([refactor](catalog) P6 iceberg: migrate to catalog SPI + cutover + remove legacy fe-core subsystem #64688)
• P7 — hive (+HMS) — HMS metadata ops, metastore-event pipeline, ACID
  transaction write path; absorbs the hudi live cutover. ([refactor](catalog) Catalog spi 11 hive #65473)
• P8 — cleanup (mostly done) — the SPI_READY_TYPES allowlist is gone, and so
  are the reverse instanceof *ExternalTable sites (only javadoc references remain).
  Still open: the legacy TableType.*_EXTERNAL_TABLE enum values. Plugin-driven
  tables now report PLUGIN_EXTERNAL_TABLE; the old constants survive in persisted
  metadata and in a few SHOW PARTITIONS / partition-TVF allowlists.

Supporting refactor & post-migration hardening (landed alongside / after the phases above):

• P3b — kerberos consolidation into fe-kerberos. ([refactor](catalog) P3b: consolidate kerberos authentication into fe-kerberos #64655)
• HMS shade slimming — replace the fat hive-catalog-shade with a slim,
  plugin-private HMS metastore-client shade (~122 MB → ~15 MB per plugin). ([refactor](catalog) fe-connector: replace fat hive-catalog-shade with a slim HMS metastore-client shade #65721)
• fe-core dead-code sweep — remove subsystems orphaned by the migration:
  the dead Trino-ported directory-listing cache ([refactor](catalog) fe-core: remove dead Trino-ported DirectoryListercache cluster #65734); the dead
  ExternalMetadataOps layer, deprecated LakeSoul catalog, and odbc/statistics
  stubs ([refactor](catalog) fe-core: remove dead ExternalMetadataOps layer, deprecated LakeSoul catalog, and odbc/statistics stubs #65736); and 6 dead datasource classes ([refactor](catalog) fe-core: remove 6 dead datasource classes orphaned by the SPI migration #65740).
• Cross-query cache restoration — restore the connector metadata / partition
  caching the SPI cutover dropped, held to behavior parity, with session=user
  cache isolation: iceberg hot-path caching + a per-statement ConnectorMetadata
  funnel ([perf](catalog) fe-connector-iceberg hot-path caching + fe-core per-statement metadata funnel, with session=user cache isolation #65785).
• Two-level external-partition derived-view cache — connector-view Cache A +
  fe-core SortedPartitionRanges reuse (Cache B). ([perf](catalog) two-level cross-query cache for external partition derived views #65829)
• fe-core residual-dependency strip — remove the last iceberg / hive / hudi
  dependencies and dead code, delegate DDL validation to connectors, drop the
  hudi_meta TVF. ([refactor](catalog) fe-core: strip residual iceberg/hive/hudi deps & dead code, delegate DDL validation to connectors, remove hudi_meta TVF #65893)
• Abstraction-mismatch review — paimon time-travel crash, partition / schema
  fidelity, SPI convergence, dead-code cleanup. ([fix](catalog) catalog-SPI abstraction-mismatch review: paimon time-travel crash, partition/schema fidelity, SPI convergence & dead-code cleanup #65970)
• Cache foundation unification — one connector cache foundation, per-statement
  hot-path memoization, iceberg case-insensitive column DDL fix. ([opt](catalog) fe-connector: unify the connector cache foundation, memoize per-statement hot paths, fix iceberg case-insensitive column DDL #66029)
• Connector public-interface cleanup — adding a connector no longer requires
  editing a common module; dead SPI surface deleted; 11 correctness fixes. ([refactor](catalog) connector public interface cleanup: no common-module edit to add a connector, dead-surface deletion, 11 correctness fixes #66135)
• fe-core surface deletion — the unreachable metastore property cluster and the
  dead storage / AWS-credential surface. ([refactor](catalog) fe-core: delete the unreachable metastore property cluster and the dead storage/AWS-credential surface #66210)
• Plugin API version gate — replace the always-true apiVersion() with a
  manifest-carried API version gate, for all four plugin families. ([feature](plugin) replace the always-true connector apiVersion() with a manifest-carried API version gate for all four plugin families #66211)
• 4.1.3 upgrade-compatibility suite — assert 4.1.3-persisted catalogs survive
  the cutover, with fixtures generated by real 4.1.3 bytecode. ([test](catalog) verify 4.1.3 catalogs survive the connector-SPI cutover, with fixtures generated by real 4.1.3 bytecode #66214)
• Developer docs + gate relocation — fe/fe-connector/README.md and
  AGENTS.md replace the branch's plan-doc/ corpus; both gates move to
  build-support/ with self-tests. ([refactor](catalog) delete unused file plan-doc and add AGENTS.md #66251)
• fe-trino-connector-common extracted so fe-common no longer depends on
  Trino.
• iceberg $position_deletes system table served through the SPI.
• Per-user REST session routing + neutral delegated-credential DTO and
  capability-gated credential injection (port of upstream [feature](fe) Support OIDC session credentials for Iceberg REST catalog #63068).

Connector status

Shared modules added for them (see the README's module map): fe-connector-metastore-api
/ -spi / -hms / -iceberg / -paimon (so a connector never hand-parses metastore
endpoint properties), fe-connector-cache, fe-connector-hms-hive-shade,
fe-connector-paimon-hive-shade, plus fe-kerberos and fe-trino-connector-common
outside this directory.

Upstream re-ports

The branch is rebased onto master continuously, and every upstream change to an
already-migrated datasource is re-ported onto the SPI rather than merged textually
(a textual merge would silently resurrect fe-core-side code paths that no longer run).
Re-ported so far: #63068, #64966, #65094, #65262, #65329, #65332, #65365, #65548,
#65676, #65782, #65784, #65834, #65870, #65904, #65955, #65984, #65991, #66004, #66008,
#66112.

User-visible changes at merge time

• External catalogs are served by connector plugins, shipped as zips under
  fe/plugins/connector/ (Config.connector_plugin_root). Existing catalogs keep
  working across upgrade; no DDL change is required.
• trino_connector_plugin_dir default moves from plugins/connectors to
  plugins/trino_plugins; the old directory is still read as a fallback.
• The deprecated lakesoul catalog type is removed — creating or replaying one fails
  with Lakesoul catalog is no longer supported.

Remaining before #64304 can merge

• P8 leftover: the legacy TableType.*_EXTERNAL_TABLE enum values.
• Documentation PR: plugin layout, connector_plugin_root, the
  trino_connector_plugin_dir default, and the lakesoul removal.
• Full CI green on [refactor](catalog) decouple external catalogs from FE core into loadable connector plugins #64304 (FE UT + external_table_p0 / external_table_p2).

This is an umbrella tracking issue; individual PRs reference it for detailed
per-phase design and review.