Repository navigation
[Enhancement] Include catalog and database MetaCacheEntry instances in unified cache statistics #65792
Description
Status after connector SPI merge (#64304)
This issue remains open after #64304 and its scope is now broader.
The unified FE endpoint still reads only engine caches registered in ExternalMetaCacheMgr. It does not include:
- the four direct FE MetaCacheEntry categories owned by ExternalCatalog and ExternalDatabase; or
- connector-owned MetaCacheEntry instances introduced by [refactor](catalog) decouple external catalogs from FE core into loadable connector plugins #64304, such as HMS table/partition caches and connector table/snapshot/file-listing caches.
The connector cache framework exposes local statistics, but the Connector SPI currently has no cache-statistics reporting surface through which fe-core can collect them. Therefore #64304 did not fix this issue; it added another cache ownership domain that the eventual solution should cover.
A follow-up implementation should remain cache-only and should enumerate only already initialized/built connector state. Statistics collection must not initialize a catalog, construct a connector sibling, perform remote I/O, or warm any cache.
Search before asking
I searched existing Apache Doris issues for external metadata cache statistics and ExternalMetaCacheMgr.getCatalogCacheStats. The closed tracking issue #60686 lists unified monitoring as a framework goal, but I did not find a dedicated issue covering this remaining gap.
Background
PR #65126 migrates the ExternalCatalog and ExternalDatabase name/object caches to MetaCacheEntry. Four cache entries are owned directly by the catalog hierarchy rather than by an engine cache registered in ExternalMetaCacheMgr:
- ExternalCatalog database names
- ExternalCatalog database objects
- ExternalDatabase table names
- ExternalDatabase table objects
MetaCacheEntry exposes statistics for these entries, but ExternalMetaCacheMgr.getCatalogCacheStats(catalogId) currently reports only engine caches registered in its catalog cache registry. As a result, the unified statistics endpoint omits the central catalog/database name and object caches.
Problem
Operators cannot observe capacity, estimated size, hits, misses, load success/failure, eviction, invalidation, or the last load error for these four cache categories through the unified per-catalog statistics path. This makes cache sizing and cache-consistency diagnosis incomplete even after the MetaCacheEntry migration.
Expected behavior
ExternalMetaCacheMgr.getCatalogCacheStats(catalogId), or the public statistics API built on it, should include both:
- Engine metadata caches registered in ExternalMetaCacheMgr.
- Direct MetaCacheEntry instances owned by ExternalCatalog and its ExternalDatabase objects.
Statistics collection must remain cache-only: it must not initialize a catalog, enumerate remote metadata, load a database/table object, or turn a cold cache hot merely to produce statistics.
Cache names should be stable and distinguish at least catalog database-name/object entries from database table-name/object entries. If table entries are aggregated across databases, the aggregation semantics should be documented; otherwise each database should have a stable owner-qualified cache name.
Suggested implementation
- Add a catalog-side statistics collector for the direct MetaCacheEntry instances.
- Let ExternalMetaCacheMgr merge those statistics with the registered engine-cache statistics.
- Traverse only already initialized/local catalog and database state.
- Preserve lifecycle safety for refresh, unregister, and replay paths.
Suggested tests
- An uninitialized catalog returns statistics without triggering initialization or a remote call.
- Initialized but cold entries are reported with zero size and remain cold.
- Hot database-name/object and table-name/object entries report their own counters and sizes.
- Multiple databases are represented or aggregated deterministically.
- Refresh/reset/unregister removes stale owners from the statistics result.
Activity