[Bug] HMS incremental events have multiple metadata cache consistency gaps #65750

Status after connector SPI merge (#64304)

Updated against master after connector SPI PR #64304 was merged and PR #65126 was rebased on top of it.

No top-level issue is closed by this update because items 3, 5, 6, and 7 remain open.

Search before asking

I searched existing Apache Doris issues and did not find an issue covering these HMS incremental-event cache consistency problems.

Version

Current status verified on Apache Doris master at af6dcff905114d7e05a6101496f5a629d8831b64, after connector SPI PR #64304 was merged. The original report was collected at 3a75387e61388bd886eeef38b794d5ed4eb298bf.

The remaining open items are master-baseline issues and are not introduced by #65126. Item 1 is fixed by #65126 pending merge; items 2 and 4 were fixed or eliminated by #64304.

What's Wrong?

The original report identified multiple independent correctness gaps in HMS incremental event handling. The status table above reflects the current code after #64304.

1. ALTER TABLE/ALTER DATABASE rename can be incorrectly cancelled by a load-through "local existence" check

Status: Fixed by the current revision of #65126 (pending merge).

AlterTableEvent.processRename() calls externalTableExistInLocal() before applying the rename:

However, HMSExternalCatalog.tableExistInLocal() eventually calls the normal load-through table lookup rather than a cache-only lookup:

After HMS has renamed old_table to new_table, querying new_table from this check can load it from HMS and return true. The event is then cancelled, and cached state for old_table may remain. AlterDatabaseEvent.processRename() has the same structural problem through catalog.getDbNullable(dbAfter.getName()), although database rename is less commonly reachable in standard Hive.

The current revision of #65126 removes the load-through target-existence checks and makes both rename handlers converge local state by unregistering the old identity and registering the new identity. Its focused FE unit tests cover cold and already-hot rename targets.

2. ADD/DROP PARTITION events do not fence an in-flight partition-values load, and DROP may skip lower-level cache invalidation

Current status: Fixed structurally by #64304. The details below describe the former legacy implementation.

Both addPartitionsCache() and dropPartitionsCache() return immediately when the partition-values entry is absent:

This causes two problems:

• If a partition-values load started before the event, the event does not invalidate/bump the entry, so the pre-event snapshot may still be published afterward.
• If the partition-values entry has been evicted while partition metadata or file listings remain cached, a DROP PARTITION event returns before invalidating those lower-level entries.

The second case can expose stale data when a partition name is later recreated with a different path or files. It also prevents the cache-miss fallback added by #65334 from being reached through this DROP event path.

3. Incremental CREATE_DATABASE/CREATE_TABLE events can bypass include/exclude visibility filters

Full names loading applies include_database_list, exclude_database_list, and include_table_list:

The incremental register paths directly update the legacy metadata cache without applying the same visibility policy:

When a names cache is already hot, an HMS CREATE event can therefore add a database/table that should remain hidden. The object can subsequently become queryable through the cached name.

4. DROP_PARTITION records a DATABASE deletion in ExternalMetaIdMgr

Current status: Obsolete/fixed by #64304. The legacy per-event ID-mapping payload path no longer exists.

DropPartitionEvent.transferToMetaIdMappings() uses META_OBJECT_TYPE_DATABASE instead of META_OBJECT_TYPE_PARTITION:

ExternalMetaIdMgr interprets this as a request to remove the entire database mapping subtree:

Current production use of the ExternalMetaIdMgr lookup APIs is limited, so the immediate query impact appears limited, but the persisted/replayed external metadata ID state is incorrect.

5. ALTER_DATABASE rename can generate an ID from the remote name instead of the canonical local name

Status: Open. This is present on the master baseline and is not introduced by #65126.

CatalogMgr.registerExternalDatabaseFromEvent() generates the event-created database ID directly from the incoming remote dbName:

Normal external-database object loading instead generates the ID from the canonical localDbName:

When lower_case_database_names = 1, or more generally whenever the incoming remote name differs from its canonical local cache key, the same database can therefore have two deterministic IDs:

• Immediately after the event: ID(remote name) -> canonical local name.
• After cache reset or normal reload: ID(canonical local name) -> canonical local name.

This makes the database ID and the local ID-to-name index unstable across refresh/reload and can break ID-based lookup consistency.

What You Expected?

• Rename events should inspect only locally published cache state and should always remove the old local name. This is fixed by the current revision of [refactor](meta cache) Refactor external metadata cache with MetaCacheEntry #65126.
• Partition events should prevent pre-event snapshots from being published and should invalidate partition/file caches even when partition-values cache state is cold.
• Incremental create events should apply the same visibility rules as full names loading.
• DROP_PARTITION should delete only the corresponding partition metadata-ID mapping.
• Incremental database registration should derive the database ID from the same canonical local name used by normal object loading.
• Incremental database registration should also publish names/object/ID navigation state under that same canonical local cache key.

How to Reproduce?

Rename cancellation (fixed by the current revision of #65126)

1. Enable HMS incremental event synchronization.
2. Ensure the target name is not locally cached.
3. Rename old_table to new_table in Hive.
4. Process the ALTER_TABLE event.
5. On the affected baseline, the existence check loads new_table from HMS and cancels the event; cached old_table state may remain.

Partition cache

1. Cache partition metadata/file listings for p=1.
2. Evict or invalidate only the table's partition-values entry, or block an in-flight partition-values load.
3. Drop p=1 through HMS and process the DROP_PARTITION event.
4. Recreate p=1 with a different location/files.
5. The old partition metadata or file listing may still be reused.

Visibility filter

1. Create an HMS catalog with an include/exclude database filter or include_table_list.
2. Warm the corresponding names cache.
3. Create a filtered-out database/table directly in Hive.
4. Process the HMS CREATE event.
5. Observe that the filtered object is inserted into the cached names and becomes visible/queryable.

External metadata ID mapping

1. Add a database/table/partition mapping to ExternalMetaIdMgr.
2. Process the partition's DROP_PARTITION mapping.
3. Observe that the database mapping subtree is removed instead of only the partition mapping.

Database ID stability

1. Configure an HMS catalog with lower_case_database_names = 1.
2. Process an ALTER_DATABASE rename event whose new remote name contains uppercase characters.
3. Observe that registerExternalDatabaseFromEvent() generates the new ID from the remote mixed-case name while the local cache key is lowercase.
4. Reset or reload the catalog so the normal database-object loader runs.
5. Observe that the reloaded database ID is generated from the lowercase local name and differs from the event-created ID.

Database local-key publication

1. Configure an HMS catalog with lower_case_database_names = 1.
2. Process a CREATE_DATABASE or ALTER_DATABASE rename event whose remote name contains uppercase characters.
3. Observe that the event-created database is published under the raw-case local key returned by buildDbForInit(remoteDbName, null, ...) rather than the canonical lowercase local key.
4. Perform a normal database lookup that first canonicalizes the name to lowercase.
5. Observe that the event-published database state no longer follows the same local-key convention as ordinary lookup and reload paths.

Suggested Fixes

1. Fixed in the current revision of [refactor](meta cache) Refactor external metadata cache with MetaCacheEntry #65126: remove the load-through rename target-existence checks and always converge the old/new local identities.
2. For ADD/DROP PARTITION, invalidate or bump the partition-values key when it is cold/loading. DROP should unconditionally invalidate partition metadata/file cache entries for each dropped partition before optionally updating a hot partition-values snapshot.
3. Reuse a shared database/table visibility predicate in both full names loading and incremental registration.
4. Change the DROP_PARTITION mapping type to META_OBJECT_TYPE_PARTITION and add an event mapping unit test.
5. Resolve the canonical local database name with a cache-only helper, and use it consistently for event registration, invalidation, and Util.genIdByName(). Add mode-1 coverage that asserts the event-created ID equals the normal reload ID.
6. Reuse that same canonical local database name when publishing the event-created database into the names snapshot, object cache, and dbIdToName map. Add mode-1 coverage that asserts event-created cache keys match the normal lookup/reload convention.

The remaining four open fixes can be implemented as separate PRs if preferred.

6. Incremental database event registration can publish a non-canonical local cache key

Status: Open. This is present on the master baseline and is not introduced by #65126.

HMSExternalCatalog.registerDatabase() builds the event-created database object through buildDbForInit(remoteDbName, null, ...):

However, ExternalCatalog.buildDbForInit() only applies fromRemoteDatabaseName(remoteDbName) when localDbName == null and does not also canonicalize the local cache key under lower_case_database_names = 1:

Normal database lookup does canonicalize the local key first:

When an HMS CREATE_DATABASE or ALTER_DATABASE rename event carries a mixed-case remote name while lower_case_database_names = 1, the event path may therefore publish a raw-case local database name into the database names snapshot, the database object cache, and the dbIdToName navigation map. Later normal mode-1 lookups will canonicalize the lookup name to lowercase first, so the event-published state no longer follows the same local-key convention as ordinary database lookup and reload paths.

This problem is closely related to item 5, but it is not identical:

• item 5 is about generating an unstable database ID from the remote name instead of the canonical local name;
• this item is about publishing the event-created database state under a non-canonical local cache key.

Both should be fixed by resolving the canonical local database name first and then using that same name consistently for event registration, cache publication, invalidation, and ID generation.

7. Database/table CREATE, DROP, and RENAME events do not invalidate connector-owned caches

Status: Open. This is present after connector SPI PR #64304 and is not introduced by #65126.

After #64304, HMS table metadata, partition metadata, column statistics, derived partition views, and file listings can be owned by the connector. The event driver currently applies database/table REGISTER, UNREGISTER, and RENAME descriptors only through FE CatalogMgr mutations. Those mutations update FE names/object/ID state and engine caches, but they do not call Connector.invalidateTable or Connector.invalidateDb.

REFRESH TABLE, REFRESH DATABASE, direct connector DDL/replay, and partition events already invalidate connector caches. Database/table incremental events are the remaining asymmetric path.

One concrete failure sequence is:

1. Query an HMS table and warm connector-side table/schema caches.
2. Drop the table remotely.
3. Recreate a table with the same database/table name but a different schema.
4. Process the DROP and CREATE events.
5. FE state converges to the recreated object, but the connector may continue serving the old table metadata until TTL or an explicit refresh.

A same-name ALTER VIEW recreate has the same risk because it is represented as an unregister/register table descriptor.

Suggested fix for item 7

• Invalidate the connector source key before UNREGISTER.
• Invalidate the connector target key before REGISTER.
• Invalidate both source and target for database/table RENAME.
• Use remote names directly from MetastoreChangeDescriptor and do not perform remote lookup or catalog warming.
• Add tests for same-name DROP+CREATE, same-name view recreate, and cross-database rename.

This item may be fixed in a follow-up PR because it is a #64304 master-baseline gap rather than a regression introduced by #65126. Until then, #65126 should describe its rename guarantee as FE metadata-cache convergence rather than complete connector-cache convergence.