Skip to content

[Bug] REPLACE TABLE may skip binlog on master FE when only table-level binlog is enabled #66571

New issue
New issue
Open
#66930
Open
[Bug] REPLACE TABLE may skip binlog on master FE when only table-level binlog is enabled#66571
#66930

Description

@Ryan19929
Ryan19929
opened on Aug 7, 2026

Search before asking

  • I had searched in the issues and found no similar issues.

Version

  • Source cluster initially ran Doris 3.0.8
  • FE was upgraded to Doris 3.1.4
  • The missing REPLACE_TABLE binlog was observed on Doris 3.1.4

What's Wrong?

When database-level binlog is disabled but the original table has binlog.enable=true, a swap=false REPLACE TABLE operation may not generate a REPLACE_TABLE binlog on the master FE.

The issue occurs when the current master FE has a cold BinlogConfigCache, for example after FE restart or master failover. During swap=false replacement, the master unregisters the original table from Catalog before BinlogManager.addReplaceTable() evaluates whether to write binlog. The event carries origTblId; the cold cache cannot resolve that removed table ID and returns false. The master then skips addBinlog(), so table-level CCR cannot observe the new REPLACE event through getBinlog. It only observes the previous dummy binlog.

FE may log:

fail to get table. db: <db>, table id: <origTblId>

What You Expected?

If the original table was binlog-enabled before REPLACE TABLE, the master FE should always emit the corresponding REPLACE_TABLE binlog. The decision must not depend on whether the old table remains in Catalog or whether BinlogConfigCache is warm.

How to Reproduce?

  1. Enable the global binlog feature.
  2. Disable DB-level binlog:
ALTER DATABASE db_name SET PROPERTIES ("binlog.enable" = "false");
  1. Enable binlog on the original table:
ALTER TABLE origin_table SET ("binlog.enable" = "true");
  1. Restart or fail over FE so that the current master has a cold BinlogConfigCache.
  2. Execute:
ALTER TABLE origin_table REPLACE WITH TABLE replacement_table PROPERTIES ("swap" = "false");
  1. Query binlog using the original table ID. The new REPLACE_TABLE event is absent.

Anything Else?

Related CCR tracking issue: #66265.

Upstream PR #48919 fixes the separate DB-level-enabled case by changing the single-table anyEnable assignment to OR logic. It is insufficient for db binlog=false plus table binlog=true, because the original table configuration is already unavailable after unregistering the table.

Suggested fix: capture the original table BinlogConfig before unregistering it, persist the snapshot in ReplaceTableOperationLog, and use it when emitting and replaying REPLACE_TABLE binlog.

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Activity

  1. Doris-Breakwater commented on Aug 7, 2026

    @Doris-Breakwater
    Doris-Breakwater
    on Aug 7, 2026
    Collaborator

    Breakwater-GitHub-Analysis-Slot: slot_449674ca69e5

    Triage verdict

    Confirmed as an actionable FE/binlog correctness bug by static code-path review. The report is internally consistent and the proposed configuration snapshot is directionally correct. No additional logs are required to establish the immediate root cause.

    At intake this issue is open with no labels, assignee, milestone, or prior comments.

    Verified facts

    • On 3.1.4-rc02 (7f5ba43de6c7f148e1dacdb7cb838394d2ec1537), Alter.processReplaceTable() calls replaceTableInternal() before constructing and logging ReplaceTableOperationLog (source).
    • For swap=false, replaceTableInternal() unregisters both names, registers only the replacement under the original name, and does not re-register origTable (source).
    • BinlogManager.addReplaceTable() attaches only origTblId, then the generic addBinlog() asks BinlogConfigCache.isEnableTable(dbId, origTblId) (source). On a miss, the cache resolves the ID from the live catalog; after swap=false that lookup returns no table and therefore no enabled table config (source).
    • The follower path has the same ordering: EditLog.loadJournal() first calls replayReplaceTable() and only afterward addReplaceTable() (source).
    • PR [fix](binlog) Wrong judgment of anyEnable for add binlog #48919 changes the single-table assignment to OR with the DB-level result. That prevents a missing table lookup from overriding dbBinlogEnable=true, but with DB binlog disabled the result still depends on resolving the removed original table. It does not fix this table-only case.
    • The same catalog-before-binlog ordering and cache-dependent predicate are still present on current upstream master (c1bab9e8cf80fc5e0c52914e9d4aa69717ea2e7a).

    This explains the missing REPLACE_TABLE event without assuming any CCR-side fault. A warm cache can mask the defect, so behavior is incorrectly cache-state-dependent.

    Fix requirements

    Capturing a copy of the original table's BinlogConfig while the table locks are held and before unregistering it, then persisting it in ReplaceTableOperationLog, is the appropriate source of truth. The implementation should also cover all consumers of that historical configuration:

    1. Live master emission and follower edit-log replay must decide from the persisted snapshot (plus DB-level enablement), not from a post-mutation catalog lookup.
    2. The resolved configuration must reach the DBBinlog/TableBinlog path as well, and remain available for table-level TTL/max-bytes/max-history GC after the original table leaves the catalog. Changing only the top-level anyEnable predicate is not sufficient as a robust fix.
    3. Checkpoint/image recovery needs explicit coverage. BinlogManager.write() stores table dummies before common binlogs, while read() currently filters a table dummy through isEnableTable() before it encounters the serialized REPLACE_TABLE record. Once the original table is absent, a restart can therefore fail to reconstruct that original table's TableBinlog even if the live FE emitted the event. Recovery should restore/use the persisted snapshot before filtering or reconstructing that table binlog.
    4. Older ReplaceTableOperationLog JSON has no snapshot. Deserialization and replay must remain null-safe; retain the legacy behavior where a reliable snapshot is unavailable rather than failing replay.

    Missing information

    Nothing is blocking bug acceptance. The exact FE build/commit identifier would only be useful for deciding the precise 3.1.x backport target and whether that build already carries a backport of #48919.

    Recommended next steps

    • Implement the persisted pre-mutation snapshot and the live/replay/image-recovery handling above.
    • Add FE tests for db=false, original table binlog enabled, swap=false, and a deliberately cold cache; assert that getBinlog(origTblId, ...) returns REPLACE_TABLE on both the writer and replay paths.
    • Add a BinlogManager serialize/read test with the original table already absent, plus warm-cache/cold-cache equivalence and legacy-log-without-snapshot compatibility tests.
    • Keep the separate swap=true/table-ID attachment concern tracked independently; this issue's verified failure is the swap=false, table-level-enable decision.

  2. goutamadwant commented on Aug 19, 2026

    @goutamadwant
    goutamadwant
    on Aug 19, 2026

    @Doris-Breakwater I prepared a fix that snapshots the original table's BinlogConfig before swap=false removes it, then reuses it for live emission, follower replay, table-level GC, and image recovery. It includes cold-cache, image round-trip, missing-database boundary, and legacy-log coverage. #66930 let me know if you have any suggestions. thanks!

  3. goutamadwant
    added a commit that references this issue on Sep 20, 2026

    [fix](fe) Preserve config for replace table binlogs

    4d1aea1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions