[qzsee]

What problem does this PR solve?

Related PR: #66378, #66389, #66447, #66451

Problem Summary:

Every round, CloudTabletRebalancer rebuilds a current and a future generation of three routing indexes:

• global: be -> tabletIds
• table: tableId -> be -> tabletIds
• partition: partitionId -> indexId -> be -> tabletIds

All inner containers were ConcurrentHashMap / ConcurrentHashMap.newKeySet() holding boxed Longs. The partition and table levels create one concurrent set per (partition index, backend) and per (table, backend), so the per-container and per-element overhead dominates. With millions of routes, these indexes become one of the largest objects on the FE heap.

Test cluster and metadata

Item	Value
Compute groups / backends	3 compute groups, 5 backends (1 + 2 + 2)
Tables	192 (77 partitioned, 2 of them with 1 rollup; 115 non-partitioned)
Partitions	~36,800
Tablets per compute group	947,075
Routes per generation (all compute groups)	2,841,225
Tablets per backend	947,075 (single-backend group) / ~473,500 (two-backend groups)
Data	none, the rebalancer only depends on metadata

Table specs (empty tables created by a script):

Buckets	Partitioned tables (count × partitions)	Non-partitioned tables
16	30 × 200, 10 × 500, 6 × 1000, 4 × 2000	50
32	8 × 200, 2 × 200 (with 1 rollup), 4 × 500, 2 × 1000, 1 × 2000	30
64	3 × 200, 2 × 500, 1 × 1000	20
128	2 × 200, 1 × 500	10
256	1 × 200	5

With this metadata, a heap dump taken while the indexes were alive showed CloudTabletRebalancer retaining 839.82 MB, which is 4.6 times CloudInternalCatalog (182.16 MB) and 16 times CloudTabletInvertedIndex (52.56 MB).

Changes

This PR replaces the inner containers with primitive fastutil collections. Scheduling logic and decisions are unchanged.

1. Partition level: ConcurrentHashMap<partitionId, Long2ObjectOpenHashMap<indexId, Long2ObjectOpenHashMap<be, LongArrayList>>>. A partition index usually has only a few tablets per backend, so a list is smaller than a set and its linear removal is cheap. Containers are created with small initial capacities.

2. Table level: ConcurrentHashMap<tableId, Long2ObjectOpenHashMap<be, LongOpenHashSet>>. A large table can have many tablets on one backend, so a hash set keeps removal O(1).

3. The partition and table indexes are only read and written by the rebalancer daemon thread (statRouteInfo, balanceImpl, warmup rollback in processFailedWarmupTasks), so their inner containers are intentionally not thread-safe and are accessed without locking. The outer maps are still ConcurrentHashMap.

4. A tablet is added to the partition / table containers only when it is newly added to the global set of the same generation. Lists are not idempotent like the sets they replace, so this keeps them free of duplicates (e.g. two replicas of one tablet routed to the same backend).

5. Global level: the inner sets become LongOpenHashSet. Unlike the other two levels, these sets are copied by other threads through getSnapshotTabletsIn*ByBeId (tablet report in CloudReportHandler, warm up in CacheHotspotManager, multi-cluster sync load in StmtExecutor.syncLoadForTablets). An open-addressing removal shifts unrelated elements, so an unguarded copy could miss a tablet that is not moving, and the tablet report would then drop a tablet that is still in use. Each set is therefore guarded by its own monitor:

   • the rebalancer thread modifies a set only while holding its monitor (fillBeToTablets, updateBeToTablets); the lock is uncontended while statRouteInfo fills the sets, because they are published afterwards by the volatile field write;
   • snapshot methods copy a set only while holding its monitor, and addAll takes fastutil's primitive copy path, so the lock is held for a few milliseconds even for a backend with ~950k tablets;
   • the rebalancer thread's own reads and getTabletNumByBackendId (size() only) do not lock.

   The lock is never nested and never held together with a table or catalog lock. A snapshot is now an exact point-in-time copy; previously the weakly consistent iteration could include or miss the tablet being moved.

6. Snapshot results change from HashSet<Long> to LongOpenHashSet and are still returned as Set<Long>. Callers only use contains / removeAll.

7. The global set presize hint is no longer capped at 1 << 16 and uses the previous round's size directly. With LongOpenHashSet the table is the whole storage: under the cap, a backend with ~950k tablets regrew 4 times every round, discarding about as much memory as the final set. An over-sized hint after a catalog shrink lasts one round and is never larger than the previous round's set, which is still alive while the new one is built.

8. Balancing helpers take Map<Long, ? extends Collection<Long>>. reservoirPick / containsTablet iterate LongCollections without boxing, and pickedTabletIds is a LongOpenHashSet.

Result

Measured on the same metadata (MAT retained heap, both generations, taken while the indexes were alive; currentRoundTabletScanCount 5,682,450 vs 5,682,462):

Field	Before	After	Change
partitionToTablets + futurePartitionToTablets	299.58 MB	96.36 MB	-67.8%
beToTabletsInTable + futureBeToTabletsInTable	235.74 MB	107.36 MB	-54.5%
beToTabletsGlobal + futureBeToTabletsGlobal	232.16 MB	100.66 MB	-56.6%
Others (mostly boxed tablet ids)	72.34 MB	3.31 MB	-95.4%
CloudTabletRebalancer retained	839.82 MB	307.69 MB	-63.4%

The global sets were the last holders of boxed tablet ids, so the boxed Longs retained by the rebalancer (about 68 MB, 24 bytes each) are gone as well. The remaining 300 MB is almost entirely the long[] tables of the three levels.

Release note

Reduce FE memory used by the cloud tablet rebalancer by storing its routing indexes in primitive collections.

Check List (For Author)

• Test
  • Regression test
  • Unit Test
    • CloudTabletRebalancerTest (37 tests), including:
      • testSnapshotDoesNotMissUnmovedTabletsWhileTabletsAreMoving: one thread keeps moving tablets between two backends while another keeps taking snapshots; every unmoved tablet must be present. It fails with the locks removed.
      • testStatRouteInfoPresizesLargeGlobalTabletSetsWithoutBound
      • testFillBeToTabletsAddsDuplicateRouteOnce
      • testPickTabletPreferCold_primitiveCandidates
    • CacheHotspotManagerTest (15 tests)
  • Manual test (add detailed scripts or steps below)
    • Metadata: see "Test cluster and metadata" above.
    • Held the rebalancer thread after statRouteInfo and took jmap -dump:live with the old and new builds on the same metadata; the retained sizes are in "Result" above.
    • TODO: compare cloud_tablet_rebalancer_last_round_duration_ms before and after.
• Behavior changed:
  • No.
• Does this need documentation?
  • No.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?