Repository navigation
Conversation
### Background The BE service registers a file cache factory in `ExecEnv` at startup. BE writer unit tests also construct S3 writers and packed-file managers, but run without starting the service. ### What problem does this PR solve? Issue Number: None Related PR: apache#68797 (enabled file caching by default), apache#69605 (initial S3 fixture fix), apache#69670 (overlapping S3 client teardown) PR Type: Test Problem Summary: S3, Hive report-writer and packed-file-manager unit tests can crash when run alone with file caching enabled. A downloader or cloud-tablet test can mask the crash by leaving caching disabled. Give the writer fixtures a populated cache for the duration of the test and restore the factory and its enable flag afterward. ### Details - **Failure:** Isolated writers dereference a missing cache factory, either while constructing an S3 writer or while uploading a packed file. In a shuffled run, the S3 buffer-boundary test also removed the suite's mock callbacks, which made later tests send requests to S3. - **Cause:** The writer fixtures relied on process-wide state set by other tests. The boundary test used its own in-memory client but also cleared all registered callbacks and left a changed buffer size behind. - **Stability evidence:** The six affected suites pass together in three shuffled runs (seeds 7301, 7302 and 7303). The new scope tests exercise a missing factory, an empty factory, reuse of a populated factory, and restoration with either initial value of the enable flag. ### Testing - Reproduced three ASAN segmentation faults with the fixtures from `315c2e0033`. `S3FileWriterTest.multi_part_io_error` and `VHivePartitionWriterReportLifecycleTest.PeriodicReportDefersMetadataAndFinalReportTransfersUploadIdentity` fail during writer construction. `PackedFileManagerTest.ProcessUploadingFilesSetsFailedWhenMetaUpdateFails` fails during the cache write. All three pass in separate processes with this change. The Azure Hive case and three additional packed-upload cases also pass in separate processes (7/7). - Ran all six suites locally with the BE unit-test entry point under ASAN. The default run passed 68/68. Setting `GTEST_SHUFFLE=1 GTEST_REPEAT=3 GTEST_RANDOM_SEED=7301` passed 68/68 in each iteration and the process exited with status 0. ```sh ./run-be-ut.sh --run --filter='ScopedFileCacheTest.*:S3FileWriterTest.*:VHivePartitionWriterReportLifecycleTest.*:PackedFileManagerTest.*:FileCacheBlockDownloaderTest.*:CloudTabletSyncMetaTest.*' ``` Clang-format 16, clang-tidy on the changed lines, and the BE build-hygiene checks passed. ### Notes for Reviewers The scope reuses an existing populated factory. Otherwise it creates a 16 MiB memory cache. The S3 and packed-file fixtures drain pending uploads before releasing the cache. The S3 fixture also releases its filesystem before AWS SDK shutdown. ### Release note None ### Check List (For Author) - Test: BE unit tests under ASAN. - Behavior changed: No - Does this need documentation: No - Pre-submission status: Ready ### AI Disclosure - Tool: OpenAI Codex (GPT-6) - Generated: Implementation, unit tests, and PR description. - Human review: Pending author review.
xylaaaaa
requested review from
Gabriel39,
gavinchou,
liaoxin01,
luwei16 and
morningman
as code owners
November 5, 2026 03:49
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 28446 ms |
Contributor
TPC-DS: Total hot run time: 151621 ms |
Contributor
ClickBench: Total hot run time: 24.51 s |
### What problem does this PR solve? Issue Number: None Related PR: apache#70243 Problem Summary: HdfsFileSystemTest.Write requests cache writes but does not set up a usable cache factory. With caching enabled, it crashes when run alone or when an earlier block-cache fixture leaves an empty factory. Use the shared scoped cache for this test, verify the writer installs its cache builder, and restore the original HDFS write batch size when the test finishes. ### Release note None ### Check List (For Author) - Test: Reproduced the HDFS crash alone and following the block-cache fixture. Both pass locally under ASAN with the fix. All 76 focused tests passed in the default order and in three shuffled runs. The local Linux x86_64 full build was stopped before test execution. - Behavior changed: No - Does this need documentation: No
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 28811 ms |
Contributor
TPC-DS: Total hot run time: 153833 ms |
Contributor
ClickBench: Total hot run time: 24.4 s |
### What problem does this PR solve? Issue Number: None Related PR: apache#70243 Problem Summary: The BE test binary loads the server default for file caching without initializing the server's cache factory. When a test leaves an empty factory in ExecEnv, dropping a tablet in TestDeltaWriterClusterKey reaches a modulo-by-zero in Rowset::clear_cache. Disable caching in the test runner's copy of be.conf so ordinary tests start with a consistent setting. Writer fixtures that exercise caching enable it with a populated factory for the duration of each test. ### Release note None ### Check List (For Author) - Test: ASAN unit tests on macOS. The failing DeltaWriter test passed alone and all 85 related tests passed together. Validated the generated test config with an explicit cache setting and no final newline. - Behavior changed: No - Does this need documentation: No
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 28437 ms |
Contributor
TPC-DS: Total hot run time: 153135 ms |
Contributor
ClickBench: Total hot run time: 24.41 s |
Contributor
Author
|
run cloud_p0 |
Contributor
Author
|
run p0 |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run cloud_p0 |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
1 similar comment
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
|
run cloud |
Contributor
|
run cloudut |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
The BE service registers a file cache factory in
ExecEnvat startup. The BE unit-test runner reads the server configuration without performing that startup step. Both writer construction and tablet cleanup can reach the file cache.What problem does this PR solve?
Issue Number: None
Related PR: #68797 (enabled file caching by default), #69605 (initial S3 fixture fix), #69670 (overlapping S3 client teardown)
PR Type: Test
Problem Summary: S3, Hive report-writer, HDFS and packed-file-manager unit tests can crash when run alone or after a fixture leaves an empty cache factory. A downloader or cloud-tablet test can mask the crash by leaving caching disabled. With caching enabled but no populated factory, dropping a tablet can also crash in rowset cleanup. Set the file-cache default to false in the test runner’s copy of
be.conf. Writer fixtures that exercise caching install a populated cache, explicitly enable it, and restore both the factory and the flag afterward.Details
TestDeltaWriterClusterKey.vec_sequence_colcan callRowset::clear_cache()with an empty factory while dropping the tablet. The test runner now writes its default to the copy ofbe.confused by the test binary.Testing
315c2e0033.S3FileWriterTest.multi_part_io_errorandVHivePartitionWriterReportLifecycleTest.PeriodicReportDefersMetadataAndFinalReportTransfersUploadIdentityfail during writer construction.PackedFileManagerTest.ProcessUploadingFilesSetsFailedWhenMetaUpdateFailsfails during the cache write. All three pass in separate processes with this change. The Azure Hive case and three additional packed-upload cases also pass in separate processes (7/7).HdfsFileSystemTest.Writeaborted with ASAN FPE inFileCacheFactory::get_by_path. Reproduced the failure locally both alone and followingBlockFileCacheTest.file_cache_profile_specialized_write_cache_counters. Both runs pass after giving the HDFS writer a scoped cache. The test still requests cache writes and verifies the writer installed its cache builder.HdfsFileSystemTest.Writepassed. The run later aborted inTestDeltaWriterClusterKey.vec_sequence_colwhile dropping a tablet:TabletManager::drop_tablet→Rowset::clear_cache→FileCacheFactory::get_by_path(ASAN FPE). On macOS the old runner reproduced the crash at the same cache lookup when the block-cache test ran immediately before the DeltaWriter case. With the new runner the same pair passes, and the DeltaWriter case also passes alone under ASAN. All 85 tests in a combined run also passed, covering the DeltaWriter, MemTable, Segment, cache and writer fixtures, including the original HDFS failure. Checked that the generated config disables caching even when the source config explicitly enabled it and had no final newline.GTEST_SHUFFLE=1 GTEST_REPEAT=3 GTEST_RANDOM_SEED=7301, and both processes exited with status 0../run-be-ut.sh --run --filter='ScopedFileCacheTest.*:S3FileWriterTest.*:VHivePartitionWriterReportLifecycleTest.*:PackedFileManagerTest.*:FileCacheBlockDownloaderTest.*:CloudTabletSyncMetaTest.*:HdfsFileSystemTest.*:BlockFileCacheTest.file_cache_profile_specialized_write_cache_counters'The macOS full suite stopped before the HDFS test on two failures that also reproduce in isolation:
ColumnVariantV2Test.TypedComparisonBoundariesMatchEncodedWithoutMaterializingandLocalExchangerTest.TestShuffleExchangerWrongMap. Both passed on Linux in build 1093099. The local Linux x86_64 build was stopped before running tests. The full Linux BE UT, build 1094205, passed on5d9cf757c6. GoogleTest reported 16,200 passed, 68 skipped and zero failures. The HDFS write and DeltaWriter tablet-cleanup cases both passed in the full run. The P0 regression and Cloud P0 regression also passed on the same commit. TeamCity reported one muted case in P0 and two in Cloud P0, including the STS role-configuration case in both jobs.Clang-format 16, clang-tidy on the changed lines, and the BE build-hygiene checks passed.
Notes for Reviewers
The scope reuses an existing populated factory. Otherwise it creates a 16 MiB memory cache. The S3 and packed-file fixtures drain pending uploads before releasing the cache. The S3 fixture also releases its filesystem before AWS SDK shutdown.
Release note
None
Check List (For Author)
AI Disclosure