[Bug] BE UT core dumps on an ASAN SEGV at pc 0x0 inside absl seed reading #67871

Activity

1. LuciferYang commented on Sep 11, 2026

   

   LuciferYang

   on Sep 11, 2026

   ContributorAuthor

   More actions

   Some follow-up that narrows this, and rules out half of the hypothesis I posted above.

   The register dump in the report identifies the callee. rdi is the buffer and rsi is 0x100, which is the 256-byte cap absl applies right before the call (size_t to_read = std::min<size_t>(buffer_size, 256); int result = getentropy(buffer, to_read);). So the jump to address 0 is the getentropy call itself, not something further down.

   The "libc has no getentropy" branch is out. The failing build's runtime libc is /lib64/libc.so.6 with Build ID 75e396c6440f382892765e8f24e2e0fc93800163, which is glibc 2.28 out of the build image, and it exports getentropy@@GLIBC_2.25.

   What is asymmetric is how the two symbols bind in an ASAN build of doris_be_test:

   2083: 10d80f2f  5 FUNC WEAK   DEFAULT 16 getentropy          # ASAN's __interceptor_trampoline_getentropy
    868: 10e4e6b0 56 FUNC GLOBAL DEFAULT 16 getrandom           # be/src/glibc-compatibility/musl/getrandom.c
   

   musl/getrandom.c is force-linked through the glibc-compatibility-explicit OBJECT library, so Doris's definition wins. musl/getentropy.c stays in the glibc-compatibility static archive and is not pulled in, so the name is left to ASAN's interceptor. A RELEASE build of the same tree does link ours (getentropy, GLOBAL, 139 bytes), so this asymmetry is specific to the sanitizer build.

   That is not yet the whole story. I built a standalone ASAN probe in the same image with the same symbol shape, Doris's getrandom.c object plus ASAN's getentropy trampoline, calling getentropy on a spawned thread, and it returns normally on this host. So the binding shape alone is not sufficient and something else about the full link matters. I could not take it further without a full ASAN link of the tree.

   The candidate fix that follows from this is one line: move musl/getentropy.c into the glibc-compatibility-explicit OBJECT library next to musl/getrandom.c, for the reason the comment there already gives about archive scan order. Then getentropy is a strong definition in the binary and the call cannot land on 0, and the sanitizer build matches what the RELEASE build already does. I have not proven that it fixes the crash, since I cannot reproduce it.
2. Doris-Breakwater commented on Sep 11, 2026

   

   Doris-Breakwater

   on Sep 11, 2026

   Collaborator

   More actions

   Initial triage

   This is a valid, high-impact BE-UT infrastructure failure. The available evidence points away from the Remote Doris cancellation logic itself and toward ASAN/libc symbol interposition during Abseil entropy initialization. The affected scope is currently x86 Linux ASAN_UT; there is not yet evidence that a non-sanitized BE or a production deployment is affected.

   The issue currently has no labels or assignee. It should be routed to the BE build/glibc-compatibility owners and treated as a high-priority CI reliability regression because one occurrence aborts the whole test binary.

   What is verified

   • The failing worker is created in FlightRemoteDorisStream::open(), but it crashes before the in-process Flight server enters DoGet. Arrow/gRPC startup is therefore the trigger surface, not proof of a Remote Doris cancellation race.
   • Doris uses Abseil 20250512.1. On glibc headers >= 2.25, that Abseil version's ReadSeedMaterialFromOSEntropy() calls getentropy(buffer, min(remaining, 256)) before falling back to /dev/urandom.
   • The failing build used Clang 20.1.8 with -fsanitize=address and GLIBC_COMPATIBILITY=ON.
   • The crash registers are consistent with the exact Abseil call above: rdi is a buffer and rsi == 0x100 (256), followed by a jump to PC 0.
   • Clang 20's static ASAN runtime supplies weak getentropy and getrandom interceptor trampolines. Its getentropy interceptor unconditionally calls the function pointer stored in __interception::real_getentropy.
   • Doris puts musl/getrandom.c in glibc-compatibility-explicit, but musl/getentropy.c remains only in the glibc-compatibility static archive. A minimal link using the same Clang 20 toolchain and these Doris sources produced a strong Doris getrandom, but a weak ASAN-trampoline getentropy; the archive member containing Doris' getentropy was not extracted. This occurs because the ASAN archive is linked with --whole-archive before user objects/libraries and its weak definition already satisfies later getentropy references.
   • PR [bugfix](glibc) fix glibc compatible bugs #67701 did not change getrandom.c, getentropy.c, their explicit/archive membership, or their relative link order. It added forced undefined references for math symbols and expanded lance_symbol_versions.c. Its timing is worth retaining as a correlation, but the current source diff does not establish it as the entropy crash's cause.

   Leading root-cause hypothesis

   The issue's interposition hypothesis is directionally strong, but the likely null target is real_getentropy rather than real_getrandom:

   1. Abseil calls the ASAN-provided weak getentropy trampoline.
   2. At ASAN startup, compiler-rt resolves the real function with dlsym(RTLD_NEXT, "getentropy").
   3. If the runtime libc does not export getentropy (for example, glibc older than 2.25), compiler-rt's fallback finds its own trampoline through RTLD_DEFAULT, rejects that self-reference, and leaves real_getentropy == nullptr.
   4. The first Abseil entropy request then calls through that null pointer, yielding PC 0 with (buffer, 256) still in the argument registers.

   This explains the observed signature and why Doris' explicit getrandom object does not by itself protect this path. It is still a hypothesis until the failing binary and its runtime libc are inspected. The reported nondeterminism may come from heterogeneous agent runtimes or differing final symbol resolution; it should not be attributed to a code race without that evidence.

   Information needed to confirm it

   Please preserve one failing doris_be_test binary (or rerun on the same image) and collect:

   getconf GNU_LIBC_VERSION
   readelf -WsW be/ut_build_ASAN/test/doris_be_test | grep -E 'getentropy|getrandom'
   nm -an be/ut_build_ASAN/test/doris_be_test | grep -E '(___interceptor_)?getentropy|(___interceptor_)?getrandom|real_getentropy|real_getrandom'
   ASAN_OPTIONS=verbosity=1 be/ut_build_ASAN/test/doris_be_test \
     --gtest_filter=RemoteDorisV2ReaderTest.RuntimeCancellationInterruptsBlockedFlightDoGet

   For a core under gdb, also inspect:

   p/x __interception::real_getentropy
   p/x __interception::real_getrandom
   info address getentropy
   info address getrandom

   The decisive result would be a weak getentropy resolving to the ASAN trampoline together with real_getentropy == 0, ideally accompanied by AddressSanitizer: failed to intercept 'getentropy'. Please collect the same symbol/glibc information from one passing agent as a comparison. The issue already contains sufficient test steps and representative logs; a Doris Profile is not relevant to this failure.

   Recommended next steps

   1. Run the symbol/glibc checks above on one failing and one passing agent before changing code.
   2. If confirmed, make Doris' compatibility getentropy definition unavoidable in the final link (for example, move musl/getentropy.c into the explicit object target or force its archive extraction), while retaining the existing syscall-backed getrandom implementation. Verify the final symbol table, not only a successful link.
   3. Add a link-level regression check requiring the intended strong getentropy/getrandom definitions in ASAN_UT, plus repeated execution of the focused test on the oldest supported glibc runtime and on a current runtime. A runtime-only test on modern glibc would miss this failure mode.
   4. Treat CI observability separately: emit the active gtest name as a TeamCity service message or otherwise persist it before test execution, because gtest XML is not completed when the process aborts.

   Breakwater-GitHub-Analysis-Slot: slot_d1b38fd4389f
3. LuciferYang commented on Sep 11, 2026

   

   LuciferYang

   on Sep 11, 2026

   ContributorAuthor

   More actions

   Reproduced outside CI, and the mechanism in the triage above is confirmed. One correction to my earlier comment first.

   Correction: my claim that the failing build's runtime libc is glibc 2.28 and exports getentropy was wrong. That Build ID came from build 1040773, which is a different crash (the Iceberg null dereference). The logs of the builds that carry this signature contain no libc Build ID at all, because ASAN could not unwind past frame 1. So the "old libc" branch was never ruled out, and it is in fact the trigger.

   What sets the date: #67752 reverted the compilation image from almalinux:8 back to centos:7, merged 2026-09-10 10:32 +0800. almalinux:8 carries glibc 2.28, which exports getentropy@@GLIBC_2.25; centos:7 carries glibc 2.17, which does not export it at all. The BE UT core dumps start that day (26 of 76 builds) and reach 51 of 63 the next, which is what a staged image refresh across agents looks like, and it also explains why the same test passes on some agents minutes apart.

   Reproduction, with no Doris build involved. A 25-line program that calls getentropy(buf, 32) on the main thread and on a spawned thread, compiled with the repo's own clang 20.1.7 and -fsanitize=address, linked four ways, then run under glibc 2.17 (centos:7, x86_64) and glibc 2.28 (the current build image):

   link	getentropy in .dynsym	glibc 2.17	glibc 2.28
   ASAN only	WEAK, 5 bytes, ASAN trampoline	SEGV at pc 0	ok
   plus Doris getrandom.c.o (BE UT today, forced object)	WEAK, 5 bytes, ASAN trampoline	SEGV at pc 0	ok
   plus libglibc-compatibility.a on the link line	WEAK, 5 bytes, ASAN trampoline	SEGV at pc 0	ok
   plus Doris getentropy.c.o forced in	GLOBAL, 637 bytes, Doris	ok	ok

   The crash is the same shape as CI's, including the register pattern, rdi a buffer and rsi the requested length, and #1 being the caller of getentropy rather than an interceptor frame:

   ==1==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x000000000000 ...)
       #0 0x000000000000  (<unknown module>)
       #1 0x5555556a5ef4  (/ent/t_archive+0x151ef4)
   rdi = 0x00007bfffcbde020  rsi = 0x0000000000000020
   

   With ASAN_OPTIONS=verbosity=1, glibc 2.17 prints AddressSanitizer: failed to intercept 'getentropy' on every one of these binaries, so real_getentropy is null in all four. Only the last one survives, because there the call binds to Doris' own definition and never reaches the interceptor.

   Two things follow for the fix. Forcing getrandom alone does not protect this path, which the second row shows, so the existing explicit object is not enough. Putting libglibc-compatibility.a on the link line is also not enough, which the third row shows: ASAN's weak definition already satisfies the reference by the time the archive is scanned, so the member is never extracted. What works is making Doris' getentropy unavoidable, that is moving musl/getentropy.c into glibc-compatibility-explicit next to musl/getrandom.c, keeping the existing HAVE_SYS_RANDOM_H guard.

   I can put up that change with the symbol table check the triage asks for. The reason to also keep the CentOS 7 runtime in the loop for any regression test is visible in the table: every variant passes on glibc 2.28, so a test that only runs on a modern runtime proves nothing here.
4. LuciferYang commented on Sep 11, 2026

   

   LuciferYang

   on Sep 11, 2026

   ContributorAuthor

   More actions

   Opened #67873 with the one-line move plus the symbol table checks. BE UT on that PR is the first end-to-end check, and since the crash is intermittent per agent one green run will not settle it; I will report what several runs look like.
5. LuciferYang commented on Sep 11, 2026

   

   LuciferYang

   on Sep 11, 2026

   ContributorAuthor

   More actions

   Fixed by #67848, merged as 859a19c, which moves musl/getentropy.c into glibc-compatibility-explicit. That is the same change the reproduction above pointed at, arrived at independently, so I closed my own #67873 as a duplicate.

   Two notes for anyone who lands here from a red BE UT.

   A build whose head predates 859a19c will still core dump, since BE UT builds the pull request head rather than a merge with master. Rebasing or merging master in is what clears it.

   The other two crashers this issue explicitly did not cover are still open as far as I know: Check failed: _seg_compact_mem_tracker != nullptr in SegcompactionWorker::~SegcompactionWorker() during DeltaWriterCancelTest (seen in build 1041858), and a null dereference in IcebergTableReader::_read_equality_delete_file (build 1040773). Both abort the whole test binary the same way, so a red BE UT with no test name attached is not automatically this issue.

   Closing.
6. LuciferYang

   closed this as completedon Sep 11, 2026