Skip to content

[fix](lance) Fix Lance JNI compatibility with glibc 2.17 #67862

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yiguolei merged 2 commits into from
Sep 13, 2026
Merged

[fix](lance) Fix Lance JNI compatibility with glibc 2.17 #67862

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1bb9171
test
zhangstar333 Sep 7, 2026
19b79a7
[fix](lance) Fix Lance JNI compatibility with glibc 2.17
zhangstar333 Sep 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1000,7 +1000,7 @@ if [[ "${BUILD_FE}" -eq 1 ]]; then

#cp -r -p "${DORIS_HOME}/docs/build/help-resource.zip" "${DORIS_OUTPUT}/fe/lib"/

# Third-party filesystem jars (JuiceFS, JindoFS) are packaged by post-build.sh
# Third-party filesystem JARs and the Lance JNI library are packaged by post-build.sh
"${DORIS_HOME}/post-build.sh" --fe --output "${DORIS_OUTPUT}"

cp -r -p "${DORIS_HOME}/webroot/static" "${DORIS_OUTPUT}/fe/webroot"/
Expand Down
117 changes: 117 additions & 0 deletions docker/thirdparties/lance-jni-helpers.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

# Keep these pins in sync with lance.version in fe/pom.xml.
LANCE_JNI_VERSION="11.0.0"
Comment thread
yiguolei marked this conversation as resolved.
Resolved
LANCE_JNI_ASSET="liblance_jni-11.0.0-linux-x86_64-glibc2.17-r1.so.gz"
LANCE_JNI_URL="https://github.com/apache/doris-thirdparty/releases/download/lance-jni-11.0.0-glibc2.17-r1/${LANCE_JNI_ASSET}"
LANCE_JNI_ARCHIVE_SHA256="f9dc713269632e26c06ea2ea20637ca16ec9d76772dc20db0bead11e5cd34a6b"
LANCE_JNI_LIBRARY_SHA256="b6540edd3bdcd76b04f96a9e78fe804a2f1bffbd75eb35abe84de9091ba432ac"

lance_jni_download() (
set -eo pipefail
local cache_dir="$1"
local archive="${cache_dir}/${LANCE_JNI_ASSET}"
local work_dir url digest
local -a urls=()

mkdir -p "${cache_dir}"
if [[ -f "${archive}" ]]; then
digest=$(sha256sum "${archive}" | awk '{print $1}')
if [[ "${digest}" == "${LANCE_JNI_ARCHIVE_SHA256}" ]]; then
printf '%s\n' "${archive}"
exit 0
fi
echo "Lance JNI cache checksum mismatch; downloading again: ${archive}" >&2
fi

work_dir=$(mktemp -d "${cache_dir}/.lance-jni-download.XXXXXX")
trap 'rm -rf -- "${work_dir}"' EXIT
if [[ -n "${REPOSITORY_URL:-}" ]]; then
urls+=("${REPOSITORY_URL%/}/${LANCE_JNI_ASSET}")
fi
urls+=("${LANCE_JNI_URL}")
for url in "${urls[@]}"; do
echo "Downloading Lance JNI from ${url}" >&2
if ! curl -fL --retry 3 --retry-delay 2 --connect-timeout 10 \
-o "${work_dir}/library.gz" "${url}"; then
continue
fi
digest=$(sha256sum "${work_dir}/library.gz" | awk '{print $1}')
if [[ "${digest}" != "${LANCE_JNI_ARCHIVE_SHA256}" ]]; then
echo "ERROR: Lance JNI archive SHA256 mismatch from ${url}: expected ${LANCE_JNI_ARCHIVE_SHA256}, got ${digest}" >&2
continue
fi
# Concurrent builds only publish complete, verified cache files.
mv -f "${work_dir}/library.gz" "${archive}"
printf '%s\n' "${archive}"
exit 0
done
echo "ERROR: failed to download and verify ${LANCE_JNI_ASSET}" >&2
exit 1
)

lance_jni_replace() (
set -eo pipefail
local output_dir="$1"
local thirdparty_dir="$2"
local target_system="$3"
local target_arch="$4"
if [[ "${target_system}" != "Linux" || "${target_arch}" != "x86_64" ]]; then
exit 0
fi

local entry="nativelib/linux-x86-64/liblance_jni.so"
local target_jar="${output_dir}/fe/lib/lance-core-${LANCE_JNI_VERSION}.jar"
local archive work_dir source_hash packaged_hash
local -a lance_jars
shopt -s nullglob
lance_jars=("${output_dir}/fe/lib/"lance-core-*.jar)

@github-actions github-actions Bot Sep 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve non-clean FE builds across the version bump

build.sh --fe does not run Maven clean by default, and dependency:copy-dependencies only adds the newly resolved version to the persistent fe/fe-core/target/lib; it does not delete lance-core-9.1.0-beta.3.jar. After a developer builds the base and then this head, build.sh copies both JARs into output/fe/lib, so this exact-one check aborts the otherwise normal incremental build. Please purge obsolete Lance JARs while synchronizing the current dependency (or clean that dependency directory) before enforcing the count.

if [[ ${#lance_jars[@]} -ne 1 || "${lance_jars[0]}" != "${target_jar}" ]]; then
echo "ERROR: expected exactly one lance-core-${LANCE_JNI_VERSION}.jar in ${output_dir}/fe/lib" >&2
exit 1
fi
if ! unzip -Z1 "${target_jar}" | grep -Fx "${entry}" >/dev/null; then
echo "ERROR: missing JNI entry ${entry} in ${target_jar}" >&2
exit 1
fi

archive=$(lance_jni_download "${thirdparty_dir}/installed/lance-jni")

@github-actions github-actions Bot Sep 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Include the JNI binary's third-party license inventory

The Doris third-party release publishes RUST_THIRD_PARTY_LICENSES.html alongside this separately rebuilt native library, but this packaging path downloads only the .so.gz. The FE output later receives the repository's dist/LICENSE-dist.txt and dist/licenses, which do not identify Lance Java/JNI 11 (their only Lance Rust inventory is for an older BE lance-c component), and lance-core-11.0.0.jar itself contains no license/notice entries. The binary distribution would therefore ship the v11 Rust code without its exact third-party notices. Please add the reviewed v11 inventory under dist/licenses, reference it from the FE section of LICENSE-dist.txt, and propagate any required NOTICE material.

work_dir=$(mktemp -d "${output_dir}/fe/lib/.lance-jni.XXXXXX")
trap 'rm -rf -- "${work_dir}"' EXIT
mkdir -p "${work_dir}/$(dirname "${entry}")"
gzip -dc "${archive}" > "${work_dir}/${entry}"
source_hash=$(sha256sum "${work_dir}/${entry}" | awk '{print $1}')
if [[ "${source_hash}" != "${LANCE_JNI_LIBRARY_SHA256}" ]]; then
echo "ERROR: Lance JNI library SHA256 mismatch: expected ${LANCE_JNI_LIBRARY_SHA256}, got ${source_hash}" >&2
exit 1
fi
# Preserve the Maven cache and only replace the output JAR after verification.
cp -p "${target_jar}" "${work_dir}/lance-core.jar"
(
cd "${work_dir}"
zip -q lance-core.jar "${entry}"
)
packaged_hash=$(unzip -p "${work_dir}/lance-core.jar" "${entry}" | sha256sum | awk '{print $1}')
if [[ "${source_hash}" != "${packaged_hash}" ]]; then
echo "ERROR: JNI checksum mismatch in ${target_jar}" >&2
exit 1
fi
mv -f "${work_dir}/lance-core.jar" "${target_jar}"
echo "Replaced ${target_jar}!/${entry} (SHA256: ${packaged_hash})"
)
42 changes: 42 additions & 0 deletions docs/lance-jni-packaging.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# Lance JNI packaging for Linux x86_64

`bash build.sh --fe` automatically replaces the Linux x86_64 JNI entry in
`lance-core-11.0.0.jar` with the glibc 2.17 rebuild. No local Lance checkout,
Rust toolchain, or `LANCE_JNI_SO` variable is required on the Doris build host.

The artifact is pinned in `docker/thirdparties/lance-jni-helpers.sh` and published
in [apache/doris-thirdparty](https://github.com/apache/doris-thirdparty/releases/tag/lance-jni-11.0.0-glibc2.17-r1).
Both the compressed archive and the extracted library have fixed SHA256 values.

The archive is cached under:

```text
thirdparty/installed/lance-jni/liblance_jni-11.0.0-linux-x86_64-glibc2.17-r1.so.gz
```

With a custom `DORIS_THIRDPARTY`, the cache is under that directory's
`installed/lance-jni/`. A valid cache is reused without network access. A corrupt
cache is downloaded again. The existing `REPOSITORY_URL` mirror is tried first
when configured, followed by the pinned GitHub Release URL. Downloads use curl;
packaging also needs gzip, zip, unzip, and sha256sum.

To update an existing FE output without compiling Doris:

```bash
bash post-build.sh --fe
# Or use a custom output directory:
bash post-build.sh --fe --output /path/to/output
```

Only the JAR entry `nativelib/linux-x86-64/liblance_jni.so` is changed. The Maven
cache, Java classes, and other native entries are preserved. Download, checksum,
or JAR version failures stop packaging and leave the original output JAR intact.
ARM64, macOS, and BE-only packaging do not download or replace this library.

The rebuild retains Lance's Haswell CPU baseline (AVX2/FMA/F16C). Its GLIBC symbol
requirements were checked statically; a full CentOS 7 FE runtime test is still
required. An already running FE must restart to load a newly packaged JNI library.

When upgrading Lance, update the release URL, filename, version, and both hashes
in the helper together with `lance.version` in `fe/pom.xml`. Offline packaging
tests are available via `python3 thirdparty/test/lance-jni-packaging-test.py`.

@github-actions github-actions Bot Sep 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Add and run the packaging test referenced here

This file is absent at the PR head, so the documented command exits with No such file or directory. The Thirdparty Script Test workflow is also skipped because its path filter does not include docker/thirdparties/**, post-build.sh, or the other changed packaging paths. Please add the promised offline test (including the stale/multiple-JAR upgrade case) and wire these paths into its CI trigger so this mandatory download/repack step is actually exercised.

2 changes: 1 addition & 1 deletion fe/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -349,7 +349,7 @@ under the License.
<!-- Please modify iceberg.version and avro.version together,
you can find avro version info in iceberg mvn repository -->
<iceberg.version>1.11.0</iceberg.version>
<lance.version>9.1.0-beta.3</lance.version>
<lance.version>11.0.0</lance.version>
<substrait.version>0.40.0</substrait.version>
<!-- 0.56.1 has bug that "SplitMode" in query response may not be set-->
<maxcompute.version>0.53.2-public</maxcompute.version>
Expand Down
10 changes: 8 additions & 2 deletions post-build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@

##############################################################
# Post-build script for packaging third-party filesystem JARs
# (JuiceFS, JindoFS) into Doris FE/BE output directories.
# (JuiceFS, JindoFS) and the Lance JNI library into Doris output directories.
#
# This script is independent of the main build and can be
# executed standalone or called from build.sh.
Expand Down Expand Up @@ -47,6 +47,7 @@ TARGET_ARCH="${TARGET_ARCH:-$(uname -m)}"
# --- Source helper scripts ---
. "${DORIS_HOME}/docker/thirdparties/juicefs-helpers.sh"
. "${DORIS_HOME}/docker/thirdparties/jindofs-helpers.sh"
. "${DORIS_HOME}/docker/thirdparties/lance-jni-helpers.sh"

# --- JuiceFS wrapper functions (migrated from build.sh) ---

Expand Down Expand Up @@ -133,7 +134,8 @@ usage() {
Usage: $0 [--fe] [--be] [--output <dir>] [--juicefs] [--jindofs]

Package third-party filesystem JARs (JuiceFS, JindoFS) into Doris output directories.
By default, no third-party JARs are packaged. Use --juicefs/--jindofs or environment variables to opt in.
Linux x86_64 FE packaging automatically installs the glibc 2.17 Lance JNI library.
Use --juicefs/--jindofs or environment variables to opt into filesystem JAR packaging.

Options:
--fe Process FE output directory
Expand Down Expand Up @@ -222,6 +224,10 @@ for target_type in fe be; do
continue
fi

if [[ "${target_type}" == "fe" ]]; then
lance_jni_replace "${OUTPUT_DIR}" "${DORIS_THIRDPARTY}" "${TARGET_SYSTEM}" "${TARGET_ARCH}"
fi

if [[ "${BUILD_JINDOFS}" == "ON" ]]; then
install_jindofs "${target_type}" "${OUTPUT_DIR}"
fi
Expand Down
Loading