Skip to content

[fix](fe) Upgrade gRPC and remove unused LZ4 dependencies - #67585

Merged
CalvinKirs merged 3 commits into
apache:masterfrom
CalvinKirs:dep-0707
Sep 8, 2026
Merged

[fix](fe) Upgrade gRPC and remove unused LZ4 dependencies#67585
CalvinKirs merged 3 commits into
apache:masterfrom
CalvinKirs:dep-0707

Conversation

@CalvinKirs

@CalvinKirs CalvinKirs commented Sep 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

What problem does this PR solve?

Related PR: #67065

Problem Summary:

Upgrade the managed gRPC BOM from 1.65.1 to 1.75.0, matching #67065 and replacing the grpc-netty-shaded version affected by CVE-2025-55163.

Exclude org.lz4:lz4-pure-java centrally from the managed odps-sdk-core and odps-sdk-table-api dependencies. Neither MaxCompute connector needs a replacement Java LZ4 dependency: the FE uses the ODPS metadata/session APIs, and the BE uses Table API Arrow readers/writers with ZSTD. The legacy SDK Tunnel LZ4 implementations are outside the active Doris data path. The BE connector retains its local Jackson exclusions.

Limit flight-sql-jdbc-driver to test scope. Only the FE test client loads the JDBC driver; production code uses the retained FlightServer and FlightSqlClient protocol libraries. The regression framework keeps its independent JDBC driver dependency.

Add a ZSTD round-trip test through the ODPS Arrow writer/reader factories, with the Java 17 direct-buffer access options required by the test JVM. This verifies 1,024 rows including a null after removing LZ4, alongside the existing isolated ODPS class-loading test.

Release note

Upgrade gRPC to 1.75.0. Stop bundling the unused Java LZ4 dependency in MaxCompute connectors and the test-only Flight SQL JDBC driver in the FE runtime distribution.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (details below)

    Validation of the final implementation in d9613e32c86:

    • EXTRA_FE_MODULES=maxcompute=be-java-extensions/max-compute-connector ./run-fe-ut.sh --run 'org.apache.doris.connector.maxcompute.*Test,org.apache.doris.maxcompute.*Test,!org.apache.doris.connector.maxcompute.OdpsLiveConnectivityTest': 146 FE and 14 BE connector tests passed, including ODPS class-loader isolation and the new ZSTD round trip. The successful test runtime classpaths were checked and contain no net.jpountz.lz4 classes.
    • build.sh --fe passed with DISABLE_BE_JAVA_EXTENSIONS=OFF, FE_MAVEN_THREADS=4, MVN_OPT=-Dmaven.build.cache.enabled=false, and --be-extension-ignore iceberg-metadata-scanner,hadoop-hudi-scanner,java-common,java-udf,jdbc-scanner,paimon-scanner,trino-connector-scanner,preload-extensions,hadoop-deps,java-writer. All 64 reactor modules passed, including Checkstyle, FE compilation and FE/BE MaxCompute packaging. Maven wall time: 2m 31s.
    • Both connector dependency trees contain no Java LZ4 artifacts. The rebuilt FE plugin zip and BE connector jar contain no Java LZ4 jars/classes. The incremental output directory retained the previous build's LZ4 jar; that stale artifact was removed and the runtime plugin directory was rechecked.
    • output/fe/lib contains grpc-netty-shaded-1.75.0.jar; no Flight SQL JDBC driver jar is present under output/fe.
    • git diff --check passed. Live MaxCompute connectivity, cluster startup and regression tests were not run.

  • Behavior changed:

    • No.
    • Yes. Dependency versions and runtime packaging change; MaxCompute compression selection is unchanged.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@CalvinKirs
[fix](fe) Upgrade gRPC and replace ODPS LZ4 dependencies
1dc2da6 
### What problem does this PR solve?

Related PR: apache#67065

Problem Summary: gRPC 1.65.1 brings grpc-netty-shaded affected by
CVE-2025-55163. Upgrade the managed gRPC BOM to 1.75.0, matching the
master PR. Both MaxCompute connectors directly exclude org.lz4:lz4-pure-java
from each ODPS SDK dependency and explicitly depend on parent-managed
at.yawk.lz4:lz4-java 1.11.1, preserving the SDK dependency closure without
relying on dependencyManagement exclusion merging. ODPS ownership has
moved from fe-common/fe-core to fe-connector-maxcompute in this branch.
Current MaxCompute reads and writes select ZSTD; the LZ4 replacement
preserves SDK dependencies rather than changing the selected codec.
Limit flight-sql-jdbc-driver to test scope: only the FE test client uses
it, while production FlightServer and FlightSqlClient use the retained
Flight protocol dependencies.

### Release note

Upgrade gRPC to 1.75.0 and replace the legacy ODPS LZ4 dependency with
lz4-java 1.11.1. Stop bundling the test-only Flight SQL JDBC driver in FE.

### Check List (For Author)

- Test: Manual validation
    - Maven validate (including Checkstyle) passed for fe-core and both
      MaxCompute connectors with their reactor dependencies.
    - Maven dependency:tree verified gRPC 1.75.0, LZ4 1.11.1, no
      lz4-pure-java in the connector trees, and JDBC driver test scope.
    - XML assertions verified exclusions on all four actual ODPS declarations.
    - git diff --check passed.
    - ./build.sh --fe was interrupted before FE compilation: missing
      thirdparty libhdfs.a triggered thirdparty reconstruction.
      Full build, unit tests and cluster regression tests were not completed.
- Behavior changed: Yes (dependency versions and FE runtime packaging;
  MaxCompute compression selection is unchanged)
- Does this need documentation: No
@hello-stephen

hello-stephen commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@CalvinKirs

CalvinKirs commented Sep 7, 2026

Copy link
Copy Markdown
Member Author

run buildall

@CalvinKirs
[refactor](fe) Centralize ODPS LZ4 exclusions in the parent POM
ded8531 
### What problem does this PR solve?

Related PR: apache#67585

Problem Summary: Both MaxCompute connectors repeated the same LZ4 exclusion
on each ODPS SDK dependency. Manage these exclusions once in fe/pom.xml for
odps-sdk-core and odps-sdk-table-api. Keep the BE connector-specific Jackson
exclusions local and retain explicit parent-versioned LZ4 replacement
dependencies in both consuming modules. The resolved dependency trees are
unchanged.

### Release note

None

### Check List (For Author)

- Test: Manual validation
    - Maven validate, including Checkstyle, passed for both connectors and
      their reactor dependencies with the build cache disabled.
    - Compared complete Maven dependency trees before and after the change
      for fe-core and both MaxCompute connectors: byte-for-byte identical.
    - Effective POM checks verified managed LZ4 exclusions and preserved
      connector-specific Jackson exclusions.
    - git diff --check passed.
    - No additional full build or test execution: this relocation leaves
      the resolved dependency graphs unchanged from the successful FE build.
- Behavior changed: No
- Does this need documentation: No
@CalvinKirs
[fix](fe) Remove unused MaxCompute LZ4 dependencies
d9613e3 
### What problem does this PR solve?

Related PR: apache#67585

Problem Summary: The explicit at.yawk.lz4 dependencies were added to preserve
the ODPS SDK dependency closure without establishing that Doris uses the
legacy Tunnel LZ4 paths. Both connectors use the Table API, and Doris data
reads and writes select ZSTD. Remove the two unnecessary dependencies while
retaining the centrally managed exclusions of org.lz4:lz4-pure-java.

Add an ODPS Arrow ZSTD round-trip test to verify actual compression and
reading after removal, with test-only Java 17 direct-buffer access options.
The existing isolated ODPS client test and the new round-trip test execute
without net.jpountz.lz4 classes on their runtime classpaths.

### Release note

Stop bundling the unused Java LZ4 dependency in the MaxCompute connectors.

### Check List (For Author)

- Test: Unit Test and Manual test
    - run-fe-ut.sh with both MaxCompute test packages: 160 tests passed,
      including isolated ODPS class loading and the new ZSTD round trip.
      The credential-gated live connectivity test was excluded.
    - build.sh --fe with BE Java extensions enabled and unrelated extensions
      ignored passed with the build cache disabled, including Checkstyle
      and FE/BE MaxCompute packaging (64 modules, Maven wall time 02:31).
    - Both resolved connector trees and successful test classpaths contain
      no Java LZ4 implementation. FE plugin zip and BE connector jar contain
      no Java LZ4 jars/classes. Removed a stale LZ4 jar from the previous
      incremental output and rechecked the FE runtime plugin directory.
    - git diff --check passed. No live MaxCompute or cluster test was run.
- Behavior changed: Yes (remove unused connector dependencies; ZSTD
  compression selection is unchanged)
- Does this need documentation: No
@CalvinKirs CalvinKirs changed the title [fix](fe) Upgrade gRPC and replace ODPS LZ4 dependencies [fix](fe) Upgrade gRPC and remove unused LZ4 dependencies Sep 7, 2026
@CalvinKirs

CalvinKirs commented Sep 7, 2026

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

hello-stephen commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@hello-stephen

hello-stephen commented Sep 7, 2026

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16750 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d9613e32c86f73606d5caf5912a3541c05387cdb, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17562	3089	3089	3089
q2	2096	296	227	227
q3	10195	860	516	516
q4	4667	250	208	208
q5	7669	563	395	395
q6	136	111	92	92
q7	519	517	396	396
q8	9236	893	932	893
q9	3403	2398	2332	2332
q10	6520	857	687	687
q11	391	200	177	177
q12	636	265	199	199
q13	18137	1555	1150	1150
q14	162	148	139	139
q15	q16	430	390	366	366
q17	1301	939	820	820
q18	3072	2246	2238	2238
q19	1270	855	739	739
q20	379	283	201	201
q21	5661	1657	1835	1657
q22	326	265	229	229
Total cold run time: 93768 ms
Total hot run time: 16750 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3471	3403	3397	3397
q2	503	391	376	376
q3	2243	2268	2184	2184
q4	1196	1161	899	899
q5	2152	2098	2109	2098
q6	170	119	89	89
q7	1027	927	849	849
q8	1590	1411	1414	1411
q9	3156	3132	3143	3132
q10	1839	1806	1635	1635
q11	361	272	253	253
q12	446	427	347	347
q13	1460	1509	1152	1152
q14	177	179	167	167
q15	q16	399	398	352	352
q17	3534	3331	3233	3233
q18	4792	4401	4762	4401
q19	868	883	882	882
q20	1013	958	847	847
q21	3853	3125	3193	3125
q22	405	343	325	325
Total cold run time: 34655 ms
Total hot run time: 31154 ms

@hello-stephen

hello-stephen commented Sep 7, 2026

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 81427 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d9613e32c86f73606d5caf5912a3541c05387cdb, data reload: false

query5	4239	407	334	334
query6	381	133	120	120
query7	4965	430	234	234
query8	294	127	118	118
query9	8680	2868	2885	2868
query10	390	220	169	169
query11	5378	1029	913	913
query12	113	74	70	70
query13	1186	444	317	317
query14	6118	2192	2106	2106
query14_1	1980	1966	1974	1966
query15	180	121	112	112
query16	924	391	356	356
query17	836	475	380	380
query18	2321	326	240	240
query19	163	137	114	114
query20	74	72	70	70
query21	200	104	89	89
query22	5367	5326	5288	5288
query23	6756	6197	5803	5803
query23_1	6200	6152	5878	5878
query24	7245	1102	779	779
query24_1	780	782	790	782
query25	429	298	251	251
query26	1227	220	132	132
query27	2803	426	257	257
query28	4691	1522	1482	1482
query29	942	441	353	353
query30	249	155	132	132
query31	829	404	331	331
query32	133	78	77	77
query33	470	220	179	179
query34	995	830	477	477
query35	393	411	343	343
query36	572	567	529	529
query37	121	82	70	70
query38	1007	844	826	826
query39	506	504	485	485
query39_1	444	454	461	454
query40	228	101	78	78
query41	59	58	57	57
query42	76	77	77	77
query43	243	243	213	213
query44	1016	534	553	534
query45	109	113	100	100
query46	791	852	554	554
query47	758	758	698	698
query48	304	298	218	218
query49	568	255	195	195
query50	764	261	195	195
query51	8207	8087	8083	8083
query52	72	64	66	64
query53	187	195	145	145
query54	224	162	170	162
query55	79	62	59	59
query56	180	156	145	145
query57	705	668	692	668
query58	196	150	154	150
query59	1251	1219	1113	1113
query60	243	199	162	162
query61	105	115	126	115
query62	360	194	169	169
query63	180	140	142	140
query64	2679	653	607	607
query65	1579	1587	1602	1587
query66	1905	251	202	202
query67	9661	9718	9592	9592
query68	2742	1144	724	724
query69	337	212	189	189
query70	689	630	642	630
query71	245	182	171	171
query72	2267	1690	1470	1470
query73	649	564	330	330
query74	1569	1210	1114	1114
query75	1167	1091	948	948
query76	2279	769	558	558
query77	249	252	210	210
query78	3905	3621	3130	3130
query79	2846	811	595	595
query80	1618	324	275	275
query81	523	157	130	130
query82	629	120	91	91
query83	273	205	191	191
query84	309	110	86	86
query85	834	350	308	308
query86	478	183	189	183
query87	994	974	887	887
query88	3006	2109	2095	2095
query89	297	197	171	171
query90	2129	125	125	125
query91	130	122	93	93
query92	97	61	69	61
query93	2296	1112	723	723
query94	645	245	232	232
query95	532	336	237	237
query96	782	578	261	261
query97	1045	1053	1050	1050
query98	182	130	134	130
query99	423	337	307	307
Total cold run time: 178901 ms
Total hot run time: 81427 ms

@hello-stephen

hello-stephen commented Sep 7, 2026

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.63 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d9613e32c86f73606d5caf5912a3541c05387cdb, data reload: false

query1	0.00	0.01	0.00
query2	0.07	0.04	0.04
query3	0.25	0.11	0.12
query4	1.60	0.10	0.10
query5	0.18	0.15	0.16
query6	1.28	0.73	0.69
query7	0.04	0.01	0.00
query8	0.05	0.03	0.03
query9	0.29	0.21	0.21
query10	0.35	0.35	0.36
query11	0.17	0.11	0.11
query12	0.14	0.12	0.12
query13	0.30	0.31	0.30
query14	0.45	0.45	0.45
query15	0.36	0.35	0.34
query16	0.23	0.23	0.23
query17	0.67	0.72	0.67
query18	0.18	0.17	0.17
query19	1.21	1.12	1.17
query20	0.02	0.01	0.00
query21	15.45	0.16	0.12
query22	5.10	0.04	0.05
query23	16.17	0.26	0.10
query24	3.00	0.32	0.27
query25	0.11	0.04	0.03
query26	0.81	0.16	0.12
query27	0.04	0.03	0.02
query28	3.70	0.57	0.26
query29	12.44	3.21	2.57
query30	0.25	0.12	0.11
query31	2.76	0.38	0.17
query32	3.54	0.32	0.22
query33	1.43	1.45	1.40
query34	15.39	2.16	1.78
query35	1.72	1.75	1.73
query36	0.46	0.29	0.28
query37	0.06	0.04	0.04
query38	0.05	0.03	0.03
query39	0.03	0.03	0.02
query40	0.11	0.08	0.07
query41	0.07	0.03	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.02
Total cold run time: 90.59 s
Total hot run time: 14.63 s

@CalvinKirs
CalvinKirs merged commit be7c013 into apache:master Sep 8, 2026
42 of 44 checks passed
@CalvinKirs
CalvinKirs deleted the dep-0707 branch September 8, 2026 01:49
@CalvinKirs CalvinKirs mentioned this pull request Sep 16, 2026
Merged
yiguolei pushed a commit that referenced this pull request Sep 16, 2026
@CalvinKirs
branch-4.1: [fix](fe) Upgrade gRPC and remove unused LZ4 dependencies (…
8bbe4c5 
…#68060)

#67585
yiguolei pushed a commit that referenced this pull request Sep 17, 2026
@CalvinKirs @yiguolei
branch-4.1: [fix](fe) Upgrade gRPC and remove unused LZ4 dependencies (…
608f2f5 
…#68060)

#67585
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes
@924060929 924060929 924060929 approved these changes

Assignees

No one assigned

Labels

dev/3.1.x dev/4.0.x dev/4.1.5-merged

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@CalvinKirs @hello-stephen @morningman @924060929 @yiguolei