Skip to content

branch-3.0: [fix](arrow-flight-sql) Separate arrow-flight-sql connection and mysql connection#51294

Merged
dataroaring merged 2 commits into
apache:branch-3.0from
xinyiZzz:branch-3.0_20250527
Jun 11, 2025
Merged

branch-3.0: [fix](arrow-flight-sql) Separate arrow-flight-sql connection and mysql connection#51294
dataroaring merged 2 commits into
apache:branch-3.0from
xinyiZzz:branch-3.0_20250527

Conversation

@xinyiZzz

@xinyiZzz xinyiZzz commented May 27, 2025

Copy link
Copy Markdown
Contributor

pick #50939

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@xinyiZzz xinyiZzz requested a review from dataroaring as a code owner May 27, 2025 09:42
@Thearas

Thearas commented May 27, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@xinyiZzz

xinyiZzz commented May 27, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@xinyiZzz
[fix](arrow-flight-sql) Separate arrow-flight-sql connection and mysq…
173bd9a 
…l connection (apache#50939)

Add FE conf `arrow_flight_max_connections` as the upper limit of
arrow-flight-sql connection, arrow-flight-sql connections and mysql
connections will be managed separately.
@xinyiZzz xinyiZzz force-pushed the branch-3.0_20250527 branch from 4619d13 to 173bd9a Compare May 27, 2025 10:03
@xinyiZzz

xinyiZzz commented May 27, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot

doris-robot commented May 27, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 40100 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 173bd9a7e5aac906fd1c4ab68afdd1ca99757eb0, data reload: false

------ Round 1 ----------------------------------
q1	18031	6853	6628	6628
q2	2059	174	178	174
q3	11562	1140	1120	1120
q4	10541	805	730	730
q5	7763	2889	2876	2876
q6	216	133	132	132
q7	987	616	592	592
q8	10652	1968	2057	1968
q9	6849	6486	6455	6455
q10	7166	2314	2305	2305
q11	471	268	257	257
q12	395	205	206	205
q13	19696	3051	3063	3051
q14	251	210	210	210
q15	505	472	455	455
q16	479	375	385	375
q17	974	593	630	593
q18	7183	6590	6643	6590
q19	1496	1025	1094	1025
q20	478	212	194	194
q21	4061	3283	3131	3131
q22	1167	1090	1034	1034
Total cold run time: 112982 ms
Total hot run time: 40100 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6701	6546	6497	6497
q2	333	233	236	233
q3	2988	2908	2947	2908
q4	2042	1870	1785	1785
q5	5707	5701	5723	5701
q6	214	126	129	126
q7	2231	1812	1785	1785
q8	3339	3527	3428	3428
q9	8846	8851	8805	8805
q10	3572	3504	3521	3504
q11	595	510	512	510
q12	811	606	620	606
q13	10412	3128	3160	3128
q14	307	271	270	270
q15	503	474	459	459
q16	481	447	448	447
q17	1827	1608	1568	1568
q18	8144	7748	7691	7691
q19	1642	1488	1450	1450
q20	2082	1841	1875	1841
q21	5536	5344	5379	5344
q22	1138	1062	1053	1053
Total cold run time: 69451 ms
Total hot run time: 59139 ms

@doris-robot

doris-robot commented May 27, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 196071 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 173bd9a7e5aac906fd1c4ab68afdd1ca99757eb0, data reload: false

query1	1304	917	924	917
query2	6233	1885	1884	1884
query3	10986	4480	4384	4384
query4	65882	33344	23506	23506
query5	5003	465	457	457
query6	311	181	170	170
query7	4874	321	308	308
query8	287	227	240	227
query9	6519	2608	2563	2563
query10	420	267	270	267
query11	15669	15270	15756	15270
query12	153	103	108	103
query13	1044	451	425	425
query14	9788	7519	7031	7031
query15	209	182	173	173
query16	7183	477	460	460
query17	1050	603	592	592
query18	1868	319	326	319
query19	215	168	160	160
query20	117	107	107	107
query21	213	109	106	106
query22	4699	4423	4485	4423
query23	34406	34758	33805	33805
query24	6194	2907	3007	2907
query25	556	447	429	429
query26	658	173	173	173
query27	1730	374	373	373
query28	3783	2179	2144	2144
query29	713	454	423	423
query30	236	155	157	155
query31	990	789	811	789
query32	64	61	61	61
query33	396	284	284	284
query34	899	508	518	508
query35	849	741	716	716
query36	1119	945	950	945
query37	114	73	70	70
query38	4045	4044	3967	3967
query39	1534	1507	1448	1448
query40	203	94	101	94
query41	51	48	47	47
query42	113	99	97	97
query43	513	468	479	468
query44	1156	802	813	802
query45	186	165	163	163
query46	1105	714	707	707
query47	2017	1930	1932	1930
query48	477	409	404	404
query49	715	406	408	406
query50	844	420	424	420
query51	7436	7396	7192	7192
query52	99	89	87	87
query53	258	185	180	180
query54	553	443	454	443
query55	77	87	78	78
query56	257	273	240	240
query57	1204	1083	1091	1083
query58	220	200	205	200
query59	3112	2859	2822	2822
query60	273	252	244	244
query61	111	109	111	109
query62	769	683	699	683
query63	217	193	187	187
query64	1374	658	661	658
query65	3241	3223	3210	3210
query66	707	300	312	300
query67	15869	15460	15497	15460
query68	3740	602	577	577
query69	432	267	275	267
query70	1173	1081	1134	1081
query71	334	263	258	258
query72	6149	4013	3967	3967
query73	750	345	347	345
query74	10123	8898	8877	8877
query75	3347	2643	2669	2643
query76	1645	1069	1130	1069
query77	501	278	282	278
query78	10762	9637	9560	9560
query79	1672	597	594	594
query80	888	435	425	425
query81	499	233	233	233
query82	1306	92	91	91
query83	244	146	149	146
query84	292	84	76	76
query85	870	309	292	292
query86	344	294	273	273
query87	4402	4226	4239	4226
query88	3808	2353	2343	2343
query89	406	290	289	289
query90	2020	184	187	184
query91	188	147	157	147
query92	63	52	51	51
query93	2265	564	550	550
query94	768	294	307	294
query95	359	262	256	256
query96	615	281	285	281
query97	3311	3167	3192	3167
query98	214	195	198	195
query99	2087	1297	1273	1273
Total cold run time: 311047 ms
Total hot run time: 196071 ms

@doris-robot

doris-robot commented May 27, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 29.97 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 173bd9a7e5aac906fd1c4ab68afdd1ca99757eb0, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.06	0.06
query4	1.64	0.10	0.10
query5	0.54	0.53	0.49
query6	1.14	0.73	0.72
query7	0.02	0.01	0.01
query8	0.04	0.03	0.04
query9	0.57	0.50	0.50
query10	0.55	0.54	0.56
query11	0.16	0.10	0.10
query12	0.15	0.11	0.10
query13	0.60	0.59	0.58
query14	0.76	0.81	0.79
query15	0.85	0.82	0.82
query16	0.41	0.38	0.41
query17	0.99	1.04	1.04
query18	0.24	0.23	0.21
query19	1.86	1.79	1.83
query20	0.01	0.01	0.01
query21	15.40	0.61	0.59
query22	3.22	2.57	1.32
query23	17.00	0.96	0.82
query24	3.19	1.22	1.80
query25	0.29	0.07	0.06
query26	0.59	0.13	0.14
query27	0.06	0.05	0.04
query28	9.55	0.50	0.52
query29	12.57	3.29	3.26
query30	0.25	0.06	0.06
query31	2.84	0.39	0.39
query32	3.25	0.48	0.45
query33	2.97	2.96	2.99
query34	17.07	4.51	4.56
query35	4.55	4.57	4.54
query36	0.68	0.50	0.48
query37	0.08	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.16	0.12	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.83 s
Total hot run time: 29.97 s

dataroaring
dataroaring previously approved these changes Jun 11, 2025
View reviewed changes

@dataroaring dataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xinyiZzz

xinyiZzz commented Jun 11, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot

doris-robot commented Jun 11, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 40006 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 969a67ae7661b6a3c3ec07e8e9bdd4781b493934, data reload: false

------ Round 1 ----------------------------------
q1	17577	7004	6635	6635
q2	2076	166	163	163
q3	10615	1065	1192	1065
q4	10564	746	735	735
q5	7737	2885	2759	2759
q6	220	136	134	134
q7	985	619	613	613
q8	9364	1971	2030	1971
q9	6654	6412	6401	6401
q10	7021	2286	2329	2286
q11	448	269	252	252
q12	397	212	209	209
q13	17773	3001	3008	3001
q14	233	198	220	198
q15	496	460	472	460
q16	478	379	376	376
q17	974	644	542	542
q18	7293	6745	6706	6706
q19	1383	1041	1139	1041
q20	501	203	208	203
q21	4072	3269	3323	3269
q22	1064	987	996	987
Total cold run time: 107925 ms
Total hot run time: 40006 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6588	6580	6583	6580
q2	330	234	236	234
q3	2927	2768	2860	2768
q4	2032	1821	1824	1821
q5	5786	5783	5763	5763
q6	210	136	132	132
q7	2161	1810	1836	1810
q8	3414	3581	3570	3570
q9	9024	8777	8940	8777
q10	3560	3557	3543	3543
q11	591	491	493	491
q12	815	589	608	589
q13	8187	3115	3232	3115
q14	305	261	266	261
q15	504	470	475	470
q16	482	458	430	430
q17	1822	1638	1610	1610
q18	8231	7736	7682	7682
q19	1694	1717	1483	1483
q20	2163	1801	1831	1801
q21	5604	5285	5482	5285
q22	1167	1044	1035	1035
Total cold run time: 67597 ms
Total hot run time: 59250 ms

@doris-robot

doris-robot commented Jun 11, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 197540 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 969a67ae7661b6a3c3ec07e8e9bdd4781b493934, data reload: false

query1	1296	906	909	906
query2	6230	1951	1907	1907
query3	10798	4431	4171	4171
query4	61131	30116	23810	23810
query5	5198	471	446	446
query6	409	170	172	170
query7	5438	315	332	315
query8	299	233	226	226
query9	8732	2626	2588	2588
query10	483	273	266	266
query11	17931	15187	16052	15187
query12	167	107	106	106
query13	1417	443	433	433
query14	11020	7426	7359	7359
query15	206	182	181	181
query16	7153	498	533	498
query17	1166	603	634	603
query18	1828	334	333	333
query19	201	154	167	154
query20	125	110	109	109
query21	200	100	99	99
query22	4584	4368	4636	4368
query23	34372	33690	33921	33690
query24	6155	2937	2955	2937
query25	523	440	414	414
query26	651	166	167	166
query27	2060	359	376	359
query28	4172	2176	2198	2176
query29	751	495	472	472
query30	238	159	156	156
query31	977	853	828	828
query32	68	60	59	59
query33	463	310	303	303
query34	918	510	521	510
query35	886	748	748	748
query36	1093	952	977	952
query37	120	71	70	70
query38	4118	4029	4034	4029
query39	1545	1469	1484	1469
query40	214	103	103	103
query41	51	51	51	51
query42	119	100	102	100
query43	544	499	511	499
query44	1232	844	836	836
query45	189	182	170	170
query46	1172	785	737	737
query47	2011	1957	1954	1954
query48	487	405	399	399
query49	742	408	447	408
query50	856	446	441	441
query51	7447	7315	7297	7297
query52	104	94	95	94
query53	273	197	192	192
query54	584	492	473	473
query55	88	84	82	82
query56	289	259	274	259
query57	1258	1185	1182	1182
query58	244	206	218	206
query59	3426	3113	3029	3029
query60	289	278	256	256
query61	109	109	105	105
query62	781	679	660	660
query63	232	191	186	186
query64	1371	673	625	625
query65	3261	3176	3192	3176
query66	704	291	291	291
query67	15798	15578	15535	15535
query68	4162	586	597	586
query69	440	267	266	266
query70	1134	1065	1110	1065
query71	347	263	256	256
query72	6360	4046	4133	4046
query73	755	350	351	350
query74	10202	9285	9345	9285
query75	3348	2640	2696	2640
query76	1950	931	1159	931
query77	515	265	276	265
query78	10582	9574	9527	9527
query79	1853	608	604	604
query80	1140	431	422	422
query81	501	223	220	220
query82	360	97	94	94
query83	179	152	138	138
query84	280	75	82	75
query85	960	305	290	290
query86	378	303	298	298
query87	4454	4238	4239	4238
query88	4036	2365	2374	2365
query89	411	293	290	290
query90	2003	186	188	186
query91	187	151	152	151
query92	61	50	48	48
query93	2354	566	572	566
query94	775	304	299	299
query95	353	288	261	261
query96	630	277	281	277
query97	3299	3104	3148	3104
query98	217	208	198	198
query99	1601	1280	1287	1280
Total cold run time: 314824 ms
Total hot run time: 197540 ms

@doris-robot

doris-robot commented Jun 11, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 30 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 969a67ae7661b6a3c3ec07e8e9bdd4781b493934, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.03
query3	0.23	0.06	0.07
query4	1.62	0.10	0.10
query5	0.54	0.52	0.49
query6	1.13	0.72	0.72
query7	0.02	0.01	0.01
query8	0.03	0.03	0.03
query9	0.57	0.50	0.49
query10	0.55	0.55	0.55
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.62	0.60	0.60
query14	0.77	0.80	0.81
query15	0.85	0.82	0.82
query16	0.38	0.38	0.40
query17	1.07	1.05	1.06
query18	0.22	0.21	0.22
query19	1.89	1.78	1.88
query20	0.01	0.01	0.01
query21	15.40	0.60	0.59
query22	2.46	2.59	1.87
query23	17.09	0.78	0.86
query24	3.04	1.22	0.86
query25	0.22	0.10	0.09
query26	0.45	0.13	0.13
query27	0.04	0.05	0.04
query28	10.61	0.53	0.46
query29	12.54	3.19	3.21
query30	0.24	0.05	0.06
query31	2.87	0.38	0.38
query32	3.28	0.45	0.45
query33	2.95	3.02	3.02
query34	17.15	4.48	4.45
query35	4.47	4.56	4.49
query36	0.67	0.48	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.17	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 104.85 s
Total hot run time: 30 s

dataroaring
dataroaring approved these changes Jun 11, 2025
View reviewed changes

@dataroaring dataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 5bcaf5c into apache:branch-3.0 Jun 11, 2025
22 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

dev/3.0.7-merged dev/3.1.0-merged

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@xinyiZzz @Thearas @doris-robot @dataroaring @morrySnow