[fix](cloud) fix file cache potential leakage by freemandealer · Pull Request #46561 · apache/doris

freemandealer · 2025-01-07T12:50:01Z

将原来的同步/异步删除 cache meta + 同步/异步删除 cache data file 多维度的删除策略降维简化：所有 cache meta 都是同步删除（除正在使用，此case处理方式见下文），data file在 critical 场景同步删除、gc 场景下异步删除

异步清理调度的优化：

之前的调度逻辑会提前中断，导致清理效率低下
甚至调度会有概率进入某些状态导致清理无法继续进行
优化 CPU 使用，避免额外无效队列遍历
增加窗口算法对异步删除 data file 进行 qps 限制

优化标记删除：

之前的标记删除机制对 TTL data file 有两个方面的空间泄漏问题
扩展应用场景：从原来只能用于 clear_cache、reset_capacity缩容，扩展任意异步删除的场景
将新的标记删除机制除应用在正在引用的数据之外，还解决了 DOWNLOADING 状态数据的删除泄漏问题

fix 删除正在引用的数据过程的多处泄漏：

之前没有机制对于正在引用的数据进行标记删除，只能放任赦免
现在配合优化后的标记删除机制，使用析构函数在释放引用后自动删除

发现并修复队列操作中存在的内存写飞隐患

reset_capacity 在迭代内部 erase容器条目，可能会导致指针悬空

其它小优化：

使用 concurrentqueue 代替之前的静态无锁队列：保持性能的同时减少队列满、进入同步删文件带来的 IO burst 及伴随的 cache lock 开销
清理弃用的 file_cache_ttl_valid_check_interval_second 配置：现在 ttl 支持 LRU 了，不用额外定时清理
多线程拆分：避免 metrics、resource limit、data file 清理、ttl 超时清理相互影响

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-01-07T12:50:06Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

freemandealer · 2025-01-07T12:50:09Z

run buildall

freemandealer · 2025-01-09T17:12:19Z

run buildall

doris-robot · 2025-01-09T18:59:33Z

TPC-H: Total hot run time: 32805 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1a55a0a5c60131985e9f28d8ee19adf9aa86b3e4, data reload: false

------ Round 1 ----------------------------------
q1	17572	6245	6104	6104
q2	2064	324	184	184
q3	10593	1212	747	747
q4	10196	875	429	429
q5	7524	2208	2000	2000
q6	205	178	157	157
q7	892	747	593	593
q8	9247	1407	1184	1184
q9	5314	4885	4856	4856
q10	6758	2296	1857	1857
q11	489	281	256	256
q12	353	356	214	214
q13	17783	3755	3061	3061
q14	232	234	236	234
q15	572	517	508	508
q16	631	631	585	585
q17	584	864	330	330
q18	7154	6462	6492	6462
q19	1923	949	546	546
q20	312	324	195	195
q21	2850	2160	2001	2001
q22	379	342	302	302
Total cold run time: 103627 ms
Total hot run time: 32805 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6359	6266	6249	6249
q2	239	331	234	234
q3	2273	2691	2308	2308
q4	1396	1834	1408	1408
q5	4356	4794	4854	4794
q6	188	184	142	142
q7	2079	1954	1807	1807
q8	2637	2898	2682	2682
q9	7325	7291	7331	7291
q10	3062	3289	2706	2706
q11	566	515	503	503
q12	755	799	622	622
q13	3572	3863	3283	3283
q14	278	304	270	270
q15	567	520	504	504
q16	641	695	647	647
q17	1243	1753	1271	1271
q18	7793	7637	7344	7344
q19	797	921	1172	921
q20	2063	2082	1912	1912
q21	5748	5152	5018	5018
q22	611	623	589	589
Total cold run time: 54548 ms
Total hot run time: 52505 ms

doris-robot · 2025-01-09T19:11:46Z

TPC-DS: Total hot run time: 195212 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1a55a0a5c60131985e9f28d8ee19adf9aa86b3e4, data reload: false

query1	1292	963	937	937
query2	6326	2380	2345	2345
query3	10972	4709	4582	4582
query4	33161	23441	23599	23441
query5	4221	612	465	465
query6	286	200	193	193
query7	3982	483	304	304
query8	307	249	242	242
query9	9422	2741	2698	2698
query10	443	304	256	256
query11	18040	15272	14926	14926
query12	153	105	101	101
query13	1537	517	376	376
query14	9610	7425	7325	7325
query15	227	206	212	206
query16	8108	612	424	424
query17	1509	787	558	558
query18	2075	431	315	315
query19	203	181	156	156
query20	119	116	144	116
query21	208	121	100	100
query22	4645	4635	4340	4340
query23	34398	33533	33212	33212
query24	6539	2485	2297	2297
query25	474	462	380	380
query26	769	300	156	156
query27	1999	490	340	340
query28	5780	2534	2535	2534
query29	591	540	415	415
query30	215	182	148	148
query31	957	877	826	826
query32	70	64	56	56
query33	488	339	295	295
query34	783	871	523	523
query35	774	829	742	742
query36	1055	1027	964	964
query37	130	106	83	83
query38	4172	4295	4109	4109
query39	1515	1488	1467	1467
query40	214	123	108	108
query41	63	55	57	55
query42	128	130	106	106
query43	530	527	489	489
query44	1306	859	855	855
query45	196	181	169	169
query46	905	1063	655	655
query47	1907	1932	1870	1870
query48	394	419	325	325
query49	737	515	383	383
query50	675	681	388	388
query51	7157	7075	6943	6943
query52	105	103	92	92
query53	235	261	183	183
query54	479	519	407	407
query55	81	89	83	83
query56	238	259	235	235
query57	1213	1222	1142	1142
query58	227	227	229	227
query59	3142	3367	2994	2994
query60	278	278	250	250
query61	114	108	109	108
query62	875	812	732	732
query63	233	188	196	188
query64	3348	1035	650	650
query65	3280	3259	3228	3228
query66	1016	419	313	313
query67	16369	15870	15656	15656
query68	8015	709	515	515
query69	482	283	253	253
query70	1224	1129	1131	1129
query71	383	287	264	264
query72	6184	3971	4013	3971
query73	659	766	363	363
query74	10108	9039	8733	8733
query75	3307	3135	2647	2647
query76	3069	1130	788	788
query77	463	361	266	266
query78	10093	10122	9286	9286
query79	3642	830	579	579
query80	680	526	444	444
query81	506	270	231	231
query82	663	161	120	120
query83	157	164	155	155
query84	250	92	68	68
query85	769	342	297	297
query86	383	317	303	303
query87	4472	4376	4434	4376
query88	4728	2205	2191	2191
query89	421	329	293	293
query90	1799	188	194	188
query91	135	137	106	106
query92	68	61	53	53
query93	2187	905	543	543
query94	686	380	313	313
query95	339	268	249	249
query96	495	610	297	297
query97	2915	2955	2818	2818
query98	215	201	195	195
query99	1666	1517	1391	1391
Total cold run time: 294163 ms
Total hot run time: 195212 ms

doris-robot · 2025-01-09T19:17:09Z

ClickBench: Total hot run time: 31.72 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1a55a0a5c60131985e9f28d8ee19adf9aa86b3e4, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.04	0.03
query3	0.24	0.07	0.07
query4	1.62	0.10	0.10
query5	0.43	0.42	0.41
query6	1.14	0.65	0.65
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.60	0.50	0.50
query10	0.55	0.57	0.54
query11	0.14	0.11	0.11
query12	0.14	0.10	0.10
query13	0.61	0.62	0.59
query14	2.73	2.79	2.74
query15	0.90	0.83	0.84
query16	0.38	0.38	0.39
query17	1.07	1.07	1.00
query18	0.23	0.21	0.21
query19	1.92	2.01	1.76
query20	0.02	0.01	0.01
query21	15.35	0.98	0.58
query22	0.75	0.90	0.77
query23	15.09	1.44	0.55
query24	3.42	1.99	1.25
query25	0.18	0.14	0.16
query26	0.30	0.15	0.14
query27	0.05	0.04	0.04
query28	13.78	1.46	1.06
query29	12.54	3.93	3.29
query30	0.26	0.09	0.08
query31	2.82	0.57	0.38
query32	3.22	0.56	0.47
query33	3.08	3.10	3.08
query34	16.74	5.08	4.50
query35	4.45	4.51	4.50
query36	0.65	0.50	0.47
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.12
query41	0.08	0.03	0.03
query42	0.04	0.03	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.05 s
Total hot run time: 31.72 s

freemandealer · 2025-01-10T01:29:08Z

run p0

freemandealer · 2025-01-10T01:33:50Z

run p0

freemandealer · 2025-01-10T15:32:49Z

run buildall

doris-robot · 2025-01-10T17:01:05Z

TPC-H: Total hot run time: 32575 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c70b01d8167d5ec77d2b2d16106dec131e82ff69, data reload: false

------ Round 1 ----------------------------------
q1	17577	6162	6086	6086
q2	2069	316	181	181
q3	10467	1238	763	763
q4	10198	896	451	451
q5	7514	2213	1990	1990
q6	211	179	149	149
q7	900	766	610	610
q8	9228	1395	1180	1180
q9	5349	4931	4903	4903
q10	6765	2295	1836	1836
q11	502	288	257	257
q12	351	357	218	218
q13	17766	3720	3056	3056
q14	239	232	213	213
q15	566	505	500	500
q16	627	634	588	588
q17	572	866	334	334
q18	7170	6502	6243	6243
q19	2202	959	549	549
q20	312	325	187	187
q21	2904	2262	1974	1974
q22	381	340	307	307
Total cold run time: 103870 ms
Total hot run time: 32575 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6324	6725	6241	6241
q2	241	329	243	243
q3	2322	2704	2331	2331
q4	1400	1850	1371	1371
q5	4288	4731	4885	4731
q6	203	176	142	142
q7	2094	2012	1849	1849
q8	2615	2873	2717	2717
q9	7356	7301	7280	7280
q10	3113	3332	2795	2795
q11	599	544	513	513
q12	708	767	620	620
q13	3525	3930	3283	3283
q14	287	295	281	281
q15	569	513	503	503
q16	645	696	652	652
q17	1247	1754	1253	1253
q18	7721	7341	7300	7300
q19	837	857	1190	857
q20	1952	2048	1940	1940
q21	5760	5228	5087	5087
q22	641	638	581	581
Total cold run time: 54447 ms
Total hot run time: 52570 ms

doris-robot · 2025-01-10T17:13:27Z

TPC-DS: Total hot run time: 196353 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c70b01d8167d5ec77d2b2d16106dec131e82ff69, data reload: false

query1	1314	965	941	941
query2	6039	2361	2410	2361
query3	11103	4917	5091	4917
query4	32851	23402	23389	23389
query5	4435	610	456	456
query6	286	208	191	191
query7	3996	511	317	317
query8	310	249	261	249
query9	9481	2705	2722	2705
query10	483	311	253	253
query11	17915	15126	15060	15060
query12	159	109	108	108
query13	1692	535	404	404
query14	10773	7893	7087	7087
query15	253	202	211	202
query16	7933	632	467	467
query17	1513	759	591	591
query18	2108	448	313	313
query19	214	208	166	166
query20	122	117	116	116
query21	205	127	109	109
query22	4679	4398	4367	4367
query23	34286	34192	33451	33451
query24	6199	2347	2361	2347
query25	486	460	416	416
query26	1046	271	165	165
query27	2251	469	345	345
query28	5459	2510	2486	2486
query29	547	596	442	442
query30	217	198	161	161
query31	941	870	812	812
query32	69	59	64	59
query33	514	366	303	303
query34	794	872	524	524
query35	820	819	747	747
query36	1038	1046	991	991
query37	129	103	82	82
query38	4137	4213	4172	4172
query39	1523	1532	1453	1453
query40	204	124	103	103
query41	51	53	56	53
query42	125	109	104	104
query43	538	553	508	508
query44	1395	866	862	862
query45	187	177	169	169
query46	890	1074	657	657
query47	1895	1920	1812	1812
query48	386	417	338	338
query49	760	502	417	417
query50	670	666	403	403
query51	6960	7133	6939	6939
query52	105	101	97	97
query53	235	258	190	190
query54	510	534	424	424
query55	79	83	80	80
query56	290	276	244	244
query57	1214	1184	1164	1164
query58	265	235	249	235
query59	3237	3370	3120	3120
query60	285	281	265	265
query61	119	127	122	122
query62	816	796	724	724
query63	234	194	192	192
query64	4040	1023	668	668
query65	3360	3250	3292	3250
query66	1197	427	313	313
query67	16714	15777	15482	15482
query68	8041	714	526	526
query69	474	293	254	254
query70	1182	1073	1149	1073
query71	421	284	261	261
query72	6214	3898	3989	3898
query73	664	750	370	370
query74	10119	8992	8870	8870
query75	4022	3184	2699	2699
query76	3642	1182	753	753
query77	761	373	308	308
query78	10854	10181	9358	9358
query79	6782	755	578	578
query80	708	563	445	445
query81	492	277	232	232
query82	565	154	118	118
query83	196	176	163	163
query84	277	97	76	76
query85	747	380	328	328
query86	349	321	301	301
query87	4463	4440	4444	4440
query88	3955	2257	2206	2206
query89	431	311	290	290
query90	2122	246	193	193
query91	140	136	113	113
query92	68	54	52	52
query93	2288	753	546	546
query94	683	414	297	297
query95	343	263	261	261
query96	499	610	290	290
query97	2933	2930	2828	2828
query98	227	211	194	194
query99	1608	1509	1372	1372
Total cold run time: 301351 ms
Total hot run time: 196353 ms

doris-robot · 2025-01-10T17:18:49Z

ClickBench: Total hot run time: 31.05 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c70b01d8167d5ec77d2b2d16106dec131e82ff69, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.04	0.03
query3	0.24	0.07	0.08
query4	1.61	0.11	0.10
query5	0.43	0.43	0.39
query6	1.16	0.65	0.67
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.58	0.52	0.50
query10	0.56	0.56	0.55
query11	0.14	0.10	0.10
query12	0.14	0.12	0.11
query13	0.61	0.62	0.59
query14	2.74	2.83	2.74
query15	0.91	0.83	0.82
query16	0.38	0.37	0.38
query17	1.01	0.99	1.02
query18	0.23	0.20	0.20
query19	2.02	1.88	1.97
query20	0.01	0.01	0.01
query21	15.36	0.96	0.59
query22	0.76	0.87	0.66
query23	15.20	1.45	0.51
query24	3.98	1.42	0.68
query25	0.22	0.22	0.06
query26	0.28	0.15	0.13
query27	0.06	0.07	0.04
query28	13.35	1.52	1.05
query29	12.57	4.06	3.36
query30	0.24	0.08	0.06
query31	2.83	0.58	0.38
query32	3.24	0.55	0.47
query33	3.06	3.09	3.12
query34	16.87	5.15	4.52
query35	4.51	4.53	4.51
query36	0.81	0.49	0.48
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.13
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.74 s
Total hot run time: 31.05 s

freemandealer · 2025-01-13T16:06:35Z

run beut

freemandealer · 2025-01-13T16:06:41Z

run buildall

freemandealer · 2025-01-14T01:13:09Z

run buildall

freemandealer · 2025-01-14T01:27:10Z

run buildall

doris-robot · 2025-01-14T02:42:25Z

TPC-H: Total hot run time: 32961 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit df8f75e713f2f268cb25aecf6ded6f410f5d7d69, data reload: false

------ Round 1 ----------------------------------
q1	17565	6350	6066	6066
q2	2054	296	170	170
q3	10421	1253	732	732
q4	10176	879	440	440
q5	7535	2204	2011	2011
q6	206	179	149	149
q7	905	756	603	603
q8	9226	1400	1207	1207
q9	5237	4907	4963	4907
q10	6800	2319	1845	1845
q11	506	280	248	248
q12	349	356	219	219
q13	17763	3685	3108	3108
q14	239	232	221	221
q15	595	507	496	496
q16	624	623	596	596
q17	588	860	328	328
q18	6988	6631	6412	6412
q19	3957	1114	684	684
q20	302	309	179	179
q21	2881	2168	2022	2022
q22	363	335	318	318
Total cold run time: 105280 ms
Total hot run time: 32961 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6654	6267	6235	6235
q2	234	325	231	231
q3	2291	2654	2370	2370
q4	1398	1790	1343	1343
q5	4369	4826	4961	4826
q6	185	175	145	145
q7	2219	1985	1832	1832
q8	2670	2814	2728	2728
q9	7291	7235	7330	7235
q10	3003	3247	2857	2857
q11	598	527	515	515
q12	692	850	660	660
q13	3467	3826	3213	3213
q14	284	322	277	277
q15	566	509	523	509
q16	671	693	665	665
q17	1229	1756	1297	1297
q18	7867	7402	7420	7402
q19	822	1135	1113	1113
q20	2044	2111	1988	1988
q21	5608	5101	5043	5043
q22	620	630	564	564
Total cold run time: 54782 ms
Total hot run time: 53048 ms

doris-robot · 2025-01-14T02:54:40Z

TPC-DS: Total hot run time: 195635 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit df8f75e713f2f268cb25aecf6ded6f410f5d7d69, data reload: false

query1	1323	971	932	932
query2	6151	2286	2286	2286
query3	10997	4648	4760	4648
query4	33276	23599	23252	23252
query5	5352	613	463	463
query6	292	210	197	197
query7	4001	483	300	300
query8	286	229	211	211
query9	9292	2682	2687	2682
query10	496	313	269	269
query11	18173	15167	15044	15044
query12	156	109	105	105
query13	1556	516	403	403
query14	9989	7363	8069	7363
query15	246	211	193	193
query16	8672	689	456	456
query17	1593	784	596	596
query18	2072	418	317	317
query19	209	200	172	172
query20	124	118	120	118
query21	210	143	118	118
query22	4680	4434	4365	4365
query23	34486	33294	33253	33253
query24	6300	2299	2344	2299
query25	482	451	409	409
query26	788	275	153	153
query27	2360	477	331	331
query28	5372	2511	2493	2493
query29	608	573	426	426
query30	214	184	165	165
query31	1014	905	852	852
query32	68	63	62	62
query33	498	350	294	294
query34	796	860	497	497
query35	826	847	757	757
query36	1034	1062	987	987
query37	138	104	78	78
query38	4121	4384	4201	4201
query39	1477	1456	1427	1427
query40	206	121	115	115
query41	55	54	55	54
query42	126	105	104	104
query43	513	534	500	500
query44	1350	833	827	827
query45	192	180	175	175
query46	876	1063	659	659
query47	1904	1854	1874	1854
query48	389	411	328	328
query49	709	501	410	410
query50	658	683	389	389
query51	7062	7059	6951	6951
query52	102	100	90	90
query53	223	250	187	187
query54	492	524	418	418
query55	85	78	79	78
query56	259	262	246	246
query57	1213	1199	1159	1159
query58	253	239	235	235
query59	3253	3181	3080	3080
query60	273	266	290	266
query61	118	149	112	112
query62	858	813	735	735
query63	234	187	190	187
query64	2761	1022	655	655
query65	3333	3259	3212	3212
query66	727	446	314	314
query67	16247	15832	15368	15368
query68	7948	723	527	527
query69	502	298	253	253
query70	1201	1063	1204	1063
query71	460	290	258	258
query72	6226	3908	3871	3871
query73	663	744	338	338
query74	10314	9161	8813	8813
query75	4236	3133	2670	2670
query76	3756	1196	783	783
query77	771	384	276	276
query78	10067	9840	9720	9720
query79	3692	797	576	576
query80	685	520	450	450
query81	491	275	235	235
query82	589	151	122	122
query83	199	168	151	151
query84	279	96	73	73
query85	804	343	299	299
query86	345	316	307	307
query87	4324	4599	4379	4379
query88	4115	2178	2161	2161
query89	401	331	295	295
query90	1884	191	188	188
query91	133	136	102	102
query92	68	57	54	54
query93	1292	866	534	534
query94	651	399	290	290
query95	323	264	254	254
query96	563	609	291	291
query97	2792	2956	2814	2814
query98	222	207	190	190
query99	1623	1506	1416	1416
Total cold run time: 295818 ms
Total hot run time: 195635 ms

doris-robot · 2025-01-14T03:00:02Z

ClickBench: Total hot run time: 31.63 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit df8f75e713f2f268cb25aecf6ded6f410f5d7d69, data reload: false

query1	0.04	0.03	0.04
query2	0.07	0.04	0.03
query3	0.24	0.07	0.07
query4	1.62	0.10	0.10
query5	0.43	0.43	0.41
query6	1.16	0.66	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.52	0.49
query10	0.56	0.55	0.56
query11	0.16	0.11	0.11
query12	0.15	0.12	0.11
query13	0.61	0.59	0.60
query14	2.83	2.81	2.86
query15	0.90	0.82	0.82
query16	0.39	0.38	0.36
query17	1.02	1.04	1.02
query18	0.24	0.21	0.20
query19	1.94	1.82	1.99
query20	0.02	0.02	0.01
query21	15.35	0.95	0.56
query22	0.76	0.96	0.96
query23	14.90	1.45	0.53
query24	2.94	1.00	1.76
query25	0.24	0.16	0.13
query26	0.22	0.14	0.13
query27	0.08	0.04	0.04
query28	13.72	1.49	1.05
query29	12.58	4.02	3.33
query30	0.26	0.09	0.07
query31	2.80	0.59	0.39
query32	3.23	0.55	0.46
query33	3.03	3.13	3.04
query34	16.68	5.14	4.47
query35	4.49	4.45	4.44
query36	0.82	0.48	0.48
query37	0.09	0.06	0.06
query38	0.04	0.03	0.04
query39	0.04	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.02	0.03
query42	0.03	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.6 s
Total hot run time: 31.63 s

freemandealer · 2025-01-14T16:15:53Z

run buildall

doris-robot · 2025-01-14T18:01:58Z

TPC-H: Total hot run time: 32917 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d734743537a2ddee168a4fd8fe6a449f86460d6c, data reload: false

------ Round 1 ----------------------------------
q1	17650	6177	6056	6056
q2	2057	305	177	177
q3	10501	1247	729	729
q4	10235	895	445	445
q5	8081	2212	1967	1967
q6	209	179	148	148
q7	919	758	607	607
q8	9234	1401	1179	1179
q9	5271	4939	4883	4883
q10	6758	2289	1874	1874
q11	476	280	264	264
q12	353	376	223	223
q13	17766	3714	3166	3166
q14	235	227	210	210
q15	585	523	536	523
q16	646	639	594	594
q17	599	866	320	320
q18	7066	6559	6504	6504
q19	3078	983	552	552
q20	310	327	186	186
q21	2804	2277	1988	1988
q22	376	334	322	322
Total cold run time: 105209 ms
Total hot run time: 32917 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6458	6221	6244	6221
q2	245	335	245	245
q3	2263	2652	2335	2335
q4	1451	1814	1378	1378
q5	4362	4824	4887	4824
q6	197	172	144	144
q7	2081	1978	1903	1903
q8	2646	2810	2742	2742
q9	7348	7244	7278	7244
q10	3113	3270	2769	2769
q11	606	550	513	513
q12	708	770	655	655
q13	3482	3913	3342	3342
q14	301	298	280	280
q15	579	530	538	530
q16	658	710	655	655
q17	1246	1765	1319	1319
q18	7941	7457	7318	7318
q19	830	1141	1108	1108
q20	2037	2010	1875	1875
q21	5682	5160	5024	5024
q22	581	574	611	574
Total cold run time: 54815 ms
Total hot run time: 52998 ms

doris-robot · 2025-01-14T18:14:07Z

TPC-DS: Total hot run time: 195071 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d734743537a2ddee168a4fd8fe6a449f86460d6c, data reload: false

query1	1343	972	943	943
query2	6300	2093	2046	2046
query3	11066	4654	4757	4654
query4	35291	23821	23232	23232
query5	3567	637	479	479
query6	288	210	195	195
query7	3989	507	309	309
query8	283	235	212	212
query9	9468	2721	2729	2721
query10	456	307	250	250
query11	17719	15677	15004	15004
query12	160	105	99	99
query13	1549	536	414	414
query14	10186	6400	6922	6400
query15	213	205	187	187
query16	7827	680	474	474
query17	1554	760	598	598
query18	2034	420	321	321
query19	215	205	170	170
query20	116	119	113	113
query21	208	122	106	106
query22	4583	4453	4571	4453
query23	34534	33817	33576	33576
query24	6595	2407	2341	2341
query25	504	443	383	383
query26	828	273	157	157
query27	2372	469	343	343
query28	5233	2506	2504	2504
query29	603	594	475	475
query30	220	200	166	166
query31	938	910	821	821
query32	73	66	66	66
query33	501	380	352	352
query34	742	868	504	504
query35	811	839	764	764
query36	1007	1026	951	951
query37	126	101	79	79
query38	4280	4260	4318	4260
query39	1498	1461	1445	1445
query40	214	114	104	104
query41	53	53	56	53
query42	122	102	109	102
query43	527	543	489	489
query44	1393	857	867	857
query45	184	180	165	165
query46	867	1050	661	661
query47	1952	1935	1781	1781
query48	401	410	339	339
query49	736	512	403	403
query50	682	664	407	407
query51	7176	6902	6966	6902
query52	104	105	95	95
query53	233	253	190	190
query54	494	535	426	426
query55	91	103	90	90
query56	287	287	265	265
query57	1224	1192	1155	1155
query58	269	249	257	249
query59	3154	3225	3165	3165
query60	283	283	267	267
query61	123	118	121	118
query62	867	806	754	754
query63	230	196	197	196
query64	3322	1060	671	671
query65	3333	3165	3151	3151
query66	802	410	304	304
query67	16188	15792	15608	15608
query68	4689	829	546	546
query69	503	300	265	265
query70	1195	1153	1101	1101
query71	395	286	256	256
query72	6022	3961	3838	3838
query73	657	751	360	360
query74	10244	9195	9097	9097
query75	3180	3142	2643	2643
query76	3159	1176	758	758
query77	464	363	283	283
query78	10153	10108	9240	9240
query79	2537	831	615	615
query80	691	520	447	447
query81	506	375	235	235
query82	220	155	118	118
query83	177	185	152	152
query84	236	93	75	75
query85	750	354	303	303
query86	386	301	297	297
query87	4367	4422	4372	4372
query88	4419	2235	2175	2175
query89	395	335	288	288
query90	1865	196	193	193
query91	139	140	114	114
query92	71	55	55	55
query93	2445	861	537	537
query94	650	399	298	298
query95	341	261	258	258
query96	508	611	283	283
query97	2812	2856	2771	2771
query98	235	202	204	202
query99	1402	1469	1362	1362
Total cold run time: 290404 ms
Total hot run time: 195071 ms

doris-robot · 2025-01-14T18:19:27Z

ClickBench: Total hot run time: 31.43 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d734743537a2ddee168a4fd8fe6a449f86460d6c, data reload: false

query1	0.05	0.03	0.03
query2	0.07	0.04	0.03
query3	0.24	0.07	0.06
query4	1.62	0.10	0.10
query5	0.42	0.42	0.40
query6	1.14	0.66	0.65
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.50	0.51
query10	0.54	0.59	0.55
query11	0.14	0.09	0.11
query12	0.14	0.11	0.11
query13	0.62	0.61	0.60
query14	2.76	2.86	2.73
query15	0.90	0.83	0.82
query16	0.39	0.38	0.38
query17	1.07	1.05	1.06
query18	0.23	0.21	0.20
query19	1.95	1.88	1.99
query20	0.02	0.01	0.01
query21	15.38	0.92	0.58
query22	0.76	0.70	0.70
query23	15.30	1.47	0.61
query24	2.84	1.25	0.92
query25	0.24	0.09	0.08
query26	0.35	0.15	0.14
query27	0.06	0.06	0.06
query28	14.01	1.51	1.05
query29	12.60	3.93	3.29
query30	0.25	0.09	0.07
query31	2.82	0.59	0.38
query32	3.22	0.55	0.47
query33	3.01	3.01	3.00
query34	16.67	5.32	4.51
query35	4.55	4.65	4.53
query36	0.65	0.50	0.50
query37	0.10	0.07	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.15	0.14	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.14 s
Total hot run time: 31.43 s

doris-robot · 2025-01-15T12:15:45Z

TPC-H: Total hot run time: 32500 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a06fe6eb71322aab158fd9acc943f87ae632ec86, data reload: false

------ Round 1 ----------------------------------
q1	17576	5593	5386	5386
q2	2045	295	182	182
q3	10410	1236	744	744
q4	10221	979	559	559
q5	7830	2391	2154	2154
q6	197	166	135	135
q7	886	748	602	602
q8	9237	1351	1159	1159
q9	5364	4880	4906	4880
q10	6891	2317	1904	1904
q11	483	291	250	250
q12	342	360	222	222
q13	17778	3692	3172	3172
q14	223	235	213	213
q15	524	457	470	457
q16	648	611	584	584
q17	585	863	324	324
q18	6824	6571	6451	6451
q19	2283	943	543	543
q20	309	322	188	188
q21	2889	2281	2086	2086
q22	372	336	305	305
Total cold run time: 103917 ms
Total hot run time: 32500 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5642	5564	5526	5526
q2	243	331	231	231
q3	2228	2659	2275	2275
q4	1430	1806	1369	1369
q5	4297	4770	4699	4699
q6	189	160	134	134
q7	2085	1962	1792	1792
q8	2644	2844	2808	2808
q9	7285	7196	7243	7196
q10	3033	3296	2795	2795
q11	589	521	483	483
q12	672	744	601	601
q13	3576	3968	3346	3346
q14	310	294	285	285
q15	521	463	465	463
q16	642	693	664	664
q17	1251	1747	1291	1291
q18	7775	7470	7234	7234
q19	855	1168	1115	1115
q20	1987	2025	1891	1891
q21	5858	5267	5011	5011
q22	625	614	592	592
Total cold run time: 53737 ms
Total hot run time: 51801 ms

doris-robot · 2025-01-15T12:20:30Z

TeamCity be ut coverage result:
Function Coverage: 41.02% (10687/26054)
Line Coverage: 31.52% (90264/286361)
Region Coverage: 30.67% (46243/150757)
Branch Coverage: 26.82% (23421/87320)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a06fe6eb71322aab158fd9acc943f87ae632ec86_a06fe6eb71322aab158fd9acc943f87ae632ec86/report/index.html

doris-robot · 2025-01-15T12:28:20Z

TPC-DS: Total hot run time: 194278 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a06fe6eb71322aab158fd9acc943f87ae632ec86, data reload: false

query1	1309	931	954	931
query2	6467	2028	1958	1958
query3	10996	4529	4371	4371
query4	61597	28751	23511	23511
query5	5562	639	466	466
query6	433	214	191	191
query7	5573	523	294	294
query8	318	238	235	235
query9	8406	2577	2565	2565
query10	475	319	258	258
query11	17590	15433	15453	15433
query12	170	126	114	114
query13	1476	542	414	414
query14	11445	6714	7266	6714
query15	224	206	186	186
query16	6816	598	519	519
query17	1127	755	618	618
query18	1714	413	329	329
query19	205	188	166	166
query20	119	112	114	112
query21	221	125	109	109
query22	4435	4702	4519	4519
query23	34088	33665	33332	33332
query24	5656	2400	2333	2333
query25	479	487	417	417
query26	689	295	158	158
query27	1806	486	338	338
query28	3681	2461	2425	2425
query29	567	558	436	436
query30	222	189	161	161
query31	931	850	819	819
query32	71	66	57	57
query33	449	367	339	339
query34	921	885	531	531
query35	806	869	792	792
query36	1023	1057	971	971
query37	119	105	77	77
query38	4316	4334	4202	4202
query39	1526	1472	1464	1464
query40	210	116	101	101
query41	52	49	51	49
query42	121	106	108	106
query43	532	533	497	497
query44	1400	841	833	833
query45	194	175	166	166
query46	915	1094	665	665
query47	1865	1911	1837	1837
query48	389	408	334	334
query49	742	507	416	416
query50	704	703	422	422
query51	7008	7137	7027	7027
query52	106	101	94	94
query53	248	269	188	188
query54	492	502	419	419
query55	89	82	89	82
query56	285	271	259	259
query57	1203	1186	1140	1140
query58	230	225	229	225
query59	2984	3078	2907	2907
query60	272	261	250	250
query61	121	121	116	116
query62	724	709	645	645
query63	224	189	190	189
query64	1480	1037	663	663
query65	3234	3176	3169	3169
query66	741	394	307	307
query67	15763	15584	15339	15339
query68	5009	851	541	541
query69	487	310	264	264
query70	1220	1153	1127	1127
query71	409	296	265	265
query72	6068	3470	3806	3470
query73	797	764	372	372
query74	9862	8666	8701	8666
query75	3227	3159	2675	2675
query76	3768	1226	796	796
query77	497	381	293	293
query78	10083	10140	9391	9391
query79	2806	815	608	608
query80	799	547	446	446
query81	502	278	244	244
query82	220	151	124	124
query83	172	170	154	154
query84	290	105	71	71
query85	745	364	305	305
query86	397	320	290	290
query87	4446	4413	4572	4413
query88	3855	2212	2175	2175
query89	414	329	338	329
query90	1684	193	193	193
query91	137	140	113	113
query92	62	62	52	52
query93	2642	899	538	538
query94	707	415	306	306
query95	321	277	258	258
query96	498	630	290	290
query97	2866	2881	2777	2777
query98	252	196	194	194
query99	1463	1375	1228	1228
Total cold run time: 313061 ms
Total hot run time: 194278 ms

doris-robot · 2025-01-15T12:33:39Z

ClickBench: Total hot run time: 30.66 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a06fe6eb71322aab158fd9acc943f87ae632ec86, data reload: false

query1	0.04	0.06	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.60	0.10	0.10
query5	0.43	0.43	0.42
query6	1.15	0.65	0.68
query7	0.02	0.01	0.01
query8	0.04	0.02	0.03
query9	0.59	0.50	0.50
query10	0.58	0.56	0.54
query11	0.14	0.10	0.11
query12	0.14	0.10	0.10
query13	0.61	0.59	0.60
query14	2.74	2.88	2.76
query15	0.91	0.83	0.83
query16	0.39	0.40	0.38
query17	1.06	0.99	1.01
query18	0.23	0.20	0.21
query19	1.94	1.74	2.00
query20	0.01	0.01	0.01
query21	15.36	0.92	0.58
query22	0.76	0.73	0.75
query23	15.25	1.37	0.65
query24	2.84	0.93	1.65
query25	0.13	0.17	0.10
query26	0.22	0.14	0.15
query27	0.06	0.04	0.05
query28	14.16	1.01	0.45
query29	12.63	4.02	3.32
query30	0.25	0.09	0.07
query31	2.83	0.62	0.40
query32	3.23	0.55	0.47
query33	3.01	3.01	3.13
query34	16.51	5.20	4.46
query35	4.48	4.46	4.52
query36	0.66	0.48	0.48
query37	0.09	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.03	0.02
query40	0.16	0.13	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.02
Total cold run time: 105.78 s
Total hot run time: 30.66 s

freemandealer · 2025-01-16T17:09:52Z

run buildall

doris-robot · 2025-01-16T18:09:57Z

TeamCity be ut coverage result:
Function Coverage: 41.53% (10819/26054)
Line Coverage: 31.91% (91379/286347)
Region Coverage: 31.06% (46817/150745)
Branch Coverage: 27.14% (23699/87308)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a8efb897ef42ffd8eb491ce3c5c8a546dd5b3b05_a8efb897ef42ffd8eb491ce3c5c8a546dd5b3b05/report/index.html

freemandealer · 2025-01-17T01:47:30Z

run external

freemandealer · 2025-01-17T01:47:36Z

run performance

doris-robot · 2025-01-17T02:09:33Z

TPC-H: Total hot run time: 32215 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a8efb897ef42ffd8eb491ce3c5c8a546dd5b3b05, data reload: false

------ Round 1 ----------------------------------
q1	17583	5645	5360	5360
q2	2043	312	176	176
q3	10501	1215	746	746
q4	10217	974	519	519
q5	7536	2386	2131	2131
q6	192	174	135	135
q7	893	757	600	600
q8	9250	1347	1130	1130
q9	5111	4945	4870	4870
q10	6842	2341	1894	1894
q11	472	271	249	249
q12	353	365	219	219
q13	17764	3660	3101	3101
q14	244	232	219	219
q15	514	474	456	456
q16	625	630	598	598
q17	546	859	327	327
q18	7097	6524	6477	6477
q19	1605	956	547	547
q20	304	320	192	192
q21	2822	2199	1957	1957
q22	373	336	312	312
Total cold run time: 102887 ms
Total hot run time: 32215 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5540	5475	5496	5475
q2	238	329	242	242
q3	2263	2642	2343	2343
q4	1340	1786	1342	1342
q5	4294	4764	4613	4613
q6	167	161	128	128
q7	2072	1953	1804	1804
q8	2623	2813	2724	2724
q9	7205	7216	7186	7186
q10	3031	3274	2790	2790
q11	573	509	482	482
q12	624	733	580	580
q13	3431	3930	3319	3319
q14	291	289	293	289
q15	528	461	485	461
q16	645	694	649	649
q17	1217	1746	1286	1286
q18	7610	7442	7196	7196
q19	788	1095	1090	1090
q20	2019	2017	1870	1870
q21	5831	5293	4906	4906
q22	595	631	559	559
Total cold run time: 52925 ms
Total hot run time: 51334 ms

doris-robot · 2025-01-17T02:22:02Z

TPC-DS: Total hot run time: 193427 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a8efb897ef42ffd8eb491ce3c5c8a546dd5b3b05, data reload: false

query1	1332	971	923	923
query2	6294	2055	2031	2031
query3	10958	4383	4396	4383
query4	62288	28343	23419	23419
query5	5408	619	454	454
query6	429	207	213	207
query7	5451	510	296	296
query8	336	245	237	237
query9	7981	2665	2654	2654
query10	444	318	267	267
query11	17194	15120	15793	15120
query12	165	113	120	113
query13	1401	573	428	428
query14	10875	6515	6547	6515
query15	210	199	201	199
query16	7247	639	508	508
query17	1073	690	576	576
query18	1809	402	302	302
query19	217	180	156	156
query20	118	114	110	110
query21	217	135	111	111
query22	4596	4741	4285	4285
query23	34621	33359	33068	33068
query24	5645	2313	2334	2313
query25	468	475	410	410
query26	652	265	159	159
query27	1649	481	321	321
query28	4315	2508	2424	2424
query29	535	550	428	428
query30	218	198	156	156
query31	922	879	847	847
query32	70	60	58	58
query33	462	364	296	296
query34	733	857	508	508
query35	826	857	747	747
query36	1023	1035	977	977
query37	129	109	81	81
query38	4357	4300	4167	4167
query39	1502	1463	1446	1446
query40	202	122	103	103
query41	53	49	54	49
query42	129	104	103	103
query43	509	516	495	495
query44	1343	836	864	836
query45	182	179	164	164
query46	884	1065	654	654
query47	1954	1962	1842	1842
query48	388	410	328	328
query49	723	491	404	404
query50	662	687	418	418
query51	7157	6912	6931	6912
query52	100	96	98	96
query53	220	248	183	183
query54	487	499	423	423
query55	82	80	83	80
query56	268	292	280	280
query57	1212	1173	1121	1121
query58	241	238	230	230
query59	3114	3047	2926	2926
query60	284	268	248	248
query61	122	118	119	118
query62	712	728	635	635
query63	221	186	184	184
query64	1259	1040	644	644
query65	3202	3139	3126	3126
query66	708	401	297	297
query67	15951	15548	15647	15548
query68	4218	771	537	537
query69	470	303	267	267
query70	1198	1133	1086	1086
query71	410	294	257	257
query72	5401	3811	3779	3779
query73	742	739	371	371
query74	9606	8983	9164	8983
query75	3224	3157	2703	2703
query76	3439	1157	776	776
query77	488	357	271	271
query78	10066	10098	9220	9220
query79	2445	788	601	601
query80	1400	519	519	519
query81	534	274	237	237
query82	348	151	116	116
query83	243	173	173	173
query84	291	91	79	79
query85	772	410	307	307
query86	383	323	290	290
query87	4509	4424	4379	4379
query88	3589	2198	2161	2161
query89	390	322	282	282
query90	1656	185	187	185
query91	133	135	108	108
query92	57	56	52	52
query93	1893	855	524	524
query94	751	409	300	300
query95	331	264	251	251
query96	492	606	282	282
query97	2791	2866	2725	2725
query98	222	202	194	194
query99	1274	1393	1265	1265
Total cold run time: 310305 ms
Total hot run time: 193427 ms

doris-robot · 2025-01-17T02:27:20Z

ClickBench: Total hot run time: 30.03 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a8efb897ef42ffd8eb491ce3c5c8a546dd5b3b05, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.04	0.03
query3	0.24	0.07	0.07
query4	1.62	0.10	0.11
query5	0.43	0.43	0.43
query6	1.14	0.67	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.59	0.50	0.50
query10	0.56	0.57	0.56
query11	0.14	0.10	0.11
query12	0.14	0.10	0.10
query13	0.60	0.60	0.60
query14	2.84	2.82	2.79
query15	0.89	0.84	0.82
query16	0.40	0.38	0.38
query17	0.98	1.05	0.97
query18	0.23	0.20	0.21
query19	1.86	1.88	1.99
query20	0.02	0.01	0.01
query21	15.38	0.88	0.58
query22	0.77	0.83	0.58
query23	15.32	1.44	0.55
query24	3.12	0.36	1.76
query25	0.21	0.22	0.12
query26	0.29	0.15	0.14
query27	0.06	0.06	0.04
query28	13.54	0.99	0.43
query29	12.58	3.99	3.27
query30	0.25	0.08	0.07
query31	2.83	0.61	0.39
query32	3.23	0.55	0.46
query33	2.97	2.96	3.05
query34	16.71	5.12	4.49
query35	4.54	4.55	4.56
query36	0.65	0.49	0.48
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.04	0.03	0.02
query40	0.16	0.13	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.03
query43	0.03	0.02	0.02
Total cold run time: 105.78 s
Total hot run time: 30.03 s

github-actions · 2025-01-18T17:56:43Z

PR approved by at least one committer and no changes requested.

gavinchou · 2025-01-18T17:57:50Z

+            int64_t duration_ns = 0;
+            SCOPED_RAW_TIMER(&duration_ns);
+            Status st = _storage->remove(key);
+            *_storage_async_remove_latency << duration_ns;


microseconds is enough, and easy to keep consistent with existing latency unit

microseconds is enough, and easy to keep consistent with existing latency unit

OK, another monitor PR is on the way. Do it later.

dataroaring

LGTM

freemandealer · 2025-01-21T04:18:41Z

+void BlockFileCache::run_background_gc() {
+    FileCacheKey key;
+    static const size_t interval_ms = 100;
+    const size_t batch_limit = config::file_cache_remove_block_qps_limit * interval_ms / 1000;


should move into while loop to be dynamically configurable

interval_ms set to interval_us = 1000us
const size_t batch_limit = std::max(config::file_cache_remove_block_qps_limit * (interval_us / 1000000.0), 1);

single thread IO bottleneck

corner cases: upper bound and lower bound

dataroaring added the dev/3.0.x label Jan 7, 2025

freemandealer force-pushed the fix-filecache-leak-v2 branch from 13739c8 to 272ecea Compare January 8, 2025 07:48

freemandealer force-pushed the fix-filecache-leak-v2 branch from 1a55a0a to c70b01d Compare January 10, 2025 15:32

freemandealer changed the title ~~[fix](cloud) fix file cache potential leakage (1/3)~~ [fix](cloud) fix file cache potential leakage Jan 10, 2025

freemandealer force-pushed the fix-filecache-leak-v2 branch from 2ead512 to f0ce039 Compare January 14, 2025 01:11

freemandealer mentioned this pull request Jan 14, 2025

[fix](cloud) fix file cache potential leakage #46404

Closed

16 tasks

freemandealer force-pushed the fix-filecache-leak-v2 branch from df8f75e to d734743 Compare January 14, 2025 16:13

Merge branch 'master' into fix-filecache-leak-v2

a8efb89

freemandealer requested a review from gavinchou as a code owner January 16, 2025 17:09

gavinchou approved these changes Jan 18, 2025

View reviewed changes

github-actions Bot added the approved Indicates a PR has been approved by one committer. label Jan 18, 2025

gavinchou reviewed Jan 18, 2025

View reviewed changes

dataroaring approved these changes Jan 21, 2025

View reviewed changes

freemandealer commented Jan 21, 2025

View reviewed changes

luwei16 approved these changes Jan 23, 2025

View reviewed changes

gavinchou merged commit 4c540f5 into apache:master Jan 23, 2025

github-actions Bot mentioned this pull request Jan 23, 2025

branch-3.0: [fix](cloud) fix file cache potential leakage #46561 #47376

Merged

dataroaring added dev/3.0.4-merged and removed dev/3.0.x labels Jan 24, 2025

gavinchou mentioned this pull request Feb 18, 2025

Release Notes 3.0.4 #48013

Closed

Uh oh!

Conversation

freemandealer commented Jan 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Jan 7, 2025

Uh oh!

freemandealer commented Jan 7, 2025

Uh oh!

freemandealer commented Jan 9, 2025

Uh oh!

doris-robot commented Jan 9, 2025

Uh oh!

doris-robot commented Jan 9, 2025

Uh oh!

doris-robot commented Jan 9, 2025

Uh oh!

freemandealer commented Jan 10, 2025

Uh oh!

freemandealer commented Jan 10, 2025

Uh oh!

freemandealer commented Jan 10, 2025

Uh oh!

doris-robot commented Jan 10, 2025

Uh oh!

doris-robot commented Jan 10, 2025

Uh oh!

doris-robot commented Jan 10, 2025

Uh oh!

freemandealer commented Jan 13, 2025

Uh oh!

freemandealer commented Jan 13, 2025

Uh oh!

freemandealer commented Jan 14, 2025

Uh oh!

freemandealer commented Jan 14, 2025

Uh oh!

doris-robot commented Jan 14, 2025

Uh oh!

doris-robot commented Jan 14, 2025

Uh oh!

doris-robot commented Jan 14, 2025

Uh oh!

freemandealer commented Jan 14, 2025

Uh oh!

doris-robot commented Jan 14, 2025

Uh oh!

doris-robot commented Jan 14, 2025

Uh oh!

doris-robot commented Jan 14, 2025

Uh oh!

doris-robot commented Jan 15, 2025

Uh oh!

doris-robot commented Jan 15, 2025

Uh oh!

doris-robot commented Jan 15, 2025

Uh oh!

doris-robot commented Jan 15, 2025

Uh oh!

freemandealer commented Jan 16, 2025

Uh oh!

doris-robot commented Jan 16, 2025

Uh oh!

freemandealer commented Jan 17, 2025

Uh oh!

freemandealer commented Jan 17, 2025

Uh oh!

doris-robot commented Jan 17, 2025

Uh oh!

doris-robot commented Jan 17, 2025

Uh oh!

doris-robot commented Jan 17, 2025

Uh oh!

github-actions Bot commented Jan 18, 2025

Uh oh!

gavinchou Jan 18, 2025

Choose a reason for hiding this comment

Uh oh!

freemandealer commented Jan 7, 2025 •

edited

Loading

freemandealer Jan 21, 2025 •

edited

Loading