Skip to content

[Opt](merge-on-write) Remove useless delete bitmaps periodically to reduce storage occupation#46696

Closed
bobhan1 wants to merge 1 commit into
apache:masterfrom
bobhan1:remove-useless-delete-bitmap
Closed

[Opt](merge-on-write) Remove useless delete bitmaps periodically to reduce storage occupation#46696
bobhan1 wants to merge 1 commit into
apache:masterfrom
bobhan1:remove-useless-delete-bitmap

Conversation

@bobhan1

@bobhan1 bobhan1 commented Jan 9, 2025

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas

Thearas commented Jan 9, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@bobhan1

bobhan1 commented Jan 9, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@bobhan1 bobhan1 marked this pull request as ready for review January 9, 2025 09:05
@doris-robot

Copy link
Copy Markdown
TPC-H: Total hot run time: 32553 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f855bdd4244f2dc423569e5d8972fe075d96bf4c, data reload: false

------ Round 1 ----------------------------------
q1	17758	6125	6073	6073
q2	2047	320	168	168
q3	10499	1297	721	721
q4	10215	872	427	427
q5	7599	2233	1980	1980
q6	206	180	146	146
q7	911	763	609	609
q8	9250	1393	1225	1225
q9	5271	4913	4891	4891
q10	6752	2318	1866	1866
q11	497	279	251	251
q12	353	350	215	215
q13	17756	3686	3015	3015
q14	240	229	212	212
q15	561	504	505	504
q16	630	619	578	578
q17	584	859	319	319
q18	7189	6574	6324	6324
q19	1944	964	554	554
q20	314	311	194	194
q21	2873	2136	1954	1954
q22	369	332	327	327
Total cold run time: 103818 ms
Total hot run time: 32553 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6265	6224	6236	6224
q2	247	323	232	232
q3	2309	2697	2318	2318
q4	1408	1765	1385	1385
q5	4344	4741	4858	4741
q6	186	175	143	143
q7	2092	2017	1814	1814
q8	2692	2739	2708	2708
q9	7196	7247	7175	7175
q10	3053	3333	2792	2792
q11	602	521	494	494
q12	664	798	663	663
q13	3468	3819	3229	3229
q14	281	300	291	291
q15	568	506	495	495
q16	659	700	655	655
q17	1236	1751	1269	1269
q18	7786	7440	7265	7265
q19	823	1126	1105	1105
q20	1976	2007	1887	1887
q21	5633	5250	4882	4882
q22	609	589	574	574
Total cold run time: 54097 ms
Total hot run time: 52341 ms

@doris-robot

Copy link
Copy Markdown
TPC-DS: Total hot run time: 188464 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f855bdd4244f2dc423569e5d8972fe075d96bf4c, data reload: false

query1	980	394	360	360
query2	6517	2350	2305	2305
query3	6703	222	211	211
query4	33612	23453	23074	23074
query5	4315	615	462	462
query6	306	207	191	191
query7	4620	505	304	304
query8	306	256	238	238
query9	9339	2643	2639	2639
query10	477	300	252	252
query11	18103	15292	15034	15034
query12	169	107	104	104
query13	1665	528	380	380
query14	10710	6831	6565	6565
query15	259	193	178	178
query16	8142	585	470	470
query17	1576	733	575	575
query18	2093	440	307	307
query19	212	167	153	153
query20	115	113	112	112
query21	210	126	100	100
query22	4379	4424	4092	4092
query23	34144	34205	33246	33246
query24	6425	2250	2222	2222
query25	502	462	383	383
query26	1214	273	154	154
query27	2027	455	336	336
query28	5089	2445	2405	2405
query29	730	535	413	413
query30	227	173	148	148
query31	967	862	813	813
query32	93	62	58	58
query33	499	346	293	293
query34	776	823	511	511
query35	784	789	729	729
query36	989	1004	930	930
query37	133	108	73	73
query38	4208	4125	4050	4050
query39	1468	1421	1416	1416
query40	208	111	97	97
query41	50	49	50	49
query42	131	106	104	104
query43	523	522	498	498
query44	1307	826	806	806
query45	178	171	162	162
query46	855	1016	623	623
query47	1863	1857	1777	1777
query48	375	398	301	301
query49	773	481	385	385
query50	628	664	380	380
query51	6925	7012	6998	6998
query52	103	99	92	92
query53	223	240	182	182
query54	476	484	406	406
query55	82	79	75	75
query56	262	278	239	239
query57	1199	1158	1109	1109
query58	236	229	226	226
query59	3159	3104	3055	3055
query60	295	258	251	251
query61	110	107	137	107
query62	842	763	724	724
query63	226	204	183	183
query64	4537	991	638	638
query65	3213	3120	3177	3120
query66	833	420	313	313
query67	15893	15826	15373	15373
query68	8825	719	514	514
query69	506	291	254	254
query70	1210	1111	1153	1111
query71	439	277	252	252
query72	6095	3843	3836	3836
query73	687	747	361	361
query74	10135	8936	9328	8936
query75	4489	3143	2641	2641
query76	3875	1176	755	755
query77	785	372	274	274
query78	10033	10166	9937	9937
query79	4980	752	626	626
query80	686	522	418	418
query81	470	269	221	221
query82	299	155	118	118
query83	206	170	158	158
query84	283	101	67	67
query85	723	347	298	298
query86	360	321	303	303
query87	4551	4371	4477	4371
query88	3654	2205	2169	2169
query89	409	323	303	303
query90	2095	187	186	186
query91	133	141	109	109
query92	70	57	53	53
query93	2344	820	519	519
query94	692	389	294	294
query95	324	257	249	249
query96	493	600	290	290
query97	2848	2934	2728	2728
query98	228	205	217	205
query99	1638	1476	1349	1349
Total cold run time: 295153 ms
Total hot run time: 188464 ms

@doris-robot

Copy link
Copy Markdown
ClickBench: Total hot run time: 31.34 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f855bdd4244f2dc423569e5d8972fe075d96bf4c, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.03	0.03
query3	0.23	0.07	0.07
query4	1.60	0.10	0.10
query5	0.41	0.42	0.40
query6	1.15	0.65	0.64
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.60	0.48	0.51
query10	0.55	0.55	0.56
query11	0.15	0.11	0.10
query12	0.14	0.11	0.10
query13	0.60	0.62	0.60
query14	2.72	2.87	2.75
query15	0.90	0.83	0.82
query16	0.40	0.39	0.37
query17	1.06	1.05	1.06
query18	0.22	0.21	0.20
query19	1.91	1.98	1.86
query20	0.01	0.02	0.01
query21	15.40	0.92	0.57
query22	0.77	0.76	0.67
query23	15.34	1.49	0.61
query24	3.35	0.97	2.94
query25	0.13	0.15	0.13
query26	0.27	0.14	0.13
query27	0.05	0.05	0.07
query28	14.12	1.44	1.04
query29	12.63	3.98	3.29
query30	0.26	0.09	0.06
query31	2.84	0.60	0.39
query32	3.24	0.55	0.47
query33	3.17	3.10	3.04
query34	16.50	5.08	4.46
query35	4.47	4.46	4.51
query36	0.65	0.48	0.49
query37	0.09	0.06	0.06
query38	0.04	0.04	0.04
query39	0.04	0.02	0.03
query40	0.18	0.13	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.52 s
Total hot run time: 31.34 s

@doris-robot

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 38.86% (10128/26062)
Line Coverage: 29.92% (85688/286399)
Region Coverage: 29.01% (43709/150665)
Branch Coverage: 25.55% (22311/87306)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f855bdd4244f2dc423569e5d8972fe075d96bf4c_f855bdd4244f2dc423569e5d8972fe075d96bf4c/report/index.html

@bobhan1

bobhan1 commented May 26, 2025

Copy link
Copy Markdown
Contributor Author

fixed in #50973

@bobhan1 bobhan1 closed this May 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants