Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[opt](inverted index) Opt Inverted Index Composite File Order #46120

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

zzzxl1993
Copy link
Contributor

@zzzxl1993 zzzxl1993 commented Dec 27, 2024

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

  1. Optimizing Composite File Order Improves File Cache Efficiency

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zzzxl1993
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32704 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6472174a9dfab849fd28bc5c4a619af9f8b48a71, data reload: false

------ Round 1 ----------------------------------
q1	17581	6206	6045	6045
q2	2044	313	173	173
q3	10406	1261	747	747
q4	10225	887	444	444
q5	7684	2220	2022	2022
q6	208	186	152	152
q7	917	756	597	597
q8	9237	1408	1196	1196
q9	5381	5063	4984	4984
q10	6763	2321	1858	1858
q11	485	276	279	276
q12	338	362	218	218
q13	17784	3629	2956	2956
q14	235	231	205	205
q15	562	515	498	498
q16	626	616	580	580
q17	572	876	333	333
q18	7092	6388	6375	6375
q19	2657	977	560	560
q20	296	318	184	184
q21	2928	2231	1990	1990
q22	364	332	311	311
Total cold run time: 104385 ms
Total hot run time: 32704 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6349	6204	6484	6204
q2	235	321	225	225
q3	2263	2652	2336	2336
q4	1454	1906	1407	1407
q5	4366	4741	4894	4741
q6	183	177	139	139
q7	2115	1998	1823	1823
q8	2632	2814	2668	2668
q9	7415	7500	7461	7461
q10	3112	3400	2772	2772
q11	579	516	512	512
q12	702	804	625	625
q13	3438	3749	3134	3134
q14	301	313	277	277
q15	577	530	502	502
q16	671	691	671	671
q17	1202	1731	1248	1248
q18	7760	7616	7525	7525
q19	793	1142	1135	1135
q20	1982	2037	1861	1861
q21	5812	5214	5084	5084
q22	594	597	584	584
Total cold run time: 54535 ms
Total hot run time: 52934 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197234 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6472174a9dfab849fd28bc5c4a619af9f8b48a71, data reload: false

query1	1264	1000	927	927
query2	6122	2484	2315	2315
query3	11065	4958	5167	4958
query4	32947	23878	23652	23652
query5	4785	613	466	466
query6	282	195	188	188
query7	3994	492	311	311
query8	312	262	246	246
query9	9442	2741	2721	2721
query10	466	302	255	255
query11	18229	15770	15046	15046
query12	161	106	105	105
query13	1656	551	426	426
query14	11109	6765	8467	6765
query15	260	216	191	191
query16	7312	666	520	520
query17	1522	762	568	568
query18	1324	372	328	328
query19	194	183	173	173
query20	124	122	114	114
query21	215	123	110	110
query22	4847	4804	4507	4507
query23	34618	33633	33731	33633
query24	6176	2286	2355	2286
query25	489	472	377	377
query26	885	277	160	160
query27	2073	495	334	334
query28	5944	2524	2470	2470
query29	626	547	420	420
query30	216	188	155	155
query31	1008	928	850	850
query32	89	55	67	55
query33	498	347	296	296
query34	774	856	521	521
query35	819	858	754	754
query36	1037	1114	984	984
query37	112	103	74	74
query38	4276	4318	4495	4318
query39	1520	1485	1481	1481
query40	218	112	100	100
query41	46	50	44	44
query42	116	136	110	110
query43	519	527	486	486
query44	1311	852	846	846
query45	191	183	176	176
query46	894	1073	666	666
query47	2024	2061	1989	1989
query48	415	416	343	343
query49	699	474	402	402
query50	626	660	399	399
query51	7379	7304	7305	7304
query52	107	102	94	94
query53	226	257	182	182
query54	467	488	412	412
query55	84	77	78	77
query56	267	252	264	252
query57	1232	1265	1202	1202
query58	252	217	215	215
query59	3216	3489	3235	3235
query60	272	269	258	258
query61	108	100	110	100
query62	861	827	772	772
query63	239	197	203	197
query64	3954	1025	652	652
query65	3321	3238	3250	3238
query66	858	410	307	307
query67	16538	15823	15460	15460
query68	9842	745	510	510
query69	495	297	255	255
query70	1258	1125	1163	1125
query71	430	275	253	253
query72	5963	3853	3764	3764
query73	1286	760	361	361
query74	9875	9052	9128	9052
query75	4697	3168	2630	2630
query76	5629	1182	771	771
query77	1028	356	272	272
query78	10127	10335	9498	9498
query79	2574	899	605	605
query80	633	514	416	416
query81	490	264	229	229
query82	663	152	128	128
query83	160	157	144	144
query84	247	90	68	68
query85	770	351	301	301
query86	338	316	310	310
query87	4646	4870	4381	4381
query88	3455	2231	2203	2203
query89	427	340	298	298
query90	1915	186	184	184
query91	131	133	107	107
query92	69	54	53	53
query93	1120	884	524	524
query94	649	412	298	298
query95	328	259	257	257
query96	481	605	283	283
query97	2745	2855	2732	2732
query98	216	204	198	198
query99	1781	1563	1446	1446
Total cold run time: 299700 ms
Total hot run time: 197234 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.31 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6472174a9dfab849fd28bc5c4a619af9f8b48a71, data reload: false

query1	0.05	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.62	0.10	0.10
query5	0.42	0.41	0.43
query6	1.18	0.64	0.65
query7	0.03	0.02	0.02
query8	0.04	0.03	0.04
query9	0.59	0.49	0.49
query10	0.57	0.59	0.56
query11	0.16	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	2.70	2.73	2.73
query15	0.89	0.82	0.83
query16	0.39	0.39	0.38
query17	1.10	1.07	1.01
query18	0.22	0.20	0.20
query19	1.98	1.78	1.99
query20	0.02	0.01	0.01
query21	15.36	0.91	0.57
query22	0.75	0.78	0.67
query23	15.32	1.43	0.50
query24	3.44	1.91	1.05
query25	0.21	0.05	0.12
query26	0.21	0.15	0.13
query27	0.05	0.04	0.05
query28	14.13	1.49	1.06
query29	12.58	4.01	3.28
query30	0.25	0.09	0.06
query31	2.84	0.62	0.39
query32	3.23	0.53	0.45
query33	3.02	3.11	3.10
query34	16.81	5.16	4.58
query35	4.56	4.47	4.50
query36	0.66	0.49	0.48
query37	0.10	0.06	0.06
query38	0.04	0.04	0.04
query39	0.04	0.03	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.04	0.02
Total cold run time: 106.93 s
Total hot run time: 31.31 s

@zzzxl1993
Copy link
Contributor Author

run buildall

@zzzxl1993
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32438 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 148112bb9ade0ab8bcb82af50337064e451f71c9, data reload: false

------ Round 1 ----------------------------------
q1	17584	6159	6180	6159
q2	2042	291	166	166
q3	10525	1213	743	743
q4	10218	882	437	437
q5	7511	2174	1953	1953
q6	204	182	146	146
q7	889	751	596	596
q8	9233	1329	1125	1125
q9	5201	4863	4943	4863
q10	6806	2304	1878	1878
q11	503	288	260	260
q12	350	367	220	220
q13	18281	3588	2967	2967
q14	244	243	217	217
q15	567	504	500	500
q16	645	633	575	575
q17	567	846	319	319
q18	6893	6366	6298	6298
q19	2797	973	555	555
q20	304	306	183	183
q21	2921	2157	1971	1971
q22	359	337	307	307
Total cold run time: 104644 ms
Total hot run time: 32438 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6353	6283	6217	6217
q2	236	329	229	229
q3	2270	2619	2289	2289
q4	1417	1833	1360	1360
q5	4363	4781	4827	4781
q6	180	181	140	140
q7	2086	1954	1801	1801
q8	2675	2872	2679	2679
q9	7326	7298	7336	7298
q10	3074	3398	2768	2768
q11	572	522	485	485
q12	677	792	603	603
q13	3357	3708	3189	3189
q14	279	293	292	292
q15	571	527	513	513
q16	648	697	662	662
q17	1240	1730	1263	1263
q18	7549	7556	7274	7274
q19	866	1158	1086	1086
q20	2079	2054	1868	1868
q21	5623	5542	4926	4926
q22	634	609	626	609
Total cold run time: 54075 ms
Total hot run time: 52332 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.88% (10118/26021)
Line Coverage: 29.90% (85578/286238)
Region Coverage: 29.01% (43708/150652)
Branch Coverage: 25.55% (22306/87294)
Coverage Report: http://coverage.selectdb-in.cc/coverage/148112bb9ade0ab8bcb82af50337064e451f71c9_148112bb9ade0ab8bcb82af50337064e451f71c9/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 197116 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 148112bb9ade0ab8bcb82af50337064e451f71c9, data reload: false

query1	1301	942	943	942
query2	6460	2516	2246	2246
query3	11112	4795	4952	4795
query4	32704	23984	23517	23517
query5	4578	625	448	448
query6	298	193	184	184
query7	3976	497	309	309
query8	290	244	225	225
query9	9301	2688	2669	2669
query10	441	300	244	244
query11	17816	15444	15406	15406
query12	153	106	103	103
query13	1585	578	453	453
query14	10039	6930	7791	6930
query15	247	214	190	190
query16	7816	646	469	469
query17	1571	785	599	599
query18	2108	405	313	313
query19	197	204	183	183
query20	130	114	115	114
query21	203	130	116	116
query22	4680	4566	4276	4276
query23	36325	33752	33556	33556
query24	6635	2340	2319	2319
query25	475	458	421	421
query26	795	252	148	148
query27	2225	494	345	345
query28	5094	2461	2453	2453
query29	608	573	464	464
query30	208	196	156	156
query31	1013	913	899	899
query32	72	64	55	55
query33	463	352	313	313
query34	783	877	528	528
query35	804	824	742	742
query36	1032	1066	939	939
query37	137	98	81	81
query38	4131	4304	4220	4220
query39	1525	1456	1465	1456
query40	207	121	104	104
query41	44	46	43	43
query42	114	100	101	100
query43	500	545	507	507
query44	1356	844	845	844
query45	182	176	170	170
query46	885	1059	664	664
query47	1970	1961	1928	1928
query48	406	433	330	330
query49	714	483	406	406
query50	686	700	403	403
query51	7377	7315	7232	7232
query52	111	106	99	99
query53	233	269	195	195
query54	514	548	443	443
query55	83	84	86	84
query56	272	282	271	271
query57	1220	1232	1176	1176
query58	263	236	254	236
query59	3130	3450	3093	3093
query60	264	273	272	272
query61	128	107	115	107
query62	835	813	762	762
query63	230	194	193	193
query64	3129	1085	667	667
query65	3346	3289	3256	3256
query66	742	416	302	302
query67	16308	15851	15666	15666
query68	8900	789	522	522
query69	470	295	255	255
query70	1221	1152	1071	1071
query71	478	285	257	257
query72	6204	3946	3791	3791
query73	658	767	362	362
query74	9905	9033	9238	9033
query75	4445	3176	2664	2664
query76	4271	1198	811	811
query77	801	372	273	273
query78	10192	10242	9544	9544
query79	2758	879	597	597
query80	583	520	447	447
query81	483	267	239	239
query82	640	152	127	127
query83	159	166	146	146
query84	244	89	76	76
query85	802	355	365	355
query86	347	330	304	304
query87	4528	4376	4338	4338
query88	3570	2203	2203	2203
query89	416	333	305	305
query90	1796	188	194	188
query91	136	135	105	105
query92	66	56	52	52
query93	999	912	534	534
query94	648	384	284	284
query95	325	266	257	257
query96	508	607	282	282
query97	2721	2847	2686	2686
query98	226	208	200	200
query99	1642	1544	1429	1429
Total cold run time: 295722 ms
Total hot run time: 197116 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.59 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 148112bb9ade0ab8bcb82af50337064e451f71c9, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.03	0.03
query3	0.23	0.07	0.06
query4	1.61	0.10	0.11
query5	0.42	0.40	0.39
query6	1.16	0.65	0.65
query7	0.02	0.01	0.02
query8	0.04	0.04	0.03
query9	0.57	0.49	0.50
query10	0.56	0.57	0.55
query11	0.14	0.10	0.10
query12	0.14	0.12	0.11
query13	0.61	0.62	0.60
query14	2.69	2.84	2.73
query15	0.90	0.83	0.83
query16	0.37	0.39	0.36
query17	1.06	1.05	1.03
query18	0.23	0.21	0.21
query19	1.88	1.73	1.96
query20	0.01	0.02	0.01
query21	15.38	0.91	0.59
query22	0.75	0.85	0.72
query23	15.17	1.47	0.55
query24	2.51	1.29	1.25
query25	0.21	0.16	0.06
query26	0.29	0.15	0.14
query27	0.09	0.04	0.04
query28	13.80	1.55	1.05
query29	12.59	3.92	3.29
query30	0.25	0.10	0.06
query31	2.82	0.62	0.39
query32	3.23	0.55	0.47
query33	3.18	3.18	3.18
query34	16.68	5.17	4.51
query35	4.53	4.46	4.48
query36	0.62	0.50	0.48
query37	0.11	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.03	0.03
query42	0.04	0.03	0.02
query43	0.04	0.03	0.02
Total cold run time: 105.36 s
Total hot run time: 31.59 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants