Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[enhance](nereids)rewrite aggregate to limit when all group by key is uniform and not null, and there is no aggregate functions #46223

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

feiniaofeiafei
Copy link
Contributor

@feiniaofeiafei feiniaofeiafei commented Dec 31, 2024

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
select 1 c1 from test group by c; -> select 1 c1 from test limit 1;
c is uniform and aggregate can be eliminated

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Dec 31, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@feiniaofeiafei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33135 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 58827db55b38f87444c8c78d1d9991fcc8bb58c9, data reload: false

------ Round 1 ----------------------------------
q1	17596	6270	6170	6170
q2	2057	308	166	166
q3	10411	1287	735	735
q4	10225	886	439	439
q5	7519	2268	2055	2055
q6	213	184	153	153
q7	937	750	625	625
q8	9244	1514	1315	1315
q9	5299	4997	5158	4997
q10	6801	2336	1900	1900
q11	491	284	263	263
q12	352	395	224	224
q13	17807	3617	2952	2952
q14	239	230	211	211
q15	562	507	503	503
q16	619	631	593	593
q17	592	887	345	345
q18	7306	6500	6370	6370
q19	1996	1004	583	583
q20	312	328	186	186
q21	2916	2194	2040	2040
q22	368	334	310	310
Total cold run time: 103862 ms
Total hot run time: 33135 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6423	6474	6457	6457
q2	244	326	230	230
q3	2285	2647	2282	2282
q4	1500	1855	1359	1359
q5	4357	4799	5089	4799
q6	196	173	140	140
q7	2151	2008	1831	1831
q8	2705	2842	2808	2808
q9	7266	7226	7285	7226
q10	3066	3333	2832	2832
q11	607	532	505	505
q12	669	760	612	612
q13	3368	3778	3061	3061
q14	282	311	291	291
q15	602	506	500	500
q16	673	687	657	657
q17	1269	1731	1252	1252
q18	7766	7505	7146	7146
q19	855	941	1258	941
q20	1917	1984	1821	1821
q21	5508	5216	4866	4866
q22	609	620	601	601
Total cold run time: 54318 ms
Total hot run time: 52217 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191379 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 58827db55b38f87444c8c78d1d9991fcc8bb58c9, data reload: false

query1	965	394	381	381
query2	6546	2357	2359	2357
query3	6706	207	214	207
query4	33362	23536	23347	23347
query5	4334	607	467	467
query6	287	206	198	198
query7	4624	488	300	300
query8	304	250	251	250
query9	9449	2728	2722	2722
query10	456	322	264	264
query11	18126	15450	15342	15342
query12	151	110	105	105
query13	1675	536	396	396
query14	10139	7731	7245	7245
query15	289	198	187	187
query16	8121	565	445	445
query17	1626	751	568	568
query18	1331	370	286	286
query19	213	173	162	162
query20	126	139	112	112
query21	202	117	106	106
query22	4506	4497	4398	4398
query23	34258	33695	33671	33671
query24	6348	2298	2249	2249
query25	496	440	404	404
query26	901	278	172	172
query27	1991	451	334	334
query28	5194	2444	2403	2403
query29	645	545	419	419
query30	235	180	150	150
query31	970	912	816	816
query32	87	59	58	58
query33	490	365	298	298
query34	758	831	509	509
query35	799	804	734	734
query36	1025	1051	965	965
query37	114	97	75	75
query38	4432	4165	4201	4165
query39	1488	1434	1455	1434
query40	203	112	98	98
query41	47	45	46	45
query42	118	99	101	99
query43	503	532	494	494
query44	1234	790	786	786
query45	183	183	174	174
query46	865	1035	634	634
query47	1931	1938	1904	1904
query48	382	407	321	321
query49	731	510	384	384
query50	624	635	387	387
query51	7131	7102	7009	7009
query52	102	97	89	89
query53	220	248	181	181
query54	480	471	404	404
query55	78	82	77	77
query56	263	252	242	242
query57	1213	1175	1145	1145
query58	240	224	227	224
query59	2993	3384	3246	3246
query60	264	262	252	252
query61	118	113	121	113
query62	890	799	776	776
query63	224	189	190	189
query64	3713	1004	652	652
query65	3288	3187	3216	3187
query66	945	422	305	305
query67	15917	15857	15577	15577
query68	9143	746	517	517
query69	471	291	251	251
query70	1229	1178	1100	1100
query71	427	281	256	256
query72	5923	3817	3901	3817
query73	664	752	355	355
query74	10252	9363	9279	9279
query75	4416	3174	2646	2646
query76	4098	1289	761	761
query77	773	365	281	281
query78	10162	10168	9377	9377
query79	3288	898	580	580
query80	728	524	442	442
query81	499	265	232	232
query82	660	150	125	125
query83	166	167	145	145
query84	238	90	74	74
query85	804	436	301	301
query86	405	310	273	273
query87	4417	4425	4430	4425
query88	4802	2229	2197	2197
query89	420	339	300	300
query90	1836	187	187	187
query91	136	150	104	104
query92	70	58	53	53
query93	1735	877	517	517
query94	665	392	288	288
query95	341	262	250	250
query96	480	611	285	285
query97	2764	2806	2661	2661
query98	233	200	197	197
query99	1643	1562	1408	1408
Total cold run time: 292557 ms
Total hot run time: 191379 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.27 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 58827db55b38f87444c8c78d1d9991fcc8bb58c9, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.24	0.07	0.07
query4	1.61	0.11	0.10
query5	0.42	0.42	0.39
query6	1.14	0.64	0.64
query7	0.03	0.02	0.01
query8	0.04	0.04	0.03
query9	0.55	0.51	0.51
query10	0.55	0.53	0.55
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.59	0.62
query14	2.81	2.78	2.72
query15	0.88	0.83	0.82
query16	0.37	0.38	0.37
query17	1.02	1.05	1.02
query18	0.23	0.20	0.20
query19	1.96	1.80	2.03
query20	0.01	0.02	0.01
query21	15.37	0.91	0.57
query22	0.78	0.72	0.66
query23	15.33	1.45	0.55
query24	3.76	1.38	0.97
query25	0.16	0.18	0.12
query26	0.25	0.16	0.14
query27	0.07	0.05	0.04
query28	13.64	1.57	1.04
query29	12.58	3.94	3.28
query30	0.25	0.09	0.06
query31	2.83	0.60	0.39
query32	3.22	0.54	0.47
query33	3.06	3.05	3.16
query34	16.70	5.11	4.52
query35	4.59	4.50	4.52
query36	0.64	0.49	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.17	0.14	0.13
query41	0.07	0.02	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.57 s
Total hot run time: 31.27 s

@feiniaofeiafei
Copy link
Contributor Author

run buildall

1 similar comment
@feiniaofeiafei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32695 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9a613239a96f40f43ffea9cf4d355c02bc79de01, data reload: false

------ Round 1 ----------------------------------
q1	17595	6118	6062	6062
q2	2046	298	175	175
q3	10424	1244	766	766
q4	10516	864	433	433
q5	9236	2205	1963	1963
q6	212	186	149	149
q7	885	756	597	597
q8	9238	1355	1168	1168
q9	5249	4909	4960	4909
q10	6766	2312	1860	1860
q11	492	282	255	255
q12	348	355	222	222
q13	17767	3587	3018	3018
q14	244	241	228	228
q15	567	507	501	501
q16	648	614	595	595
q17	581	857	355	355
q18	6944	6438	6406	6406
q19	1569	964	545	545
q20	315	315	189	189
q21	2912	2179	1999	1999
q22	366	342	300	300
Total cold run time: 104920 ms
Total hot run time: 32695 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6234	6256	6200	6200
q2	241	323	226	226
q3	2269	2643	2327	2327
q4	1450	1829	1343	1343
q5	4402	4758	4862	4758
q6	192	195	145	145
q7	2051	1954	1814	1814
q8	2666	2853	2705	2705
q9	7284	7254	7320	7254
q10	3083	3366	2764	2764
q11	565	519	492	492
q12	676	813	634	634
q13	3358	3725	3073	3073
q14	278	321	269	269
q15	581	535	497	497
q16	681	694	653	653
q17	1249	1728	1288	1288
q18	7654	7386	7329	7329
q19	858	1225	1065	1065
q20	2010	2019	1902	1902
q21	5869	5357	4900	4900
q22	646	632	605	605
Total cold run time: 54297 ms
Total hot run time: 52243 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196823 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9a613239a96f40f43ffea9cf4d355c02bc79de01, data reload: false

query1	1281	957	943	943
query2	6485	2378	2365	2365
query3	11033	4644	4583	4583
query4	32984	23676	23454	23454
query5	4554	620	455	455
query6	278	203	176	176
query7	3981	497	308	308
query8	297	236	244	236
query9	9230	2631	2617	2617
query10	468	311	254	254
query11	17972	15475	15284	15284
query12	165	117	108	108
query13	1577	543	401	401
query14	9069	6897	8303	6897
query15	265	210	206	206
query16	8609	628	469	469
query17	1635	771	616	616
query18	2151	416	325	325
query19	201	184	166	166
query20	117	129	129	129
query21	209	129	111	111
query22	4485	4579	4623	4579
query23	34252	33527	33810	33527
query24	6466	2297	2334	2297
query25	492	468	394	394
query26	811	278	163	163
query27	2149	471	344	344
query28	5681	2439	2429	2429
query29	671	590	432	432
query30	209	184	152	152
query31	989	898	858	858
query32	87	62	66	62
query33	475	347	324	324
query34	788	864	536	536
query35	828	835	736	736
query36	1029	1077	984	984
query37	123	106	79	79
query38	4388	4409	4154	4154
query39	1527	1476	1472	1472
query40	221	119	101	101
query41	46	47	43	43
query42	123	107	111	107
query43	534	542	507	507
query44	1367	836	836	836
query45	187	187	172	172
query46	906	1066	649	649
query47	2012	1996	1951	1951
query48	399	433	325	325
query49	715	484	396	396
query50	655	688	399	399
query51	7241	7156	7162	7156
query52	103	109	97	97
query53	237	266	192	192
query54	488	507	416	416
query55	82	79	80	79
query56	263	264	252	252
query57	1279	1232	1223	1223
query58	246	235	231	231
query59	3222	3307	3310	3307
query60	282	268	267	267
query61	110	107	106	106
query62	884	798	774	774
query63	230	196	196	196
query64	3286	1039	665	665
query65	3309	3285	3306	3285
query66	921	420	308	308
query67	16654	15738	15511	15511
query68	9699	776	527	527
query69	476	295	249	249
query70	1192	1148	1151	1148
query71	452	320	258	258
query72	5997	3808	3801	3801
query73	668	754	368	368
query74	10339	9208	8978	8978
query75	4592	3105	2672	2672
query76	4500	1193	775	775
query77	915	445	285	285
query78	10307	10539	9363	9363
query79	5415	871	593	593
query80	724	517	436	436
query81	476	270	228	228
query82	320	149	122	122
query83	191	160	149	149
query84	281	89	75	75
query85	804	357	302	302
query86	357	317	281	281
query87	4644	4710	4508	4508
query88	3895	2254	2172	2172
query89	436	328	303	303
query90	2119	184	185	184
query91	127	137	103	103
query92	67	54	50	50
query93	3381	878	537	537
query94	671	380	293	293
query95	352	261	241	241
query96	502	612	273	273
query97	2779	2824	2728	2728
query98	232	208	204	204
query99	1645	1578	1441	1441
Total cold run time: 302568 ms
Total hot run time: 196823 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.21 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9a613239a96f40f43ffea9cf4d355c02bc79de01, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.04	0.03
query3	0.24	0.08	0.07
query4	1.60	0.11	0.11
query5	0.43	0.41	0.41
query6	1.17	0.65	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.59	0.50	0.54
query10	0.56	0.57	0.54
query11	0.15	0.10	0.10
query12	0.14	0.12	0.11
query13	0.60	0.59	0.59
query14	2.81	2.83	2.75
query15	0.89	0.81	0.82
query16	0.39	0.38	0.38
query17	1.06	1.06	0.99
query18	0.24	0.20	0.21
query19	1.94	1.89	2.03
query20	0.01	0.00	0.01
query21	15.36	0.90	0.57
query22	0.76	0.99	0.64
query23	15.17	1.43	0.57
query24	3.25	0.91	1.96
query25	0.20	0.26	0.11
query26	0.24	0.14	0.14
query27	0.05	0.07	0.04
query28	13.70	1.51	1.05
query29	12.55	3.96	3.23
query30	0.25	0.09	0.08
query31	2.81	0.59	0.40
query32	3.23	0.54	0.47
query33	3.10	3.08	3.08
query34	16.63	5.09	4.50
query35	4.45	4.43	4.45
query36	0.64	0.48	0.47
query37	0.10	0.07	0.06
query38	0.04	0.04	0.03
query39	0.03	0.03	0.02
query40	0.16	0.14	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 105.86 s
Total hot run time: 31.21 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants