Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](hive)fix hive catalog can't read acid table when quety rowid column. #46130

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

hubgeter
Copy link
Contributor

@hubgeter hubgeter commented Dec 29, 2024

What problem does this PR solve?

Problem Summary:
fix case :

create table in hive : 
CREATE TABLE `baby`(
  `rowid` int,
  `fname` varchar(50),
  `gender` varchar(2),
  `totcount` int)
STORED AS ORC
TBLPROPERTIES (
  'orc.compress'='snappy',
  'bucketing_version'='2',
  'transactional_properties'='default',
  'transactional'='true')

then query in doris : 
select rowid,fname,gender,totcount from baby limit 2;

You will get the following error: 
[INTERNAL_ERROR]cur path:  xxxx . Failed to create orc row reader. 
reason = Invalid column selected row.rowId. 
Valid names are bucket, currentTransaction, operation, originalTransaction, row, row.fname, row.gender, row.rowid, row.totcount, rowId

The reason is that the hive transaction table will add columns to the orc file to implement the delete & acid operation. The corresponding data in the column is stored in the row struct. The file structure here is as follows:

struct< operation:int, originalTransaction:bigint, bucket:int, rowId:bigint, currentTransaction:bigint, 
row:struct<rowid:int,fname:varchar(50),gender:varchar(2),totcount:int>>

Release note

Fixed the issue where an error would occur when reading Hive transaction tables with rowid, bucket, originalTransaction columns

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hubgeter
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32463 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e4da2c05729235e9bac8e7cc2fff575bbddc51f5, data reload: false

------ Round 1 ----------------------------------
q1	17583	6109	6004	6004
q2	2042	302	165	165
q3	10456	1276	746	746
q4	10284	871	435	435
q5	8863	2122	1938	1938
q6	207	180	148	148
q7	895	779	624	624
q8	9258	1373	1188	1188
q9	5434	4938	5003	4938
q10	6824	2295	1847	1847
q11	479	283	256	256
q12	340	358	219	219
q13	17778	3564	2946	2946
q14	242	236	218	218
q15	554	518	493	493
q16	640	617	598	598
q17	567	850	330	330
q18	7064	6455	6365	6365
q19	2366	965	553	553
q20	303	310	189	189
q21	2825	2172	1953	1953
q22	356	333	310	310
Total cold run time: 105360 ms
Total hot run time: 32463 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6273	6170	6177	6170
q2	240	336	229	229
q3	2244	2633	2316	2316
q4	1421	1819	1350	1350
q5	4307	4716	4800	4716
q6	182	180	147	147
q7	2071	1965	1813	1813
q8	2630	2804	2678	2678
q9	7302	7306	7207	7207
q10	3054	3344	2751	2751
q11	557	516	481	481
q12	649	780	640	640
q13	3389	3741	3114	3114
q14	273	315	282	282
q15	587	519	516	516
q16	650	706	644	644
q17	1255	1731	1246	1246
q18	7666	7388	7297	7297
q19	819	882	1148	882
q20	1974	2021	1894	1894
q21	5651	5377	4821	4821
q22	595	638	623	623
Total cold run time: 53789 ms
Total hot run time: 51817 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.89% (10118/26015)
Line Coverage: 29.88% (85499/286147)
Region Coverage: 29.02% (43709/150640)
Branch Coverage: 25.55% (22300/87290)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e4da2c05729235e9bac8e7cc2fff575bbddc51f5_e4da2c05729235e9bac8e7cc2fff575bbddc51f5/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 196545 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e4da2c05729235e9bac8e7cc2fff575bbddc51f5, data reload: false

query1	1315	938	949	938
query2	6503	2245	2321	2245
query3	10933	4596	4854	4596
query4	33284	23643	23788	23643
query5	4729	603	459	459
query6	281	203	207	203
query7	3999	481	305	305
query8	310	251	243	243
query9	9565	2741	2744	2741
query10	479	309	247	247
query11	18557	15320	15128	15128
query12	168	115	109	109
query13	1704	586	445	445
query14	12061	8987	7051	7051
query15	256	209	193	193
query16	7625	583	500	500
query17	1496	741	575	575
query18	1372	370	321	321
query19	226	196	162	162
query20	127	139	109	109
query21	217	131	116	116
query22	4634	4644	4431	4431
query23	34247	33517	33574	33517
query24	6173	2317	2303	2303
query25	459	440	411	411
query26	716	282	156	156
query27	2089	476	336	336
query28	5510	2483	2478	2478
query29	537	551	432	432
query30	229	185	152	152
query31	968	922	856	856
query32	73	59	59	59
query33	471	381	311	311
query34	829	883	527	527
query35	806	823	756	756
query36	1022	1062	990	990
query37	121	104	78	78
query38	4297	4326	4231	4231
query39	1526	1504	1487	1487
query40	217	116	104	104
query41	45	46	43	43
query42	121	105	114	105
query43	534	517	498	498
query44	1328	864	834	834
query45	191	172	160	160
query46	895	1079	680	680
query47	1965	1967	1944	1944
query48	387	424	324	324
query49	697	474	412	412
query50	655	673	396	396
query51	7299	7195	7332	7195
query52	100	99	95	95
query53	233	266	190	190
query54	494	497	415	415
query55	84	83	81	81
query56	257	306	252	252
query57	1244	1262	1139	1139
query58	235	219	238	219
query59	3133	3216	3106	3106
query60	275	273	258	258
query61	112	110	104	104
query62	847	802	747	747
query63	227	203	193	193
query64	2619	1012	654	654
query65	3325	3340	3261	3261
query66	711	413	315	315
query67	16562	15809	15549	15549
query68	9588	749	521	521
query69	502	310	254	254
query70	1257	1187	1142	1142
query71	455	285	256	256
query72	5912	3845	3826	3826
query73	694	740	364	364
query74	10287	9033	9196	9033
query75	4413	3151	2662	2662
query76	5552	1189	766	766
query77	1049	355	293	293
query78	10171	10158	9309	9309
query79	2926	897	600	600
query80	776	516	434	434
query81	476	266	232	232
query82	242	158	121	121
query83	199	162	144	144
query84	281	93	67	67
query85	813	369	300	300
query86	339	324	280	280
query87	4736	4387	4270	4270
query88	3599	2246	2203	2203
query89	419	338	290	290
query90	2020	193	190	190
query91	136	131	102	102
query92	65	57	52	52
query93	1897	874	526	526
query94	678	406	274	274
query95	340	272	257	257
query96	486	603	285	285
query97	2693	2807	2741	2741
query98	217	202	190	190
query99	1671	1575	1448	1448
Total cold run time: 299595 ms
Total hot run time: 196545 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.14 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e4da2c05729235e9bac8e7cc2fff575bbddc51f5, data reload: false

query1	0.04	0.05	0.04
query2	0.06	0.04	0.03
query3	0.24	0.06	0.07
query4	1.62	0.11	0.10
query5	0.41	0.43	0.42
query6	1.18	0.66	0.66
query7	0.03	0.02	0.01
query8	0.04	0.04	0.03
query9	0.59	0.50	0.52
query10	0.56	0.55	0.55
query11	0.15	0.10	0.11
query12	0.14	0.11	0.11
query13	0.61	0.62	0.59
query14	2.85	2.81	2.74
query15	0.91	0.83	0.82
query16	0.39	0.39	0.38
query17	1.07	1.07	1.06
query18	0.21	0.21	0.21
query19	1.90	1.80	1.99
query20	0.02	0.01	0.01
query21	15.38	0.97	0.59
query22	0.76	0.86	0.68
query23	15.25	1.48	0.52
query24	2.94	1.25	0.77
query25	0.23	0.21	0.18
query26	0.41	0.16	0.14
query27	0.06	0.05	0.06
query28	13.33	1.53	1.05
query29	12.55	3.91	3.24
query30	0.25	0.10	0.06
query31	2.82	0.58	0.38
query32	3.27	0.55	0.46
query33	3.08	3.08	3.09
query34	16.94	5.14	4.47
query35	4.50	4.45	4.44
query36	0.67	0.49	0.50
query37	0.09	0.07	0.06
query38	0.04	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.14
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 105.95 s
Total hot run time: 31.14 s

@hubgeter hubgeter changed the title [fix](hive)fix hive catalog can't read acid table when column = rowid [fix](hive)fix hive catalog can't read acid table when quety rowid column. Dec 29, 2024
@hubgeter hubgeter marked this pull request as ready for review December 29, 2024 12:42
@morningman
Copy link
Contributor

This pull request primarily focuses on enhancing the OrcReader class to support special column names in transactional Hive tables. The changes include modifying various methods to handle column name transformations and adding new test cases to verify the functionality.

Key changes include:

Enhancements to OrcReader for special column names:

  • Modified _init_orc_cols and _init_select_types methods to handle special column names by adding pre and recursion parameters. (be/src/vec/exec/format/orc/vorc_reader.cpp, [1] [2]
  • Updated _make_orc_literal, _check_slot_can_push_down, and other predicate-building methods to use transformed column names if they exist in _table_col_to_file_col. (be/src/vec/exec/format/orc/vorc_reader.cpp, [1] [2] [3] [4] [5] [6] [7]
  • Adjusted _fill_batch_vec method to handle special column names during batch processing. (be/src/vec/exec/format/orc/vorc_reader.cpp, be/src/vec/exec/format/orc/vorc_reader.cppL2071-R2120)

Support for transactional Hive reader:

  • Added _table_col_to_file_col mapping in TransactionalHiveReader to translate table column names to file column names. (be/src/vec/exec/format/table/transactional_hive_reader.cpp, [1] [2] [3]

Test cases:

  • Added new test cases in run25.hql and test_transactional_hive.out to verify the handling of special column names in transactional Hive tables. (docker/thirdparties/docker-compose/hive/scripts/create_preinstalled_scripts/run25.hql, [1] regression-test/data/external_table_p0/hive/test_transactional_hive.out, [2]

@hubgeter
Copy link
Contributor Author

run buildall

1 similar comment
@hubgeter
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32847 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit de08108c149ada052012925267835616e13d073d, data reload: false

------ Round 1 ----------------------------------
q1	17583	6251	6086	6086
q2	2045	305	179	179
q3	10406	1212	794	794
q4	10206	877	435	435
q5	7525	2208	1999	1999
q6	206	178	149	149
q7	897	756	615	615
q8	9230	1375	1187	1187
q9	5306	4988	4945	4945
q10	6790	2337	1896	1896
q11	485	292	263	263
q12	354	375	221	221
q13	18126	3694	2991	2991
q14	232	233	203	203
q15	565	513	492	492
q16	620	631	602	602
q17	581	861	338	338
q18	6870	6349	6405	6349
q19	1998	973	589	589
q20	305	328	186	186
q21	2947	2241	2018	2018
q22	378	336	310	310
Total cold run time: 103655 ms
Total hot run time: 32847 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6284	6245	6232	6232
q2	242	328	238	238
q3	2190	2720	2328	2328
q4	1435	1847	1394	1394
q5	4362	4752	4800	4752
q6	187	180	147	147
q7	2088	1967	1830	1830
q8	2668	2852	2708	2708
q9	7311	7244	7283	7244
q10	3087	3363	2857	2857
q11	590	512	492	492
q12	645	727	628	628
q13	3424	3740	3158	3158
q14	301	314	284	284
q15	573	540	524	524
q16	656	699	671	671
q17	1220	1742	1276	1276
q18	7841	7502	7513	7502
q19	903	1215	1114	1114
q20	2001	2030	1922	1922
q21	5894	5286	4833	4833
q22	626	599	579	579
Total cold run time: 54528 ms
Total hot run time: 52713 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.89% (10120/26024)
Line Coverage: 29.89% (85533/286197)
Region Coverage: 29.00% (43707/150690)
Branch Coverage: 25.53% (22296/87322)
Coverage Report: http://coverage.selectdb-in.cc/coverage/de08108c149ada052012925267835616e13d073d_de08108c149ada052012925267835616e13d073d/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 197678 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit de08108c149ada052012925267835616e13d073d, data reload: false

query1	1303	995	946	946
query2	6464	2400	2298	2298
query3	11104	4774	4933	4774
query4	33166	23548	23755	23548
query5	3878	657	456	456
query6	272	228	213	213
query7	3973	499	302	302
query8	290	250	233	233
query9	9458	2751	2735	2735
query10	421	307	243	243
query11	17926	15366	15181	15181
query12	160	107	103	103
query13	1573	539	420	420
query14	9426	6926	7783	6926
query15	334	215	196	196
query16	8683	643	483	483
query17	1546	790	615	615
query18	2159	429	348	348
query19	198	206	173	173
query20	120	124	116	116
query21	207	134	111	111
query22	4601	4668	4508	4508
query23	35355	33672	33546	33546
query24	6729	2330	2379	2330
query25	484	473	395	395
query26	1202	285	158	158
query27	2791	488	329	329
query28	5457	2492	2522	2492
query29	735	561	427	427
query30	207	191	150	150
query31	983	973	849	849
query32	86	66	57	57
query33	501	364	304	304
query34	824	863	543	543
query35	812	822	752	752
query36	1038	1074	980	980
query37	123	110	80	80
query38	4346	4354	4160	4160
query39	1534	1545	1624	1545
query40	211	122	106	106
query41	45	46	44	44
query42	140	109	111	109
query43	528	532	495	495
query44	1348	829	822	822
query45	185	177	181	177
query46	913	1084	691	691
query47	2000	2001	1902	1902
query48	395	428	346	346
query49	758	501	413	413
query50	651	679	431	431
query51	7372	7187	7283	7187
query52	106	104	92	92
query53	242	264	189	189
query54	497	494	420	420
query55	88	86	77	77
query56	265	283	259	259
query57	1253	1275	1159	1159
query58	242	227	229	227
query59	3150	3228	3139	3139
query60	271	274	268	268
query61	118	108	106	106
query62	887	791	747	747
query63	239	196	196	196
query64	4270	1048	673	673
query65	3369	3296	3296	3296
query66	793	409	314	314
query67	16723	15974	15623	15623
query68	10288	773	522	522
query69	485	298	257	257
query70	1220	1162	1155	1155
query71	433	287	253	253
query72	5908	3845	3906	3845
query73	1889	766	366	366
query74	10367	9324	9187	9187
query75	4447	3158	2700	2700
query76	5621	1224	793	793
query77	1035	371	282	282
query78	10209	10162	9435	9435
query79	2904	910	612	612
query80	726	522	471	471
query81	481	269	233	233
query82	244	153	125	125
query83	197	169	155	155
query84	303	92	143	92
query85	751	355	303	303
query86	342	326	297	297
query87	4443	4590	4563	4563
query88	3601	2258	2229	2229
query89	442	326	304	304
query90	1991	190	192	190
query91	138	148	104	104
query92	61	54	51	51
query93	1870	891	544	544
query94	690	391	295	295
query95	326	263	255	255
query96	492	606	282	282
query97	2696	2765	2651	2651
query98	226	205	196	196
query99	1655	1582	1398	1398
Total cold run time: 304009 ms
Total hot run time: 197678 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.46 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit de08108c149ada052012925267835616e13d073d, data reload: false

query1	0.04	0.05	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.62	0.10	0.10
query5	0.41	0.43	0.43
query6	1.14	0.66	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.60	0.50	0.52
query10	0.56	0.58	0.56
query11	0.15	0.10	0.10
query12	0.14	0.10	0.11
query13	0.60	0.59	0.60
query14	2.72	2.87	2.76
query15	0.90	0.83	0.82
query16	0.39	0.39	0.38
query17	1.00	1.06	1.10
query18	0.23	0.20	0.22
query19	1.98	1.73	1.98
query20	0.01	0.01	0.01
query21	15.38	0.90	0.57
query22	0.77	0.82	0.60
query23	15.29	1.42	0.55
query24	3.03	1.24	1.16
query25	0.18	0.22	0.13
query26	0.30	0.15	0.14
query27	0.04	0.05	0.06
query28	13.97	1.50	1.05
query29	12.58	3.96	3.30
query30	0.25	0.08	0.06
query31	2.83	0.59	0.38
query32	3.23	0.53	0.46
query33	3.10	3.04	3.13
query34	16.48	5.11	4.52
query35	4.52	4.55	4.51
query36	0.65	0.49	0.47
query37	0.09	0.06	0.05
query38	0.04	0.04	0.03
query39	0.03	0.03	0.02
query40	0.16	0.14	0.12
query41	0.07	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.91 s
Total hot run time: 31.46 s

@hubgeter
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32371 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e4da2c05729235e9bac8e7cc2fff575bbddc51f5, data reload: false

------ Round 1 ----------------------------------
q1	17600	6173	6104	6104
q2	2045	324	173	173
q3	10622	1217	764	764
q4	10256	881	448	448
q5	8177	2204	1933	1933
q6	212	180	149	149
q7	899	758	600	600
q8	9244	1326	1123	1123
q9	5125	4915	4884	4884
q10	6795	2313	1873	1873
q11	504	293	258	258
q12	339	364	224	224
q13	17750	3573	2896	2896
q14	235	229	210	210
q15	557	506	493	493
q16	652	617	593	593
q17	558	846	325	325
q18	6809	6433	6301	6301
q19	1776	972	562	562
q20	313	306	195	195
q21	2836	2120	1952	1952
q22	361	340	311	311
Total cold run time: 103665 ms
Total hot run time: 32371 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6265	6249	6188	6188
q2	234	326	241	241
q3	2252	2665	2332	2332
q4	1380	1809	1336	1336
q5	4313	4715	4775	4715
q6	191	176	149	149
q7	2066	1930	1808	1808
q8	2608	2808	2691	2691
q9	7239	7187	7320	7187
q10	3058	3389	2891	2891
q11	588	525	505	505
q12	646	726	579	579
q13	3394	3780	3140	3140
q14	298	293	287	287
q15	576	518	513	513
q16	659	691	657	657
q17	1218	1717	1283	1283
q18	7812	7425	7320	7320
q19	835	1176	1099	1099
q20	1999	2029	1907	1907
q21	5810	5343	4985	4985
q22	631	624	586	586
Total cold run time: 54072 ms
Total hot run time: 52399 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.89% (10120/26024)
Line Coverage: 29.88% (85508/286215)
Region Coverage: 29.01% (43721/150703)
Branch Coverage: 25.54% (22301/87334)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e4da2c05729235e9bac8e7cc2fff575bbddc51f5_e4da2c05729235e9bac8e7cc2fff575bbddc51f5/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 196559 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e4da2c05729235e9bac8e7cc2fff575bbddc51f5, data reload: false

query1	1281	949	949	949
query2	6492	2237	2296	2237
query3	10934	4697	4669	4669
query4	32910	23556	23288	23288
query5	4403	611	467	467
query6	280	203	203	203
query7	3971	487	303	303
query8	302	257	229	229
query9	9455	2727	2727	2727
query10	466	296	243	243
query11	17905	15277	15208	15208
query12	165	112	110	110
query13	1589	542	408	408
query14	10172	7840	7167	7167
query15	234	204	183	183
query16	7292	647	433	433
query17	1570	777	625	625
query18	1443	383	348	348
query19	219	206	180	180
query20	126	117	117	117
query21	209	133	112	112
query22	4770	4499	4461	4461
query23	34671	33256	33694	33256
query24	6455	2308	2345	2308
query25	470	469	377	377
query26	980	270	154	154
query27	2110	485	336	336
query28	5286	2525	2473	2473
query29	616	549	416	416
query30	201	184	148	148
query31	983	937	856	856
query32	71	62	52	52
query33	499	369	320	320
query34	763	859	534	534
query35	819	855	784	784
query36	1029	1058	951	951
query37	117	99	69	69
query38	4152	4379	4046	4046
query39	1499	1481	1513	1481
query40	208	115	97	97
query41	44	46	44	44
query42	117	107	102	102
query43	513	521	480	480
query44	1341	834	818	818
query45	179	172	169	169
query46	885	1088	656	656
query47	2013	1961	1948	1948
query48	395	424	321	321
query49	731	470	424	424
query50	623	675	388	388
query51	7303	7437	7325	7325
query52	106	98	95	95
query53	224	264	196	196
query54	514	509	424	424
query55	81	83	79	79
query56	258	248	277	248
query57	1271	1229	1148	1148
query58	240	227	237	227
query59	3212	3252	3200	3200
query60	289	297	245	245
query61	107	107	109	107
query62	860	803	750	750
query63	231	198	192	192
query64	3649	1031	673	673
query65	3314	3263	3255	3255
query66	886	416	303	303
query67	16444	15848	15595	15595
query68	9713	785	540	540
query69	484	295	258	258
query70	1207	1105	1116	1105
query71	447	293	257	257
query72	5937	3938	3858	3858
query73	1109	755	375	375
query74	9875	9077	9069	9069
query75	4587	3147	2668	2668
query76	5598	1183	778	778
query77	1042	368	272	272
query78	10188	10204	9429	9429
query79	2608	919	608	608
query80	690	516	429	429
query81	482	271	229	229
query82	253	149	123	123
query83	184	251	159	159
query84	298	98	71	71
query85	727	381	301	301
query86	344	304	305	304
query87	4630	4399	4486	4399
query88	3943	2233	2217	2217
query89	420	331	288	288
query90	2003	193	192	192
query91	142	138	106	106
query92	66	57	57	57
query93	1128	876	548	548
query94	699	396	279	279
query95	323	265	260	260
query96	487	613	279	279
query97	2700	2808	2698	2698
query98	215	208	202	202
query99	1667	1537	1464	1464
Total cold run time: 297143 ms
Total hot run time: 196559 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.48 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e4da2c05729235e9bac8e7cc2fff575bbddc51f5, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.04	0.04
query3	0.23	0.07	0.06
query4	1.61	0.10	0.11
query5	0.41	0.43	0.42
query6	1.15	0.65	0.64
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.49	0.53
query10	0.55	0.55	0.54
query11	0.15	0.10	0.11
query12	0.14	0.11	0.11
query13	0.64	0.61	0.59
query14	2.72	2.76	2.71
query15	0.89	0.84	0.82
query16	0.39	0.40	0.38
query17	1.06	1.04	1.01
query18	0.23	0.20	0.20
query19	1.89	1.89	2.03
query20	0.01	0.02	0.01
query21	15.38	0.90	0.57
query22	0.76	0.77	0.78
query23	15.15	1.48	0.60
query24	3.16	1.03	1.22
query25	0.24	0.17	0.16
query26	0.23	0.15	0.14
query27	0.06	0.05	0.05
query28	13.80	1.53	1.05
query29	12.60	4.02	3.29
query30	0.25	0.10	0.06
query31	2.81	0.62	0.38
query32	3.23	0.54	0.46
query33	3.06	3.05	3.04
query34	16.76	5.12	4.45
query35	4.52	4.45	4.52
query36	0.65	0.51	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.15	0.13	0.12
query41	0.08	0.02	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.04
Total cold run time: 105.95 s
Total hot run time: 31.48 s

@hubgeter
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32433 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cd70af777b91f998def71c2dc56a599df7af2c79, data reload: false

------ Round 1 ----------------------------------
q1	17572	6140	6065	6065
q2	2042	293	160	160
q3	10437	1229	719	719
q4	10253	885	443	443
q5	8145	2222	2010	2010
q6	203	179	148	148
q7	895	767	620	620
q8	9238	1322	1098	1098
q9	5162	4883	4860	4860
q10	6753	2286	1870	1870
q11	501	275	258	258
q12	350	346	217	217
q13	17750	3608	2942	2942
q14	232	221	204	204
q15	548	508	489	489
q16	624	619	592	592
q17	555	850	325	325
q18	6836	6407	6467	6407
q19	2007	957	566	566
q20	300	306	187	187
q21	2830	2141	1953	1953
q22	353	337	300	300
Total cold run time: 103586 ms
Total hot run time: 32433 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6264	6185	6192	6185
q2	243	328	229	229
q3	2217	2639	2302	2302
q4	1368	1828	1352	1352
q5	4312	4758	4697	4697
q6	184	179	144	144
q7	2055	1960	1794	1794
q8	2607	2786	2679	2679
q9	7309	7272	7300	7272
q10	3074	3337	2819	2819
q11	584	504	493	493
q12	624	716	562	562
q13	3540	3826	3095	3095
q14	295	302	269	269
q15	573	535	514	514
q16	645	696	673	673
q17	1196	1729	1255	1255
q18	7555	7432	7366	7366
q19	805	1166	1030	1030
q20	2034	2052	1891	1891
q21	5745	5343	4944	4944
q22	597	600	571	571
Total cold run time: 53826 ms
Total hot run time: 52136 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.89% (10122/26024)
Line Coverage: 29.88% (85519/286216)
Region Coverage: 29.01% (43717/150699)
Branch Coverage: 25.55% (22310/87332)
Coverage Report: http://coverage.selectdb-in.cc/coverage/cd70af777b91f998def71c2dc56a599df7af2c79_cd70af777b91f998def71c2dc56a599df7af2c79/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 196009 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cd70af777b91f998def71c2dc56a599df7af2c79, data reload: false

query1	1279	963	917	917
query2	6462	2324	2131	2131
query3	10988	4649	4801	4649
query4	32733	23675	23389	23389
query5	3628	620	437	437
query6	270	205	194	194
query7	3978	494	308	308
query8	317	237	234	234
query9	9575	2749	2741	2741
query10	445	309	240	240
query11	18098	15380	15186	15186
query12	159	102	101	101
query13	1571	554	413	413
query14	10306	7637	6892	6892
query15	265	230	186	186
query16	8097	616	484	484
query17	1555	755	580	580
query18	2158	401	318	318
query19	187	181	199	181
query20	129	118	113	113
query21	207	125	104	104
query22	4608	4613	4742	4613
query23	34346	33622	33864	33622
query24	6469	2364	2362	2362
query25	470	465	381	381
query26	712	270	150	150
query27	1980	469	331	331
query28	5651	2522	2501	2501
query29	653	581	444	444
query30	215	190	157	157
query31	981	949	855	855
query32	90	56	60	56
query33	468	360	301	301
query34	742	868	501	501
query35	848	836	755	755
query36	1028	1033	959	959
query37	115	100	79	79
query38	4298	4275	4298	4275
query39	1529	1462	1445	1445
query40	203	113	106	106
query41	47	42	42	42
query42	116	105	111	105
query43	504	533	514	514
query44	1365	826	842	826
query45	181	181	179	179
query46	887	1055	660	660
query47	2017	1995	1907	1907
query48	381	409	323	323
query49	696	470	397	397
query50	633	695	391	391
query51	7244	7420	7136	7136
query52	112	100	99	99
query53	233	249	183	183
query54	488	518	430	430
query55	94	82	79	79
query56	273	261	266	261
query57	1208	1253	1153	1153
query58	251	238	226	226
query59	3183	3283	3055	3055
query60	269	262	255	255
query61	107	106	105	105
query62	920	818	775	775
query63	221	194	193	193
query64	3171	1032	681	681
query65	3339	3258	3310	3258
query66	848	408	320	320
query67	16360	16000	15546	15546
query68	9715	755	522	522
query69	471	289	255	255
query70	1201	1074	1148	1074
query71	426	275	275	275
query72	6314	3930	3784	3784
query73	654	748	360	360
query74	10064	9354	8866	8866
query75	4512	3150	2673	2673
query76	4719	1172	747	747
query77	827	353	277	277
query78	10260	10185	9375	9375
query79	3415	901	579	579
query80	620	505	411	411
query81	508	277	224	224
query82	682	231	119	119
query83	198	162	146	146
query84	276	81	81	81
query85	786	353	301	301
query86	398	323	288	288
query87	4529	4643	4337	4337
query88	4890	2222	2211	2211
query89	415	331	292	292
query90	1794	188	186	186
query91	177	136	101	101
query92	66	53	50	50
query93	2187	938	522	522
query94	655	398	285	285
query95	330	265	258	258
query96	479	609	272	272
query97	2748	2825	2714	2714
query98	230	205	196	196
query99	1626	1551	1438	1438
Total cold run time: 299133 ms
Total hot run time: 196009 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.78 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cd70af777b91f998def71c2dc56a599df7af2c79, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.60	0.11	0.11
query5	0.40	0.40	0.43
query6	1.16	0.65	0.65
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.64	0.49	0.53
query10	0.56	0.57	0.55
query11	0.14	0.10	0.11
query12	0.15	0.11	0.10
query13	0.61	0.61	0.60
query14	2.71	2.74	2.77
query15	0.90	0.82	0.82
query16	0.38	0.39	0.38
query17	1.06	1.06	1.03
query18	0.22	0.21	0.20
query19	1.97	1.88	2.01
query20	0.01	0.01	0.01
query21	15.37	0.88	0.58
query22	0.74	0.79	0.68
query23	15.32	1.48	0.49
query24	3.11	0.59	1.73
query25	0.12	0.19	0.08
query26	0.34	0.15	0.14
query27	0.06	0.04	0.05
query28	13.93	1.53	1.05
query29	12.57	3.93	3.24
query30	0.25	0.09	0.07
query31	2.82	0.58	0.38
query32	3.23	0.53	0.46
query33	3.06	3.18	3.04
query34	16.33	5.00	4.47
query35	4.51	4.44	4.45
query36	0.64	0.50	0.48
query37	0.09	0.06	0.05
query38	0.04	0.04	0.04
query39	0.04	0.02	0.02
query40	0.16	0.14	0.14
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.78 s
Total hot run time: 30.78 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants