Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[cherry-pick](branch-2.0) Pick "[Fix](Rowset Id) Use a randomly generated rowset ID to handle memory write failures (#42949) #46102

Open
wants to merge 2 commits into
base: branch-2.0
Choose a base branch
from

Conversation

Yukang-Lian
Copy link
Collaborator

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Pick #42949 #46074

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Yukang-Lian
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49097 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 47586ac61aad984bd5157c18f4fe6c35efcf35f2, data reload: false

------ Round 1 ----------------------------------
q1	17625	4404	4383	4383
q2	2065	157	148	148
q3	10265	1870	1940	1870
q4	10308	1254	1313	1254
q5	8511	3878	3882	3878
q6	236	125	123	123
q7	2065	1569	1593	1569
q8	9311	2747	2727	2727
q9	10319	10025	9842	9842
q10	8618	3491	3565	3491
q11	425	254	247	247
q12	471	296	305	296
q13	18356	3978	4034	3978
q14	354	324	337	324
q15	520	457	447	447
q16	551	456	463	456
q17	1141	974	952	952
q18	7349	6861	6818	6818
q19	1717	1544	1510	1510
q20	507	330	314	314
q21	4395	4198	4083	4083
q22	509	387	398	387
Total cold run time: 115618 ms
Total hot run time: 49097 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4341	4302	4325	4302
q2	327	223	218	218
q3	4192	4150	4172	4150
q4	2768	2738	2764	2738
q5	7205	7138	7110	7110
q6	242	122	118	118
q7	3272	2818	2873	2818
q8	4345	4490	4515	4490
q9	13618	13604	13566	13566
q10	4272	4265	4257	4257
q11	770	700	688	688
q12	1041	880	876	876
q13	6951	3734	3774	3734
q14	453	442	443	442
q15	500	450	460	450
q16	640	608	597	597
q17	3809	3815	3861	3815
q18	8759	8738	8839	8738
q19	1738	1661	1710	1661
q20	2388	2089	2135	2089
q21	8450	8411	8424	8411
q22	1028	960	937	937
Total cold run time: 81109 ms
Total hot run time: 76205 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 212534 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 47586ac61aad984bd5157c18f4fe6c35efcf35f2, data reload: false

query1	957	422	384	384
query2	6527	2137	1976	1976
query3	6925	207	202	202
query4	23360	21639	21630	21630
query5	19743	6509	6516	6509
query6	292	221	238	221
query7	4336	303	315	303
query8	281	280	243	243
query9	3136	2718	2628	2628
query10	468	315	300	300
query11	16063	15320	14957	14957
query12	125	82	76	76
query13	1025	453	444	444
query14	17250	13190	13451	13190
query15	356	218	233	218
query16	6490	288	273	273
query17	1699	947	933	933
query18	899	309	309	309
query19	220	154	160	154
query20	108	96	102	96
query21	191	100	95	95
query22	5246	5068	5060	5060
query23	34140	33293	33560	33293
query24	7875	6394	6310	6310
query25	516	432	446	432
query26	1272	168	166	166
query27	2419	302	301	301
query28	6126	2308	2265	2265
query29	2793	2709	2869	2709
query30	241	168	169	168
query31	964	755	736	736
query32	74	75	63	63
query33	447	267	271	267
query34	870	508	466	466
query35	1133	923	906	906
query36	1187	1411	1152	1152
query37	181	62	61	61
query38	3095	2941	2881	2881
query39	1364	1330	1319	1319
query40	311	94	97	94
query41	45	39	49	39
query42	87	87	93	87
query43	634	525	606	525
query44	1156	736	741	736
query45	251	231	228	228
query46	1228	958	966	958
query47	1886	1740	1763	1740
query48	518	408	412	408
query49	664	376	379	376
query50	855	612	577	577
query51	4779	4670	4665	4665
query52	83	78	88	78
query53	229	182	189	182
query54	2671	2487	2449	2449
query55	92	83	86	83
query56	238	205	214	205
query57	1286	1169	1179	1169
query58	222	211	207	207
query59	3370	3164	3193	3164
query60	226	200	207	200
query61	97	93	106	93
query62	807	449	476	449
query63	207	180	174	174
query64	3484	1633	1496	1496
query65	3659	3537	3562	3537
query66	803	390	396	390
query67	15732	16644	15248	15248
query68	9406	679	647	647
query69	498	272	269	269
query70	1703	1371	1335	1335
query71	410	322	317	317
query72	6749	4884	4816	4816
query73	767	314	329	314
query74	6317	5903	5816	5816
query75	5197	3685	3683	3683
query76	5097	1146	1168	1146
query77	858	267	269	267
query78	12616	12731	11920	11920
query79	11378	663	651	651
query80	847	396	405	396
query81	490	236	238	236
query82	1102	100	98	98
query83	170	133	137	133
query84	264	69	73	69
query85	867	312	311	311
query86	339	303	297	297
query87	3188	3030	3048	3030
query88	5271	2350	2328	2328
query89	405	313	285	285
query90	1900	218	217	217
query91	156	134	127	127
query92	66	53	51	51
query93	5846	639	604	604
query94	750	209	207	207
query95	2029	1940	1978	1940
query96	669	328	340	328
query97	6500	6402	6405	6402
query98	230	225	208	208
query99	2756	886	833	833
Total cold run time: 320855 ms
Total hot run time: 212534 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 47586ac61aad984bd5157c18f4fe6c35efcf35f2, data reload: false

query1	0.02	0.03	0.02
query2	0.07	0.03	0.03
query3	0.25	0.05	0.05
query4	1.76	0.06	0.05
query5	0.53	0.52	0.52
query6	1.23	0.62	0.62
query7	0.01	0.01	0.01
query8	0.04	0.02	0.02
query9	0.53	0.48	0.47
query10	0.55	0.53	0.53
query11	0.13	0.09	0.09
query12	0.12	0.09	0.09
query13	0.63	0.62	0.60
query14	0.79	0.80	0.77
query15	0.79	0.76	0.74
query16	0.38	0.37	0.36
query17	1.01	1.03	1.02
query18	0.23	0.24	0.26
query19	1.93	1.86	1.85
query20	0.02	0.01	0.01
query21	15.44	0.57	0.55
query22	2.55	2.27	1.88
query23	17.26	1.02	1.00
query24	4.69	0.95	1.51
query25	0.34	0.07	0.06
query26	0.57	0.15	0.16
query27	0.06	0.03	0.04
query28	8.08	0.78	0.78
query29	12.68	2.34	2.20
query30	0.63	0.63	0.53
query31	2.82	0.39	0.37
query32	3.35	0.50	0.49
query33	3.05	3.07	3.07
query34	15.23	4.77	4.80
query35	4.88	4.82	4.84
query36	1.06	1.03	1.01
query37	0.05	0.05	0.05
query38	0.04	0.02	0.02
query39	0.03	0.01	0.02
query40	0.16	0.14	0.14
query41	0.07	0.02	0.01
query42	0.02	0.01	0.02
query43	0.02	0.01	0.02
Total cold run time: 104.1 s
Total hot run time: 31 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 47586ac61aad984bd5157c18f4fe6c35efcf35f2 with default session variables
Stream load json:         21 seconds loaded 2358488459 Bytes, about 107 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants