Skip to content

[Exec](performance) change colocate execution parallel num#60677

Merged
HappenLee merged 1 commit intoapache:masterfrom
HappenLee:new_cache
Feb 13, 2026
Merged

[Exec](performance) change colocate execution parallel num#60677
HappenLee merged 1 commit intoapache:masterfrom
HappenLee:new_cache

Conversation

@HappenLee
Copy link
Contributor

@HappenLee HappenLee commented Feb 11, 2026

What problem does this PR solve?

Problem Summary:

Previously, the parallelism for colocate execution was calculated based on bucket count, which might not fully utilize available resources. This PR changes the parallelism calculation logic to:

  1. Use tablet count instead of bucket count for more accurate resource utilization
  2. Add a new session variable colocate_max_parallel_num to control the maximum parallelism
  3. Calculate parallelism as min(max(tablet_num, fragment.getParallelExecNum()), colocate_max_parallel_num) to ensure both resource utilization and system stability

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@HappenLee
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 28705 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 04b78600ac6a0e75e787b7073c14ead2e1136d90, data reload: false

------ Round 1 ----------------------------------
q1	17634	4603	4323	4323
q2	2105	391	237	237
q3	9992	794	500	500
q4	4678	342	243	243
q5	7576	1173	994	994
q6	179	168	145	145
q7	779	819	648	648
q8	9274	1413	1322	1322
q9	4853	4783	4666	4666
q10	6770	1869	1635	1635
q11	485	265	241	241
q12	692	561	462	462
q13	17760	4235	3373	3373
q14	230	234	216	216
q15	884	810	804	804
q16	720	727	671	671
q17	715	881	417	417
q18	6141	5388	5175	5175
q19	1260	981	617	617
q20	506	505	387	387
q21	4714	1844	1386	1386
q22	344	301	243	243
Total cold run time: 98291 ms
Total hot run time: 28705 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4409	4313	4348	4313
q2	284	357	267	267
q3	1799	2155	1722	1722
q4	861	1181	779	779
q5	4054	4372	4292	4292
q6	177	175	139	139
q7	1708	1623	1504	1504
q8	2452	2658	2462	2462
q9	7642	7475	7337	7337
q10	2663	2827	2427	2427
q11	530	451	409	409
q12	484	555	437	437
q13	4039	4406	3746	3746
q14	299	302	313	302
q15	862	822	824	822
q16	726	778	739	739
q17	1180	1511	1324	1324
q18	7245	6976	6726	6726
q19	903	849	848	848
q20	2074	2191	2041	2041
q21	4111	3459	3265	3265
q22	510	468	402	402
Total cold run time: 49012 ms
Total hot run time: 46303 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.33 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 04b78600ac6a0e75e787b7073c14ead2e1136d90, data reload: false

query1	0.05	0.05	0.05
query2	0.11	0.04	0.05
query3	0.26	0.09	0.08
query4	1.66	0.12	0.11
query5	0.27	0.25	0.24
query6	1.17	0.74	0.68
query7	0.04	0.03	0.03
query8	0.05	0.04	0.04
query9	0.60	0.51	0.50
query10	0.57	0.56	0.55
query11	0.15	0.10	0.10
query12	0.14	0.11	0.10
query13	0.64	0.62	0.62
query14	1.10	1.07	1.06
query15	0.88	0.87	0.86
query16	0.39	0.39	0.40
query17	1.11	1.17	1.12
query18	0.23	0.20	0.20
query19	2.02	1.97	2.09
query20	0.02	0.01	0.02
query21	15.40	0.29	0.16
query22	5.44	0.05	0.05
query23	16.13	0.30	0.12
query24	1.94	0.35	0.19
query25	0.10	0.08	0.05
query26	0.14	0.14	0.13
query27	0.09	0.07	0.07
query28	3.02	1.16	0.97
query29	12.56	3.89	3.15
query30	0.28	0.13	0.13
query31	2.84	0.62	0.40
query32	3.24	0.60	0.49
query33	3.39	3.31	3.37
query34	16.52	5.29	4.74
query35	4.80	4.74	4.83
query36	0.65	0.50	0.49
query37	0.11	0.07	0.07
query38	0.08	0.04	0.04
query39	0.05	0.04	0.03
query40	0.20	0.16	0.16
query41	0.08	0.04	0.03
query42	0.05	0.03	0.03
query43	0.05	0.04	0.03
Total cold run time: 98.62 s
Total hot run time: 28.33 s

@HappenLee
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 29112 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0e50d3c6146a54a44e7d444e4fc6c0b4c9aadf92, data reload: false

------ Round 1 ----------------------------------
q1	17577	4519	4306	4306
q2	2002	342	236	236
q3	10153	773	507	507
q4	4670	354	241	241
q5	7567	1194	1023	1023
q6	177	173	146	146
q7	793	857	646	646
q8	9288	1449	1337	1337
q9	4895	4705	4666	4666
q10	6761	1870	1622	1622
q11	441	269	237	237
q12	714	557	462	462
q13	17771	4221	3420	3420
q14	229	234	216	216
q15	891	815	792	792
q16	709	718	686	686
q17	706	862	400	400
q18	5923	5463	5314	5314
q19	1170	993	615	615
q20	513	498	392	392
q21	4720	1951	1546	1546
q22	373	316	302	302
Total cold run time: 98043 ms
Total hot run time: 29112 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4611	4544	4534	4534
q2	276	352	261	261
q3	1824	2218	1755	1755
q4	885	1195	769	769
q5	4037	4381	4322	4322
q6	183	180	143	143
q7	1804	1665	1534	1534
q8	2468	2859	2566	2566
q9	7643	7419	7327	7327
q10	2727	2858	2480	2480
q11	504	433	427	427
q12	498	582	450	450
q13	3916	4358	3672	3672
q14	281	287	268	268
q15	850	817	809	809
q16	697	727	699	699
q17	1187	1530	1348	1348
q18	7060	6933	6810	6810
q19	930	936	1058	936
q20	2099	2178	2076	2076
q21	4109	3435	3309	3309
q22	494	459	441	441
Total cold run time: 49083 ms
Total hot run time: 46936 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.28 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0e50d3c6146a54a44e7d444e4fc6c0b4c9aadf92, data reload: false

query1	0.05	0.04	0.04
query2	0.09	0.04	0.04
query3	0.25	0.09	0.08
query4	1.62	0.12	0.11
query5	0.26	0.25	0.25
query6	1.17	0.68	0.66
query7	0.03	0.03	0.03
query8	0.05	0.04	0.04
query9	0.56	0.50	0.48
query10	0.54	0.55	0.55
query11	0.14	0.09	0.10
query12	0.15	0.11	0.10
query13	0.63	0.61	0.61
query14	1.06	1.05	1.06
query15	0.87	0.87	0.87
query16	0.39	0.40	0.39
query17	1.11	1.09	1.10
query18	0.22	0.21	0.21
query19	2.00	1.94	2.01
query20	0.02	0.02	0.01
query21	15.42	0.25	0.15
query22	5.19	0.06	0.05
query23	15.67	0.28	0.11
query24	1.84	1.16	0.29
query25	0.06	0.08	0.06
query26	0.15	0.14	0.13
query27	0.09	0.08	0.05
query28	4.58	1.15	0.96
query29	12.56	4.06	3.24
query30	0.27	0.13	0.12
query31	2.81	0.64	0.40
query32	3.23	0.58	0.49
query33	3.26	3.22	3.21
query34	16.15	5.36	4.74
query35	4.78	4.82	4.78
query36	0.64	0.51	0.49
query37	0.13	0.07	0.07
query38	0.08	0.04	0.04
query39	0.04	0.03	0.03
query40	0.19	0.16	0.15
query41	0.09	0.04	0.03
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 98.52 s
Total hot run time: 28.28 s

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 58.33% (14/24) 🎉
Increment coverage report
Complete coverage report

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 12, 2026
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@HappenLee HappenLee merged commit 90ffecd into apache:master Feb 13, 2026
30 of 31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants