Skip to content

[Exec](profile) add max row count one backend in materialization profile #59728

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
HappenLee merged 1 commit into from
Jan 12, 2026
Merged

[Exec](profile) add max row count one backend in materialization profile #59728

HappenLee merged 1 commit into from
Jan 12, 2026
+21 −0

Conversation

@HappenLee
Copy link
Contributor

@HappenLee HappenLee commented Jan 9, 2026 •
edited
Loading

What problem does this PR solve?

feat(materialization): track and expose max rows per backend metric

  • Add _backend_rows_count to MaterializationSharedState to count the number of rows assigned to each backend during block distribution.
  • Introduce _max_rows_per_backend to record the maximum row count handled by any single backend in the current batch.
  • Update merge_multi_response() to compute _max_rows_per_backend after row assignment.
  • Expose this metric via a new runtime profile counter MaxRowsPerBackend in MaterializationLocalState.
  • This helps monitor data skew across backends and aids in performance debugging for materialization-heavy queries.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 9, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@HappenLee
Copy link
Contributor Author

HappenLee commented Jan 9, 2026

run buildall

@doris-robot
Copy link

doris-robot commented Jan 9, 2026

TPC-H: Total hot run time: 32266 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a7c4d93595cf63dfe17601d676a6ab5da4e636a0, data reload: false

------ Round 1 ----------------------------------
q1	17629	4220	4091	4091
q2	2051	363	241	241
q3	10130	1290	753	753
q4	10209	850	317	317
q5	7529	2103	1884	1884
q6	188	175	144	144
q7	955	837	670	670
q8	9291	1441	1153	1153
q9	4859	4546	4688	4546
q10	6760	1828	1404	1404
q11	526	304	299	299
q12	716	745	592	592
q13	17776	3854	3085	3085
q14	292	306	277	277
q15	586	502	515	502
q16	729	697	661	661
q17	663	824	536	536
q18	6706	6592	7009	6592
q19	1154	1034	643	643
q20	432	390	269	269
q21	3289	2669	2583	2583
q22	1175	1096	1024	1024
Total cold run time: 103645 ms
Total hot run time: 32266 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4473	4262	4331	4262
q2	327	422	331	331
q3	2273	2809	2436	2436
q4	1418	1899	1461	1461
q5	4408	4376	4364	4364
q6	215	167	129	129
q7	1947	2154	1731	1731
q8	2541	2389	2373	2373
q9	7160	7026	7089	7026
q10	2476	2665	2328	2328
q11	552	484	456	456
q12	747	805	652	652
q13	3652	4159	3343	3343
q14	281	302	274	274
q15	542	543	493	493
q16	654	695	655	655
q17	1130	1315	1343	1315
q18	7557	7411	7195	7195
q19	871	803	833	803
q20	1888	1962	1831	1831
q21	4681	4299	4162	4162
q22	1083	1012	960	960
Total cold run time: 50876 ms
Total hot run time: 48580 ms

@doris-robot
Copy link

doris-robot commented Jan 9, 2026

TPC-DS: Total hot run time: 173255 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a7c4d93595cf63dfe17601d676a6ab5da4e636a0, data reload: false

query5	4507	594	427	427
query6	347	240	253	240
query7	4234	461	273	273
query8	349	273	252	252
query9	8805	2646	2615	2615
query10	515	389	321	321
query11	15281	15264	14921	14921
query12	203	116	114	114
query13	1261	501	385	385
query14	6821	2974	2804	2804
query14_1	2616	2639	2617	2617
query15	198	193	169	169
query16	993	479	443	443
query17	1070	649	543	543
query18	2697	435	332	332
query19	228	222	203	203
query20	128	121	116	116
query21	214	140	126	126
query22	3753	4150	3957	3957
query23	16090	15509	15243	15243
query23_1	15377	15533	15418	15418
query24	7282	1560	1196	1196
query24_1	1202	1176	1206	1176
query25	562	473	424	424
query26	1249	268	161	161
query27	2748	462	287	287
query28	4509	2160	2149	2149
query29	824	566	478	478
query30	313	246	218	218
query31	810	653	557	557
query32	83	81	75	75
query33	549	358	324	324
query34	910	892	528	528
query35	722	778	698	698
query36	848	893	806	806
query37	132	94	81	81
query38	2765	2686	2681	2681
query39	770	761	730	730
query39_1	708	722	725	722
query40	220	136	122	122
query41	73	73	67	67
query42	110	104	107	104
query43	436	442	458	442
query44	1309	764	716	716
query45	189	186	179	179
query46	855	972	589	589
query47	1439	1509	1367	1367
query48	312	318	241	241
query49	606	428	331	331
query50	634	266	205	205
query51	3771	3807	3772	3772
query52	107	109	101	101
query53	288	331	278	278
query54	288	255	251	251
query55	80	73	72	72
query56	283	280	302	280
query57	1082	1012	954	954
query58	264	255	255	255
query59	1955	2044	1993	1993
query60	314	321	299	299
query61	155	158	155	155
query62	402	353	328	328
query63	304	264	273	264
query64	4932	1302	972	972
query65	3833	3761	3777	3761
query66	1366	417	292	292
query67	14893	15196	15107	15107
query68	3433	1025	701	701
query69	477	349	318	318
query70	1032	895	911	895
query71	349	309	281	281
query72	6033	3371	3512	3371
query73	757	721	303	303
query74	8781	8752	8605	8605
query75	2827	2861	2469	2469
query76	3857	1066	642	642
query77	524	390	274	274
query78	9558	9689	9219	9219
query79	1521	925	580	580
query80	650	556	501	501
query81	521	265	235	235
query82	212	143	112	112
query83	270	253	241	241
query84	257	113	103	103
query85	893	512	447	447
query86	391	304	300	300
query87	2837	2877	2777	2777
query88	3181	2225	2226	2225
query89	385	360	325	325
query90	2008	151	143	143
query91	174	160	143	143
query92	78	67	61	61
query93	1159	906	538	538
query94	558	321	291	291
query95	579	322	351	322
query96	577	458	207	207
query97	2307	2393	2290	2290
query98	243	214	197	197
query99	568	580	600	580
Total cold run time: 249205 ms
Total hot run time: 173255 ms

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 12, 2026

skip check_coverage

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 12, 2026
@github-actions
Copy link
Contributor

github-actions bot commented Jan 12, 2026

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jan 12, 2026

PR approved by anyone and no changes requested.

@HappenLee HappenLee merged commit 7a1653b into apache:master Jan 12, 2026
43 of 46 checks passed
github-actions bot pushed a commit that referenced this pull request Jan 12, 2026
@HappenLee
[Exec](profile) add max row count one backend in materialization prof…
6689273 
…ile (#59728)

- Add `_backend_rows_count` to `MaterializationSharedState` to count the
number of rows assigned to each backend during block distribution.
- Introduce `_max_rows_per_backend` to record the maximum row count
handled by any single backend in the current batch.
- Update `merge_multi_response()` to compute `_max_rows_per_backend`
after row assignment.
- Expose this metric via a new runtime profile counter
`MaxRowsPerBackend` in `MaterializationLocalState`.
- This helps monitor data skew across backends and aids in performance
debugging for materialization-heavy queries.
@github-actions github-actions bot mentioned this pull request Jan 12, 2026
Merged
yiguolei pushed a commit that referenced this pull request Jan 13, 2026
@github-actions @HappenLee
branch-4.0: [Exec](profile) add max row count one backend in material…
93aa5f4 
…ization profile #59728 (#59758)

Cherry-picked from #59728

Co-authored-by: HappenLee <[email protected]>
zzzxl1993 pushed a commit to zzzxl1993/doris that referenced this pull request Jan 13, 2026
@HappenLee @zzzxl1993
[Exec](profile) add max row count one backend in materialization prof…
a3e166f 
…ile (apache#59728)

- Add `_backend_rows_count` to `MaterializationSharedState` to count the
number of rows assigned to each backend during block distribution.
- Introduce `_max_rows_per_backend` to record the maximum row count
handled by any single backend in the current batch.
- Update `merge_multi_response()` to compute `_max_rows_per_backend`
after row assignment.
- Expose this metric via a new runtime profile counter
`MaxRowsPerBackend` in `MaterializationLocalState`.
- This helps monitor data skew across backends and aids in performance
debugging for materialization-heavy queries.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@BiteTheDDDDt BiteTheDDDDt BiteTheDDDDt approved these changes

@yiguolei yiguolei yiguolei approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/4.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@HappenLee @hello-stephen @doris-robot @BiteTheDDDDt @yiguolei