[opt](scanner) set number of file scanner to max_scanners_concurrency #59622

morningman · 2026-01-07T07:28:16Z

What problem does this PR solve?

Problem Summary:

For external tables, each scanner is not bound to a specific split. Instead, when a scanner is scheduled,
it dynamically fetches the next scan range from a unified split source for scanning.
Therefore, the number of scanners only needs to match max_scanners_concurrency to ensure full-speed execution.

It also fix a profile issue.
Before:

PerScannerRunningTime: [7.341us, 15.372us, 5.987us, 9.738us, 10.630us, 21.631us, 7.539us, 7.586us, 6.247us, 12.755us, 10.989us, 12.221us, 18.952us, 3.450us, 7.805us, 12.291us, 1s282ms, 1s242ms, 1s263ms, 1s363ms, 1s228ms, 1s283ms, 1s267ms, 1s273ms, 1s177ms, 1s271ms, 1s197ms, 1s351ms, 1s357ms, 1s460ms, 1s253ms, 4s469ms, ]

The time is not even

After:

- PerScannerRunningTime: [287.588ms, 324.711ms, 282.664ms, 299.930ms, 269.238ms, 321.864ms, 314.268ms, 309.313ms, 315.368ms, 332.571ms, 290.192ms, 278.908ms, 335.692ms, 275.525ms, 322.447ms, 346.342ms, ]

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-01-07T07:28:22Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

morningman · 2026-01-07T07:28:36Z

run buildall

…59622

doris-robot · 2026-01-07T08:46:06Z

BE UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.24% (18976/35644)
Line Coverage	39.20% (175759/448421)
Region Coverage	33.77% (136253/403448)
Branch Coverage	34.73% (58770/169214)

hello-stephen · 2026-01-07T10:27:01Z

BE Regression && UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.86% (25736/34844)
Line Coverage	61.21% (273728/447202)
Region Coverage	56.19% (228999/407567)
Branch Coverage	58.02% (98490/169763)

github-actions · 2026-01-08T14:29:35Z

PR approved by anyone and no changes requested.

github-actions · 2026-01-08T14:44:53Z

PR approved by at least one committer and no changes requested.

hello-stephen · 2026-01-08T14:46:56Z

BE Regression && UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.86% (25736/34844)
Line Coverage	61.21% (273728/447202)
Region Coverage	56.19% (228999/407567)
Branch Coverage	58.02% (98490/169763)

morningman · 2026-01-08T15:12:04Z

run buildall

doris-robot · 2026-01-08T15:47:57Z

TPC-H: Total hot run time: 32006 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a0a0ffe8d74e2660e29f1d1db5574b1570d2a6f9, data reload: false

------ Round 1 ----------------------------------
q1	17618	4167	4047	4047
q2	2068	360	235	235
q3	10147	1271	724	724
q4	10204	810	311	311
q5	7522	2025	1830	1830
q6	186	166	138	138
q7	923	800	650	650
q8	9273	1311	1143	1143
q9	4854	4526	4548	4526
q10	6799	1797	1368	1368
q11	509	313	280	280
q12	713	734	587	587
q13	17825	3819	3086	3086
q14	296	310	288	288
q15	591	519	514	514
q16	710	691	633	633
q17	668	724	584	584
q18	6704	6388	6756	6388
q19	1377	1041	670	670
q20	425	374	260	260
q21	3255	2676	2696	2676
q22	1173	1068	1086	1068
Total cold run time: 103840 ms
Total hot run time: 32006 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4347	4220	4257	4220
q2	343	414	331	331
q3	2299	2813	2455	2455
q4	1415	1874	1472	1472
q5	4536	4270	4483	4270
q6	208	166	135	135
q7	2099	1879	1783	1783
q8	2546	2453	2451	2451
q9	7023	7091	6969	6969
q10	2464	2714	2249	2249
q11	633	472	465	465
q12	738	763	609	609
q13	3448	3833	3090	3090
q14	282	299	262	262
q15	519	507	486	486
q16	649	652	602	602
q17	1109	1258	1352	1258
q18	7685	7294	7289	7289
q19	877	847	851	847
q20	1908	1993	1829	1829
q21	4625	4472	4192	4192
q22	1070	1027	957	957
Total cold run time: 50823 ms
Total hot run time: 48221 ms

doris-robot · 2026-01-08T15:58:45Z

TPC-DS: Total hot run time: 171890 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a0a0ffe8d74e2660e29f1d1db5574b1570d2a6f9, data reload: false

query5	4509	581	423	423
query6	326	227	224	224
query7	4216	481	270	270
query8	339	253	238	238
query9	8784	2653	2683	2653
query10	527	380	308	308
query11	15363	15034	14908	14908
query12	175	114	113	113
query13	1269	503	387	387
query14	6209	3008	2748	2748
query14_1	2715	2665	2668	2665
query15	208	205	179	179
query16	1012	455	451	451
query17	1111	687	589	589
query18	2471	443	340	340
query19	228	229	195	195
query20	124	119	114	114
query21	215	137	119	119
query22	3755	3991	3699	3699
query23	15910	15589	15196	15196
query23_1	15256	15366	15277	15277
query24	7385	1655	1175	1175
query24_1	1206	1190	1202	1190
query25	568	482	431	431
query26	1239	266	158	158
query27	2767	450	289	289
query28	4565	2143	2129	2129
query29	811	571	467	467
query30	316	236	208	208
query31	788	637	537	537
query32	79	67	65	65
query33	541	347	284	284
query34	901	892	523	523
query35	719	765	672	672
query36	869	908	838	838
query37	132	93	80	80
query38	2799	2720	2677	2677
query39	770	757	719	719
query39_1	707	738	709	709
query40	217	134	117	117
query41	68	65	61	61
query42	108	104	101	101
query43	469	474	405	405
query44	1319	720	723	720
query45	184	188	185	185
query46	847	963	590	590
query47	1357	1377	1318	1318
query48	306	322	240	240
query49	603	433	326	326
query50	642	280	204	204
query51	3763	3734	3766	3734
query52	102	107	97	97
query53	295	333	271	271
query54	279	263	243	243
query55	74	78	78	78
query56	289	300	298	298
query57	1056	1058	921	921
query58	268	254	244	244
query59	2058	2114	2082	2082
query60	321	336	304	304
query61	201	155	157	155
query62	380	346	332	332
query63	297	270	269	269
query64	4968	1296	976	976
query65	3815	3740	3746	3740
query66	1458	444	311	311
query67	15155	14645	14985	14645
query68	6341	1001	700	700
query69	508	342	304	304
query70	1064	963	961	961
query71	385	295	276	276
query72	6022	3493	3533	3493
query73	753	722	301	301
query74	8730	8740	8593	8593
query75	3032	2815	2442	2442
query76	3916	1059	651	651
query77	528	375	276	276
query78	9792	9625	9143	9143
query79	1582	904	589	589
query80	606	576	478	478
query81	525	273	227	227
query82	393	141	113	113
query83	270	254	234	234
query84	263	127	107	107
query85	924	524	456	456
query86	388	298	322	298
query87	2903	2859	2818	2818
query88	4372	2203	2223	2203
query89	397	357	335	335
query90	2171	158	156	156
query91	180	170	140	140
query92	78	63	62	62
query93	1738	898	530	530
query94	576	330	253	253
query95	580	326	347	326
query96	579	480	205	205
query97	2302	2401	2317	2317
query98	244	202	196	196
query99	638	583	537	537
Total cold run time: 254613 ms
Total hot run time: 171890 ms

doris-robot · 2026-01-08T16:24:20Z

BE UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	53.02% (18974/35784)
Line Coverage	39.09% (175849/449881)
Region Coverage	33.68% (136235/404479)
Branch Coverage	34.68% (58848/169708)

morningman · 2026-01-09T02:52:51Z

run check_coverage

hello-stephen · 2026-01-09T03:16:58Z

BE Regression && UT Coverage Report

Increment line coverage 100.00% (6/6) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.89% (25846/34979)
Line Coverage	61.33% (275145/448648)
Region Coverage	56.18% (229545/408595)
Branch Coverage	58.11% (98940/170257)

…_concurrency(#59622) (#59696) bp #59622

kaka11chen

LGTM

morningman added a commit that referenced this pull request Jan 7, 2026

[opt](scanner) set number of file scanner to max_scanners_concurrency #…

b884481

…59622

morningman added dev/3.1.x dev/4.0.x and removed dev/3.1.x labels Jan 8, 2026

suxiaogang223 approved these changes Jan 8, 2026

View reviewed changes

github-actions bot added the reviewed label Jan 8, 2026

hubgeter approved these changes Jan 8, 2026

View reviewed changes

CalvinKirs approved these changes Jan 8, 2026

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 8, 2026

[opt](scanner) set number of file scanner to max_scanners_concurrency

a0a0ffe

morningman force-pushed the scanner_num branch from 2062e17 to a0a0ffe Compare January 8, 2026 15:11

morningman mentioned this pull request Jan 8, 2026

branch-4.0: [opt](scanner) set number of file scanner to max_scanners_concurrency(#59622) #59696

Merged

yiguolei pushed a commit that referenced this pull request Jan 9, 2026

branch-4.0: [opt](scanner) set number of file scanner to max_scanners…

f47e162

…_concurrency(#59622) (#59696) bp #59622

yiguolei added dev/4.0.3-merged and removed dev/4.0.x labels Jan 9, 2026

morningman closed this Jan 9, 2026

morningman reopened this Jan 9, 2026

kaka11chen approved these changes Jan 9, 2026

View reviewed changes

morningman merged commit eb61c26 into apache:master Jan 9, 2026
39 of 41 checks passed

[opt](scanner) set number of file scanner to max_scanners_concurrency #59622

[opt](scanner) set number of file scanner to max_scanners_concurrency #59622

Uh oh!

Conversation

morningman commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jan 7, 2026

Uh oh!

morningman commented Jan 7, 2026

Uh oh!

doris-robot commented Jan 7, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Jan 7, 2026

BE Regression && UT Coverage Report

Uh oh!

github-actions bot commented Jan 8, 2026

Uh oh!

github-actions bot commented Jan 8, 2026

Uh oh!

hello-stephen commented Jan 8, 2026

BE Regression && UT Coverage Report

Uh oh!

morningman commented Jan 8, 2026

Uh oh!

doris-robot commented Jan 8, 2026

Uh oh!

doris-robot commented Jan 8, 2026

Uh oh!

doris-robot commented Jan 8, 2026

BE UT Coverage Report

Uh oh!

morningman commented Jan 9, 2026

Uh oh!

hello-stephen commented Jan 9, 2026

BE Regression && UT Coverage Report

Uh oh!

kaka11chen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

morningman commented Jan 7, 2026 •

edited

Loading