Skip to content

[fix](compaction) Check schema version before ordered data compaction #59570

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
yiguolei merged 1 commit into from
Jan 6, 2026
Merged

[fix](compaction) Check schema version before ordered data compaction #59570

yiguolei merged 1 commit into from
Jan 6, 2026

Conversation

@zhiqiang-hhhh
Copy link
Contributor

@zhiqiang-hhhh zhiqiang-hhhh commented Jan 5, 2026 •
edited
Loading

What problem does this PR solve?

Ordered data compaction should not be used to accelerate the merging of rowsets that have undergone a light schema change; otherwise, it may result in schema-inconsistent segments within the newly generated rowset.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 5, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zhiqiang-hhhh zhiqiang-hhhh changed the title [fix 1 [fix](compaction) Check schema version before ordered data compaction Jan 5, 2026
@zhiqiang-hhhh
Copy link
Contributor Author

zhiqiang-hhhh commented Jan 5, 2026

run buildall

@Hastyshell Hastyshell requested a review from luwei16 January 5, 2026 12:11
liaoxin01
liaoxin01 approved these changes Jan 5, 2026
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 5, 2026
@github-actions
Copy link
Contributor

github-actions bot commented Jan 5, 2026

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jan 5, 2026

PR approved by anyone and no changes requested.

airborne12
airborne12 approved these changes Jan 5, 2026
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@doris-robot
Copy link

doris-robot commented Jan 5, 2026

TPC-H: Total hot run time: 32399 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 88d157a59900b1c6f87713842486026ae4453921, data reload: false

------ Round 1 ----------------------------------
q1	17623	4251	4045	4045
q2	2086	367	247	247
q3	10086	1317	745	745
q4	10202	840	317	317
q5	7475	2150	1930	1930
q6	194	178	146	146
q7	983	818	650	650
q8	9268	1483	1254	1254
q9	4876	4629	4599	4599
q10	6760	1819	1416	1416
q11	537	293	297	293
q12	680	717	604	604
q13	17760	3922	3109	3109
q14	289	292	269	269
q15	580	515	508	508
q16	728	692	624	624
q17	677	821	591	591
q18	6732	6465	7006	6465
q19	1123	1038	655	655
q20	443	396	260	260
q21	3329	2722	2615	2615
q22	1117	1114	1057	1057
Total cold run time: 103548 ms
Total hot run time: 32399 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4334	4728	4264	4264
q2	336	402	324	324
q3	2345	2825	2473	2473
q4	1494	2096	1446	1446
q5	4360	4190	4349	4190
q6	230	179	136	136
q7	2006	1911	1676	1676
q8	2789	2383	2375	2375
q9	6884	7073	7236	7073
q10	2493	2722	2356	2356
q11	547	476	454	454
q12	771	740	612	612
q13	3783	4203	3451	3451
q14	300	299	286	286
q15	538	513	495	495
q16	642	694	620	620
q17	1155	1342	1358	1342
q18	7926	7239	7277	7239
q19	869	776	840	776
q20	1912	1941	1787	1787
q21	4596	4241	4126	4126
q22	1093	1015	980	980
Total cold run time: 51403 ms
Total hot run time: 48481 ms

@doris-robot
Copy link

doris-robot commented Jan 5, 2026

TPC-DS: Total hot run time: 172848 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 88d157a59900b1c6f87713842486026ae4453921, data reload: false

query5	4815	586	461	461
query6	330	233	211	211
query7	4247	472	258	258
query8	331	237	235	235
query9	8772	2682	2673	2673
query10	520	372	312	312
query11	15262	15159	14858	14858
query12	177	122	111	111
query13	1243	501	368	368
query14	6393	2992	2695	2695
query14_1	2629	2610	2617	2610
query15	206	194	169	169
query16	1014	483	457	457
query17	1039	662	532	532
query18	2624	422	329	329
query19	218	212	202	202
query20	129	113	111	111
query21	212	136	120	120
query22	4184	4164	3936	3936
query23	16033	15680	15584	15584
query23_1	15497	15458	15342	15342
query24	7350	1566	1203	1203
query24_1	1209	1196	1198	1196
query25	580	472	420	420
query26	1243	281	162	162
query27	2744	455	292	292
query28	4541	2153	2126	2126
query29	861	592	436	436
query30	315	239	207	207
query31	795	616	525	525
query32	79	69	64	64
query33	517	327	275	275
query34	905	862	518	518
query35	761	795	692	692
query36	891	871	798	798
query37	135	96	81	81
query38	2664	2664	2665	2664
query39	777	756	752	752
query39_1	701	709	695	695
query40	212	132	114	114
query41	67	63	65	63
query42	103	109	105	105
query43	446	471	420	420
query44	1374	729	723	723
query45	185	177	174	174
query46	877	990	601	601
query47	1332	1442	1296	1296
query48	302	330	242	242
query49	611	424	338	338
query50	654	280	202	202
query51	3782	3776	3731	3731
query52	104	108	102	102
query53	279	326	268	268
query54	278	250	257	250
query55	78	75	68	68
query56	278	288	282	282
query57	962	1028	939	939
query58	267	245	238	238
query59	2074	2138	2062	2062
query60	306	308	292	292
query61	191	155	157	155
query62	409	361	316	316
query63	299	268	281	268
query64	5002	1305	971	971
query65	3825	3728	3722	3722
query66	1383	430	318	318
query67	15174	15843	14928	14928
query68	2701	1026	733	733
query69	433	356	326	326
query70	1037	948	934	934
query71	316	306	278	278
query72	5834	3642	3707	3642
query73	613	736	307	307
query74	8801	8738	8577	8577
query75	2791	2847	2496	2496
query76	2930	1049	654	654
query77	525	376	287	287
query78	9723	10178	9185	9185
query79	1151	848	610	610
query80	1566	609	536	536
query81	547	262	230	230
query82	428	145	107	107
query83	371	247	233	233
query84	265	121	99	99
query85	931	506	446	446
query86	418	325	325	325
query87	2824	2872	2745	2745
query88	3269	2252	2267	2252
query89	392	366	331	331
query90	2028	147	139	139
query91	180	161	141	141
query92	73	70	62	62
query93	1061	880	535	535
query94	651	327	285	285
query95	567	373	300	300
query96	595	455	204	204
query97	2350	2419	2272	2272
query98	215	209	204	204
query99	610	581	512	512
Total cold run time: 249349 ms
Total hot run time: 172848 ms

@doris-robot
Copy link

doris-robot commented Jan 5, 2026

ClickBench: Total hot run time: 26.87 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 88d157a59900b1c6f87713842486026ae4453921, data reload: false

query1	0.05	0.05	0.05
query2	0.11	0.05	0.05
query3	0.26	0.09	0.08
query4	1.60	0.11	0.12
query5	0.27	0.27	0.26
query6	1.15	0.67	0.65
query7	0.03	0.02	0.02
query8	0.06	0.04	0.04
query9	0.58	0.50	0.50
query10	0.55	0.53	0.54
query11	0.14	0.09	0.10
query12	0.14	0.10	0.11
query13	0.60	0.59	0.59
query14	0.95	0.95	0.95
query15	0.81	0.78	0.78
query16	0.39	0.40	0.42
query17	1.10	1.03	1.07
query18	0.24	0.21	0.21
query19	1.85	1.90	1.83
query20	0.02	0.01	0.02
query21	15.45	0.27	0.14
query22	5.15	0.05	0.04
query23	15.83	0.30	0.11
query24	1.02	0.88	0.33
query25	0.08	0.06	0.07
query26	0.14	0.14	0.13
query27	0.08	0.05	0.05
query28	3.21	1.08	0.88
query29	12.63	3.88	3.17
query30	0.27	0.15	0.12
query31	2.84	0.64	0.40
query32	3.24	0.57	0.46
query33	2.99	3.07	3.02
query34	16.83	5.08	4.41
query35	4.52	4.44	4.46
query36	0.66	0.51	0.49
query37	0.11	0.07	0.06
query38	0.08	0.04	0.03
query39	0.05	0.03	0.03
query40	0.16	0.13	0.13
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.03	0.04
Total cold run time: 96.42 s
Total hot run time: 26.87 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 5, 2026

BE UT Coverage Report

Increment line coverage 81.82% (9/11) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.25% (18976/35639)
Line Coverage 39.23% (176052/448812)
Region Coverage 33.73% (136136/403638)
Branch Coverage 34.71% (58769/169316)

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 5, 2026

BE Regression && UT Coverage Report

Increment line coverage 81.82% (9/11) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 72.20% (25156/34840)
Line Coverage 58.95% (263835/447594)
Region Coverage 53.71% (218996/407758)
Branch Coverage 55.36% (94041/169865)

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 5, 2026

BE Regression && UT Coverage Report

Increment line coverage 81.82% (9/11) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 72.20% (25156/34840)
Line Coverage 58.94% (263834/447594)
Region Coverage 53.69% (218939/407758)
Branch Coverage 55.36% (94037/169865)

Hastyshell
Hastyshell approved these changes Jan 6, 2026
View reviewed changes
Copy link
Collaborator

@Hastyshell Hastyshell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yiguolei yiguolei merged commit 52a178a into apache:master Jan 6, 2026
43 of 44 checks passed
@zhiqiang-hhhh zhiqiang-hhhh deleted the fix-ordered-data-compaction branch January 7, 2026 01:59
@zhiqiang-hhhh zhiqiang-hhhh mentioned this pull request Jan 7, 2026
Merged
zhiqiang-hhhh added a commit to zhiqiang-hhhh/doris that referenced this pull request Jan 7, 2026
@zhiqiang-hhhh
[fix](compaction) Check schema version before ordered data compaction (…
0ac9ab7 
…apache#59570)

Ordered data compaction should not be used to accelerate the merging of
rowsets that have undergone a light schema change; otherwise, it may
result in schema-inconsistent segments within the newly generated
rowset.

None

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [x] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [x] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
yiguolei pushed a commit that referenced this pull request Jan 8, 2026
@zhiqiang-hhhh
branch-4.0: [fix](compaction) Check schema version before ordered dat…
d1ebe95 
…a compaction #59570 (#59620)

cherry pick from #59570
@zhiqiang-hhhh zhiqiang-hhhh mentioned this pull request Jan 8, 2026
Open
yiguolei pushed a commit that referenced this pull request Jan 8, 2026
@zhiqiang-hhhh
Revert "branch-4.0: [fix](compaction) Check schema version before ord…
635ed6b 
…ered data compaction #59570" (#59661)

Reverts #59620
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@airborne12 airborne12 airborne12 approved these changes

@yiguolei yiguolei yiguolei approved these changes

@liaoxin01 liaoxin01 liaoxin01 approved these changes

@luwei16 luwei16 Awaiting requested review from luwei16

+1 more reviewer

@Hastyshell Hastyshell Hastyshell approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@zhiqiang-hhhh @hello-stephen @doris-robot @airborne12 @yiguolei @liaoxin01 @Hastyshell