Skip to content

Conversation

@Dino123-maker
Copy link

@Dino123-maker Dino123-maker commented Aug 20, 2025

What problem does this PR solve?

Issue Number: close: #53017

Problem Summary:
Add sync table command for async group commit, it will return when group commit finishes.

Release note

Add support and regression test for SYNC TABLE command for async group commit.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Dino123-maker Dino123-maker changed the title [enhancement] add sync table command [enhancement](nereids) add sync table command Aug 20, 2025
@Dino123-maker
Copy link
Author

run buildall

1 similar comment
@Dino123-maker
Copy link
Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33922 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 48a3d6fb8cb38c4f2802744182ee93cc028c32bb, data reload: false

------ Round 1 ----------------------------------
q1	17577	5186	5132	5132
q2	1935	292	179	179
q3	10294	1286	757	757
q4	10218	1007	522	522
q5	7651	2437	2334	2334
q6	176	162	133	133
q7	893	751	593	593
q8	9315	1308	1049	1049
q9	6851	5141	5070	5070
q10	6884	2380	1959	1959
q11	489	287	278	278
q12	350	354	219	219
q13	17776	3693	3047	3047
q14	232	235	210	210
q15	561	481	475	475
q16	417	429	366	366
q17	598	847	357	357
q18	7608	7170	7191	7170
q19	1227	924	565	565
q20	361	340	216	216
q21	3776	3229	2315	2315
q22	1070	1036	976	976
Total cold run time: 106259 ms
Total hot run time: 33922 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5153	5138	5112	5112
q2	247	316	218	218
q3	2192	2686	2282	2282
q4	1401	1795	1323	1323
q5	4205	4615	4560	4560
q6	218	165	129	129
q7	2026	1952	1837	1837
q8	2624	2601	2570	2570
q9	7312	7244	7220	7220
q10	3109	3619	2850	2850
q11	574	498	479	479
q12	733	794	660	660
q13	3572	3955	3311	3311
q14	432	302	293	293
q15	515	475	479	475
q16	459	475	440	440
q17	1179	1545	1335	1335
q18	7984	7727	7640	7640
q19	820	824	1013	824
q20	2036	2071	2028	2028
q21	5102	4498	4431	4431
q22	1067	1036	991	991
Total cold run time: 52960 ms
Total hot run time: 51008 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185052 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 48a3d6fb8cb38c4f2802744182ee93cc028c32bb, data reload: false

query1	1001	388	414	388
query2	6540	1758	1779	1758
query3	6737	229	217	217
query4	26491	23082	22958	22958
query5	4387	627	510	510
query6	326	223	206	206
query7	4626	489	287	287
query8	281	240	221	221
query9	8581	2864	2862	2862
query10	477	310	280	280
query11	15780	15014	14782	14782
query12	167	112	107	107
query13	1658	557	430	430
query14	8558	5755	5743	5743
query15	201	189	174	174
query16	7263	625	490	490
query17	1182	730	575	575
query18	1983	420	301	301
query19	187	181	171	171
query20	131	122	115	115
query21	217	122	111	111
query22	4026	4088	4061	4061
query23	34153	33337	33316	33316
query24	8096	2341	2342	2341
query25	546	458	409	409
query26	1236	274	159	159
query27	2743	502	338	338
query28	4392	2270	2230	2230
query29	765	550	439	439
query30	289	217	194	194
query31	879	848	725	725
query32	85	79	79	79
query33	542	375	340	340
query34	787	900	535	535
query35	785	853	763	763
query36	976	1038	885	885
query37	122	107	89	89
query38	4086	3960	3925	3925
query39	1485	1421	1386	1386
query40	208	121	113	113
query41	59	57	52	52
query42	122	114	111	111
query43	516	485	489	485
query44	1317	861	859	859
query45	180	175	165	165
query46	863	994	635	635
query47	1766	1797	1710	1710
query48	395	410	342	342
query49	711	483	384	384
query50	635	681	415	415
query51	4032	4161	4042	4042
query52	111	109	102	102
query53	231	248	197	197
query54	595	577	525	525
query55	89	90	87	87
query56	321	312	292	292
query57	1218	1188	1133	1133
query58	286	261	263	261
query59	2661	2759	2685	2685
query60	342	327	329	327
query61	124	131	154	131
query62	795	748	654	654
query63	229	189	192	189
query64	4441	1065	683	683
query65	4295	4234	4203	4203
query66	1150	407	334	334
query67	15597	15200	15099	15099
query68	7892	909	568	568
query69	477	326	288	288
query70	1214	1136	1096	1096
query71	448	324	334	324
query72	5625	4814	4776	4776
query73	703	613	353	353
query74	9097	9139	8979	8979
query75	3798	3113	2635	2635
query76	3651	1140	735	735
query77	786	391	320	320
query78	9472	9699	8887	8887
query79	2191	800	613	613
query80	603	546	476	476
query81	470	246	222	222
query82	422	139	107	107
query83	245	256	238	238
query84	241	94	82	82
query85	845	372	337	337
query86	354	324	303	303
query87	4242	4254	4114	4114
query88	3168	2226	2218	2218
query89	398	337	292	292
query90	1925	222	225	222
query91	135	145	116	116
query92	88	71	68	68
query93	1193	993	644	644
query94	690	403	311	311
query95	389	321	306	306
query96	486	585	279	279
query97	2635	2665	2553	2553
query98	231	218	217	217
query99	1463	1371	1258	1258
Total cold run time: 271633 ms
Total hot run time: 185052 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.96 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 48a3d6fb8cb38c4f2802744182ee93cc028c32bb, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.25	0.08	0.08
query4	1.62	0.10	0.11
query5	0.41	0.42	0.40
query6	1.17	0.65	0.65
query7	0.03	0.02	0.02
query8	0.05	0.03	0.04
query9	0.61	0.53	0.51
query10	0.56	0.57	0.58
query11	0.16	0.11	0.11
query12	0.15	0.11	0.12
query13	0.62	0.60	0.60
query14	0.81	0.82	0.83
query15	0.87	0.85	0.84
query16	0.38	0.40	0.40
query17	1.05	1.07	1.04
query18	0.20	0.19	0.20
query19	1.96	1.80	1.80
query20	0.01	0.01	0.01
query21	15.43	0.93	0.58
query22	0.77	1.16	0.80
query23	14.77	1.37	0.58
query24	6.63	1.11	1.15
query25	0.50	0.16	0.13
query26	0.65	0.16	0.13
query27	0.06	0.05	0.06
query28	9.22	0.96	0.43
query29	12.57	3.86	3.27
query30	3.10	3.04	3.00
query31	2.82	0.58	0.38
query32	3.24	0.55	0.48
query33	3.04	3.11	3.18
query34	16.11	5.53	4.85
query35	4.91	4.87	4.92
query36	0.67	0.50	0.50
query37	0.10	0.07	0.06
query38	0.06	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.13	0.14
query41	0.08	0.02	0.03
query42	0.04	0.03	0.02
query43	0.04	0.04	0.04
Total cold run time: 106.03 s
Total hot run time: 32.96 s

@dataroaring
Copy link
Contributor

Wait wal empty may be time consuming, because data may comes into doris continuously. We should wait for finishing of started group commit backgroup loads.

felixwluo
felixwluo previously approved these changes Sep 12, 2025
Copy link
Contributor

@felixwluo felixwluo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 12, 2025
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@Dino123-maker Dino123-maker changed the title [enhancement](nereids) add sync table command [enhancement] (nereids) add sync table command Sep 12, 2025
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 17, 2025
@Dino123-maker
Copy link
Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 1498 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit aa9a6a48cba5661abf58596d139cec6de4516402, data reload: false

------ Round 1 ----------------------------------
q1	17652	35	19	19
q2	2599	25	24	24
q3	10721	17	16	16
q4	11190	14	14	14
q5	8123	18	13	13
q6	231	13	12	12
q7	927	24	22	22
q8	9439	14	12	12
q9	9092	23	13	13
q10	7022	13	12	12
q11	711	22	19	19
q12	357	12	11	11
q13	18620	12	11	11
q14	245	14	11	11
q15	619	12	11	11
q16	1049	978	953	953
q17	568	13	12	12
q18	7729	12	11	11
q19	1175	13	11	11
q20	1989	374	259	259
q21	9185	21	22	21
q22	1074	12	11	11
Total cold run time: 120317 ms
Total hot run time: 1498 ms

----- Round 2, with runtime_filter_mode=off -----
q1	12	11	11	11
q2	21	20	19	19
q3	11	12	11	11
q4	15	11	11	11
q5	13	11	16	11
q6	12	11	10	10
q7	19	19	19	19
q8	11	11	11	11
q9	10	11	10	10
q10	10	9	10	9
q11	18	20	19	19
q12	10	10	10	10
q13	10	10	10	10
q14	10	10	24	10
q15	15	11	10	10
q16	1049	1090	1027	1027
q17	12	10	10	10
q18	10	11	10	10
q19	10	10	10	10
q20	1830	1978	1865	1865
q21	20	20	19	19
q22	10	9	10	9
Total cold run time: 3138 ms
Total hot run time: 3131 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 2798 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit aa9a6a48cba5661abf58596d139cec6de4516402, data reload: false

query1	1083	19	11	11
query2	7157	17	15	15
query3	7626	12	10	10
query4	27109	14	14	14
query5	4517	12	12	12
query6	404	11	11	11
query7	5405	12	10	10
query8	358	19	18	18
query9	9224	12	10	10
query10	736	12	11	11
query11	15519	10	10	10
query12	175	11	10	10
query13	1719	11	10	10
query14	10876	15	13	13
query15	232	11	10	10
query16	7165	11	10	10
query17	1451	10	11	10
query18	2880	11	10	10
query19	205	10	9	9
query20	137	11	10	10
query21	223	10	10	10
query22	4099	10	9	9
query23	33567	17	13	13
query24	9475	11	13	11
query25	727	9	9	9
query26	1043	10	9	9
query27	3409	10	8	8
query28	6334	11	10	10
query29	1231	11	9	9
query30	652	10	11	10
query31	1766	11	11	11
query32	123	11	10	10
query33	1254	10	9	9
query34	1637	823	511	511
query35	915	11	11	11
query36	985	9	8	8
query37	117	10	8	8
query38	3548	9	8	8
query39	1541	740	766	740
query40	228	9	9	9
query41	80	13	12	12
query42	162	11	11	11
query43	505	10	8	8
query44	1333	10	9	9
query45	202	11	9	9
query46	895	9	9	9
query47	1792	11	9	9
query48	403	9	9	9
query49	1133	11	10	10
query50	775	10	10	10
query51	3957	10	9	9
query52	119	12	10	10
query53	236	11	12	11
query54	699	9	8	8
query55	95	10	10	10
query56	358	11	11	11
query57	1183	10	10	10
query58	321	11	10	10
query59	2606	11	9	9
query60	393	10	10	10
query61	212	12	11	11
query62	823	10	9	9
query63	289	11	11	11
query64	4186	14	11	11
query65	4289	11	11	11
query66	1332	11	10	10
query67	16709	11	13	11
query68	3492	12	12	12
query69	615	11	10	10
query70	1435	10	9	9
query71	422	374	363	363
query72	7479	14	11	11
query73	530	11	11	11
query74	9544	10	9	9
query75	3540	13	12	12
query76	2580	11	11	11
query77	1033	13	12	12
query78	9849	23	9	9
query79	1416	9	8	8
query80	946	11	9	9
query81	1205	11	10	10
query82	314	9	9	9
query83	336	10	9	9
query84	392	10	9	9
query85	2395	9	9	9
query86	931	12	9	9
query87	3971	10	10	10
query88	3094	12	11	11
query89	449	9	9	9
query90	2289	9	7	7
query91	182	9	10	9
query92	94	10	9	9
query93	2036	10	9	9
query94	2153	10	10	10
query95	517	10	9	9
query96	420	9	9	9
query97	3098	10	9	9
query98	242	233	254	233
query99	1518	10	8	8
Total cold run time: 290456 ms
Total hot run time: 2798 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 0.06 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit aa9a6a48cba5661abf58596d139cec6de4516402, data reload: false

query1	0.06	0.01	0.01
query2	0.11	0.01	0.01
query3	0.27	0.01	0.01
query4	1.75	0.00	0.01
query5	0.29	0.01	0.00
query6	1.65	0.00	0.01
query7	0.05	0.01	0.00
query8	0.07	0.01	0.01
query9	0.63	0.00	0.00
query10	0.61	0.00	0.00
query11	0.18	0.00	0.00
query12	0.16	0.00	0.01
query13	0.63	0.01	0.00
query14	1.07	0.01	0.01
query15	0.90	0.00	0.00
query16	0.41	0.00	0.00
query17	1.02	0.01	0.00
query18	0.22	0.01	0.00
query19	1.90	0.00	0.00
query20	0.02	0.01	0.00
query21	15.36	0.00	0.01
query22	6.40	0.01	0.00
query23	16.01	0.00	0.00
query24	1.35	0.00	0.00
query25	0.22	0.01	0.00
query26	0.18	0.00	0.00
query27	0.11	0.00	0.00
query28	1.03	0.00	0.00
query29	13.30	0.01	0.01
query30	0.32	0.01	0.00
query31	2.23	0.00	0.01
query32	5.87	0.00	0.00
query33	4.31	0.01	0.00
query34	7.40	0.00	0.00
query35	5.95	0.00	0.00
query36	0.70	0.00	0.00
query37	0.11	0.00	0.00
query38	0.08	0.01	0.00
query39	0.05	0.00	0.00
query40	0.19	0.00	0.00
query41	0.09	0.00	0.01
query42	0.06	0.00	0.01
query43	0.05	0.00	0.01
Total cold run time: 93.37 s
Total hot run time: 0.06 s

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 2.27% (1/44) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 75.00% (33/44) 🎉
Increment coverage report
Complete coverage report

@dataroaring
Copy link
Contributor

run buildall

long tableId = table.getId();
GlobalTransactionMgrIface txnMgr = Env.getCurrentGlobalTransactionMgr();
// get table transaction ids
Map<Long, List<Long>> txnIdToTableIds = txnMgr.getDbRunningTransInfo(dbId);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in cloud mode, getDbRunningTransInfo is not implemented and return an empty map


@Override
public RedirectStatus toRedirectStatus() {
return null;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should return FORWARD_WITH_SYNC

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, you're right, sync table should execute by master

@Dino123-maker
Copy link
Author

run buildall

@Dino123-maker
Copy link
Author

run buildall

@doris-robot
Copy link

TPC-DS: Total hot run time: 190846 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 27f026939921b771c81e73d06b2442bc45a8289a, data reload: false

query1	1100	440	429	429
query2	6586	1717	1686	1686
query3	6758	220	225	220
query4	26062	23832	23469	23469
query5	5549	637	479	479
query6	344	261	255	255
query7	4660	509	303	303
query8	312	271	272	271
query9	8729	2581	2576	2576
query10	547	354	293	293
query11	15558	15531	14879	14879
query12	188	121	113	113
query13	1689	578	441	441
query14	12718	9359	9248	9248
query15	259	189	179	179
query16	7814	684	504	504
query17	1611	795	731	731
query18	2669	481	390	390
query19	286	217	187	187
query20	138	135	132	132
query21	232	137	121	121
query22	4694	4725	4682	4682
query23	34672	33898	33716	33716
query24	8286	2499	2545	2499
query25	588	533	494	494
query26	1273	294	170	170
query27	3068	534	356	356
query28	4294	2221	2167	2167
query29	812	644	502	502
query30	344	235	213	213
query31	944	864	771	771
query32	93	69	80	69
query33	585	390	343	343
query34	823	892	532	532
query35	792	843	754	754
query36	989	1026	953	953
query37	124	112	93	93
query38	3572	3584	3462	3462
query39	1495	1430	1430	1430
query40	217	123	117	117
query41	65	58	61	58
query42	119	110	111	110
query43	488	502	457	457
query44	1318	827	823	823
query45	192	184	173	173
query46	840	1003	632	632
query47	1794	1822	1746	1746
query48	396	422	309	309
query49	772	526	407	407
query50	658	690	408	408
query51	3899	3926	4131	3926
query52	108	104	98	98
query53	240	263	197	197
query54	601	602	554	554
query55	90	91	88	88
query56	323	318	311	311
query57	1215	1209	1129	1129
query58	289	274	280	274
query59	2514	2633	2555	2555
query60	374	338	331	331
query61	157	158	151	151
query62	785	745	677	677
query63	236	198	190	190
query64	4404	1194	898	898
query65	4056	3946	3992	3946
query66	1063	432	334	334
query67	15430	15241	15114	15114
query68	7588	949	590	590
query69	506	333	302	302
query70	1366	1300	1275	1275
query71	496	360	324	324
query72	5515	4878	4869	4869
query73	599	567	356	356
query74	9178	9144	8915	8915
query75	4363	3339	2833	2833
query76	4832	1205	762	762
query77	963	403	321	321
query78	9716	9838	8951	8951
query79	1426	929	597	597
query80	674	589	500	500
query81	493	270	225	225
query82	454	170	137	137
query83	296	279	251	251
query84	299	113	97	97
query85	883	511	436	436
query86	343	289	293	289
query87	3831	3787	3643	3643
query88	2877	2260	2280	2260
query89	418	344	299	299
query90	2024	232	226	226
query91	170	180	195	180
query92	101	72	69	69
query93	1158	1001	649	649
query94	701	423	346	346
query95	402	326	317	317
query96	499	562	282	282
query97	2974	2983	2875	2875
query98	243	226	224	224
query99	1454	1424	1321	1321
Total cold run time: 280742 ms
Total hot run time: 190846 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.77 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 27f026939921b771c81e73d06b2442bc45a8289a, data reload: false

query1	0.06	0.04	0.04
query2	0.09	0.05	0.06
query3	0.25	0.08	0.08
query4	1.60	0.12	0.12
query5	0.27	0.27	0.25
query6	1.15	0.66	0.65
query7	0.03	0.03	0.03
query8	0.06	0.04	0.04
query9	0.63	0.52	0.52
query10	0.57	0.58	0.58
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.64	0.61	0.61
query14	1.05	1.02	1.02
query15	0.87	0.88	0.87
query16	0.40	0.40	0.40
query17	1.06	1.12	1.05
query18	0.21	0.20	0.19
query19	1.95	1.87	1.83
query20	0.02	0.02	0.02
query21	15.42	0.97	0.57
query22	0.75	1.18	0.93
query23	14.77	1.46	0.62
query24	7.28	0.77	1.01
query25	0.49	0.13	0.18
query26	0.70	0.16	0.14
query27	0.07	0.06	0.05
query28	9.62	1.37	0.94
query29	12.57	4.04	3.31
query30	0.29	0.14	0.13
query31	2.83	0.59	0.39
query32	3.24	0.56	0.49
query33	3.13	3.09	3.08
query34	16.20	5.44	4.82
query35	4.96	4.96	4.94
query36	0.69	0.51	0.51
query37	0.10	0.07	0.07
query38	0.07	0.05	0.04
query39	0.04	0.03	0.03
query40	0.18	0.14	0.14
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 104.78 s
Total hot run time: 30.77 s

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 78.26% (36/46) 🎉
Increment coverage report
Complete coverage report

@dataroaring
Copy link
Contributor

Please handle cloud mode .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Enhancement] sync groupcomimt

6 participants