Skip to content

Conversation

@JNSimba
Copy link
Member

@JNSimba JNSimba commented Jan 12, 2026

What problem does this PR solve?

Related PR: #59461

  1. PostgreSQL uses slots for data consumption, but only one client can use a slot at a time. Therefore, after consuming data from the WAL phase, the slot needs to be closed. This doesn't affect MySQL, but it can be closed to avoid consuming connections.
  2. Create pg slot first when create job
  3. fix unstable case

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@JNSimba
Copy link
Member Author

JNSimba commented Jan 12, 2026

run buildall

@dataroaring
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31764 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2fa81dc0487f2371d61b7b2d8b1c31809a57b32a, data reload: false

------ Round 1 ----------------------------------
q1	17623	4317	4043	4043
q2	2048	365	241	241
q3	10147	1272	710	710
q4	10219	863	314	314
q5	7543	2100	1806	1806
q6	195	170	138	138
q7	925	795	653	653
q8	9268	1356	1140	1140
q9	4936	4602	4573	4573
q10	6806	1806	1410	1410
q11	539	294	291	291
q12	690	740	564	564
q13	17790	3820	3066	3066
q14	290	289	270	270
q15	595	511	511	511
q16	661	670	625	625
q17	668	737	570	570
q18	6587	6361	6671	6361
q19	1173	1060	669	669
q20	407	370	252	252
q21	3205	2506	2635	2506
q22	1124	1114	1051	1051
Total cold run time: 103439 ms
Total hot run time: 31764 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4305	4351	4250	4250
q2	342	398	326	326
q3	2275	2783	2408	2408
q4	1433	1916	1441	1441
q5	4387	4494	4376	4376
q6	211	170	130	130
q7	2068	1940	1757	1757
q8	2522	2304	2457	2304
q9	7123	7305	7031	7031
q10	2561	2700	2253	2253
q11	553	474	477	474
q12	733	766	625	625
q13	3517	3864	3110	3110
q14	280	291	268	268
q15	523	493	489	489
q16	609	652	608	608
q17	1110	1393	1391	1391
q18	7319	7299	7318	7299
q19	815	790	831	790
q20	1891	1954	1828	1828
q21	4642	4324	4218	4218
q22	1099	1046	1014	1014
Total cold run time: 50318 ms
Total hot run time: 48390 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172827 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2fa81dc0487f2371d61b7b2d8b1c31809a57b32a, data reload: false

query5	4384	608	438	438
query6	328	222	209	209
query7	4224	462	268	268
query8	340	248	233	233
query9	8779	2626	2600	2600
query10	500	363	323	323
query11	15220	15279	15041	15041
query12	171	117	113	113
query13	1266	493	376	376
query14	6207	3006	2767	2767
query14_1	2664	2611	2685	2611
query15	198	196	176	176
query16	992	474	478	474
query17	1126	695	598	598
query18	2507	447	346	346
query19	227	232	203	203
query20	121	121	117	117
query21	212	142	120	120
query22	3937	4099	3816	3816
query23	15854	15667	15340	15340
query23_1	15411	15455	15478	15455
query24	7448	1544	1176	1176
query24_1	1185	1197	1192	1192
query25	565	480	428	428
query26	1245	270	155	155
query27	2760	470	293	293
query28	4598	2122	2137	2122
query29	814	564	462	462
query30	314	241	222	222
query31	825	637	547	547
query32	79	70	69	69
query33	559	361	296	296
query34	936	883	519	519
query35	722	820	662	662
query36	873	888	798	798
query37	142	102	83	83
query38	2764	2731	2690	2690
query39	806	768	729	729
query39_1	730	709	701	701
query40	214	129	112	112
query41	65	61	60	60
query42	106	103	100	100
query43	430	439	437	437
query44	1310	719	725	719
query45	187	184	181	181
query46	841	950	587	587
query47	1410	1501	1418	1418
query48	300	317	234	234
query49	598	412	339	339
query50	638	274	196	196
query51	3887	3788	3748	3748
query52	105	108	101	101
query53	295	321	269	269
query54	281	286	264	264
query55	82	75	74	74
query56	305	280	283	280
query57	1004	1029	879	879
query58	269	243	282	243
query59	2074	2224	2111	2111
query60	314	311	294	294
query61	162	189	168	168
query62	395	369	304	304
query63	294	260	267	260
query64	4955	1310	994	994
query65	3799	3798	3724	3724
query66	1433	420	315	315
query67	15267	14720	15010	14720
query68	6978	965	714	714
query69	493	336	295	295
query70	1085	984	973	973
query71	363	299	278	278
query72	6087	3409	3385	3385
query73	778	716	293	293
query74	8876	8848	8578	8578
query75	2847	2803	2471	2471
query76	3440	1059	621	621
query77	511	361	273	273
query78	9563	9983	9227	9227
query79	1253	905	599	599
query80	614	573	480	480
query81	512	258	227	227
query82	218	148	116	116
query83	263	264	242	242
query84	256	120	105	105
query85	927	529	464	464
query86	388	329	274	274
query87	2893	2870	2761	2761
query88	3129	2213	2218	2213
query89	404	363	341	341
query90	2186	154	146	146
query91	175	161	145	145
query92	88	71	63	63
query93	1654	920	531	531
query94	567	316	256	256
query95	570	323	357	323
query96	571	451	206	206
query97	2350	2369	2331	2331
query98	211	201	194	194
query99	601	596	484	484
Total cold run time: 253182 ms
Total hot run time: 172827 ms

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 0.00% (0/2) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 0.00% (0/2) 🎉
Increment coverage report
Complete coverage report

@JNSimba
Copy link
Member Author

JNSimba commented Jan 13, 2026

run cloud_p0

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 0.00% (0/2) 🎉
Increment coverage report
Complete coverage report

# Conflicts:
#	fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/StreamingMultiTblTask.java
@JNSimba
Copy link
Member Author

JNSimba commented Jan 13, 2026

run buildall

@JNSimba
Copy link
Member Author

JNSimba commented Jan 13, 2026

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31642 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 01d0b0d0ee11a0389ee2a193476af72656d361ce, data reload: false

------ Round 1 ----------------------------------
q1	17633	4220	4017	4017
q2	2083	339	226	226
q3	10183	1258	730	730
q4	10206	844	311	311
q5	7533	2029	1809	1809
q6	181	165	137	137
q7	934	777	644	644
q8	9260	1413	1045	1045
q9	4849	4539	4480	4480
q10	6793	1795	1396	1396
q11	495	277	262	262
q12	685	740	596	596
q13	17786	3777	3076	3076
q14	308	294	268	268
q15	573	517	507	507
q16	661	675	654	654
q17	635	782	488	488
q18	6663	6456	6677	6456
q19	1374	1067	669	669
q20	411	375	280	280
q21	3188	2564	2524	2524
q22	1085	1157	1067	1067
Total cold run time: 103519 ms
Total hot run time: 31642 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4348	4257	4284	4257
q2	313	432	340	340
q3	2339	2778	2415	2415
q4	1405	1807	1475	1475
q5	4659	4328	4360	4328
q6	215	169	129	129
q7	1991	1908	1787	1787
q8	2468	2583	2345	2345
q9	6928	7160	7204	7160
q10	2515	2644	2226	2226
q11	558	467	477	467
q12	713	756	645	645
q13	3475	3965	3078	3078
q14	264	294	255	255
q15	522	487	481	481
q16	625	654	589	589
q17	1095	1343	1388	1343
q18	7286	7160	7204	7160
q19	814	785	768	768
q20	1870	1961	1875	1875
q21	4415	4186	4136	4136
q22	1057	1040	982	982
Total cold run time: 49875 ms
Total hot run time: 48241 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 171994 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 01d0b0d0ee11a0389ee2a193476af72656d361ce, data reload: false

query5	4378	621	482	482
query6	335	227	201	201
query7	4207	451	249	249
query8	348	247	222	222
query9	8643	2882	2846	2846
query10	478	363	311	311
query11	15299	15258	14734	14734
query12	179	118	115	115
query13	1260	472	365	365
query14	6323	2938	2732	2732
query14_1	2645	2586	2582	2582
query15	206	191	173	173
query16	1005	483	453	453
query17	1080	638	556	556
query18	2531	425	320	320
query19	221	221	193	193
query20	124	118	117	117
query21	216	139	119	119
query22	3818	3828	4070	3828
query23	16061	15470	15293	15293
query23_1	15554	15446	15463	15446
query24	7234	1517	1115	1115
query24_1	1173	1135	1181	1135
query25	538	440	403	403
query26	1240	264	160	160
query27	2759	446	268	268
query28	4591	2153	2153	2153
query29	798	553	454	454
query30	309	241	211	211
query31	830	614	541	541
query32	87	77	78	77
query33	531	380	312	312
query34	932	882	548	548
query35	736	757	683	683
query36	929	941	828	828
query37	125	94	82	82
query38	2779	2658	2666	2658
query39	784	746	743	743
query39_1	706	730	734	730
query40	213	131	115	115
query41	70	62	61	61
query42	105	105	105	105
query43	454	444	412	412
query44	1290	718	726	718
query45	184	181	170	170
query46	820	934	574	574
query47	1423	1503	1325	1325
query48	300	306	224	224
query49	604	414	328	328
query50	602	275	208	208
query51	3691	3888	3778	3778
query52	103	111	96	96
query53	286	319	268	268
query54	286	273	264	264
query55	81	80	74	74
query56	296	296	311	296
query57	1050	989	939	939
query58	267	256	253	253
query59	2125	2052	1944	1944
query60	332	344	315	315
query61	161	148	154	148
query62	381	353	312	312
query63	296	266	259	259
query64	4976	1274	966	966
query65	3856	3775	3782	3775
query66	1429	427	315	315
query67	14791	15812	14792	14792
query68	2649	1021	715	715
query69	463	351	319	319
query70	1011	939	933	933
query71	326	306	298	298
query72	6120	3377	3566	3377
query73	588	723	303	303
query74	8816	8717	8579	8579
query75	2753	2809	2463	2463
query76	2815	1058	645	645
query77	538	382	313	313
query78	9646	9943	9159	9159
query79	931	885	565	565
query80	1429	580	488	488
query81	546	267	229	229
query82	411	157	115	115
query83	376	252	242	242
query84	248	115	92	92
query85	1190	521	437	437
query86	399	302	328	302
query87	2871	2913	2820	2820
query88	2915	2109	2091	2091
query89	389	356	317	317
query90	1906	161	159	159
query91	169	160	138	138
query92	80	71	72	71
query93	1074	904	527	527
query94	643	339	300	300
query95	575	333	370	333
query96	549	481	204	204
query97	2373	2386	2278	2278
query98	213	208	201	201
query99	600	568	504	504
Total cold run time: 247294 ms
Total hot run time: 171994 ms

@JNSimba
Copy link
Member Author

JNSimba commented Jan 13, 2026

run p0

@JNSimba
Copy link
Member Author

JNSimba commented Jan 13, 2026

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32020 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8426e972918ddf9563c6f4c196577b1f860d3c79, data reload: false

------ Round 1 ----------------------------------
q1	17613	4234	4058	4058
q2	2064	352	246	246
q3	10127	1247	712	712
q4	10215	856	321	321
q5	7520	2034	1881	1881
q6	189	172	139	139
q7	934	789	694	694
q8	9257	1396	1077	1077
q9	4859	4675	4640	4640
q10	6766	1811	1431	1431
q11	521	322	290	290
q12	677	746	600	600
q13	17784	3867	3056	3056
q14	287	299	285	285
q15	574	518	513	513
q16	685	672	609	609
q17	659	765	537	537
q18	6460	6457	6732	6457
q19	1042	1027	626	626
q20	409	402	252	252
q21	3214	2609	2555	2555
q22	1091	1120	1041	1041
Total cold run time: 102947 ms
Total hot run time: 32020 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4354	4367	4241	4241
q2	333	416	304	304
q3	2241	2804	2427	2427
q4	1419	1878	1403	1403
q5	4600	4300	4473	4300
q6	213	166	127	127
q7	2019	1843	1811	1811
q8	2542	2372	2338	2338
q9	7174	7352	7040	7040
q10	2477	2763	2309	2309
q11	537	486	443	443
q12	728	810	585	585
q13	3620	4068	3288	3288
q14	284	330	296	296
q15	537	534	504	504
q16	698	735	629	629
q17	1168	1319	1339	1319
q18	7644	7215	7264	7215
q19	804	774	794	774
q20	1866	1950	1776	1776
q21	4515	4269	4180	4180
q22	1107	1030	954	954
Total cold run time: 50880 ms
Total hot run time: 48263 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172939 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8426e972918ddf9563c6f4c196577b1f860d3c79, data reload: false

query5	5212	613	478	478
query6	336	224	211	211
query7	4228	451	255	255
query8	347	278	227	227
query9	8718	2855	2806	2806
query10	538	387	332	332
query11	15259	15193	14959	14959
query12	177	116	118	116
query13	1255	466	378	378
query14	6907	2943	2749	2749
query14_1	2639	2678	2672	2672
query15	211	197	174	174
query16	1037	496	422	422
query17	1087	649	533	533
query18	2702	419	319	319
query19	217	217	220	217
query20	124	127	117	117
query21	214	137	115	115
query22	4039	3910	4011	3910
query23	16164	15478	15501	15478
query23_1	15496	15501	15483	15483
query24	6964	1528	1150	1150
query24_1	1153	1125	1156	1125
query25	527	453	401	401
query26	1246	268	173	173
query27	2714	440	277	277
query28	4527	2175	2152	2152
query29	775	552	455	455
query30	318	243	212	212
query31	809	621	565	565
query32	82	79	78	78
query33	545	358	312	312
query34	910	889	533	533
query35	722	785	681	681
query36	904	934	847	847
query37	131	96	89	89
query38	2740	2803	2716	2716
query39	771	756	727	727
query39_1	707	724	706	706
query40	219	140	129	129
query41	94	87	66	66
query42	107	104	106	104
query43	443	419	401	401
query44	1319	751	753	751
query45	192	183	182	182
query46	829	948	569	569
query47	1464	1424	1366	1366
query48	303	328	228	228
query49	630	455	353	353
query50	603	288	207	207
query51	3807	3774	3787	3774
query52	109	113	100	100
query53	290	336	276	276
query54	303	282	280	280
query55	83	80	77	77
query56	332	338	329	329
query57	1038	1029	895	895
query58	284	280	282	280
query59	2120	2068	2022	2022
query60	361	351	338	338
query61	189	186	184	184
query62	409	363	317	317
query63	303	277	280	277
query64	4952	1362	981	981
query65	3832	3749	3782	3749
query66	1393	424	319	319
query67	15445	15896	14641	14641
query68	5576	966	697	697
query69	504	374	317	317
query70	1034	906	850	850
query71	358	309	295	295
query72	6041	3370	3518	3370
query73	769	715	306	306
query74	8736	8818	8695	8695
query75	2799	2799	2453	2453
query76	3427	1067	656	656
query77	538	386	321	321
query78	9738	9909	9104	9104
query79	1152	931	562	562
query80	675	583	491	491
query81	497	264	235	235
query82	227	150	117	117
query83	275	258	241	241
query84	259	110	91	91
query85	913	509	455	455
query86	354	285	328	285
query87	2840	2846	2758	2758
query88	2977	2117	2104	2104
query89	382	354	309	309
query90	2022	170	160	160
query91	169	164	145	145
query92	79	75	74	74
query93	1042	874	522	522
query94	587	355	310	310
query95	587	345	365	345
query96	545	478	197	197
query97	2355	2348	2317	2317
query98	217	211	214	211
query99	578	590	523	523
Total cold run time: 252555 ms
Total hot run time: 172939 ms

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 0.00% (0/35) 🎉
Increment coverage report
Complete coverage report

@JNSimba
Copy link
Member Author

JNSimba commented Jan 13, 2026

run nonConcurrent

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 0.00% (0/35) 🎉
Increment coverage report
Complete coverage report

@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jan 13, 2026
@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes PostgreSQL consumer data handling in multi-backend scenarios by ensuring proper slot management and connection cleanup. The changes address the issue where PostgreSQL slots can only be used by one client at a time, necessitating proper initialization and cleanup strategies.

Changes:

  • Added early PostgreSQL slot creation during job initialization to prevent conflicts in multi-backend scenarios
  • Moved connection cleanup from mid-processing to the finishSplitRecords method to properly release connections after each read cycle
  • Refactored tests to use proper assertions and polling mechanisms instead of fixed sleep delays

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
PostgresSourceReader.java Added log message when replication slot already exists
MySqlSourceReader.java Moved binlog reader cleanup from mid-processing to finishSplitRecords method
JdbcIncrementalSourceReader.java Moved binlog reader cleanup from mid-processing to finishSplitRecords method
ClientController.java Added new /api/initReader endpoint for early source reader initialization
JdbcSourceOffsetProvider.java Added initSourceReader method to initialize readers for latest mode scenarios
StreamingMultiTblTask.java Enhanced logging to include backend information for better debugging
test_streaming_postgres_job_priv.groovy Refactored test to expect early failure without replication privileges and use polling instead of sleep
test_streaming_mysql_job_priv.groovy Replaced fixed sleep with polling mechanism to wait for task completion

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link
Contributor

@caoliang-web caoliang-web left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JNSimba JNSimba merged commit f1cbbfe into apache:master Jan 13, 2026
35 of 39 checks passed
github-actions bot pushed a commit that referenced this pull request Jan 13, 2026
### What problem does this PR solve?

Related PR: #59461

1. PostgreSQL uses slots for data consumption, but only one client can
use a slot at a time. Therefore, after consuming data from the WAL
phase, the slot needs to be closed. This doesn't affect MySQL, but it
can be closed to avoid consuming connections.
2. Create pg slot first when create job
3. fix unstable case
yiguolei pushed a commit that referenced this pull request Jan 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/4.0.3-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants