[Bug] "FATAL","XX000","Unexpected internal error (assert.c:48)"

[ryapandt]

Apache Cloudberry version

2.0.0

What happened

An SQL query triggers a database-reset error—sporadic and not consistently reproducible.
Error log is as follows

2025-11-29 00:01:02.479579 CST,"jyzsuser","pasobi",p4062425,th1425586304,"172.18.46.87","36862",2025-11-29 00:00:38 CST,0,con5825,cmd1,seg-1,,,,sx1,"FATAL","XX000","Unexpected internal error (assert.c:48)","AssertImply failed(""!(!(inCursor) || (dtxContextInfo->distributedXid != ((DistributedTransactionId) 0) && gp_command_count == MySessionState->latestCursorCommandId))"", File: ""cdbdtxcontextinfo.c"", Line: 63)",,,,,"/* jyzsv2-DB:master*/
/* jyzsv2-schema:jyzs */
 select u.gid usergid,u.name username, a.userlevel::int::varchar,a.userlevelname,a.userlvpgid::int orgDef,a.usercode ,
               a.storetype ,a.bsortcode ,a.brandcode ,a.storerange, b.tel mobile,coalesce(usernewrole,userrole) permissrole,fu.loginname
        from v_jyzs_empstaion a
        left join faemp b on a.usergid=b.gid
        left join v_jyzs_user u on a.usergid=u.gid
        left join fauser fu on a.usergid=fu.gid
        where usergid=$1::int order by userlevel,userlvpgid limit 1",0,,"assert.c",48,"Stack trace:
1    0x7f974e4673d2 libpostgres.so errstart + 0x202
2    0x7f974ddd48d5 libpostgres.so ExceptionalCondition + 0x87
3    0x7f974e50eb9c libpostgres.so DtxContextInfo_CreateOnMaster + 0x21c
4    0x7f974e4fb8b7 libpostgres.so qdSerializeDtxContextInfo + 0x127
5    0x7f974e4fc8f9 libpostgres.so CdbDispatchPlan + 0x539
6    0x7f974e0a6acb libpostgres.so standard_ExecutorStart + 0xe3b
7    0x7f974e303b67 libpostgres.so PortalStart + 0x467
8    0x7f974e2ff6f5 libpostgres.so <symbol not found> + 0x4e2ff6f5
9    0x7f974e300bc9 libpostgres.so PostgresMain + 0xe49
10   0x7f974e247112 libpostgres.so <symbol not found> + 0x4e247112
11   0x7f974e2481f1 libpostgres.so PostmasterMain + 0xe61
12   0x4027eb postgres main (main.c:200)
13   0x7f974d2b27e5 libc.so.6 __libc_start_main + 0xe5
14   0x40298e postgres _start + 0x2e
"

then "the database system is resetting"

2025-11-29 00:01:13.336536 CST,"jyzsuser","pasobi",p4064992,th1425586304,"172.18.46.87","53582",2025-11-29 00:01:13 CST,0,,,seg-1,,,,,"FATAL","57P03","the database system is resetting",,,,,,,0,,"postmaster.c",2796

What you think should happen instead

No response

How to reproduce

none

Operating System

rocky 8.10

Anything else

No response

Are you willing to submit PR?

• Yes, I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct.

[zhangyue1818]

Based on stack trace analysis, the issue is most likely triggered by the ereport(FATAL, ...) call at line 59 in cdbdtxcontextinfo.c. Could you please check whether the environment where the problem occurs matches any of the following scenarios:

• A long-running application that maintains persistent database connections without ever disconnecting
• An application executing SQL statements in an infinite loop
• Active stress testing

[ryapandt]

Based on stack trace analysis, the issue is most likely triggered by the ereport(FATAL, ...) call at line 59 in cdbdtxcontextinfo.c. Could you please check whether the environment where the problem occurs matches any of the following scenarios:根据堆栈跟踪分析，问题最可能是由cdbdtxcontextinfo.c.第 59 行的 ereport(FATAL, ...)调用触发的。请问您能否检查问题发生的环境是否与以下场景中的任何一种匹配：

• A long-running application that maintains persistent database connections without ever disconnecting一个长时间运行的应用程序，它维护着持久的数据库连接而从未断开连接。
• An application executing SQL statements in an infinite loop一个应用程序在无限循环中执行 SQL 语句
• Active stress testing  进行主动压力测试

None of the above scenarios

[ryapandt]

We have again encountered a similar issue, and the following logs may be helpful.

2025-11-29 15:00:09.372516 CST,"gpadmin","postgres",p633510,th-1126913920,"172.18.1.47","61002",2025-11-29 15:00:05 CST,0,con8270,cmd993,seg-1,,,,sx1,"LOG","00000","serializable isolation requested, falling back to repeatable read until serializable is supported in Cloudberry",,,,,,"
          SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;

          -- distribute catalog table from coordinator, so that we can avoid to gather
          CREATE TEMPORARY TABLE _tmp_coordinator ON COMMIT DROP AS
              SELECT gp_segment_id AS segid, loid,pageno FROM pg_largeobject DISTRIBUTED BY (segid);
          SELECT loid,pageno, total, array_agg(segid order by segid) as segids
          FROM (
              SELECT segid, loid,pageno, count(*) as total
              FROM (
                   select segid, loid,pageno FROM _tmp_coordinator
                   union all
                   select gp_segment_id as segid, loid,pageno FROM gp_dist_random('pg_largeobject')
              ) all_segments
              GROUP BY segid, loid,pageno
              HAVING count(*) > 1
          ) rowresult
          GROUP BY loid,pageno, total
          ",0,,"variable.c",569,
2025-11-29 15:00:11.838488 CST,"jyzsuser","pasobi",p631667,th-1126913920,"172.18.46.87","39686",2025-11-29 14:58:37 CST,0,con8226,cmd139,seg-1,,dx23142391,,sx1,"LOG","00000","DEADLOCK HAZARD: Updating active_motion_id from 1 to 3",,,,,,"/* jyzsv2-DB:master*/
/* jyzsv2-schema:jyzs */
 with showInvTemp as (
        select case when position(b.userlevel::int::varchar in property_value)>0 and position(coalesce(b.usernewrole,'null') in property_value) <=0 then b.userlevel::int::varchar else '不可看库存' end as if_show_inv
        from ads_jyzs_globalconfigs a left join v_jyzs_empstaion b on b.usergid = $1::int  where property_name = 'if_show_inv' limit 1),
        gprofit_disable_tmp as (

                select case when position($2::varchar in property_value)>0 or position(coalesce(b.usernewrole,'null') in property_value)>0 then 'false' else 'true' end d_gprofit_disable
                from ads_jyzs_globalconfigs a left join v_jyzs_empstaion b on b.usergid = $3::int where property_name='gp_show_groupid' limit 1

        )
        select * from gprofit_disable_tmp,showInvTemp",0,,"nodeMotion.c",143,
2025-11-29 15:00:22.201043 CST,"gpadmin","postgres",p633510,th-1126913920,"172.18.1.47","61002",2025-11-29 15:00:05 CST,0,con8270,cmd1014,seg-1,,,,sx1,"ERROR","XX000","Unexpected internal error (assert.c:48)  (seg11 slice1 172.18.1.49:6001 pid=2719241) (assert.c:48)","FailedAssertion(""remote_typmod < remapper->typmodmapsize"", File: ""tupleremap.c"", Line: 255)",,,,,"
          SELECT oid,
                 case when coordinator is null then segids
                      when segids is null then array[coordinator.segid]
                      else coordinator.segid || segids end as segids
          FROM
          (
            SELECT oid, array_agg(gp_segment_id order by gp_segment_id) as segids
            FROM gp_dist_random('pg_class')  GROUP BY oid
          ) as seg
          FULL OUTER JOIN
          (
            SELECT gp_segment_id as segid, oid FROM pg_class
          ) as coordinator
          USING (oid)
          WHERE coordinator.segid is null
             OR segids is null
             OR NOT segids @> (select array_agg(content::int4) from gp_segment_configuration WHERE content >= 0)
          ",0,,"assert.c",48,"Stack trace:
1    0x7fddb62263d2 libpostgres.so errstart + 0x202
2    0x7fddb622c2f6 libpostgres.so ThrowErrorData + 0x16
3    0x7fddb62b7010 libpostgres.so CdbDispatchHandleError + 0x1b0
4    0x7fddb5e77fc3 libpostgres.so mppExecutorCleanup + 0x83
5    0x7fddb5e6455d libpostgres.so standard_ExecutorRun + 0x8cd
6    0x7fddb5e646a5 libpostgres.so ExecutorRun + 0xc5
7    0x7fddb60c197f libpostgres.so <symbol not found> + 0xb60c197f
8    0x7fddb60c374f libpostgres.so PortalRun + 0x1af
9    0x7fddb60bc6d0 libpostgres.so exec_simple_query + 0x1c0
10   0x7fddb60c10da libpostgres.so PostgresMain + 0x235a
11   0x7fddb6006112 libpostgres.so <symbol not found> + 0xb6006112
12   0x7fddb60071f1 libpostgres.so PostmasterMain + 0xe61
13   0x4027eb postgres main (main.c:200)
14   0x7fddb50717e5 libc.so.6 __libc_start_main + 0xe5
15   0x40298e postgres _start + 0x2e
"
2025-11-29 15:00:22.201105 CST,"gpadmin","postgres",p633510,th-1126913920,"172.18.1.47","61002",2025-11-29 15:00:05 CST,0,con8270,cmd1014,seg-1,,,,sx1,"LOG","00000","An exception was encountered during the execution of statement:
          SELECT oid,
                 case when coordinator is null then segids
                      when segids is null then array[coordinator.segid]
                      else coordinator.segid || segids end as segids
          FROM
          (
            SELECT oid, array_agg(gp_segment_id order by gp_segment_id) as segids
            FROM gp_dist_random('pg_class')  GROUP BY oid
          ) as seg
          FULL OUTER JOIN
          (
            SELECT gp_segment_id as segid, oid FROM pg_class
          ) as coordinator
          USING (oid)
          WHERE coordinator.segid is null
             OR segids is null
             OR NOT segids @> (select array_agg(content::int4) from gp_segment_configuration WHERE content >= 0)
          ",,,,,,"
          SELECT oid,
                 case when coordinator is null then segids
                      when segids is null then array[coordinator.segid]
                      else coordinator.segid || segids end as segids
          FROM
          (
            SELECT oid, array_agg(gp_segment_id order by gp_segment_id) as segids
            FROM gp_dist_random('pg_class')  GROUP BY oid
          ) as seg
          FULL OUTER JOIN
          (
            SELECT gp_segment_id as segid, oid FROM pg_class
          ) as coordinator
          USING (oid)
          WHERE coordinator.segid is null
             OR segids is null
             OR NOT segids @> (select array_agg(content::int4) from gp_segment_configuration WHERE content >= 0)
          ",0,,"elog.c",2129,
2025-11-29 15:00:22.201857 CST,"gpadmin","postgres",p633510,th-1126913920,"172.18.1.47","61002",2025-11-29 15:00:05 CST,0,con8370,,seg-1,,,,sx1,"LOG","00000","The previous session was reset because its gang was disconnected (session id = 8270). The new session id = 8370",,,,,,,0,,"cdbgang.c",816,
2025-11-29 15:00:22.201891 CST,"gpadmin","postgres",p634555,th-1126913920,"[local]",,2025-11-29 15:00:09 CST,0,con8270,,seg-1,,,,,"LOG","00000","Process interrupt for 'query cancel pending' (postgres.c:5636)",,,,,,,0,,"postgres.c",4216,
2025-11-29 15:00:22.211477 CST,"gpadmin","postgres",p633510,th-1126913920,"172.18.1.47","61002",2025-11-29 15:00:05 CST,0,con8370,cmd2,seg-1,,,,sx1,"LOG","00000","segment is in reset/recovery mode (seg11 172.18.1.49:6001)",,,,,,"
          SELECT adrelid,adnum,
                 case when coordinator is null then segids
                      when segids is null then array[coordinator.segid]
                      else coordinator.segid || segids end as segids
          FROM
          (
            SELECT adrelid,adnum, array_agg(gp_segment_id order by gp_segment_id) as segids
            FROM gp_dist_random('pg_attrdef')  GROUP BY adrelid,adnum
          ) as seg
          FULL OUTER JOIN
          (
            SELECT gp_segment_id as segid, adrelid,adnum FROM pg_attrdef
          ) as coordinator
          USING (adrelid,adnum)
          WHERE coordinator.segid is null
             OR segids is null
             OR NOT segids @> (select array_agg(content::int4) from gp_segment_configuration WHERE content >= 0)
          ",0,,"cdbgang_async.c",256,          

[zhangyue1818]

Could you please provide a reproducible script so I can replicate the issue locally? :-)